feat(worker): log exception messages and per-command start/end (mxaccessgw#184)
R21-O3: WorkerApplication.Run's failure lines logged only the exception type. Each now also carries exception_message, scrubbed by the new WorkerLogRedactor.RedactMessage: the value of any name=value / name: value pair whose name looks sensitive and any mxgw_ API key token are replaced, line breaks are flattened and the text is capped at 512 characters. The AppDomain unhandled-exception hook stays type-only, since an exception from an arbitrary thread can carry command values. R21-O7: StaCommandDispatcher.ExecuteQueuedCommandAsync logs WorkerCommandStarted (correlation id, method, queued_ms) and WorkerCommandCompleted (status, HRESULT, elapsed_ms) when a CommandLogger is set. IWorkerLogger has no Debug level, so the lines are opt-in through MXGATEWAY_WORKER_LOG_COMMANDS=1|true (WorkerCommandLogging), which the worker inherits from the gateway process environment. No payloads or diagnostic messages are logged. Docs updated. (mxaccessgw#184) Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -1162,7 +1162,9 @@ Worker logs should include:
|
|||||||
- protocol version,
|
- protocol version,
|
||||||
- gateway handshake result,
|
- gateway handshake result,
|
||||||
- MXAccess COM creation result,
|
- MXAccess COM creation result,
|
||||||
- command start/end with correlation id,
|
- command start/end with correlation id (`WorkerCommandStarted` /
|
||||||
|
`WorkerCommandCompleted`, opt-in through `MXGATEWAY_WORKER_LOG_COMMANDS`; see
|
||||||
|
`docs/WorkerBootstrap.md`),
|
||||||
- HRESULT/status summary,
|
- HRESULT/status summary,
|
||||||
- event family and sequence,
|
- event family and sequence,
|
||||||
- queue overflow,
|
- queue overflow,
|
||||||
|
|||||||
@@ -124,7 +124,7 @@ public static WorkerBootstrapResult Failure(WorkerExitCode exitCode, IEnumerable
|
|||||||
|
|
||||||
### Unhandled-exception hook
|
### Unhandled-exception hook
|
||||||
|
|
||||||
The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because they can carry tag values or credentials passed through a command, the same rule every other worker failure line follows. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
|
The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because an exception from an arbitrary thread (a COM callback, a command path) can carry tag values or credentials passed through a command. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
|
||||||
|
|
||||||
## Environment Abstraction
|
## Environment Abstraction
|
||||||
|
|
||||||
@@ -181,6 +181,12 @@ private void Write(
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Per-command start/end lines
|
||||||
|
|
||||||
|
`StaCommandDispatcher` can log one `WorkerCommandStarted` and one `WorkerCommandCompleted` line per STA command. Both carry `session_id`, `correlation_id`, and `command_method`; the start line adds `queued_ms` (time since the gateway enqueued the command) and the end line adds `protocol_status`, `hresult`, and `elapsed_ms` (time on the STA). Neither carries the command payload or the reply's diagnostic message, so tag values, secured-write credentials and provider detail stay out of the log.
|
||||||
|
|
||||||
|
The lines are off by default because a busy session dispatches many commands and `IWorkerLogger` has no Debug level to put them behind. Setting the environment variable `MXGATEWAY_WORKER_LOG_COMMANDS` to `1` or `true` turns them on (`WorkerCommandLogging`). The gateway launcher does not set it; a worker inherits it from the gateway process environment, so setting it on the gateway service enables it for every worker that service launches.
|
||||||
|
|
||||||
### What the redactor redacts and why
|
### What the redactor redacts and why
|
||||||
|
|
||||||
`gateway.md` "Security" requires that the worker never log raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations. The bootstrap nonce is also a credential: anyone who reads it can impersonate the worker to the gateway pipe. `WorkerLogRedactor` enforces this by replacing values whose field name contains any of these substrings (case-insensitive) with the literal `[redacted]`:
|
`gateway.md` "Security" requires that the worker never log raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations. The bootstrap nonce is also a credential: anyone who reads it can impersonate the worker to the gateway pipe. `WorkerLogRedactor` enforces this by replacing values whose field name contains any of these substrings (case-insensitive) with the literal `[redacted]`:
|
||||||
@@ -220,7 +226,7 @@ if (!result.Succeeded)
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`.
|
On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`. Each of those failure lines carries `exit_code`, `exception_type`, and `exception_message`. The message is passed through `WorkerLogRedactor.RedactMessage` first, which scrubs the value of any `name=value` or `name: value` pair whose name looks sensitive (nonce, secret, password, token, credential, API key) and any `mxgw_` API key token, flattens line breaks, and caps the text at 512 characters. Field-name redaction alone cannot see inside free text.
|
||||||
|
|
||||||
## Related Documentation
|
## Related Documentation
|
||||||
|
|
||||||
|
|||||||
@@ -102,6 +102,47 @@ public sealed class WorkerApplicationTests
|
|||||||
|
|
||||||
Assert.Equal((int)WorkerExitCode.ProtocolViolation, exitCode);
|
Assert.Equal((int)WorkerExitCode.ProtocolViolation, exitCode);
|
||||||
Assert.Equal("WorkerPipeProtocolFailure", logger.Entries[1].EventName);
|
Assert.Equal("WorkerPipeProtocolFailure", logger.Entries[1].EventName);
|
||||||
|
Assert.Equal("Bad nonce.", logger.Entries[1].Fields["exception_message"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Every failure line in <c>Run</c> carries the exception message, scrubbed by
|
||||||
|
/// <see cref="WorkerLogRedactor.RedactMessage"/> so a credential embedded in it never
|
||||||
|
/// reaches the log (mxaccessgw#184).
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="exceptionKind">Which catch block the thrown exception lands in.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData("io")]
|
||||||
|
[InlineData("protocol")]
|
||||||
|
[InlineData("endofstream")]
|
||||||
|
[InlineData("overflow")]
|
||||||
|
[InlineData("unexpected")]
|
||||||
|
public void Run_WhenPipeClientFails_LogsRedactedExceptionMessage(string exceptionKind)
|
||||||
|
{
|
||||||
|
const string message = "pipe failed password=hunter2 for mxgw_dev_s3cr3t";
|
||||||
|
Exception exception = exceptionKind switch
|
||||||
|
{
|
||||||
|
"io" => new System.IO.IOException(message),
|
||||||
|
"protocol" => new WorkerFrameProtocolException(WorkerFrameProtocolErrorCode.InvalidEnvelope, message),
|
||||||
|
"endofstream" => new WorkerFrameProtocolException(WorkerFrameProtocolErrorCode.EndOfStream, message),
|
||||||
|
"overflow" => new WorkerEventQueueFaultException(WorkerFaultCategory.QueueOverflow, message),
|
||||||
|
_ => new InvalidOperationException(message),
|
||||||
|
};
|
||||||
|
MemoryWorkerLogger logger = new();
|
||||||
|
|
||||||
|
ZB.MOM.WW.MxGateway.Worker.WorkerApplication.Run(
|
||||||
|
ValidArgs(),
|
||||||
|
CreateEnvironment("nonce-secret"),
|
||||||
|
logger,
|
||||||
|
new ThrowingPipeClient(exception));
|
||||||
|
|
||||||
|
MemoryWorkerLogEntry entry = logger.Entries[1];
|
||||||
|
Assert.Equal("Error", entry.Level);
|
||||||
|
string logged = Assert.IsType<string>(entry.Fields["exception_message"]);
|
||||||
|
Assert.Contains("pipe failed", logged);
|
||||||
|
Assert.Contains("password=[redacted]", logged);
|
||||||
|
Assert.DoesNotContain("hunter2", logged);
|
||||||
|
Assert.DoesNotContain("s3cr3t", logged);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -44,4 +44,35 @@ public sealed class WorkerLogRedactorTests
|
|||||||
Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("password_value", "secret"));
|
Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("password_value", "secret"));
|
||||||
Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("secured_write_token", "secret"));
|
Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("secured_write_token", "secret"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies <see cref="WorkerLogRedactor.RedactMessage"/> scrubs credential values and API key
|
||||||
|
/// tokens out of free text while keeping the rest of the message.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void RedactMessage_ScrubsSensitiveAssignmentsAndApiKeyTokens()
|
||||||
|
{
|
||||||
|
string redacted = WorkerLogRedactor.RedactMessage(
|
||||||
|
"Login failed: Password=\"hunter 2\", api_key: abc123; token=xyz for mxgw_dev_s3cr3t on tag Area1.Tank");
|
||||||
|
|
||||||
|
Assert.DoesNotContain("hunter", redacted);
|
||||||
|
Assert.DoesNotContain("abc123", redacted);
|
||||||
|
Assert.DoesNotContain("xyz", redacted);
|
||||||
|
Assert.DoesNotContain("s3cr3t", redacted);
|
||||||
|
Assert.Contains("Password=[redacted]", redacted);
|
||||||
|
Assert.Contains("api_key: [redacted]", redacted);
|
||||||
|
Assert.Contains("Area1.Tank", redacted);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies messages are flattened to one line, capped, and null-safe.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void RedactMessage_FlattensLineBreaksCapsLengthAndHandlesNull()
|
||||||
|
{
|
||||||
|
Assert.Equal(string.Empty, WorkerLogRedactor.RedactMessage(null));
|
||||||
|
Assert.Equal("a b c", WorkerLogRedactor.RedactMessage("a\r\nb\nc"));
|
||||||
|
|
||||||
|
string capped = WorkerLogRedactor.RedactMessage(new string('x', WorkerLogRedactor.MaxMessageLength + 100));
|
||||||
|
Assert.StartsWith(new string('x', WorkerLogRedactor.MaxMessageLength), capped);
|
||||||
|
Assert.EndsWith("...[truncated]", capped);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ using System.Runtime.InteropServices;
|
|||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||||
|
using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
||||||
using ZB.MOM.WW.MxGateway.Worker.Sta;
|
using ZB.MOM.WW.MxGateway.Worker.Sta;
|
||||||
|
using ZB.MOM.WW.MxGateway.Worker.Tests.Bootstrap;
|
||||||
using ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
|
using ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.MxGateway.Worker.Tests.Sta;
|
namespace ZB.MOM.WW.MxGateway.Worker.Tests.Sta;
|
||||||
@@ -191,6 +193,85 @@ public sealed class StaCommandDispatcherTests
|
|||||||
await Task.WhenAll(current, pending);
|
await Task.WhenAll(current, pending);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// With a command logger set, each STA command logs a start line and an end line carrying its
|
||||||
|
/// correlation id, method, status summary and latency (mxaccessgw#184).
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_WithCommandLogger_LogsStartAndEndWithCorrelationIdAndLatency()
|
||||||
|
{
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
runtime.Start();
|
||||||
|
MemoryWorkerLogger logger = new();
|
||||||
|
StaCommandDispatcher dispatcher = new(runtime, new RecordingCommandExecutor())
|
||||||
|
{
|
||||||
|
CommandLogger = logger,
|
||||||
|
};
|
||||||
|
|
||||||
|
await dispatcher.DispatchAsync(CreateCommand("correlation-1", MxCommandKind.Register));
|
||||||
|
|
||||||
|
Assert.Equal(2, logger.Entries.Count);
|
||||||
|
MemoryWorkerLogEntry started = logger.Entries[0];
|
||||||
|
Assert.Equal("WorkerCommandStarted", started.EventName);
|
||||||
|
Assert.Equal("Information", started.Level);
|
||||||
|
Assert.Equal("correlation-1", started.Fields["correlation_id"]);
|
||||||
|
Assert.Equal("Register", started.Fields["command_method"]);
|
||||||
|
Assert.Equal("session-1", started.Fields["session_id"]);
|
||||||
|
MemoryWorkerLogEntry completed = logger.Entries[1];
|
||||||
|
Assert.Equal("WorkerCommandCompleted", completed.EventName);
|
||||||
|
Assert.Equal("correlation-1", completed.Fields["correlation_id"]);
|
||||||
|
Assert.Equal("Register", completed.Fields["command_method"]);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok.ToString(), completed.Fields["protocol_status"]);
|
||||||
|
double elapsed = Assert.IsType<double>(completed.Fields["elapsed_ms"]);
|
||||||
|
Assert.True(elapsed >= 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The end line of a failed command carries the status summary and HRESULT but never the
|
||||||
|
/// exception's message, which can echo provider detail.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_WithCommandLogger_WhenExecutorThrows_LogsStatusWithoutMessage()
|
||||||
|
{
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
runtime.Start();
|
||||||
|
MemoryWorkerLogger logger = new();
|
||||||
|
StaCommandDispatcher dispatcher = new(
|
||||||
|
runtime,
|
||||||
|
new ThrowingCommandExecutor(new COMException("provider detail", unchecked((int)0x80070057))))
|
||||||
|
{
|
||||||
|
CommandLogger = logger,
|
||||||
|
};
|
||||||
|
|
||||||
|
await dispatcher.DispatchAsync(CreateCommand("correlation-1", MxCommandKind.Register));
|
||||||
|
|
||||||
|
MemoryWorkerLogEntry completed = Assert.Single(logger.Entries, entry => entry.EventName == "WorkerCommandCompleted");
|
||||||
|
Assert.Equal(ProtocolStatusCode.MxaccessFailure.ToString(), completed.Fields["protocol_status"]);
|
||||||
|
Assert.Equal("0x80070057", completed.Fields["hresult"]);
|
||||||
|
Assert.DoesNotContain(
|
||||||
|
logger.Entries.SelectMany(entry => entry.Fields.Values),
|
||||||
|
value => value?.ToString()?.Contains("provider detail") == true);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies the opt-in environment value is parsed strictly.</summary>
|
||||||
|
/// <param name="value">Raw environment value.</param>
|
||||||
|
/// <param name="expected">Whether logging should be enabled.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData(null, false)]
|
||||||
|
[InlineData("", false)]
|
||||||
|
[InlineData("0", false)]
|
||||||
|
[InlineData("false", false)]
|
||||||
|
[InlineData("yes", false)]
|
||||||
|
[InlineData("1", true)]
|
||||||
|
[InlineData("true", true)]
|
||||||
|
[InlineData(" TRUE ", true)]
|
||||||
|
public void WorkerCommandLogging_IsEnabledValue_AcceptsOnlyOneOrTrue(string? value, bool expected)
|
||||||
|
{
|
||||||
|
Assert.Equal(expected, WorkerCommandLogging.IsEnabledValue(value));
|
||||||
|
}
|
||||||
|
|
||||||
private static StaCommand CreateCommand(
|
private static StaCommand CreateCommand(
|
||||||
string correlationId,
|
string correlationId,
|
||||||
MxCommandKind kind,
|
MxCommandKind kind,
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
using System;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gate for the per-command <c>WorkerCommandStarted</c> / <c>WorkerCommandCompleted</c> log
|
||||||
|
/// lines. <see cref="IWorkerLogger"/> has no Debug level, and a pair of lines for every STA
|
||||||
|
/// command is too much for the default log, so the lines are opt-in through an environment
|
||||||
|
/// variable. The gateway's launcher does not set it; the worker inherits it from the gateway
|
||||||
|
/// process environment, so setting it on the gateway service turns the lines on for every worker
|
||||||
|
/// that service launches.
|
||||||
|
/// </summary>
|
||||||
|
public static class WorkerCommandLogging
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Environment variable that enables per-command logging when set to <c>1</c> or
|
||||||
|
/// <c>true</c> (case-insensitive). Any other value, or none, leaves it off.
|
||||||
|
/// </summary>
|
||||||
|
public const string EnvironmentVariable = "MXGATEWAY_WORKER_LOG_COMMANDS";
|
||||||
|
|
||||||
|
/// <summary>Gets whether per-command logging is enabled for this process.</summary>
|
||||||
|
/// <returns><c>true</c> when <see cref="EnvironmentVariable"/> enables it.</returns>
|
||||||
|
public static bool IsEnabled()
|
||||||
|
{
|
||||||
|
return IsEnabledValue(Environment.GetEnvironmentVariable(EnvironmentVariable));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Interprets a raw <see cref="EnvironmentVariable"/> value.</summary>
|
||||||
|
/// <param name="value">The variable's value, or <c>null</c> when unset.</param>
|
||||||
|
/// <returns><c>true</c> for <c>1</c> or <c>true</c>; otherwise <c>false</c>.</returns>
|
||||||
|
public static bool IsEnabledValue(string? value)
|
||||||
|
{
|
||||||
|
string trimmed = value?.Trim() ?? string.Empty;
|
||||||
|
return trimmed == "1" || string.Equals(trimmed, "true", StringComparison.OrdinalIgnoreCase);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
|
using System.Text.RegularExpressions;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
||||||
|
|
||||||
@@ -24,6 +25,50 @@ public static class WorkerLogRedactor
|
|||||||
"api_key",
|
"api_key",
|
||||||
];
|
];
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Longest exception message <see cref="RedactMessage"/> passes through; the rest is cut and
|
||||||
|
/// marked, so one runaway message cannot flood the worker's stderr log.
|
||||||
|
/// </summary>
|
||||||
|
public const int MaxMessageLength = 512;
|
||||||
|
|
||||||
|
// A "name=value" or "name: value" pair whose name contains a sensitive part, e.g.
|
||||||
|
// "password=hunter2" or "ApiKey: mxgw_k_s". The name is kept; the value is replaced.
|
||||||
|
private static readonly Regex SensitiveAssignmentPattern = new(
|
||||||
|
@"(?<name>[\w.-]*(?:nonce|secret|password|token|credential|apikey|api_key)[\w.-]*)(?<sep>\s*[=:]\s*)(?<value>""[^""]*""|'[^']*'|[^\s,;]+)",
|
||||||
|
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Compiled);
|
||||||
|
|
||||||
|
// A gateway API key token (mxgw_<key-id>_<secret>) anywhere in the text.
|
||||||
|
private static readonly Regex ApiKeyTokenPattern = new(
|
||||||
|
@"mxgw_[A-Za-z0-9_-]+",
|
||||||
|
RegexOptions.CultureInvariant | RegexOptions.Compiled);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Makes an exception message safe to log. Field-name redaction cannot see inside a free-text
|
||||||
|
/// message, so this scrubs the content: the value of any <c>name=value</c> or
|
||||||
|
/// <c>name: value</c> pair whose name looks sensitive, and any gateway API key token. Line
|
||||||
|
/// breaks are flattened so the message stays on one log line, and the result is capped at
|
||||||
|
/// <see cref="MaxMessageLength"/> characters.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="message">Exception message to scrub; <c>null</c> yields an empty string.</param>
|
||||||
|
/// <returns>The scrubbed message.</returns>
|
||||||
|
public static string RedactMessage(string? message)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrEmpty(message))
|
||||||
|
{
|
||||||
|
return string.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
string redacted = SensitiveAssignmentPattern.Replace(
|
||||||
|
message,
|
||||||
|
match => match.Groups["name"].Value + match.Groups["sep"].Value + RedactedValue);
|
||||||
|
redacted = ApiKeyTokenPattern.Replace(redacted, RedactedValue);
|
||||||
|
redacted = redacted.Replace("\r\n", " ").Replace('\r', ' ').Replace('\n', ' ');
|
||||||
|
|
||||||
|
return redacted.Length <= MaxMessageLength
|
||||||
|
? redacted
|
||||||
|
: redacted.Substring(0, MaxMessageLength) + "...[truncated]";
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Redacts sensitive field values from a log field dictionary.
|
/// Redacts sensitive field values from a log field dictionary.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ public sealed class WorkerPipeSession
|
|||||||
options,
|
options,
|
||||||
() => Process.GetCurrentProcess().Id,
|
() => Process.GetCurrentProcess().Id,
|
||||||
new WorkerPipeSessionOptions(),
|
new WorkerPipeSessionOptions(),
|
||||||
() => new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump)),
|
() => CreateDefaultRuntimeSession(WorkerCommandLogging.IsEnabled() ? logger : null),
|
||||||
logger,
|
logger,
|
||||||
stream)
|
stream)
|
||||||
{
|
{
|
||||||
@@ -98,7 +98,7 @@ public sealed class WorkerPipeSession
|
|||||||
options,
|
options,
|
||||||
processIdProvider,
|
processIdProvider,
|
||||||
new WorkerPipeSessionOptions(),
|
new WorkerPipeSessionOptions(),
|
||||||
() => new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump)),
|
() => CreateDefaultRuntimeSession(commandLogger: null),
|
||||||
logger: null)
|
logger: null)
|
||||||
{
|
{
|
||||||
}
|
}
|
||||||
@@ -138,6 +138,19 @@ public sealed class WorkerPipeSession
|
|||||||
_sessionOptions.Validate();
|
_sessionOptions.Validate();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Creates the production runtime session: MXAccess on its own STA with the wnwrap alarm
|
||||||
|
/// consumer. <paramref name="commandLogger"/> receives the per-command start/end lines when
|
||||||
|
/// non-null; the caller decides that through <see cref="WorkerCommandLogging"/>.
|
||||||
|
/// </summary>
|
||||||
|
private static MxAccessStaSession CreateDefaultRuntimeSession(IWorkerLogger? commandLogger)
|
||||||
|
{
|
||||||
|
return new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump))
|
||||||
|
{
|
||||||
|
CommandLogger = commandLogger,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Runs the worker session, completing the handshake and processing messages until cancellation.</summary>
|
/// <summary>Runs the worker session, completing the handshake and processing messages until cancellation.</summary>
|
||||||
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
|
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
|
||||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
|||||||
@@ -251,6 +251,14 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
internal Action<string>? AlarmPollWarningSink { get; set; }
|
internal Action<string>? AlarmPollWarningSink { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Logger handed to the STA command dispatcher for the per-command start/end lines, or
|
||||||
|
/// <c>null</c> to leave them off (see <see cref="Bootstrap.WorkerCommandLogging"/>). Set
|
||||||
|
/// before <see cref="StartAsync(string, int, CancellationToken)"/>, which creates the
|
||||||
|
/// dispatcher.
|
||||||
|
/// </summary>
|
||||||
|
internal Bootstrap.IWorkerLogger? CommandLogger { get; set; }
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Resolves the write-completion wait from the launcher-provided
|
/// Resolves the write-completion wait from the launcher-provided
|
||||||
/// environment variable; a missing or invalid value falls back to
|
/// environment variable; a missing or invalid value falls back to
|
||||||
@@ -363,7 +371,10 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
// closes over staRuntime so it always pumps the pump
|
// closes over staRuntime so it always pumps the pump
|
||||||
// tied to the apartment that owns this session.
|
// tied to the apartment that owns this session.
|
||||||
pumpStep: () => staRuntime.PumpPendingMessages(),
|
pumpStep: () => staRuntime.PumpPendingMessages(),
|
||||||
writeCompletionTimeout: WriteCompletionTimeout));
|
writeCompletionTimeout: WriteCompletionTimeout))
|
||||||
|
{
|
||||||
|
CommandLogger = this.CommandLogger,
|
||||||
|
};
|
||||||
|
|
||||||
return session.CreateWorkerReady(workerProcessId);
|
return session.CreateWorkerReady(workerProcessId);
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
|
using System.Diagnostics;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||||
|
using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
||||||
using ZB.MOM.WW.MxGateway.Worker.Conversion;
|
using ZB.MOM.WW.MxGateway.Worker.Conversion;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.MxGateway.Worker.Sta;
|
namespace ZB.MOM.WW.MxGateway.Worker.Sta;
|
||||||
@@ -72,6 +74,14 @@ public sealed class StaCommandDispatcher
|
|||||||
this.maxPendingCommands = maxPendingCommands;
|
this.maxPendingCommands = maxPendingCommands;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the logger for the per-command <c>WorkerCommandStarted</c> /
|
||||||
|
/// <c>WorkerCommandCompleted</c> lines. <c>null</c> (the default) turns them off: they are one
|
||||||
|
/// pair per STA command, so they are opt-in rather than always-on (see
|
||||||
|
/// <see cref="WorkerCommandLogging"/>). Set it before commands are dispatched.
|
||||||
|
/// </summary>
|
||||||
|
public IWorkerLogger? CommandLogger { get; set; }
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Gets the count of pending commands in the queue.
|
/// Gets the count of pending commands in the queue.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
@@ -263,22 +273,90 @@ public sealed class StaCommandDispatcher
|
|||||||
}
|
}
|
||||||
|
|
||||||
SetCurrentCommand(command.CorrelationId);
|
SetCurrentCommand(command.CorrelationId);
|
||||||
|
IWorkerLogger? logger = CommandLogger;
|
||||||
|
Stopwatch? stopwatch = null;
|
||||||
|
if (logger is not null)
|
||||||
|
{
|
||||||
|
LogCommandStarted(logger, command);
|
||||||
|
stopwatch = Stopwatch.StartNew();
|
||||||
|
}
|
||||||
|
|
||||||
|
MxCommandReply completedReply;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
MxCommandReply reply = await staRuntime
|
MxCommandReply reply = await staRuntime
|
||||||
.InvokeAsync(() => commandExecutor.Execute(command))
|
.InvokeAsync(() => commandExecutor.Execute(command))
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
queuedCommand.Complete(NormalizeReply(command, reply));
|
completedReply = NormalizeReply(command, reply);
|
||||||
}
|
}
|
||||||
catch (Exception exception)
|
catch (Exception exception)
|
||||||
{
|
{
|
||||||
queuedCommand.Complete(CreateExceptionReply(command, exception));
|
completedReply = CreateExceptionReply(command, exception);
|
||||||
}
|
}
|
||||||
finally
|
finally
|
||||||
{
|
{
|
||||||
SetCurrentCommand(string.Empty);
|
SetCurrentCommand(string.Empty);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (logger is not null && stopwatch is not null)
|
||||||
|
{
|
||||||
|
LogCommandCompleted(logger, command, completedReply, stopwatch.Elapsed);
|
||||||
|
}
|
||||||
|
|
||||||
|
queuedCommand.Complete(completedReply);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Identity and timing only: never the command payload, whose values (and WriteSecured
|
||||||
|
// credentials) must not reach the log.
|
||||||
|
private static void LogCommandStarted(IWorkerLogger logger, StaCommand command)
|
||||||
|
{
|
||||||
|
Dictionary<string, object?> fields = new()
|
||||||
|
{
|
||||||
|
["session_id"] = command.SessionId,
|
||||||
|
["correlation_id"] = command.CorrelationId,
|
||||||
|
["command_method"] = command.MethodName,
|
||||||
|
};
|
||||||
|
if (command.EnqueueTimestamp is not null)
|
||||||
|
{
|
||||||
|
double queuedMilliseconds = (DateTime.UtcNow - command.EnqueueTimestamp.ToDateTime()).TotalMilliseconds;
|
||||||
|
fields["queued_ms"] = Math.Max(0, Math.Round(queuedMilliseconds, 3));
|
||||||
|
}
|
||||||
|
|
||||||
|
TryLog(() => logger.Information("WorkerCommandStarted", fields));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Status summary only: the protocol status code and HRESULT, not the diagnostic message,
|
||||||
|
// which can echo provider detail.
|
||||||
|
private static void LogCommandCompleted(
|
||||||
|
IWorkerLogger logger,
|
||||||
|
StaCommand command,
|
||||||
|
MxCommandReply reply,
|
||||||
|
TimeSpan elapsed)
|
||||||
|
{
|
||||||
|
Dictionary<string, object?> fields = new()
|
||||||
|
{
|
||||||
|
["session_id"] = command.SessionId,
|
||||||
|
["correlation_id"] = command.CorrelationId,
|
||||||
|
["command_method"] = command.MethodName,
|
||||||
|
["protocol_status"] = (reply.ProtocolStatus?.Code ?? ProtocolStatusCode.Unspecified).ToString(),
|
||||||
|
["hresult"] = reply.HasHresult ? $"0x{reply.Hresult:X8}" : string.Empty,
|
||||||
|
["elapsed_ms"] = Math.Round(elapsed.TotalMilliseconds, 3),
|
||||||
|
};
|
||||||
|
|
||||||
|
TryLog(() => logger.Information("WorkerCommandCompleted", fields));
|
||||||
|
}
|
||||||
|
|
||||||
|
// A diagnostics line must never fail the command it describes.
|
||||||
|
private static void TryLog(Action log)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
log();
|
||||||
|
}
|
||||||
|
catch
|
||||||
|
{
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private void SetCurrentCommand(string correlationId)
|
private void SetCurrentCommand(string correlationId)
|
||||||
|
|||||||
@@ -40,10 +40,14 @@ public static class WorkerApplication
|
|||||||
/// CLR's crash code.
|
/// CLR's crash code.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// Logs the exception type only — never its message or stack, which
|
/// Logs the exception type only — never its message or stack. An
|
||||||
/// can carry tag values or credentials passed through a command — the
|
/// exception reaching this hook can come from any thread, including a
|
||||||
/// same rule every other worker failure line follows. A logging
|
/// COM callback or a command path whose message can carry tag values
|
||||||
/// failure is swallowed so the exit always happens.
|
/// or credentials, so it gets none of the content that
|
||||||
|
/// <see cref="Run(string[], IWorkerEnvironment, IWorkerLogger, IWorkerPipeClient)"/>'s
|
||||||
|
/// own catch blocks log through <see cref="WorkerLogRedactor.RedactMessage"/>
|
||||||
|
/// (those see only pipe-session failures). A logging failure is
|
||||||
|
/// swallowed so the exit always happens.
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
/// <param name="logger">Worker logger.</param>
|
/// <param name="logger">Worker logger.</param>
|
||||||
/// <param name="exceptionObject">The unhandled exception object (not necessarily an <see cref="Exception"/>).</param>
|
/// <param name="exceptionObject">The unhandled exception object (not necessarily an <see cref="Exception"/>).</param>
|
||||||
@@ -171,6 +175,7 @@ public static class WorkerApplication
|
|||||||
["exit_code"] = WorkerExitCode.PipeConnectionFailed,
|
["exit_code"] = WorkerExitCode.PipeConnectionFailed,
|
||||||
["error_code"] = exception.ErrorCode,
|
["error_code"] = exception.ErrorCode,
|
||||||
["exception_type"] = exception.GetType().FullName,
|
["exception_type"] = exception.GetType().FullName,
|
||||||
|
["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message),
|
||||||
});
|
});
|
||||||
|
|
||||||
return (int)WorkerExitCode.PipeConnectionFailed;
|
return (int)WorkerExitCode.PipeConnectionFailed;
|
||||||
@@ -182,6 +187,7 @@ public static class WorkerApplication
|
|||||||
["exit_code"] = WorkerExitCode.ProtocolViolation,
|
["exit_code"] = WorkerExitCode.ProtocolViolation,
|
||||||
["error_code"] = exception.ErrorCode,
|
["error_code"] = exception.ErrorCode,
|
||||||
["exception_type"] = exception.GetType().FullName,
|
["exception_type"] = exception.GetType().FullName,
|
||||||
|
["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message),
|
||||||
});
|
});
|
||||||
|
|
||||||
return (int)WorkerExitCode.ProtocolViolation;
|
return (int)WorkerExitCode.ProtocolViolation;
|
||||||
@@ -193,6 +199,7 @@ public static class WorkerApplication
|
|||||||
{
|
{
|
||||||
["exit_code"] = WorkerExitCode.EventQueueOverflow,
|
["exit_code"] = WorkerExitCode.EventQueueOverflow,
|
||||||
["exception_type"] = exception.GetType().FullName,
|
["exception_type"] = exception.GetType().FullName,
|
||||||
|
["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message),
|
||||||
});
|
});
|
||||||
|
|
||||||
return (int)WorkerExitCode.EventQueueOverflow;
|
return (int)WorkerExitCode.EventQueueOverflow;
|
||||||
@@ -203,6 +210,7 @@ public static class WorkerApplication
|
|||||||
{
|
{
|
||||||
["exit_code"] = WorkerExitCode.PipeConnectionFailed,
|
["exit_code"] = WorkerExitCode.PipeConnectionFailed,
|
||||||
["exception_type"] = exception.GetType().FullName,
|
["exception_type"] = exception.GetType().FullName,
|
||||||
|
["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message),
|
||||||
});
|
});
|
||||||
|
|
||||||
return (int)WorkerExitCode.PipeConnectionFailed;
|
return (int)WorkerExitCode.PipeConnectionFailed;
|
||||||
@@ -213,6 +221,7 @@ public static class WorkerApplication
|
|||||||
{
|
{
|
||||||
["exit_code"] = WorkerExitCode.UnexpectedFailure,
|
["exit_code"] = WorkerExitCode.UnexpectedFailure,
|
||||||
["exception_type"] = exception.GetType().FullName,
|
["exception_type"] = exception.GetType().FullName,
|
||||||
|
["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message),
|
||||||
});
|
});
|
||||||
|
|
||||||
return (int)WorkerExitCode.UnexpectedFailure;
|
return (int)WorkerExitCode.UnexpectedFailure;
|
||||||
|
|||||||
Reference in New Issue
Block a user