fix(CLI-40): scrub the credential from the redacted error's structured reply, route MXACCESS_FAILURE to MxAccess (Rust), fix Go Subscribe terminal-error drop

Code-review follow-up on the CLI-40/41/44 branch.

ISSUE 1 (all five, critical): the message-only scrub still leaked the
server-echoed credential through the redacted error's structured reply accessor
(.NET Reply/Statuses, Java reply()/protocolStatus(), Go MxAccessError.Reply via
errors.As, Rust reply()/into_reply(), Python raw_reply). The redacted error now
carries a scrubbed clone of the reply (protocol_status.message,
diagnostic_message, statuses[].diagnostic_text), with per-language tests asserting
the reply accessor no longer contains the credential.

ISSUE 2 (Rust, critical): ensure_command_success routed MXACCESS_FAILURE to
Error::Command (unlike the other four clients), bypassing attach_secrets and
leaking via derived Debug/Display. MXACCESS_FAILURE now routes to Error::MxAccess,
fixing the cross-client inconsistency.

ISSUE 3 (Go, important): the CLI-44 terminal send was unconditionally
non-blocking, dropping a genuine terminal error under a full buffer on the
never-drop SubscribeEvents path. It is now reserved-slot-non-blocking only for the
cancel-on-overflow path and blocking for the never-drop path.

New shared fixture authenticate-user.echoed-credential-mxaccess-failure.reply.json
wired into all five suites. Minors: whitespace-secret guard on .NET/Java redact
helpers; Java preserves exception subtype on redaction; redaction-helper unit
tests (Go/Java/.NET). Docs (ClientBehaviorFixtures.md, ClientLibrariesDesign.md)
updated to make the structured-field claim true.
This commit is contained in:
Joseph Doherty
2026-08-07 07:04:56 -04:00
parent dc7fd16dd5
commit 0d874f91ee
25 changed files with 1190 additions and 88 deletions
File diff suppressed because one or more lines are too long
@@ -0,0 +1,96 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Unit tests for <see cref="MxGatewaySecretRedaction"/> — the exact-substring scrub applied to
/// diagnostic text and rebuilt exceptions before they leave the client on a failure path.
/// </summary>
public sealed class MxGatewaySecretRedactionTests
{
[Fact]
public void Redact_ReplacesEveryOccurrenceOfSecret()
{
string result = MxGatewaySecretRedaction.Redact(
"pw=hunter2 retry pw=hunter2 again hunter2",
"hunter2");
Assert.DoesNotContain("hunter2", result, StringComparison.Ordinal);
Assert.Equal("pw=<redacted> retry pw=<redacted> again <redacted>", result);
}
[Fact]
public void Redact_ScrubsBothSecretsWhenOneIsSubstringOfTheOther()
{
// "secret" is a substring of "secretPassword"; both must be fully scrubbed regardless of
// supplied order — no residual leak of either verbatim value.
string result = MxGatewaySecretRedaction.Redact(
"a=secretPassword b=secret",
"secret",
"secretPassword");
Assert.DoesNotContain("secretPassword", result, StringComparison.Ordinal);
Assert.DoesNotContain("secret", result, StringComparison.Ordinal);
}
[Fact]
public void Redact_WithNullSecretsArray_ReturnsMessageUnchanged()
{
const string message = "nothing to scrub here";
string result = MxGatewaySecretRedaction.Redact(message, null!);
Assert.Equal(message, result);
}
[Fact]
public void Redact_WithEmptySecretsArray_ReturnsMessageUnchanged()
{
const string message = "nothing to scrub here";
string result = MxGatewaySecretRedaction.Redact(message);
Assert.Equal(message, result);
}
[Fact]
public void Redact_IgnoresWhitespaceOnlySecret()
{
// A whitespace-only secret must not over-redact the internal spaces of the message.
const string message = "user operator logged in";
string result = MxGatewaySecretRedaction.Redact(message, " ");
Assert.Equal(message, result);
}
[Fact]
public void Redacted_PreservesConcreteSubtypeAndDoesNotChainSecretBearingOriginal()
{
const string secret = "hunter2";
Exception transportCause = new InvalidOperationException("transport reset");
MxGatewaySessionException original = new(
$"session rejected credential '{secret}'",
"session-1",
"correlation-1",
new ProtocolStatus { Code = ProtocolStatusCode.SessionNotReady, Message = $"echoed '{secret}'" },
hResult: -1,
statuses: [new MxStatusProxy { DiagnosticText = $"denied '{secret}'" }],
innerException: transportCause);
MxGatewayException redacted = MxGatewaySecretRedaction.Redacted(original, secret);
// Concrete runtime type is preserved.
Assert.IsType<MxGatewaySessionException>(redacted);
// The secret is gone from the message and every structured accessor.
Assert.DoesNotContain(secret, redacted.Message, StringComparison.Ordinal);
Assert.DoesNotContain(secret, redacted.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(secret, redacted.ProtocolStatus!.Message, StringComparison.Ordinal);
Assert.All(redacted.Statuses, status =>
Assert.DoesNotContain(secret, status.DiagnosticText, StringComparison.Ordinal));
Assert.Contains("<redacted>", redacted.Message, StringComparison.Ordinal);
// The secret-bearing original is NOT chained; the original's transport cause is carried.
Assert.NotSame(original, redacted.InnerException);
Assert.Same(transportCause, redacted.InnerException);
}
}
@@ -32,6 +32,44 @@ public sealed class MxGatewaySessionReplyContractTests
Assert.DoesNotContain(password, exception.ToString(), StringComparison.Ordinal);
}
/// <summary>
/// CLI-40: the redacted exception must not leak the echoed credential through any structured
/// accessor either — <see cref="MxAccessException.Reply"/> (protocol message, diagnostic
/// message, and each MXSTATUS_PROXY diagnostic text) and <see cref="MxGatewayException.Statuses"/>
/// all carry the server-echoed credential verbatim before the fix. Both the OK+negative-HRESULT
/// and the MXACCESS_FAILURE reply route to <see cref="MxAccessException"/>, so both must scrub.
/// </summary>
/// <param name="fixture">The echoed-credential reply fixture to drive.</param>
[Theory]
[InlineData("authenticate-user.echoed-credential.reply.json")]
[InlineData("authenticate-user.echoed-credential-mxaccess-failure.reply.json")]
public async Task AuthenticateUserAsync_RedactsEchoedCredentialInStructuredAccessors(string fixture)
{
const string password = "sup3rSecretVerify9f3a2b";
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(ReadReplyFixture(fixture));
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
MxAccessException exception = await Assert.ThrowsAsync<MxAccessException>(
async () => await session.AuthenticateUserAsync(12, "operator", password));
Assert.DoesNotContain(password, exception.Message, StringComparison.Ordinal);
Assert.Contains("<redacted>", exception.Message, StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.Reply.ProtocolStatus.Message, StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.Reply.DiagnosticMessage, StringComparison.Ordinal);
foreach (MxStatusProxy status in exception.Reply.Statuses)
{
Assert.DoesNotContain(password, status.DiagnosticText, StringComparison.Ordinal);
}
foreach (MxStatusProxy status in exception.Statuses)
{
Assert.DoesNotContain(password, status.DiagnosticText, StringComparison.Ordinal);
}
}
/// <summary>
/// CLI-41: an OK reply that carries neither the typed AuthenticateUser payload nor an
/// int32 return_value is a malformed reply, surfaced as a typed exception rather than an NRE.
@@ -1,3 +1,5 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client;
/// <summary>
@@ -13,9 +15,10 @@ internal static class MxGatewaySecretRedaction
private const string Marker = "<redacted>";
/// <summary>
/// Replaces every non-null, non-empty secret in <paramref name="secrets"/> with the
/// redaction marker (ordinal comparison). Returns the message unchanged when it is null or
/// empty, or when no usable secret is supplied.
/// Replaces every usable secret in <paramref name="secrets"/> with the redaction marker
/// (ordinal comparison). Returns the message unchanged when it is null or empty, or when no
/// usable secret is supplied. A secret that is null, empty, or whitespace-only is ignored so
/// it cannot over-redact ordinary separator characters in the message.
/// </summary>
/// <param name="message">The diagnostic message to scrub.</param>
/// <param name="secrets">The secret values to remove from the message.</param>
@@ -30,7 +33,7 @@ internal static class MxGatewaySecretRedaction
string result = message;
foreach (string? secret in secrets)
{
if (!string.IsNullOrEmpty(secret))
if (!string.IsNullOrWhiteSpace(secret))
{
result = result.Replace(secret, Marker, StringComparison.Ordinal);
}
@@ -39,6 +42,81 @@ internal static class MxGatewaySecretRedaction
return result;
}
/// <summary>
/// Returns a scrubbed clone of <paramref name="reply"/>: the protocol-status message, the
/// reply-level diagnostic message, and each MXSTATUS_PROXY diagnostic text have every verbatim
/// secret replaced with the redaction marker. The original is left untouched. MXAccess can echo
/// a submitted credential into any of these fields, so a redacted exception must carry the
/// scrubbed reply rather than the secret-bearing original.
/// </summary>
/// <param name="reply">The reply to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A scrubbed clone of the reply.</returns>
internal static MxCommandReply RedactReply(MxCommandReply reply, params string?[] secrets)
{
ArgumentNullException.ThrowIfNull(reply);
MxCommandReply clone = reply.Clone();
if (clone.ProtocolStatus is not null)
{
clone.ProtocolStatus.Message = Redact(clone.ProtocolStatus.Message, secrets);
}
clone.DiagnosticMessage = Redact(clone.DiagnosticMessage, secrets);
foreach (MxStatusProxy status in clone.Statuses)
{
status.DiagnosticText = Redact(status.DiagnosticText, secrets);
}
return clone;
}
/// <summary>
/// Returns a scrubbed clone of <paramref name="status"/> (its message with every verbatim
/// secret removed), or <see langword="null"/> when the input is null.
/// </summary>
/// <param name="status">The protocol status to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A scrubbed clone, or <see langword="null"/>.</returns>
internal static ProtocolStatus? RedactStatus(ProtocolStatus? status, params string?[] secrets)
{
if (status is null)
{
return null;
}
ProtocolStatus clone = status.Clone();
clone.Message = Redact(clone.Message, secrets);
return clone;
}
/// <summary>
/// Returns a list of scrubbed clones of <paramref name="statuses"/> — each MXSTATUS_PROXY's
/// diagnostic text has every verbatim secret removed. The originals are left untouched.
/// </summary>
/// <param name="statuses">The statuses to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A list of scrubbed clones.</returns>
internal static IReadOnlyList<MxStatusProxy> RedactStatuses(
IReadOnlyList<MxStatusProxy> statuses,
params string?[] secrets)
{
if (statuses is null || statuses.Count is 0)
{
return statuses ?? [];
}
MxStatusProxy[] result = new MxStatusProxy[statuses.Count];
for (int i = 0; i < statuses.Count; i++)
{
MxStatusProxy clone = statuses[i].Clone();
clone.DiagnosticText = Redact(clone.DiagnosticText, secrets);
result[i] = clone;
}
return result;
}
/// <summary>
/// Returns an exception equivalent to <paramref name="ex"/> but with any verbatim secret
/// scrubbed from its message. When nothing changes, the original exception is returned
@@ -57,28 +135,95 @@ internal static class MxGatewaySecretRedaction
ArgumentNullException.ThrowIfNull(ex);
string redacted = Redact(ex.Message, secrets);
if (string.Equals(redacted, ex.Message, StringComparison.Ordinal))
bool messageChanged = !string.Equals(redacted, ex.Message, StringComparison.Ordinal);
Exception? cause = ex.InnerException;
// MxAccessException derives its structured fields from the raw reply, so scrubbing must
// clone and redact that reply — the message alone changing is not enough, because the reply
// can carry the echoed secret even when the message does not.
if (ex is MxAccessException access)
{
if (!messageChanged && !ReplyContainsSecret(access.Reply, secrets))
{
return ex;
}
return new MxAccessException(redacted, RedactReply(access.Reply, secrets), cause);
}
// Other subtypes carry the secret through ProtocolStatus.Message and Statuses[].DiagnosticText.
if (!messageChanged
&& !ContainsSecret(ex.ProtocolStatus?.Message, secrets)
&& !StatusesContainSecret(ex.Statuses, secrets))
{
return ex;
}
Exception? cause = ex.InnerException;
ProtocolStatus? status = RedactStatus(ex.ProtocolStatus, secrets);
IReadOnlyList<MxStatusProxy> statuses = RedactStatuses(ex.Statuses, secrets);
return ex switch
{
MxAccessException access => new MxAccessException(redacted, access.Reply, cause),
MxGatewaySessionException => new MxGatewaySessionException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayWorkerException => new MxGatewayWorkerException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayAuthenticationException => new MxGatewayAuthenticationException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayAuthorizationException => new MxGatewayAuthorizationException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayMalformedReplyException => new MxGatewayMalformedReplyException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayCommandException => new MxGatewayCommandException(
redacted, ex.SessionId, ex.CorrelationId, ex.ProtocolStatus, ex.HResultCode, ex.Statuses, cause),
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
_ => new MxGatewayException(redacted, cause),
};
}
private static bool ContainsSecret(string? text, string?[] secrets)
{
if (string.IsNullOrEmpty(text) || secrets is null)
{
return false;
}
foreach (string? secret in secrets)
{
if (!string.IsNullOrWhiteSpace(secret) && text.Contains(secret, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static bool StatusesContainSecret(IReadOnlyList<MxStatusProxy> statuses, string?[] secrets)
{
if (statuses is null)
{
return false;
}
foreach (MxStatusProxy status in statuses)
{
if (ContainsSecret(status.DiagnosticText, secrets))
{
return true;
}
}
return false;
}
private static bool ReplyContainsSecret(MxCommandReply reply, string?[] secrets)
{
if (reply is null)
{
return false;
}
return ContainsSecret(reply.ProtocolStatus?.Message, secrets)
|| ContainsSecret(reply.DiagnosticMessage, secrets)
|| StatusesContainSecret(reply.Statuses, secrets);
}
}
@@ -264,6 +264,74 @@ func TestEventsFullBufferTerminalErrorKeepsRootCause(t *testing.T) {
}
}
// TestSubscribeEventsFullBufferDeliversTerminalError is the CLI-44 regression for
// the never-drop Subscribe path. SubscribeEvents/SubscribeEventsAfter use
// cancelWhenResultBufferFull=false, so ordinary sends are blocking and uncapped and
// can fill every slot in the results channel — including the reserved terminal slot.
// A genuine terminal Recv error must still be delivered as the final result, never
// silently dropped. The server sends eventBufferSize+eventBufferReservedSlots events
// (filling every slot) and then returns a genuine gRPC error; with an unconditional
// non-blocking terminal send the error is dropped, so this fails red until the send
// path blocks for the never-drop mode.
func TestSubscribeEventsFullBufferDeliversTerminalError(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: eventBufferSize + eventBufferReservedSlots,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
subscription, err := session.SubscribeEvents(context.Background())
if err != nil {
t.Fatalf("SubscribeEvents() error = %v", err)
}
defer subscription.Close()
<-fake.streamStarted
// Wait for the server to finish sending every event and return the terminal
// error, so the producer goroutine has filled every buffered slot before the
// terminal result is processed. That is what makes the dropped-terminal bug
// observable: with the buffer full, an unconditional non-blocking send discards
// the terminal error.
select {
case <-fake.streamDone:
case <-time.After(2 * time.Second):
t.Fatal("event stream did not stop after terminal error")
}
time.Sleep(250 * time.Millisecond)
// Drain fully. Every data event, then the terminal gRPC error as the final
// result, must arrive; the channel must not close without yielding it.
events := subscription.Events()
var last EventResult
gotResult := false
for {
select {
case res, ok := <-events:
if !ok {
if !gotResult {
t.Fatal("events channel closed without yielding any result")
}
var gwErr *GatewayError
if !errors.As(last.Err, &gwErr) {
t.Fatalf("final event result err is %T (%v), want the terminal *GatewayError; it was dropped", last.Err, last.Err)
}
if code := status.Code(last.Err); code != codes.Internal {
t.Fatalf("final event result gRPC code = %s, want %s", code, codes.Internal)
}
return
}
last = res
gotResult = true
case <-time.After(2 * time.Second):
t.Fatal("events channel did not close after terminal error")
}
}
}
func TestEventsSurfacesReplayGapSentinelAsTypedSignal(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
@@ -127,3 +127,71 @@ func TestAuthenticateUserScrubsEchoedCredentialFromError(t *testing.T) {
t.Fatalf("surfaced error missing redaction marker: %q", message)
}
}
// TestAuthenticateUserScrubsEchoedCredentialFromStructuredReply is the CLI-40
// follow-up: redacting only the rendered Error() string is not enough. The typed
// *MxAccessError still carries the raw command reply, whose ProtocolStatus.Message,
// DiagnosticMessage, and Statuses[].DiagnosticText echo the credential verbatim. A
// logger dumping structured fields would reintroduce the leak, so the reply the
// typed error carries must be a scrubbed clone. Both the OK+negative-HRESULT and the
// MXACCESS_FAILURE fixtures route to *MxAccessError (via EnsureProtocolSuccess), so
// both must be scrubbed identically.
func TestAuthenticateUserScrubsEchoedCredentialFromStructuredReply(t *testing.T) {
const credential = "sup3rSecretVerify9f3a2b"
fixtures := []string{
"authenticate-user.echoed-credential.reply.json",
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
}
for _, fixture := range fixtures {
t.Run(fixture, func(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, fixture),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
_, err := session.AuthenticateUser(context.Background(), 12, "operator", credential)
if err == nil {
t.Fatal("AuthenticateUser() error = nil, want an MXAccess failure")
}
var mxErr *MxAccessError
if !errors.As(err, &mxErr) {
t.Fatalf("AuthenticateUser() error = %v (%T), want *MxAccessError", err, err)
}
reply := mxErr.Reply
if reply == nil {
t.Fatal("MxAccessError.Reply is nil, want the scrubbed command reply")
}
if got := reply.GetProtocolStatus().GetMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.ProtocolStatus.Message leaked the credential: %q", got)
}
if got := reply.GetDiagnosticMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.DiagnosticMessage leaked the credential: %q", got)
}
for i, status := range reply.GetStatuses() {
if got := status.GetDiagnosticText(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.Statuses[%d].DiagnosticText leaked the credential: %q", i, got)
}
}
// The wrapped CommandError's status/reply must be scrubbed too.
if mxErr.Command != nil {
if got := mxErr.Command.Status.GetMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Command.Status.Message leaked the credential: %q", got)
}
if cmdReply := mxErr.Command.Reply; cmdReply != nil {
if got := cmdReply.GetDiagnosticMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Command.Reply.DiagnosticMessage leaked the credential: %q", got)
}
}
}
if got := err.Error(); strings.Contains(got, credential) {
t.Fatalf("rendered error leaked the credential: %q", got)
}
})
}
}
+95 -6
View File
@@ -6,6 +6,7 @@ import (
"strings"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/protobuf/proto"
)
// redactedSecretMarker is the placeholder substituted for credential material in
@@ -49,20 +50,108 @@ func (e *secretRedactingError) Unwrap() error {
return e.err
}
// redactSecrets wraps err so any occurrence of a non-empty secret in the surfaced
// message is redacted, while errors.As / errors.Is still reach the wrapped typed
// error. It returns nil unchanged and skips wrapping when no non-empty secret is
// supplied, so non-secret-bearing calls keep their original error verbatim.
// scrubReplyStrings returns a clone of reply with every non-empty secret replaced
// by redactedSecretMarker in the free-text fields a gateway diagnostic could echo a
// credential into: ProtocolStatus.Message, DiagnosticMessage, and each
// Statuses[].DiagnosticText. It clones with proto.Clone so the caller's original
// reply is never mutated. A nil reply, or an empty/whitespace-only secret set, is a
// no-op (nil in, nil out; a clone otherwise).
func scrubReplyStrings(reply *pb.MxCommandReply, secrets []string) *pb.MxCommandReply {
if reply == nil {
return nil
}
clone, ok := proto.Clone(reply).(*pb.MxCommandReply)
if !ok {
return reply
}
for _, secret := range secrets {
if secret == "" {
continue
}
if clone.GetProtocolStatus() != nil {
clone.ProtocolStatus.Message = strings.ReplaceAll(clone.GetProtocolStatus().GetMessage(), secret, redactedSecretMarker)
}
clone.DiagnosticMessage = strings.ReplaceAll(clone.GetDiagnosticMessage(), secret, redactedSecretMarker)
for _, status := range clone.GetStatuses() {
status.DiagnosticText = strings.ReplaceAll(status.GetDiagnosticText(), secret, redactedSecretMarker)
}
}
return clone
}
// scrubProtocolStatusMessage returns a clone of status with every non-empty secret
// redacted from its Message, leaving the original untouched.
func scrubProtocolStatusMessage(status *ProtocolStatus, secrets []string) *ProtocolStatus {
if status == nil {
return nil
}
clone, ok := proto.Clone(status).(*ProtocolStatus)
if !ok {
return status
}
for _, secret := range secrets {
if secret != "" {
clone.Message = strings.ReplaceAll(clone.GetMessage(), secret, redactedSecretMarker)
}
}
return clone
}
// redactSecrets scrubs a non-empty secret set from the error it surfaces. When the
// wrapped error is a typed *MxAccessError or *CommandError it is rebuilt carrying
// scrubbed clones of its reply and protocol status, so a caller logging the typed
// error's structured fields cannot reintroduce the credential the rendered message
// hides. The rebuilt (or original, for other error types) value is then wrapped in
// secretRedactingError as a belt-and-suspenders scrub of any remaining rendered
// text. errors.As / errors.Is still reach the typed error through the wrapper. It
// returns nil unchanged and skips all work when no non-empty secret is supplied, so
// non-secret-bearing calls keep their original error verbatim.
func redactSecrets(err error, secrets ...string) error {
if err == nil {
return nil
}
hasSecret := false
for _, secret := range secrets {
if secret != "" {
return &secretRedactingError{err: err, secrets: secrets}
hasSecret = true
break
}
}
return err
if !hasSecret {
return err
}
rebuilt := rebuildScrubbedError(err, secrets)
return &secretRedactingError{err: rebuilt, secrets: secrets}
}
// rebuildScrubbedError rebuilds the typed error carrying scrubbed clones of any
// command reply / protocol status it holds, so credential text never survives in the
// error's structured fields. Non-reply-bearing error types are returned unchanged.
func rebuildScrubbedError(err error, secrets []string) error {
switch typed := err.(type) {
case *MxAccessError:
return &MxAccessError{
Command: scrubCommandError(typed.Command, secrets),
Reply: scrubReplyStrings(typed.Reply, secrets),
}
case *CommandError:
return scrubCommandError(typed, secrets)
default:
return err
}
}
// scrubCommandError rebuilds a CommandError with a scrubbed Status and Reply.
func scrubCommandError(cmd *CommandError, secrets []string) *CommandError {
if cmd == nil {
return nil
}
return &CommandError{
Op: cmd.Op,
Status: scrubProtocolStatusMessage(cmd.Status, secrets),
Reply: scrubReplyStrings(cmd.Reply, secrets),
}
}
// ErrSlowConsumer is the terminal error sent on the Events/EventsAfter
@@ -0,0 +1,115 @@
package mxgateway
import (
"errors"
"strings"
"testing"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
)
// TestScrubReplyStringsRedactsEveryOccurrence covers the multi-occurrence case:
// one secret appearing across ProtocolStatus.Message, DiagnosticMessage, and every
// Statuses[].DiagnosticText must be fully redacted with no residue.
func TestScrubReplyStringsRedactsEveryOccurrence(t *testing.T) {
const secret = "hunter2"
reply := &pb.MxCommandReply{
ProtocolStatus: &pb.ProtocolStatus{Message: "rejected hunter2 then hunter2 again"},
DiagnosticMessage: "echoed hunter2 back",
Statuses: []*pb.MxStatusProxy{
{DiagnosticText: "first hunter2"},
{DiagnosticText: "second hunter2 and hunter2"},
},
}
scrubbed := scrubReplyStrings(reply, []string{secret})
if strings.Contains(scrubbed.GetProtocolStatus().GetMessage(), secret) {
t.Fatalf("ProtocolStatus.Message still contains the secret: %q", scrubbed.GetProtocolStatus().GetMessage())
}
if strings.Contains(scrubbed.GetDiagnosticMessage(), secret) {
t.Fatalf("DiagnosticMessage still contains the secret: %q", scrubbed.GetDiagnosticMessage())
}
for i, status := range scrubbed.GetStatuses() {
if strings.Contains(status.GetDiagnosticText(), secret) {
t.Fatalf("Statuses[%d].DiagnosticText still contains the secret: %q", i, status.GetDiagnosticText())
}
}
if !strings.Contains(scrubbed.GetProtocolStatus().GetMessage(), redactedSecretMarker) {
t.Fatalf("ProtocolStatus.Message missing redaction marker: %q", scrubbed.GetProtocolStatus().GetMessage())
}
// The original reply must be untouched (scrubReplyStrings clones).
if !strings.Contains(reply.GetDiagnosticMessage(), secret) {
t.Fatal("scrubReplyStrings mutated the original reply instead of cloning it")
}
}
// TestScrubReplyStringsRedactsOverlappingSecrets covers two secrets where one is a
// substring of the other: both must be fully redacted, with no partial leak of the
// longer secret's non-shared remainder.
func TestScrubReplyStringsRedactsOverlappingSecrets(t *testing.T) {
const shortSecret = "pass"
const longSecret = "password123"
reply := &pb.MxCommandReply{
DiagnosticMessage: "value was password123 and also pass",
}
scrubbed := scrubReplyStrings(reply, []string{longSecret, shortSecret})
got := scrubbed.GetDiagnosticMessage()
if strings.Contains(got, shortSecret) {
t.Fatalf("scrubbed message still contains a secret substring %q: %q", shortSecret, got)
}
if strings.Contains(got, longSecret) {
t.Fatalf("scrubbed message still contains %q: %q", longSecret, got)
}
// "123" is the longer secret's remainder past the shared "pass" prefix; it must
// not survive as a partial leak.
if strings.Contains(got, "123") {
t.Fatalf("scrubbed message leaked the longer secret's remainder: %q", got)
}
}
// TestRedactSecretsEmptyOrNilLeavesErrorUnchanged confirms the no-secret paths keep
// the original typed error verbatim (no wrapping, no scrubbed clone).
func TestRedactSecretsEmptyOrNilLeavesErrorUnchanged(t *testing.T) {
base := &MxAccessError{Reply: &pb.MxCommandReply{DiagnosticMessage: "boom"}}
if got := redactSecrets(base); got != error(base) {
t.Fatalf("redactSecrets with no secrets = %v, want the original error unchanged", got)
}
if got := redactSecrets(base, ""); got != error(base) {
t.Fatalf("redactSecrets with only an empty secret = %v, want the original error unchanged", got)
}
if got := redactSecrets(nil, "secret"); got != nil {
t.Fatalf("redactSecrets(nil, ...) = %v, want nil", got)
}
}
// TestRedactSecretsRebuildsTypedCommandError confirms a *CommandError (non-MXAccess
// path) is rebuilt with a scrubbed Status and Reply, and errors.As still reaches it.
func TestRedactSecretsRebuildsTypedCommandError(t *testing.T) {
const secret = "topSecretValue"
base := &CommandError{
Op: "write secured",
Status: &pb.ProtocolStatus{Message: "rejected topSecretValue"},
Reply: &pb.MxCommandReply{DiagnosticMessage: "echoed topSecretValue"},
}
redacted := redactSecrets(base, secret)
var cmdErr *CommandError
if !errors.As(redacted, &cmdErr) {
t.Fatalf("redactSecrets result %T does not unwrap to *CommandError", redacted)
}
if strings.Contains(cmdErr.Status.GetMessage(), secret) {
t.Fatalf("CommandError.Status.Message leaked the secret: %q", cmdErr.Status.GetMessage())
}
if strings.Contains(cmdErr.Reply.GetDiagnosticMessage(), secret) {
t.Fatalf("CommandError.Reply.DiagnosticMessage leaked the secret: %q", cmdErr.Reply.GetDiagnosticMessage())
}
if strings.Contains(redacted.Error(), secret) {
t.Fatalf("rendered error leaked the secret: %q", redacted.Error())
}
}
+25 -13
View File
@@ -1073,8 +1073,8 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
// A genuine terminal stream error must be reported as itself, even
// when the data slots are full. Routing it through sendEventResult
// would let the overflow branch substitute ErrSlowConsumer and lose
// the real gRPC status, so send it directly on the reserved slot.
sendTerminalEventResult(results, EventResult{Err: &GatewayError{Op: "stream events", Err: err}})
// the real gRPC status, so send it directly, bypassing that branch.
sendTerminalEventResult(streamCtx, results, EventResult{Err: &GatewayError{Op: "stream events", Err: err}}, cancelWhenResultBufferFull)
return
}
}()
@@ -1093,20 +1093,32 @@ func ensureBulkSize(name string, length int) error {
return nil
}
// sendTerminalEventResult enqueues a terminal EventResult with a non-blocking
// send. The eventBufferReservedSlots reserve (beyond the eventBufferSize data
// slots) guarantees the send lands unless a terminal result was already
// enqueued; because this goroutine is the sole producer, at most one terminal
// send ever races for the reserved slot, so the select default only fires when
// the reserve is already spent — never dropping a first terminal error.
// sendTerminalEventResult enqueues a terminal EventResult, bypassing
// sendEventResult's overflow branch so a genuine stream error is reported verbatim
// rather than relabeled as ErrSlowConsumer. How it sends depends on the mode:
//
// Unlike sendEventResult, this bypasses the overflow branch: a genuine terminal
// stream error is reported verbatim even when the data slots are full, rather
// than being relabeled as ErrSlowConsumer.
func sendTerminalEventResult(results chan<- EventResult, result EventResult) {
// - cancelWhenBufferFull=true (Events/EventsAfter): ordinary data sends are capped
// at eventBufferSize, leaving eventBufferReservedSlots free, so a non-blocking
// send always lands the terminal result. Because this goroutine is the sole
// producer, at most one terminal send ever races for the reserved slot, so the
// select default only fires when the reserve is already spent — never dropping a
// first terminal error.
// - cancelWhenBufferFull=false (SubscribeEvents/SubscribeEventsAfter, never-drop):
// ordinary data sends are uncapped and blocking, so every slot including the
// reserve can hold data. A non-blocking send would then hit the full buffer and
// silently drop the terminal error, breaking the never-drop contract; instead
// block until the consumer drains a slot (or the stream context is cancelled).
func sendTerminalEventResult(ctx context.Context, results chan<- EventResult, result EventResult, cancelWhenBufferFull bool) {
if cancelWhenBufferFull {
select {
case results <- result:
default:
}
return
}
select {
case results <- result:
default:
case <-ctx.Done():
}
}
@@ -63,9 +63,9 @@ public final class MxGatewaySecrets {
* into a free-form failure message.
*
* @param message the message to scrub, may be {@code null}
* @param secrets the exact secret substrings to remove; {@code null}/empty
* entries and a {@code null} array are ignored
* @return {@code message} unchanged when it is {@code null} or no non-empty
* @param secrets the exact secret substrings to remove; {@code null}, empty,
* and blank (whitespace-only) entries and a {@code null} array are ignored
* @return {@code message} unchanged when it is {@code null} or no non-blank
* secret is supplied, otherwise the message with every secret occurrence
* replaced by {@code "<redacted>"}
*/
@@ -76,9 +76,11 @@ public final class MxGatewaySecrets {
String result = message;
for (String secret : secrets) {
if (secret != null && !secret.isEmpty()) {
result = result.replace(secret, "<redacted>");
if (secret == null || secret.isBlank()) {
// A blank "secret" would over-redact real whitespace; skip it.
continue;
}
result = result.replace(secret, "<redacted>");
}
return result;
}
@@ -31,6 +31,7 @@ import mxaccess_gateway.v1.MxaccessGateway.MxSparseElement;
import mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy;
import mxaccess_gateway.v1.MxaccessGateway.MxValue;
import mxaccess_gateway.v1.MxaccessGateway.OpenSessionReply;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatus;
import mxaccess_gateway.v1.MxaccessGateway.ReadBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.RegisterCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemBulkCommand;
@@ -1055,29 +1056,142 @@ public final class MxGatewaySession implements AutoCloseable {
* it into a diagnostic; this guarantees it never survives in the exception
* text a caller might log.
*
* <p>On failure the original exception's message is scrubbed with
* {@link MxGatewaySecrets#redactExact}. If nothing changed (the common case
* where the message never carried the secret), the original exception is
* <p>On failure both the exception's message <em>and</em> its structured
* context (the {@link ProtocolStatus} and {@link MxCommandReply} a caller can
* inspect and log) are scrubbed with {@link MxGatewaySecrets#redactExact}: the
* gateway echoes the credential into {@code protocolStatus.message},
* {@code reply.diagnosticMessage}, and each {@code statuses[i].diagnosticText}.
* If nothing carried the secret (the common case) the original exception is
* rethrown untouched. Otherwise it is re-thrown as the same concrete type
* carrying the redacted message; the secret-bearing original is not chained
* as a cause, so it cannot leak through a printed stack trace.
* carrying the redacted message and scrubbed context; the secret-bearing
* original is not chained as a cause, so it cannot leak through a printed
* stack trace.
*/
private MxCommandReply invokeCommandRedacted(MxCommand command, String... secrets) {
try {
return invokeCommand(command);
} catch (MxGatewayException ex) {
String original = ex.getMessage();
String redacted = MxGatewaySecrets.redactExact(original, secrets);
if (redacted == null || redacted.equals(original)) {
String redactedMessage = MxGatewaySecrets.redactExact(original, secrets);
boolean messageChanged = redactedMessage != null && !redactedMessage.equals(original);
ProtocolStatus status = protocolStatusOf(ex);
ProtocolStatus scrubbedStatus = scrubProtocolStatus(status, secrets);
boolean statusChanged = status != null && !status.equals(scrubbedStatus);
MxCommandReply reply = replyOf(ex);
MxCommandReply scrubbedReply = scrubReply(reply, secrets);
boolean replyChanged = reply != null && !reply.equals(scrubbedReply);
if (!messageChanged && !statusChanged && !replyChanged) {
throw ex;
}
if (ex instanceof MxAccessException mx) {
throw new MxAccessException(redacted, mx.protocolStatus(), mx.reply(), null);
}
throw new MxGatewayException(redacted);
String message = messageChanged ? redactedMessage : original;
throw rebuildRedacted(ex, message, scrubbedStatus, scrubbedReply);
}
}
/**
* Extracts the {@link ProtocolStatus} an exception carries, if any, so it can
* be scrubbed and re-attached to the rebuilt exception.
*/
private static ProtocolStatus protocolStatusOf(MxGatewayException ex) {
if (ex instanceof MxGatewayCommandException command) {
return command.protocolStatus();
}
if (ex instanceof MxGatewaySessionException session) {
return session.protocolStatus();
}
if (ex instanceof MxGatewayWorkerException worker) {
return worker.protocolStatus();
}
return null;
}
/**
* Extracts the raw {@link MxCommandReply} an exception carries, if any.
*/
private static MxCommandReply replyOf(MxGatewayException ex) {
if (ex instanceof MxGatewayCommandException command) {
return command.reply();
}
return null;
}
/**
* Rebuilds a gateway exception of the same concrete type with a redacted
* message and already-scrubbed context, mirroring the .NET client's
* type-switch. Only a truly-unknown subtype collapses to the base
* {@link MxGatewayException}. The original (secret-bearing) exception is
* deliberately not chained as a cause.
*/
private static MxGatewayException rebuildRedacted(
MxGatewayException ex, String message, ProtocolStatus status, MxCommandReply reply) {
if (ex instanceof MxAccessException) {
return new MxAccessException(message, status, reply, null);
}
if (ex instanceof MxGatewayCommandException) {
return new MxGatewayCommandException(message, status, reply, null);
}
if (ex instanceof MxGatewaySessionException) {
return new MxGatewaySessionException(message, status, null);
}
if (ex instanceof MxGatewayWorkerException) {
return new MxGatewayWorkerException(message, status, null);
}
if (ex instanceof MxGatewayMalformedReplyException) {
return new MxGatewayMalformedReplyException(message);
}
if (ex instanceof MxGatewayAuthenticationException) {
return new MxGatewayAuthenticationException(message, null);
}
if (ex instanceof MxGatewayAuthorizationException) {
return new MxGatewayAuthorizationException(message, null);
}
return new MxGatewayException(message);
}
/**
* Produces a scrubbed clone of a command reply, removing any exact secret the
* gateway echoed into {@code protocolStatus.message},
* {@code diagnosticMessage}, or a status's {@code diagnosticText}.
*
* @param reply the reply to scrub, or {@code null}
* @param secrets the exact secrets to strip
* @return {@code null} when {@code reply} is {@code null}, otherwise a clone
* with every echoed secret replaced by the redaction marker
*/
private static MxCommandReply scrubReply(MxCommandReply reply, String... secrets) {
if (reply == null) {
return null;
}
MxCommandReply.Builder builder = reply.toBuilder();
if (builder.hasProtocolStatus()) {
builder.setProtocolStatus(scrubProtocolStatus(builder.getProtocolStatus(), secrets));
}
builder.setDiagnosticMessage(MxGatewaySecrets.redactExact(builder.getDiagnosticMessage(), secrets));
for (int index = 0; index < builder.getStatusesCount(); index++) {
MxStatusProxy.Builder status = builder.getStatuses(index).toBuilder();
status.setDiagnosticText(MxGatewaySecrets.redactExact(status.getDiagnosticText(), secrets));
builder.setStatuses(index, status);
}
return builder.build();
}
/**
* Produces a scrubbed clone of a protocol status, removing any exact secret
* the gateway echoed into its free-form {@code message}.
*/
private static ProtocolStatus scrubProtocolStatus(ProtocolStatus status, String... secrets) {
if (status == null) {
return null;
}
return status.toBuilder()
.setMessage(MxGatewaySecrets.redactExact(status.getMessage(), secrets))
.build();
}
/**
* Extracts the string payload of a secured-write value so it can be scrubbed
* from an echoed failure message. Only string-kind values carry a
@@ -20,6 +20,20 @@ public final class MxGatewaySessionException extends MxGatewayException {
this.protocolStatus = protocolStatus;
}
/**
* Creates a new session exception with an already-built, verbatim message.
* Used to re-surface a failure with a redacted message while preserving the
* (already scrubbed) protocol status.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status returned by the gateway
* @param cause underlying error, or {@code null}
*/
protected MxGatewaySessionException(String message, ProtocolStatus protocolStatus, Throwable cause) {
super(message, cause);
this.protocolStatus = protocolStatus;
}
/**
* Returns the gateway protocol status that triggered this exception.
*
@@ -20,6 +20,20 @@ public final class MxGatewayWorkerException extends MxGatewayException {
this.protocolStatus = protocolStatus;
}
/**
* Creates a new worker exception with an already-built, verbatim message.
* Used to re-surface a failure with a redacted message while preserving the
* (already scrubbed) protocol status.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status returned by the gateway
* @param cause underlying error, or {@code null}
*/
protected MxGatewayWorkerException(String message, ProtocolStatus protocolStatus, Throwable cause) {
super(message, cause);
this.protocolStatus = protocolStatus;
}
/**
* Returns the gateway protocol status that triggered this exception.
*
@@ -2,6 +2,7 @@ package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -28,11 +29,23 @@ final class MxGatewayCredentialReplyTests {
@Test
void authenticateUserRedactsEchoedCredentialFromReplyDrivenError() throws Exception {
MxCommandReply reply = loadReply("authenticate-user.echoed-credential.reply.json");
assertCredentialFullyRedacted(
"authenticate-user.echoed-credential.reply.json", "auth-echo-session");
}
@Test
void authenticateUserRedactsEchoedCredentialFromMxAccessFailureReply() throws Exception {
assertCredentialFullyRedacted(
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
"auth-echo-failure-session");
}
private static void assertCredentialFullyRedacted(String fixture, String sessionId) throws Exception {
MxCommandReply reply = loadReply(fixture);
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "auth-echo-session");
MxGatewaySession session = MxGatewaySession.forSessionId(client, sessionId);
MxAccessException error = assertThrows(
MxAccessException.class,
@@ -42,6 +55,22 @@ final class MxGatewayCredentialReplyTests {
"credential echoed by the gateway must not survive in the surfaced message");
assertTrue(error.getMessage().contains("<redacted>"),
"the echoed credential must be replaced with the redaction marker");
// The rebuilt exception must not re-expose the credential through the
// structured reply/protocolStatus a caller can inspect and log.
MxCommandReply surfaced = error.reply();
assertNotNull(surfaced, "the redacted exception must preserve a reply for inspection");
assertFalse(surfaced.getProtocolStatus().getMessage().contains(CREDENTIAL),
"reply protocol status message must not leak the echoed credential");
assertFalse(surfaced.getDiagnosticMessage().contains(CREDENTIAL),
"reply diagnostic message must not leak the echoed credential");
for (int index = 0; index < surfaced.getStatusesCount(); index++) {
assertFalse(surfaced.getStatusesList().get(index).getDiagnosticText().contains(CREDENTIAL),
"reply status diagnostic text must not leak the echoed credential");
}
assertNotNull(error.protocolStatus(), "the redacted exception must preserve a protocol status");
assertFalse(error.protocolStatus().getMessage().contains(CREDENTIAL),
"exception protocol status must not leak the echoed credential");
}
}
@@ -0,0 +1,50 @@
package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import org.junit.jupiter.api.Test;
final class MxGatewaySecretsTests {
@Test
void redactExactReplacesEveryOccurrenceOfASecret() {
String message = "verify s3cr3t, retry s3cr3t, done s3cr3t";
String result = MxGatewaySecrets.redactExact(message, "s3cr3t");
assertFalse(result.contains("s3cr3t"), "no occurrence of the secret may survive");
assertEquals("verify <redacted>, retry <redacted>, done <redacted>", result);
}
@Test
void redactExactFullyRedactsOverlappingSecretsWhenOneIsASubstringOfTheOther() {
String message = "password=hunter2 token=hunter2extra";
String result = MxGatewaySecrets.redactExact(message, "hunter2extra", "hunter2");
assertFalse(result.contains("hunter2"), "both the secret and its superstring must be fully redacted");
assertEquals("password=<redacted> token=<redacted>", result);
}
@Test
void redactExactWithNoSecretsReturnsMessageUnchanged() {
String message = "nothing to scrub here";
assertEquals(message, MxGatewaySecrets.redactExact(message));
}
@Test
void redactExactToleratesNullMessage() {
assertNull(MxGatewaySecrets.redactExact(null, "secret"));
}
@Test
void redactExactIgnoresBlankSecretSoRealSpacesAreNotOverRedacted() {
String message = "keep these spaces intact";
String result = MxGatewaySecrets.redactExact(message, " ", "");
assertEquals(message, result);
}
}
@@ -0,0 +1,22 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-authenticate-echoed-mxaccess-failure",
"kind": "MX_COMMAND_KIND_AUTHENTICATE_USER",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_MXACCESS_FAILURE",
"message": "MXAccess AuthenticateUser rejected credential 'sup3rSecretVerify9f3a2b'."
},
"hresult": -2147024891,
"statuses": [
{
"success": 0,
"category": "MX_STATUS_CATEGORY_SECURITY_ERROR",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_NMX",
"detail": 5,
"rawCategory": 8,
"rawDetectedBy": 5,
"diagnosticText": "Authentication failed for password 'sup3rSecretVerify9f3a2b'."
}
],
"diagnosticMessage": "MXAccess echoed the credential 'sup3rSecretVerify9f3a2b' back in its failure diagnostic."
}
@@ -53,7 +53,14 @@
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.echoed-credential.reply.json",
"expectation": "When a gateway/MXAccess diagnostic echoes the caller's credential back, the surfaced error redacts the exact secret and never leaks the verbatim value."
"expectation": "When a gateway/MXAccess diagnostic echoes the caller's credential back (OK envelope, negative HRESULT), the surfaced error redacts the exact secret from both the rendered message and the structured reply accessors."
},
{
"id": "command-reply.authenticate-user.echoed-credential-mxaccess-failure",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.echoed-credential-mxaccess-failure.reply.json",
"expectation": "The same echoed-credential redaction holds when the reply is coded PROTOCOL_STATUS_CODE_MXACCESS_FAILURE, which every client routes to its MXAccess error type."
},
{
"id": "command-reply.authenticate-user.missing-payload",
@@ -919,19 +919,47 @@ def _value_secrets(value: MxValueInput) -> list[str]:
def _redact_error(error: MxGatewayError, secrets: Sequence[str | None]) -> None:
"""Scrub secret substrings from a raised error's message in place.
"""Scrub secret substrings from a raised error's message and reply in place.
Rewrites ``error.args[0]`` (the message returned by ``str(error)``) through
the shared :func:`~zb_mom_ww_mxgateway.auth.redact_secret` seam so credential
text can never reach logs or be re-raised to a caller. The
``protocol_status`` / ``raw_reply`` context is left untouched those hold the
gateway's own fields, which never echo the client-supplied secret.
text can never reach logs or be re-raised to a caller.
A misbehaving MXAccess provider can echo the client-supplied credential back
verbatim in its failure diagnostics, so ``error.raw_reply`` (the protobuf
reply) can carry the secret in ``protocol_status.message``,
``diagnostic_message``, and each ``statuses[].diagnostic_text``. A logger
dumping those structured fields would reintroduce the leak the message scrub
closes. When there is a secret to scrub and a reply is attached, this rebinds
``error.raw_reply`` to a scrubbed deep copy so the raised exception carries no
credential text on any surface. The clone leaves the original reply untouched.
"""
scrubbed = [secret for secret in secrets if secret]
if not scrubbed:
return
if error.args and isinstance(error.args[0], str):
error.args = (redact_secret(error.args[0], scrubbed), *error.args[1:])
if error.raw_reply is not None:
error.raw_reply = _redact_reply(error.raw_reply, scrubbed)
def _redact_reply(reply: pb.MxCommandReply, secrets: Sequence[str]) -> pb.MxCommandReply:
"""Return a deep copy of *reply* with credential text scrubbed from diagnostics.
Operates on a clone so the caller's original reply object is never mutated.
Only the free-text diagnostic fields that can echo a client-supplied secret
are scrubbed; the structured/enum fields the gateway itself sets are left as-is.
"""
clone = type(reply)()
clone.CopyFrom(reply)
if clone.protocol_status.message:
clone.protocol_status.message = redact_secret(clone.protocol_status.message, secrets)
if clone.diagnostic_message:
clone.diagnostic_message = redact_secret(clone.diagnostic_message, secrets)
for status in clone.statuses:
if status.diagnostic_text:
status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
return clone
from .client import GatewayClient # noqa: E402
+19 -3
View File
@@ -82,15 +82,31 @@ async def test_add_buffered_item_missing_payload_raises_malformed_reply() -> Non
assert captured.value.raw_reply is reply
@pytest.mark.parametrize(
"fixture",
[
"command-replies/authenticate-user.echoed-credential.reply.json",
"command-replies/authenticate-user.echoed-credential-mxaccess-failure.reply.json",
],
)
@pytest.mark.asyncio
async def test_authenticate_user_echoed_credential_is_scrubbed() -> None:
async def test_authenticate_user_echoed_credential_is_scrubbed(fixture: str) -> None:
credential = "sup3rSecretVerify9f3a2b"
reply = _load_reply("command-replies/authenticate-user.echoed-credential.reply.json")
reply = _load_reply(fixture)
session, _ = await _session_with([reply])
with pytest.raises(MxAccessError) as captured:
await session.authenticate_user(12, "operator", credential)
message = str(captured.value)
exc = captured.value
message = str(exc)
assert credential not in message
assert "[redacted]" in message
# The credential must not survive in the structured protobuf context either:
# a logger dumping raw_reply's fields would otherwise reintroduce the leak.
assert exc.raw_reply is not None
assert credential not in exc.raw_reply.protocol_status.message
assert credential not in exc.raw_reply.diagnostic_message
for status in exc.raw_reply.statuses:
assert credential not in status.diagnostic_text
@@ -140,10 +140,19 @@ async def test_write_secured_surfaces_native_failure_without_prior_authenticate(
with pytest.raises(MxAccessError) as captured:
await session.write_secured(12, 34, secret_value, current_user_id=5, verifier_user_id=6)
# Native failure is surfaced (not "fixed") and the raw reply is preserved...
assert captured.value.raw_reply is failure
# ...but the credential-sensitive value is scrubbed from the surfaced message.
# Native failure is surfaced (not "fixed"): the raw reply's structure is
# preserved so callers still see the native verdict...
raw = captured.value.raw_reply
assert raw is not None
assert raw.kind == pb.MX_COMMAND_KIND_WRITE_SECURED
assert raw.hresult == -2147217407
assert raw.protocol_status.code == pb.PROTOCOL_STATUS_CODE_MXACCESS_FAILURE
# ...but the credential-sensitive value is scrubbed from the surfaced message
# AND from the reply's echoed diagnostics, so a logger dumping raw_reply's
# structured fields cannot reintroduce the leak.
assert secret_value not in str(captured.value)
assert secret_value not in raw.protocol_status.message
assert "[redacted]" in raw.protocol_status.message
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_WRITE_SECURED
assert command.write_secured.current_user_id == 5
+24 -7
View File
@@ -346,10 +346,17 @@ impl From<tonic::Status> for Error {
/// Promote a non-OK protocol status carried inside an [`MxCommandReply`]
/// to an [`Error::Command`].
///
/// [`ProtocolStatusCode::MxaccessFailure`] is deliberately **not** a
/// command-level failure here: it signals an MXAccess-level rejection, so it
/// falls through to [`ensure_mxaccess_success`] and surfaces as
/// [`Error::MxAccess`] — matching the .NET, Java, Go, and Python clients. Every
/// other non-`Ok` code stays [`Error::Command`].
///
/// # Errors
///
/// Returns [`Error::Command`] when `reply.protocol_status` is missing or
/// reports any code other than [`ProtocolStatusCode::Ok`].
/// reports any code other than [`ProtocolStatusCode::Ok`] or
/// [`ProtocolStatusCode::MxaccessFailure`].
pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, Error> {
let code = reply
.protocol_status
@@ -357,7 +364,7 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
.and_then(|status| ProtocolStatusCode::try_from(status.code).ok())
.unwrap_or(ProtocolStatusCode::Unspecified);
if code == ProtocolStatusCode::Ok {
if code == ProtocolStatusCode::Ok || code == ProtocolStatusCode::MxaccessFailure {
Ok(reply)
} else {
Err(Box::new(CommandError::new(reply)).into())
@@ -368,9 +375,12 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
/// [`MxCommandReply`] to an [`Error::MxAccess`].
///
/// This is the second reply check applied to the typed command path, after
/// [`ensure_command_success`] confirms the protocol envelope is `Ok`. It
/// enforces MXAccess parity: a reply can carry an `Ok` protocol envelope while
/// MXAccess itself rejected the operation. Following COM semantics, only a
/// [`ensure_command_success`] confirms the protocol envelope is `Ok` (or a
/// [`ProtocolStatusCode::MxaccessFailure`] the first check lets fall through).
/// It enforces MXAccess parity: a reply can carry an `Ok` protocol envelope
/// while MXAccess itself rejected the operation, and a
/// [`ProtocolStatusCode::MxaccessFailure`] envelope is itself an MXAccess-level
/// failure regardless of `hresult`. Following COM semantics, only a
/// **negative** `hresult` is a failure — positive codes such as `S_FALSE = 1`
/// are success. A `MXSTATUS_PROXY` entry is treated as a failure when its
/// `category` is not [`MxStatusCategory::Ok`]; the `success` member mirrors the
@@ -383,17 +393,24 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
///
/// # Errors
///
/// Returns [`Error::MxAccess`] when `reply.hresult` is negative or any
/// Returns [`Error::MxAccess`] when the reply's protocol code is
/// [`ProtocolStatusCode::MxaccessFailure`], `reply.hresult` is negative, or any
/// `reply.statuses` entry reports a category other than
/// [`MxStatusCategory::Ok`].
pub fn ensure_mxaccess_success(reply: MxCommandReply) -> Result<MxCommandReply, Error> {
let protocol_code = reply
.protocol_status
.as_ref()
.and_then(|status| ProtocolStatusCode::try_from(status.code).ok())
.unwrap_or(ProtocolStatusCode::Unspecified);
let mxaccess_failure = protocol_code == ProtocolStatusCode::MxaccessFailure;
let hresult_failure = reply.hresult.is_some_and(|hresult| hresult < 0);
let status_failure = reply
.statuses
.iter()
.any(|status| status.category != MxStatusCategory::Ok as i32);
if hresult_failure || status_failure {
if mxaccess_failure || hresult_failure || status_failure {
Err(Box::new(MxAccessError::new(reply)).into())
} else {
Ok(reply)
+35 -5
View File
@@ -11,7 +11,7 @@
use std::sync::atomic::{AtomicU64, Ordering};
use crate::client::{EventStream, GatewayClient};
use crate::error::{ensure_protocol_success, Error};
use crate::error::{ensure_protocol_success, Error, MxAccessError};
use crate::generated::mxaccess_gateway::v1::mx_command::Payload;
use crate::generated::mxaccess_gateway::v1::mx_command_reply;
use crate::generated::mxaccess_gateway::v1::{
@@ -1117,16 +1117,46 @@ fn string_secret(value: &MxValue) -> Vec<String> {
}
/// Attach caller-supplied exact secrets to an [`Error::MxAccess`] before it
/// propagates, so its `Display` scrubs any occurrence of the credential (e.g. a
/// password MXAccess echoed back verbatim). Any other error variant is returned
/// unchanged.
/// propagates. This both scrubs the stored reply's caller-readable string
/// fields (so `reply()`/`into_reply()` cannot recover a credential MXAccess
/// echoed back verbatim) and keeps the secrets on the error as a
/// belt-and-suspenders for `Display`/`Debug`. Any other error variant is
/// returned unchanged.
fn attach_secrets(error: Error, secrets: Vec<String>) -> Error {
match error {
Error::MxAccess(boxed) => Error::MxAccess(Box::new(boxed.with_secrets(secrets))),
Error::MxAccess(boxed) => {
let mut reply = boxed.into_reply();
scrub_reply_strings(&mut reply, &secrets);
Error::MxAccess(Box::new(MxAccessError::new(reply).with_secrets(secrets)))
}
other => other,
}
}
/// Replace every non-empty secret occurrence with `<redacted>` in the reply's
/// caller-readable string fields — `protocol_status.message`,
/// `diagnostic_message`, and each `statuses[i].diagnostic_text`. A caller
/// reading the structured reply back off an [`Error::MxAccess`] would otherwise
/// reintroduce the leak that `Display`/`Debug` already close.
fn scrub_reply_strings(reply: &mut MxCommandReply, secrets: &[String]) {
for secret in secrets {
if secret.is_empty() {
continue;
}
if let Some(status) = reply.protocol_status.as_mut() {
status.message = status.message.replace(secret.as_str(), "<redacted>");
}
reply.diagnostic_message = reply
.diagnostic_message
.replace(secret.as_str(), "<redacted>");
for status in &mut reply.statuses {
status.diagnostic_text = status
.diagnostic_text
.replace(secret.as_str(), "<redacted>");
}
}
}
fn suspend_status(reply: MxCommandReply) -> Result<MxStatus, Error> {
match reply.payload {
Some(mx_command_reply::Payload::Suspend(suspend)) => suspend
+111 -3
View File
@@ -84,8 +84,10 @@ async fn session_helpers_build_commands_and_preserve_command_errors() {
.write(12, 34, ClientMxValue::int32(123), 0)
.await
.unwrap_err();
let Error::Command(error) = error else {
panic!("write failure should preserve the raw command reply: {error:?}");
// A MXACCESS_FAILURE-coded reply is an MXAccess-level failure, routed to
// Error::MxAccess (matching .NET/Java/Go/Python) rather than Error::Command.
let Error::MxAccess(error) = error else {
panic!("MXACCESS_FAILURE reply should route to Error::MxAccess: {error:?}");
};
assert_eq!(
error.reply().protocol_status.as_ref().unwrap().code,
@@ -841,6 +843,92 @@ async fn authenticate_user_scrubs_exact_caller_credential_echoed_in_diagnostic()
);
}
/// Drive `authenticate_user` against a canned reply that echoes the caller's
/// credential in every string field, then assert the surfaced
/// [`Error::MxAccess`] leaks it nowhere — neither through the structured reply a
/// caller can read back (`reply().protocol_status.message`,
/// `reply().diagnostic_message`, `reply().statuses[i].diagnostic_text`) nor
/// through `Display`/`Debug`.
async fn assert_authenticate_user_scrubs_structured_reply(fixture: &str) {
let credential = "sup3rSecretVerify9f3a2b";
let state = Arc::new(FakeState::default());
*state.invoke_override.lock().await = Some(InvokeOverride::CannedReply(Box::new(
command_reply_fixture(fixture),
)));
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let error = session
.authenticate_user(7, "verifier", credential)
.await
.unwrap_err();
let Error::MxAccess(mx_access) = &error else {
panic!("{fixture}: credential-echoed reply must route to Error::MxAccess, got {error:?}");
};
// The structured reply a caller can read back must be scrubbed too — the raw
// MxCommandReply otherwise reintroduces the leak Display/Debug already close.
let reply = mx_access.reply();
if let Some(status) = reply.protocol_status.as_ref() {
assert!(
!status.message.contains(credential),
"{fixture}: credential leaked via reply().protocol_status.message: {}",
status.message
);
}
assert!(
!reply.diagnostic_message.contains(credential),
"{fixture}: credential leaked via reply().diagnostic_message: {}",
reply.diagnostic_message
);
for (index, status) in reply.statuses.iter().enumerate() {
assert!(
!status.diagnostic_text.contains(credential),
"{fixture}: credential leaked via reply().statuses[{index}].diagnostic_text: {}",
status.diagnostic_text
);
}
let display = error.to_string();
let debug = format!("{error:?}");
assert!(
!display.contains(credential),
"{fixture}: credential leaked into Display: {display}"
);
assert!(
!debug.contains(credential),
"{fixture}: credential leaked into Debug: {debug}"
);
assert!(
display.contains("<redacted>"),
"{fixture}: Display must mark the redaction: {display}"
);
}
#[tokio::test]
async fn authenticate_user_scrubs_credential_from_structured_reply_ok_protocol_variant() {
// OK protocol envelope + negative hresult: already Error::MxAccess before
// ISSUE 2, but the stored reply's string fields still leaked the credential.
assert_authenticate_user_scrubs_structured_reply(
"authenticate-user.echoed-credential.reply.json",
)
.await;
}
#[tokio::test]
async fn authenticate_user_scrubs_credential_from_structured_reply_mxaccess_failure_variant() {
// PROTOCOL_STATUS_CODE_MXACCESS_FAILURE: before ISSUE 2 this landed in
// Error::Command (unscrubbed, raw Display/Debug) — the red-first case.
assert_authenticate_user_scrubs_structured_reply(
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
)
.await;
}
#[tokio::test]
async fn authenticate_user_maps_missing_payload_reply_to_malformed_reply() {
// CLI-41: an OK reply with neither a typed AuthenticateUser payload nor a
@@ -1548,15 +1636,35 @@ fn command_reply_fixture(file_name: &str) -> MxCommandReply {
})
});
// Honor the fixture's real protocol status (code + message) so a canned
// reply can drive the MXACCESS_FAILURE routing path, not just an OK
// envelope. Falls back to an OK envelope when the fixture omits it.
let protocol_status = fixture.get("protocolStatus").map_or_else(
|| ok_status("command ok"),
|status| {
let code_name = status["code"].as_str().unwrap_or("PROTOCOL_STATUS_CODE_OK");
ProtocolStatus {
code: ProtocolStatusCode::from_str_name(code_name)
.unwrap_or_else(|| panic!("unknown protocol status code {code_name}"))
as i32,
message: status["message"].as_str().unwrap_or_default().to_owned(),
}
},
);
MxCommandReply {
session_id: fixture["sessionId"].as_str().unwrap_or_default().to_owned(),
correlation_id: fixture["correlationId"]
.as_str()
.unwrap_or_default()
.to_owned(),
protocol_status: Some(ok_status("command ok")),
protocol_status: Some(protocol_status),
hresult: fixture["hresult"].as_i64().map(|hresult| hresult as i32),
statuses,
diagnostic_message: fixture["diagnosticMessage"]
.as_str()
.unwrap_or_default()
.to_owned(),
return_value,
..MxCommandReply::default()
}
+13 -8
View File
@@ -77,7 +77,8 @@ credential-redaction contracts for the credential-bearing helpers:
| Fixture | Reply | Expected behavior |
|---|---|---|
| `authenticate-user.echoed-credential.reply.json` | OK envelope, negative `hresult`, and the caller's credential echoed into `protocolStatus.message`, `statuses[0].diagnosticText`, and `diagnosticMessage` | the surfaced error redacts the exact secret (never leaks the verbatim value) |
| `authenticate-user.echoed-credential.reply.json` | OK envelope, negative `hresult`, and the caller's credential echoed into `protocolStatus.message`, `statuses[0].diagnosticText`, and `diagnosticMessage` | the surfaced error redacts the exact secret from **both** the rendered message and the structured reply accessors (never leaks the verbatim value) |
| `authenticate-user.echoed-credential-mxaccess-failure.reply.json` | the same echo, but coded `PROTOCOL_STATUS_CODE_MXACCESS_FAILURE` | identical redaction; confirms every client routes the MXAccess-failure protocol code to its MXAccess error type and scrubs it |
| `authenticate-user.missing-payload.reply.json` | OK envelope, no `AuthenticateUser` payload, no `return_value` | a typed malformed-reply error, never a proto3 default `0` and never an NRE |
| `authenticate-user.return-value-only.reply.json` | OK envelope, `return_value.int32_value = 7`, no typed payload | the id resolves to `7` via the legacy `return_value` compatibility path |
@@ -91,13 +92,17 @@ The rules those fixtures lock in are:
It never surfaces a proto3 default `0` and never throws a null-reference.
- **Credential redaction (CLI-40).** The credential-bearing helpers
(`AuthenticateUser`, `WriteSecured`/`WriteSecured2`) scrub the exact secret
values they were called with from any surfaced error text, replacing each
occurrence with the client's redaction marker. This is defense-in-depth on top
of the by-construction guarantee that exceptions carry reply-derived text, not
the request. The marker is `<redacted>` in the Go, Rust, and Java clients and
`[redacted]` in the Python client and the .NET CLI; the assertion each suite
makes is that the surfaced message no longer contains the credential and does
contain the client's marker.
values they were called with from any surfaced error — both the rendered
message text **and** the structured reply the error still exposes (a
server-echoed credential lives in `protocolStatus.message` and
`statuses[].diagnosticText`, which the error's raw-reply accessor would
otherwise re-expose to a logger dumping structured fields). The redacted error
therefore carries a scrubbed clone of the reply. This is defense-in-depth on
top of the by-construction guarantee that exceptions carry reply-derived text,
not the request. The marker is `<redacted>` in the Go, Rust, and Java clients
and `[redacted]` in the Python client and the .NET CLI; each suite asserts that
neither the surfaced message nor the exposed reply still contains the
credential, and that the message contains the client's marker.
## Event Streams
+9 -3
View File
@@ -138,10 +138,16 @@ secured payloads route through each client's secret-redaction seam so they never
reach logs, exception text, or `ToString`/`Debug`/`Display` — the value is carried
only on the wire. In addition to that by-construction guarantee (exceptions carry
reply-derived text, not the request), every client scrubs the **exact** secret
values it was called with from any surfaced error text as defense-in-depth, so a
values it was called with from any surfaced error as defense-in-depth, so a
gateway or MXAccess diagnostic that echoes a credential back cannot leak it
(CLI-40). Each client's test suite asserts a distinctive credential is absent
from any surfaced error and that the redaction marker is present.
(CLI-40). The scrub covers **both** the rendered message and the structured reply
the error still exposes (`protocolStatus.message`, `statuses[].diagnosticText`,
`diagnosticMessage`): the redacted error carries a scrubbed clone of the reply so
a logger dumping the exception's structured fields cannot reintroduce the leak.
This holds regardless of whether the reply is coded `OK` (with a failing HRESULT)
or `MXACCESS_FAILURE` — every client routes both to its MXAccess error type. Each
client's test suite asserts the distinctive credential is absent from both the
surfaced message and the exposed reply, and that the redaction marker is present.
Shipped in all five clients (.NET / Go / Rust / Python / Java).