Compare commits

...

45 Commits

Author SHA1 Message Date
Joseph Doherty 815e58d28b docs(tracking): record WRK-22/24/25/27 + IPC-26 windev evidence (377 pass)
ci / java (push) Successful in 2m7s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m12s
ci / portable (push) Failing after 4m12s
2026-08-07 07:52:44 -04:00
Joseph Doherty 8df35cd63a fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
ci / java (push) Successful in 2m7s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / portable (push) Failing after 4m6s
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).

WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.

WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).

WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.

Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
2026-08-07 07:50:38 -04:00
Joseph Doherty 10534ec906 docs(TST-27,WRK-26,CLI-42,CLI-43,IPC-28): P1 doc-drift batch, discharges IPC-29
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m10s
ci / portable (push) Failing after 3m53s
TST-27: docs/GatewayConfiguration.md's ShowTagValues row no longer says
"Reserved" — it now states what false (default) does (DashboardEventBroadcaster
blanks tag values from a deep-cloned MxEvent before the SignalR events-hub
mirror), the security relevance (no per-session hub ACL yet, so this
redaction is the only thing between a low-trust Viewer and other sessions'
tag values), and the honest scope limit (does not cover /browse).

WRK-26 (discharges IPC-29): docs/MxAccessWorkerInstanceDesign.md's "Outbound
Queues" section rewritten from the stale five-level priority list to the
two-class Control/Event scheduler actually shipped, with the collapsed-
decision rationale, and the overflow paragraph rewritten to the implemented
fail-fast. docs/WorkerFrameProtocol.md gained a "Write Scheduling And
Sequencing" section describing HEAD truthfully: WRK-23's peek-stamp-commit
sequencing is live, WRK-25's event-batch flush coalescing is not (the drain
loop still awaits each event write individually), and WRK-22's cancellation
tombstone is not yet defined (noted as pending, not documented as shipped).

CLI-42: clients/rust/README.md and docs/ClientPackaging.md document the
vendored Rust proto layout matching build.rs — repo-path-first resolution
falling back to clients/rust/protos/, the check-codegen.ps1 Check 3 refresh
rule, and why cargo package/publish run without --no-verify.

CLI-43: docs/style-guides/JavaStyleGuide.md now says Java 17 (Ignition 8.3
baseline), mirroring CLI-12's wording, matching the shipped build.gradle.

IPC-28: docs/Grpc.md's exception-mapping prose gained CommandTooLarge ->
ResourceExhausted, and the Invoke section gained the oversized-payload
sentence, cross-referencing GatewayConfiguration.md's headroom rule.

Tracking: TST-27, WRK-26, CLI-42, CLI-43, IPC-28 flipped to Done and IPC-29
marked discharged-by-WRK-26 in 00-tracking.md and the 20/30/50/60 domain
registers, with a 2026-08-07 change-log entry.

Doc-only change; no source, proto, or test edits.
2026-08-07 07:26:58 -04:00
Joseph Doherty 97f79e79ef Merge remote-tracking branch 'origin/fix/wrk-21-drain-cluster'
ci / windows-x86 (push) Successful in 1m27s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m11s
ci / portable (push) Failing after 4m7s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
#	docs/MxAccessWorkerInstanceDesign.md
2026-08-07 07:18:58 -04:00
Joseph Doherty 758277bc62 docs(tracking): record WRK-21 review follow-ups (monotonic budget, guarded fallback) with windev evidence
ci / windows-x86 (push) Successful in 1m19s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m15s
ci / portable (push) Successful in 9m30s
2026-08-07 07:13:46 -04:00
Joseph Doherty 6bc3f9b991 fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback
ci / windows-x86 (push) Successful in 1m16s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / portable (push) Successful in 9m31s
Review follow-ups on the WRK-21 cluster.

1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
   64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
   floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
   byte-heavy event, so DrainEvents truncated on every call and the drain-until-
   empty loop never terminated. It now takes the max of (frameMax - reserve) and
   frameMax/2, so the budget is monotonic and never below half the frame max. New
   test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
   byte-heavy queue at the exact validator floor and asserts it drains to empty
   with no head ever reported oversized.

2. The reply-too-large fallback write is now itself size-guarded
   (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
   at a pathologically tiny negotiated max below the gateway's floor the fallback
   could also throw MessageTooLarge and — uncaught — kill the session, defeating the
   "no diagnostics command is session-fatal" invariant. It now log-and-swallows;
   comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.

3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
   already includes the event's tag and length prefix (the same shape the reply's
   repeated events field packs), so the 8 bytes is pure slack over an already-
   conservative estimate, not compensation for a missing wrapper.
2026-08-07 07:09:13 -04:00
Joseph Doherty 34db678635 Merge branch 'fix/cli-40-41-44'
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m14s
ci / portable (push) Failing after 4m12s
2026-08-07 07:08:27 -04:00
Joseph Doherty 0d874f91ee fix(CLI-40): scrub the credential from the redacted error's structured reply, route MXACCESS_FAILURE to MxAccess (Rust), fix Go Subscribe terminal-error drop
Code-review follow-up on the CLI-40/41/44 branch.

ISSUE 1 (all five, critical): the message-only scrub still leaked the
server-echoed credential through the redacted error's structured reply accessor
(.NET Reply/Statuses, Java reply()/protocolStatus(), Go MxAccessError.Reply via
errors.As, Rust reply()/into_reply(), Python raw_reply). The redacted error now
carries a scrubbed clone of the reply (protocol_status.message,
diagnostic_message, statuses[].diagnostic_text), with per-language tests asserting
the reply accessor no longer contains the credential.

ISSUE 2 (Rust, critical): ensure_command_success routed MXACCESS_FAILURE to
Error::Command (unlike the other four clients), bypassing attach_secrets and
leaking via derived Debug/Display. MXACCESS_FAILURE now routes to Error::MxAccess,
fixing the cross-client inconsistency.

ISSUE 3 (Go, important): the CLI-44 terminal send was unconditionally
non-blocking, dropping a genuine terminal error under a full buffer on the
never-drop SubscribeEvents path. It is now reserved-slot-non-blocking only for the
cancel-on-overflow path and blocking for the never-drop path.

New shared fixture authenticate-user.echoed-credential-mxaccess-failure.reply.json
wired into all five suites. Minors: whitespace-secret guard on .NET/Java redact
helpers; Java preserves exception subtype on redaction; redaction-helper unit
tests (Go/Java/.NET). Docs (ClientBehaviorFixtures.md, ClientLibrariesDesign.md)
updated to make the structured-field claim true.
2026-08-07 07:04:56 -04:00
Joseph Doherty c9925688f5 docs(tracking): record the WRK-21 cluster as Done with windev evidence
ci / java (push) Successful in 2m28s
ci / windows-x86 (push) Failing after 1m28s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 8m38s
Change-log row for 2026-08-07: what landed for WRK-21/WRK-28/WRK-23/IPC-30, why
IPC-23 stays In progress (proto-comment/doc wave pending), and the verification
evidence — macOS NonWindows build + validator tests, and the documented windev
path (scripts/ci/windev-worker-ci.ps1 -Mode test) at a256560: x86 Worker build
clean, Worker.Tests 367 passed / 0 failed / 11 skipped.
2026-08-07 06:54:24 -04:00
Joseph Doherty 2aac29618e Merge branch 'fix/sec-33-34'
ci / java (push) Successful in 2m16s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m25s
ci / portable (push) Successful in 8m32s
2026-08-07 06:51:31 -04:00
Joseph Doherty a2565604df test(WRK-21): keep the drain-to-empty walk inside the pipe harness envelope
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m12s
ci / portable (push) Successful in 8m2s
PipePair runs both ends of a duplex pipe in one process with blocking
FlushFileBuffers under every frame write, so it wedges after roughly 85 large
round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer,
where the no-loss property actually lives, and keep the pipe walk at 1,000 events
(29 replies) so it still proves the split end to end. Also give the truncation
test's budget slack: item handle 0 is a proto3 default and is not serialized, so
the probe measurement is a lower bound on the fixture's per-event cost.
2026-08-07 06:50:17 -04:00
Joseph Doherty 193daa9ee8 fix(SEC-33,SEC-34): address code review — missed docs, key-id guard comment, test consolidation
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
  default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
  (SnapshotCachePath override is optional, not required; blank seeds a rooted host
  default, no silent no-op) and repoint the option-validation item at the new
  GalaxyRepositoryOptionsValidator.

SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.

Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
2026-08-07 06:49:24 -04:00
Joseph Doherty dc7fd16dd5 fix(CLI-40,CLI-41,CLI-44): exact-secret scrub, uniform malformed-reply contract, Go terminal-error mislabel
CLI-40: port the exact-secret credential scrub to Rust/Java/.NET (Go/Python
already did it). AuthenticateUser/WriteSecured(2) helpers now redact the exact
caller-supplied secret from any surfaced error, as defense-in-depth on top of the
by-construction guarantee. Rust hand-writes a redacting Debug (derived Debug would
leak the reply); Java/.NET rebuild the same exception type with the redacted
message and do not carry the secret-bearing original forward (so ToString/stack
traces stay clean too).

CLI-41: uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/
AddBufferedItem across all five clients — typed payload, else a present int32
return_value, else a typed malformed-reply error. Fixes Go/Java silent-0, .NET
NRE, and Rust's own internal inconsistency.

CLI-44: the Go event goroutine's Recv-error path now uses a non-blocking
sendTerminalEventResult on the reserved slot, so a genuine terminal stream error
is reported as itself instead of being mislabeled ErrSlowConsumer under overflow.

Riders from the CLI-37/38 review: (a) .NET ToDiagnosticSummary and Python
_mxaccess_message surface the raw success member (diagnostics-only parity with
Rust); (b) the status-conversion fixture carries an independent wantSuccess
boolean and the Go/.NET fixture tests assert against it instead of recomputing
the formula under test.

Shared fixtures (authenticate-user.{echoed-credential,missing-payload,
return-value-only}.reply.json) + manifest + ClientBehaviorFixtures.md +
ClientLibrariesDesign.md updated in the same change. Tracking: CLI-40/41/44 -> Done.
2026-08-07 06:42:40 -04:00
Joseph Doherty 7e7f7cad84 fix(SEC-33,SEC-34): host-meaningful path rooting; verification-cache invalidate race
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
  Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
  appsettings.json; the CommonApplicationData-derived code defaults take over. The
  Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
  (SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
  SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
  AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
  wrote it relative to the test bin CWD. The three real-host-start tests now pin
  SqlitePath to a temp path.

SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).

GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.

SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.

Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
2026-08-07 06:36:01 -04:00
Joseph Doherty 7c2eaf09e2 test(WRK-21): size the byte-heavy drain fixture for the pipe harness
ci / java (push) Successful in 2m12s
ci / portable (push) Successful in 8m23s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 11m46s
PipePair has no continuous read pump — the test thread drains the pipe only
while it sits in ReadUntilAsync — so multi-megabyte DrainEvents frames
interleaved with the heartbeat loop wedge both ends inside FlushFileBuffers,
each waiting for the other to read. Negotiate a 128 KiB frame maximum instead:
the 10,000 byte-heavy events still overflow it many times over, so every
assertion (bounded reply, reported truncation, no event loss across repeated
drains, surviving session) is unchanged.
2026-08-07 06:25:15 -04:00
Joseph Doherty d2bb32d97b Merge branch 'fix/gwc-28-29-30-polish'
ci / java (push) Successful in 2m44s
ci / windows-x86 (push) Successful in 1m3s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 17m13s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:20:14 -04:00
Joseph Doherty 404f7cd993 docs(archreview): close GWC-28, GWC-29, GWC-30, TST-28
Flip the four findings to Done in the 2026-07-12 tracking registers
(Gateway core + Testing) and in the per-domain registers of
10-gateway-core.md and 60-testing-docs-gaps.md; append the 2026-08-07
change-log row recording what shipped, the pre-fix red for GWC-28, and
the TST-28 mutation check.
2026-08-07 06:15:45 -04:00
Joseph Doherty eeee3e48a3 fix(GWC-30): reuse the frame reader's length-prefix scratch buffer
ReadAsync allocated a fresh 4-byte array per inbound frame; the GWC-08
pass pooled the payload buffer but left the prefix. Replaced with a
per-instance scratch field — the reader is single-consumer by
construction (one read loop per WorkerClient, handshake reads complete
before the loop starts), so a per-instance buffer is safe and the
non-reentrancy that makes it safe is now stated on the class. Pooling
four bytes via ArrayPool would cost more than the allocation it saves.

Tests: WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_
ParsesEveryFrame reads five frames of differing payload length through
one reader, so a stale prefix carried between calls would misparse.
2026-08-07 06:15:39 -04:00
Joseph Doherty a044f92c5d fix(GWC-29): drop the wasted request clone on the Invoke hot path
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.

Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.

The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.

Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
2026-08-07 06:15:31 -04:00
Joseph Doherty f27eb28063 fix(GWC-28): stamp gateway envelope sequence at write, not construction
CreateEnvelope stamped Sequence with an interlocked increment when the
envelope was built, so two concurrent InvokeAsync callers could take 1
and 2 and then enqueue in the order 2, 1 — non-monotonic on the wire,
breaking gateway.md's "monotonic per sender" contract. Benign today
(neither side validates inbound sequence, old GWC-10 still open) but it
would fault healthy sessions the moment worker-side validation lands.

WriteLoopAsync now stamps immediately before _writer.WriteAsync. It is
the outbound channel's single consumer (SingleReader = true), so wire
order and stamp order are the same thing by construction and
_nextSequence drops to a plain ulong with no interlocking. This mirrors
the worker's WRK-04 fix, which the gateway half never received.

Also adds TST-28: a [Theory] pinning that GatewayHello.MaxFrameBytes
carries the configured worker-frame maximum (default + 2 MiB override).
The adoption half is asserted only in the Windows-only worker suite, so
a regression to sending 0 — "older gateway, use default" to the worker —
would silently downgrade the negotiated IPC-02 limit with every CI test
still green. Mutation-checked (hard-coded 0 fails both cases).

Tests: WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequences
OnTheWire (32 parallel invokes; failed 3/3 pre-fix) and
.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes.
2026-08-07 06:15:20 -04:00
Joseph Doherty 3f854d6cbf Merge branch 'fix/sec-31-32-limiter'
ci / windows-x86 (push) Successful in 1m21s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 9m4s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:13:14 -04:00
Joseph Doherty 5b681ee59b fix(SEC-31,SEC-32): identify a probe-slot reservation by version, not by timestamp
ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.

WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.

Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
2026-08-07 06:10:14 -04:00
Joseph Doherty c836899d62 chore: untrack accidentally-committed agent worktree gitlinks; gitignore them
ci / java (push) Successful in 2m25s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / portable (push) Successful in 8m41s
2026-08-07 06:09:34 -04:00
Joseph Doherty 9825c69d92 Merge branch 'fix/cli-45-credential-envvar'
ci / java (push) Successful in 2m51s
ci / windows-x86 (push) Successful in 1m21s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 9m41s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:09:03 -04:00
Joseph Doherty 9357ff2dd4 Merge branch 'fix/cli-37-38-conformance'
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m14s
ci / portable (push) Successful in 9m13s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:08:12 -04:00
Joseph Doherty 37cb3b0df8 fix(CLI-45): standardize the CLI credential env var and fail fast on empty passwords
All five client CLIs now share one credential contract for `authenticate-user`:
flags `--password` / `--password-env` (Go: `-password` / `-password-env`) with
default env `MXGATEWAY_VERIFY_PASSWORD`, resolution flag-then-env, and a resolved
credential that is missing *or empty* is a usage error naming the flag and the
variable. The value is never echoed and never reaches the wire.

Go and Java previously sent an empty credential when the variable was unset,
turning a misconfigured environment into a real MXAccess authentication attempt.
Go now returns the guard error before dialing; Java throws a picocli
ParameterException instead of falling back to "". Python's `--password-env`
gained the canonical default and its UsageError names the resolved variable.
Rust treats an empty flag or env value as missing, with the resolution extracted
into a testable `resolve_verify_user_password`. .NET adopts the canonical flags
and keeps `--verify-user-password`, `--verify-user-password-env`, and
MXGATEWAY_VERIFY_USER_PASSWORD as deprecated aliases for one release.

Docs same commit: CrossLanguageSmokeMatrix.md gains the credential contract and
the per-CLI subcommand-coverage table (the documented-not-fixed half of the
finding); all five READMEs name the canonical variable and the fail-fast rule,
and the .NET README carries the deprecation note. Tracking flipped to Done in
both remediation registers with a change-log row.

No .proto changed; no generated code regenerated.
2026-08-07 06:05:00 -04:00
Joseph Doherty 6092172694 Merge branch 'fix/gwc-26-27-alarm-attach'
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m4s
ci / portable (push) Successful in 7m8s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
#	archreview/2026-07-12/remediation/10-gateway-core.md
2026-08-07 06:02:17 -04:00
Joseph Doherty d6b2f24c3f fix(CLI-37,CLI-38): make status/HRESULT reply validation conformant across all five clients
One cross-client conformance pass; also closes first-cycle CLI-08.

CLI-37: an MxStatusProxy entry is a failure iff `category !=
MX_STATUS_CATEGORY_OK`. The proto contract has always said so — `success` is
the raw 16-bit COM member carried verbatim for diagnostics, not a boolean — but
four clients branched on `success` alone and .NET required both, so the same
gateway reply produced opposite verdicts per language. An absent entry stays
success; a present entry with an UNSPECIFIED category is a failure, because the
worker always maps a category and an unmapped one is not proven OK.

CLI-38: a reply fails on HRESULT iff `hresult` is present and negative, so
positive COM success codes such as S_FALSE (1) pass. .NET/Go/Java used `!= 0`,
which errored on a parity-preserving S_FALSE that Python and Rust accepted.
This makes the existing ClientLibrariesDesign.md claim true rather than
rewriting the doc to describe the divergence.

Four shared fixtures pin both rules cross-client, and each language suite also
carries a table test for the two edges a fixture cannot express (absent entry,
UNSPECIFIED category). A Java test fake that built a status with a bare
`setSuccess(1)` and no category is fixed — under the category rule that reply
was never a success.
2026-08-07 06:00:58 -04:00
Joseph Doherty 09ccd9561f docs(GWC-26): record alarm feed repairs as at-least-once; share the channel worker fake
Code-review follow-up on fix/gwc-26-27-alarm-attach.

ApplyReconcile's snapshot-derived feed repairs are at-least-once, not
exactly-once: a reconcile reads the worker's current state while the matching
live transition may still be buffered in the monitor's lease, so both broadcast
and the duplicates are indistinguishable on the alarm feed. This pre-dates the
acked-state delta — the Raise/Clear presence repair has always had it, since
nothing serializes a reconcile pass against the in-flight live stream — so
closing it (serialization or timestamp dedup) stays out of scope for a P2 fix.
Documented instead, with the consumer contract stated explicitly (apply
transitions idempotently, never as an increment or toggle):

- ApplyReconcile gains a "Delivery semantics" comment.
- gateway.md softens the "defense in depth" prose to state the semantics.
- docs/Sessions.md carries the same caveat on the alarm-feed description.
- Tracker change-log records it as a known pre-existing characteristic and a
  candidate finding for the next review cycle.

Also hoists the ChannelWorkerClient fake — duplicated across the three alarm
test files — into TestSupport/, dropping the usings it took with it.
2026-08-07 06:00:16 -04:00
Joseph Doherty acebe18773 fix(SEC-31,SEC-32): make probe admission atomic and stop Reset clearing a shared fallback partition
Two defects found in code review of the limiter rework.

Probe admission was check-then-act across two lock scopes: Check() read
"probe due" under lock(state), released it, then re-acquired to advance
NextProbeAtTicks. A burst of requests arriving together at an interval boundary
could therefore all observe the slot as due and all be admitted, handing the
verifier the very burst the interval exists to bound. The claim is now a single
critical section (TryConsumeProbe). The two layers are still claimed one at a
time — holding two per-state locks at once would need a global lock ordering to
stay deadlock-free — so a slot claimed on the composite partition is compensated
via ReleaseProbe when the aggregate then refuses, which otherwise silently spent
the partition's next slot and pushed the legitimate holder out by a full
interval.

Reset() removed whatever partition the caller resolved to, including the
address's shared fallback partition when the caller's key id had been collapsed
into it by the per-peer cap (or when the token was junk-shaped). That bucket also
carries failures contributed by other key ids from the same address, so one
successful authentication became a reset button for an in-progress spray. Reset
now clears only a partition the caller owns (effectiveKeyId == presented key id);
the shared bucket decays by window expiry instead, and the caller still recovers
through probe admission. The key's aggregate is cleared either way, as designed.

Also applied from the review: closure-free GetOrAdd overload on _partitions, and
a remarks paragraph acknowledging the best-effort O(n) eviction scan under
sustained overflow. Threading the resolved partition key from Check through to
RecordFailure/Reset was declined: Check resolves with mint:false and RecordFailure
with mint:true, and the two can legitimately differ when a concurrent caller fills
the per-peer cap in between — reusing Check's key would record into the wrong
partition and bypass the cap, which is not worth saving one string concat.

Tests (limiter suite 11 -> 14): ProbeAdmission_UnderConcurrentArrivals_
GrantsExactlyOneSlot (200 rounds x 8 barrier-released threads at the boundary),
ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, and
Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. The latter two were
confirmed as genuine reds against the unfixed code; the concurrency test is a
guard — it is deterministically green on the fixed structure but did not
reproduce the original nanosecond-wide window on its own.
2026-08-07 05:57:26 -04:00
Joseph Doherty 1a75f61ebe test(GWC-26): deflake ApplyReconcileBroadcastsAcknowledgeDelta
Seeding the cache with a live Raise transition raced the first reconcile: when
the reconcile snapshot populated the cache first, the still-buffered live Raise
was applied — and broadcast — after the test's feed subscriber had registered,
so the exactly-one-transition assertion saw two. Seed through a reconcile pass
instead (forced by a provider-mode probe, as the acked step already did) so no
live transition is ever in flight. Verified: 5 consecutive full alarm-monitor
runs green, and the test still fails (timeout) with the ApplyReconcile
acked-delta branch removed.
2026-08-07 05:49:34 -04:00
Joseph Doherty cf66ebbcfb Merge branch 'fix/gwc-25-replaygap-trio'
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m10s
ci / windows-x86 (push) Successful in 1m30s
ci / portable (push) Successful in 7m39s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
#	archreview/2026-07-12/remediation/10-gateway-core.md
2026-08-07 05:49:13 -04:00
Joseph Doherty 44b8e37900 fix(GWC-25,CLI-35,CLI-36): make the empty-ring ReplayGap resumable end to end
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.

GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".

CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.

CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.

Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.

Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
  WhenRingEmptiedByAge, ...WithRetentionDisabled, and
  ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
  SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
  takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.

GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
2026-08-07 05:46:00 -04:00
Joseph Doherty 59a76da70b Merge branch 'fix/gwc-24-staging-bound'
ci / java (push) Successful in 3m4s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 8m49s
ci / windows-x86 (push) Failing after 6m59s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 05:41:32 -04:00
Joseph Doherty 3b6a239ed6 fix(GWC-26): attach the alarm monitor's lease before SubscribeAlarms
RunMonitorAsync issued SubscribeAlarms and the first reconcile before the
internal distributor subscriber was attached (via ISessionManager
.ReadAlarmEventsAsync). The pump has been running since MarkReady started the
dashboard mirror and only fans to subscribers registered at fan-out time, so
every transition raised in that two-round-trip window bypassed the alarm feed —
and a missed Acknowledge was never repaired, because ApplyReconcile broadcast
presence deltas only.

- The monitor now takes the internal lease directly from its session BEFORE
  SubscribeAlarms and drains it after the first reconcile; window transitions
  buffer in the lease's bounded channel. Processing them after ApplyReconcile is
  order-safe (ApplyTransition handles alarms the snapshot already placed).
- ISessionManager.ReadAlarmEventsAsync removed — zero remaining callers.
- ApplyReconcile broadcasts an Acknowledge feed transition when a both-present
  alarm's state advanced to ActiveAcked. This is a feed-level repair on the
  AlarmFeedMessage/StreamAlarms surface rebuilt from the worker's own snapshot,
  not MxEvent emission, so the "never synthesize events" rule is untouched;
  the reasoning is recorded on ApplyReconcile.

The alarm-monitor test fakes now hand the monitor a real Ready GatewaySession
with a dashboard mirror, which is what makes the window reproducible.

Docs: docs/Sessions.md and gateway.md alarm-monitor ordering notes.

Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-26
2026-08-07 05:40:33 -04:00
Joseph Doherty df710e18a9 fix(SEC-31,SEC-32): re-partition the API-key failure limiter on (peer, key id) with probe admission
The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).

ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.

SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.

Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.

Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
2026-08-07 05:39:10 -04:00
Joseph Doherty 33ba612ddd fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
2026-08-07 05:38:23 -04:00
Joseph Doherty d4154e340c fix(GWC-24): bound the worker event staging channel and unify the depth gauge
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
2026-08-07 05:35:07 -04:00
Joseph Doherty 1a63fdd7db fix(GWC-27): gate AttachInternalEventSubscriber on session readiness
AttachInternalEventSubscriber ran EnsureDistributorCreated / Register /
StartPumpIfRequested with no state check, unlike AttachEventSubscriber. A
premature attach would start the pump against a not-yet-Ready worker; the pump
source throws SessionNotReady, PumpAsync completes every subscriber with that
error and latches the distributor, and _eventDistributorStarted is never reset —
so the session would reach Ready with permanently dead event streaming.

Mirror AttachEventSubscriber's gate: check _state/_workerClient.State under
_syncRoot and throw SessionManagerException(SessionNotReady) before the
distributor is created, keeping the distributor calls outside the lock.

Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-27
2026-08-07 05:28:35 -04:00
Joseph Doherty ddb382c137 fix(TST-29): retire oldtasks.md; delete root docs-review artifacts
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m21s
ci / java (push) Successful in 2m5s
ci / portable (push) Successful in 7m31s
Migrate the durable session-resilience governance record (Phase 5
orphan-worker reattach deferred-not-planned, EnableOrphanReattach
does not yet exist, settled Phase-4 Viewer-default decision) from
oldtasks.md into a new "Session-Resilience Epic Scope" entry in
docs/DesignDecisions.md, repoint CLAUDE.md and stillpending.md's
oldtasks.md references to the new home / tasks.json, and git rm
oldtasks.md now that it has no unique content left. Flip TST-29 to
Done in the archreview tracking registers.

The five untracked root docs-review artifacts (MxAccessGateway-docs-*,
MxGatewayClient-docs-*) are absent from this worktree; they must be
deleted from the main working tree separately (gitignored, no repo
impact).
2026-08-07 05:25:42 -04:00
Joseph Doherty ead921cace docs: truth sweep — Galaxy adoption, Auth 0.1.5, redaction seam, resolved A2 caveats
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m16s
ci / java (push) Successful in 3m21s
ci / portable (push) Successful in 8m21s
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 01:57:01 -04:00
Joseph Doherty 47c0b646a9 fix(logging): redact command values on the shared ILogRedactor seam
ci / windows-x86 (push) Failing after 18s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m13s
ci / portable (push) Successful in 7m6s
GatewayLogRedactor.RedactCommandValue had no production caller. Its only
one was GatewayLogRedactorAdapter on the abandoned
feat/adopt-zb-telemetry-serilog branch; when that work was re-implemented
on main as GatewayLogRedactorSeam the identity half was carried over and
the command-value half was not. Four unit tests kept the policy green, so
it read as wired while masking nothing.

No live leak today — no log statement currently emits a CommandValue
property — but the next one to do so would have written credential-bearing
MXAccess payloads (AuthenticateUser, WriteSecured, WriteSecured2) to every
sink in the clear, with passing tests suggesting otherwise.

Ports the missing half onto the seam: a non-null CommandValue is masked via
the existing policy, gated on CommandMethod. Value logging stays off — the
seam exposes no opt-in — so ordinary values are masked too, matching
RedactCommandValue's default. A null value stays null rather than becoming
the placeholder, and the property is never invented when absent.

Five tests added, three of which were red first on the leak itself
(operator01:hunter2 reaching the assertion unmasked). The other two pin
the null and absent guards. Identity redaction is untouched.

Full NonWindows suite: 785 pass, 45 pre-existing macOS NamedPipe-harness
failures unchanged from baseline (verified by stashing this change).
Build 0 warnings.
2026-07-27 17:01:04 -04:00
Joseph Doherty aecc50a14b docs(claude): add Sister Projects section + cross-repo index propagation rule
ci / java (push) Successful in 2m2s
ci / windows-x86 (push) Failing after 29s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 7m20s
2026-07-27 15:25:14 -04:00
Joseph Doherty 2d54ace5d9 chore(health): bump ZB.MOM.WW.Health to 0.2.0
Family version-matrix alignment. No behaviour change — mxgw registers no Akka
checks, and 0.2.0's per-entry `data` object is emitted only when a check
publishes some, so its health payloads are byte-identical.

Note: this repo uses inline package pins, NOT central package management (there is
no Directory.Packages.props), so the version lives in the Server csproj.

Verified: Server builds 0 warnings.
Part of scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Task 2.3.
2026-07-24 05:54:40 -04:00
Joseph Doherty 8f7ee492ba chore(secrets): bump to Secrets 0.2.3 - visible delete modal (scadaproj#2)
ci / windows-x86 (push) Successful in 1m17s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m29s
ci / portable (push) Successful in 7m59s
0.2.3's Secrets.Ui ships ConfirmDeleteModal's own styles under
collision-proof zb-secrets-* class names. This host links no Bootstrap so
it never exhibited the invisible-modal defect, but it takes the fixed
line for parity; also rides over 0.2.1/0.2.2 (Akka-replicator fixes -
inert here, no replicator in use). Tests: 780 pass, 45 fail on macOS both
before and after the bump (NamedPipeServerStream multi-instance is
Windows-only - the fake-worker pipe harness cannot run on this platform);
zero delta from the bump.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 01:22:23 -04:00
158 changed files with 10328 additions and 939 deletions
+3
View File
@@ -152,3 +152,6 @@ generated-scratch/
*-docs-issues.md
*-docs-fixed.md
*-docs-final.md
# Agent worktrees (subagent isolation) — never commit
.claude/worktrees/
+25 -16
View File
@@ -156,8 +156,11 @@ accept a `browseSubtreeGlobs` param, so either fix is small plumbing:
Delete mxaccessgw's own `Galaxy/GalaxyRepositoryServiceCollectionExtensions.cs` registrations.
4. **Option validation** — the shared lib **binds only, ships no validator** (deliberate). mxaccessgw
already validates Galaxy options via `Configuration/GatewayOptionsValidator.cs` — **keep that**; it
stays the owner of fail-fast validation, exactly as HistorianGateway's `ConfigPreflight` does.
stays the owner of fail-fast Galaxy validation. **Updated 2026-08-07 (SEC-33):** this is now a dedicated
`Configuration/GalaxyRepositoryOptionsValidator.cs` (registered as `IValidateOptions<GalaxyRepositoryOptions>`
with `ValidateOnStart`), which enforces a valid, host-rooted `SnapshotCachePath` when `PersistSnapshot`
is true — exactly as HistorianGateway's `ConfigPreflight` does. (The original handoff pointed at
`GatewayOptionsValidator.cs`, but that validator does not see the lib-bound `GalaxyRepositoryOptions`.)
5. **Health check** — keep mxaccessgw's existing Galaxy-SQL readiness check; read the connection
string from the same `MxGateway:Galaxy` section the lib binds (HistorianGateway does this with a raw
@@ -189,20 +192,26 @@ be **deleted**. **Keep** the mxaccessgw-specific ones that exercise behavior the
## Post-adoption notes / caveats
- **Deployment config (NSSM):** the deployed services (`MxAccessGw` on 10.100.0.48; the wonder host) read
config from **NSSM environment variables, not `appsettings.json`**. The lib's `SnapshotCachePath` default
is empty (persistence no-ops). `appsettings.json` sets `MxGateway:Galaxy:SnapshotCachePath` +
`PersistSnapshot`, but the deployments must carry `MxGateway__Galaxy__SnapshotCachePath` and
`MxGateway__Galaxy__PersistSnapshot` in their NSSM env on redeploy, or snapshot persistence silently
no-ops in production.
- **Pre-existing NU1903 (unrelated):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
2.1.11 advisory (GHSA-2m69-gcr7-jv3q, no upstream patch) that breaks the build under `TreatWarningsAsErrors`
— already red on `main`. Resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
(its own commit). Remove the suppression once a patched e_sqlite3 ships.
- **Pre-existing IntegrationTests break (unrelated, NOT fixed here):** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
constructs `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
the IntegrationTests project does not compile (already broken on `main`, independent of Galaxy). The Galaxy
live tests there were rebound to the lib and compile in isolation, but the project won't build until that
unrelated call site is fixed. Track separately.
config from **NSSM environment variables, not `appsettings.json`**. **Updated 2026-08-07 (SEC-33):** the
lib's own `SnapshotCachePath` default is empty (would no-op persistence), but mxaccessgw no longer relies
on it. `appsettings.json` no longer sets `SnapshotCachePath` at all; instead the gateway seeds a
`CommonApplicationData`-derived default (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on the Windows
hosts) when the bound value is blank, and `GalaxyRepositoryOptionsValidator` fails startup if
`PersistSnapshot` is true with a non-rooted/invalid path. So `MxGateway__Galaxy__SnapshotCachePath` in the
NSSM env is now **optional** (an override), not required — a deployment that omits it gets the rooted host
default and persistence works; it no longer silently no-ops. `MxGateway__Galaxy__PersistSnapshot` still
governs whether persistence runs at all.
- **Pre-existing NU1903 (unrelated) — ✅ RESOLVED (2026-07-18, commit `2f0cfe3`):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
2.1.11 advisory (GHSA-2m69-gcr7-jv3q, at the time no upstream patch) that breaks the build under `TreatWarningsAsErrors`
already red on `main`. Initially resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
(its own commit). The patched e_sqlite3 (2.1.12) has since shipped: the suppression was **removed** and the
patched native lib pinned intentionally (`src/Directory.Build.props` now documents this in place of the suppression).
- **Pre-existing IntegrationTests break (unrelated, NOT fixed here) — ✅ RESOLVED since:** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
constructed `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
the IntegrationTests project did not compile (already broken on `main`, independent of Galaxy). The call site
has since been fixed to match the 3-arg ctor (`sessionManager, options, metrics` —
`Grpc/EventStreamService.cs:11-14`; call site at `WorkerLiveMxAccessSmokeTests.cs:~1580`) and the project compiles
(verified 2026-08-07).
- **No republish needed for the lib test additions:** the browse-projector / deploy-notifier / refresh-service
tests were added to the lib AFTER 0.2.0 was published; tests aren't shipped, so 0.2.0 is unchanged.
+37 -1
View File
@@ -15,6 +15,42 @@ The architecture is a two-process design — read `gateway.md` before making str
The worker must do all MXAccess COM calls on its dedicated STA thread, and the STA loop must pump Windows messages (`MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage`) so MXAccess events deliver. A plain blocking queue on an STA is not enough.
## Sister Projects (scadaproj umbrella)
`mxaccessgw` is one of a family of related SCADA / OT / Wonderware / OPC UA **sister
projects** cloned as sibling directories under `~/Desktop/`. They are **separate repos
and separate processes**, coupled at runtime over wire protocols (gRPC + OPC UA) — not
by project/compile references — and share the `ZB.MOM.WW.*` product namespace. **mxaccessgw
is the linchpin**: the only component that loads 32-bit MXAccess COM, so the others depend
on it over gRPC rather than touching COM directly.
- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) — the
umbrella/index workspace that aggregates the whole family (purpose, location, stack,
primary commands per project) and hosts the shared `ZB.MOM.WW.*` libraries, the dev/test
GLAuth (`10.100.0.35:3893`, source of truth `scadaproj/infra/glauth/`), and the shared
**`ZB.MOM.WW.GalaxyRepository`** package this gateway consumes. **This repo is indexed
there** — see the MxAccessGateway entry in `../scadaproj/CLAUDE.md`.
- **`~/Desktop/OtOpcUa`** (`lmxopcua`, [`../OtOpcUa/CLAUDE.md`](../OtOpcUa/CLAUDE.md)) — OPC
UA server whose in-process `GalaxyDriver` **depends on this gateway** for live Galaxy
read/write/subscribe and Galaxy Repository browse.
- **`~/Desktop/ScadaBridge`** ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) —
distributed SCADA platform whose Data Connection Layer has a dedicated **MxGateway
adapter** that talks to this gateway directly (native MxAccess data + A&C alarms),
bypassing OtOpcUa.
- **`~/Desktop/HistorianGateway`** (`historiangw`,
[`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process gRPC
sidecar **patterned on this gateway** (session model, dashboard shell, auth interceptor,
Galaxy SQL browse) but with no COM / no x86 worker. It also consumes the shared
`ZB.MOM.WW.GalaxyRepository` package — the same library mxaccessgw adopted (2026-06-25),
so the `galaxy_repository.v1` wire contract is served from one implementation.
**Propagate cross-repo changes to the umbrella index.** When a fact the index records about
mxaccessgw changes here — remote/push status, the `.proto` contracts (`mxaccess_gateway.proto`,
`mxaccess_worker.proto`, `galaxy_repository.proto`), the two-process architecture, shared-lib
consumption, or per-project commands — update the **MxAccessGateway entry in
[`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index
never drifts from this repo. (Mirrors the same rule in the peer repos.)
## Build, Test, Run
```powershell
@@ -76,7 +112,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
- **Style guides** in `docs/style-guides/` are authoritative. Follow `CSharpStyleGuide.md` for gateway/worker/.NET-client code: file-scoped namespaces, `sealed` by default, `Async` suffix on Task-returning methods, MXAccess-aligned names (`MxStatusProxy`, `ServerHandle`, `ItemHandle`, `HResult`).
- **MXAccess parity is the contract.** Don't "fix" surprising MXAccess behavior (e.g., `WriteSecured` failing before a value-bearing NMX body, distinct `OperationComplete` semantics, invalid-handle exceptions) unless the client explicitly opts into a non-parity mode. The installed MXAccess COM component is the baseline.
- **Don't synthesize events.** The gateway forwards only events the worker emits; it never invents `OperationComplete` from write completion or command replies.
- **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `oldtasks.md` (session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`.
- **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `docs/DesignDecisions.md` (Session-Resilience Epic Scope, session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`.
- **Gateway restart does not reattach orphan workers.** The first version terminates orphaned workers on startup; do not design code paths that assume reattachment.
- **No Blazor UI component libraries.** Dashboard uses local Bootstrap CSS/JS only — do not introduce MudBlazor, Radzen, FluentUI, etc.
- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, and `AuthenticateUser` credentials must never reach logs. Value logging is opt-in and redacted.
@@ -14,7 +14,7 @@
| SEC-06 | Done | **Partial** | Production hard-stop verified: `GatewayOptionsValidator.cs:161-165` (`Transport==None` in Production → startup error). Docs: `docs/GatewayConfiguration.md:244-248` (env-var override `MxGateway__Ldap__ServiceAccountPassword` documented). | **The committed dev service-account password is still in the repo at `appsettings.json:29`** (`Ldap.ServiceAccountPassword`) and has not been rotated — the doc itself says it "should be rotated". The transport guard shipped; the credential-removal/rotation half of the remediation did not. → **SEC-36** |
| SEC-07 | Done | **Yes** | `Security/Authorization/GatewayGrpcScopeResolver.cs:23` (`QueryActiveAlarmsRequest => GatewayScopes.EventsRead`); both tests now construct the real type (`Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:330,350`). | |
| SEC-08 | Done | **Yes** | `Security/Authentication/CachingApiKeyVerifier.cs` (15 s success-only TTL cache keyed on SHA-256 of the presented token, `:96-120`; only successes cached `:110-117`); `Security/Authentication/CoalescingMarkApiKeyStore.cs:76-113` (≤1 `last_used` write/key/60 s); wired as decorators in `Security/Authentication/AuthStoreServiceCollectionExtensions.cs:89-98`; invalidation on dashboard revoke/rotate/delete at `Dashboard/DashboardApiKeyManagementService.cs:104,144,190`; per-call constraints JSON deserialize removed via blob cache (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:22-45`). Tests exist (`Tests/Security/Authentication/CachingApiKeyVerifierTests.cs`). | New surface reviewed in depth — see SEC-34 (staleness/race, bounded) below. |
| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.4 (documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.5 (pins bumped 0.1.4→0.1.5 in `e107019`, a transitive-dependency security fix only — the expiry enforcement is unchanged; documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
| SEC-11 | Done | **Yes, with new defects** | Login: fixed-window per-remote-IP limiter policy (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:27-41`), applied to POST `/auth/login` (`:83`), registered + 429 (`GatewayApplication.cs:111-126`), middleware in pipeline (`GatewayApplication.cs:46`), test `Tests/Gateway/Dashboard/DashboardLoginRateLimitTests.cs`. gRPC: `Security/Authorization/ApiKeyFailureLimiter.cs` checked **before** the store read (`GatewayGrpcAuthorizationInterceptor.cs:72-77`), failure recorded `:89`, reset on success `:97`; interceptor test asserts the short-circuit (`GatewayGrpcAuthorizationInterceptorTests.cs:395-402`). | The limiter exists and is enforced, but its key-id partitioning creates an unauthenticated lockout DoS (**SEC-31**) and its LRU eviction is flushable (**SEC-32**). |
| SEC-12 | Done | **Yes** | `Dashboard/DashboardSessionAdminService.cs`: canonical `AuditEvent`s `dashboard-close-session`/`dashboard-kill-worker` (`:37-40`), written on Denied (`:65,147`), Success (`:89-96,171-178`), and every Failure arm (`:105,116,132,187,198,214`), category `SessionAdmin`, actor/remote/correlation captured (`:238-261`), via `IAuditWriter` (same store as API-key events). | Audit-event completeness is good: denied, not-found, faulted, and unexpected paths all emit. |
| SEC-20 | Done | **Yes** | `Metrics/GatewayMetrics.cs:374``_heartbeatFailuresCounter.Add(1)` with no `session_id` tag (rationale comment `:370-373`). | The in-memory per-session map remains dashboard-only. |
File diff suppressed because one or more lines are too long
@@ -8,19 +8,19 @@ This document turns the 2026-07-12 re-review's **new** Gateway Server Core findi
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| GWC-24 | Medium | P1 | M | GWC-21 (coord) | Not started | Unbounded event staging channel: sustained slow drain grows memory silently and invisibly |
| GWC-25 | Medium | P0 | S | CLI-35/36 (coord) | Not started | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
| GWC-26 | Low | P2 | M | GWC-27 | Not started | Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed, missed Acknowledge never repaired |
| GWC-27 | Low | P2 | S | GWC-26 | Not started | `AttachInternalEventSubscriber` bypasses the readiness gate; premature attach poisons the distributor permanently |
| GWC-28 | Low | P2 | S | GWC-10 (coord) | Not started | Gateway→worker envelope `sequence` stamped at creation, not at write — non-monotonic on the wire under concurrent invokes |
| GWC-29 | Low | — | S | — | Not started | `Invoke` deep-clones the entire request only to discard the cloned command |
| GWC-30 | Info | — | S | — | Not started | Frame reader allocates a fresh 4-byte length-prefix array per frame |
| GWC-24 | Medium | P1 | M | GWC-21 (coord) | Done | Unbounded event staging channel: sustained slow drain grows memory silently and invisibly |
| GWC-25 | Medium | P0 | S | CLI-35/36 (coord) | Done | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
| GWC-26 | Low | P2 | M | GWC-27 | Done | Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed, missed Acknowledge never repaired |
| GWC-27 | Low | P2 | S | GWC-26 | Done | `AttachInternalEventSubscriber` bypasses the readiness gate; premature attach poisons the distributor permanently |
| GWC-28 | Low | P2 | S | GWC-10 (coord) | Done | Gateway→worker envelope `sequence` stamped at creation, not at write — non-monotonic on the wire under concurrent invokes |
| GWC-29 | Low | — | S | — | Done | `Invoke` deep-clones the entire request only to discard the cloned command |
| GWC-30 | Info | — | S | — | Done | Frame reader allocates a fresh 4-byte length-prefix array per frame |
Dependency notes: GWC-26 and GWC-27 both change the internal-subscriber attach path (`GatewaySession.AttachInternalEventSubscriber` and its `SessionManager`/alarm-monitor callers) — land GWC-27's readiness gate first (or in the same commit), then GWC-26's reorder, so the reordered monitor attach is proven against the gate. GWC-24 is the direct successor of the prior cycle's GWC-04 backpressure fix and raises the value of the still-open GWC-21 (making `EventChannelFullModeTimeout` configurable); GWC-28 is the gateway half of the worker's WRK-04 fix and must be coordinated with the still-open GWC-10 if inbound sequence enforcement is ever added. GWC-25 is server-complete on its own, but the end-to-end reconnect story also needs the client-domain CLI-35/36 fixes (Python CLI crashes on the sentinel, Go CLI destroys it).
---
## GWC-24 — Unbounded event staging channel: sustained slow drain grows memory silently and invisibly `Medium` · `P1`
## GWC-24 — Unbounded event staging channel: sustained slow drain grows memory silently and invisibly `Medium` · `P1` · **Done (2026-08-07)**
**Finding.** The GWC-04 remediation decoupled the read loop from event backpressure by staging events into `_eventStaging`, an **unbounded** channel (`Workers/WorkerClient.cs:93-100`). `StageWorkerEvent`'s `TryWrite` therefore always succeeds (`:565-573`), and the sustained-overflow `ProtocolViolation` fault fires only when a *single* timed `WriteAsync` against the bounded `_events` exceeds `EventChannelFullModeTimeout` (default 5 s, `:610-647`). The queue-depth gauge counts only `_events``_eventQueueDepth` is incremented in `EnqueueWorkerEventAsync` (`:616`, `:627`) and decremented in `ReadEventsCoreAsync` (`:289`), so staged-but-unqueued events are invisible to `SetWorkerEventQueueDepth`. The field comment (`:28-33`) claims staging "only fills during the bounded EventChannelFullModeTimeout window", which is true only for a full consumer stall, not for a consumer that drains slower than the worker produces while each individual write still completes inside the window.
@@ -16,14 +16,14 @@ members, no positional records). The worker builds and tests only on the Windows
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Not started | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Not started | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Not started | WRK-12 flush coalescing never engages on the event hot path |
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Not started | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Done | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-24 | Low | — | S | — | Done | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Done | WRK-12 flush coalescing never engages on the event hot path |
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
---
@@ -12,14 +12,14 @@ All `path:line` citations were re-verified against the working tree at `4f5371f`
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| IPC-23 | Medium | P0 | S¹ | WRK-21 | Not started | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
| IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
| IPC-26 | Low | P2 | S¹ | WRK-22 | Not started | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
| IPC-26 | Low | P2 | S¹ | WRK-22 | Done (mechanics landed in WRK-22) | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
| IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
| IPC-28 | Low | — | S | — | Not started | `docs/Grpc.md` omits the `CommandTooLarge``ResourceExhausted` mapping |
| IPC-29 | Low | — | S | — | Not started | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Not started | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
| IPC-28 | Low | — | S | — | Done | `docs/Grpc.md` omits the `CommandTooLarge``ResourceExhausted` mapping |
| IPC-29 | Low | — | S | — | Done (discharged by WRK-26) | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Done | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
| IPC-31 | Info | — | — | — | N/A | Gateway stamps sequence at creation, worker at write — accepted divergence; sequence is documented diagnostic-only (`gateway.md:328-330`); revisit only if sequence ever becomes load-bearing |
| IPC-32 | Info | — | S | IPC-25 | Not started | `check-codegen.ps1` check labels miscounted (folded into the IPC-25 script edit) |
@@ -10,11 +10,11 @@ Repo rules that bind every entry: docs change in the same commit as the source (
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| SEC-31 | Medium | P0 | M | — | Not started | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
| SEC-32 | Low | P0 | S | SEC-31 | Not started | Failure-limiter LRU is flushable by junk-token spray; token prefix never validated |
| SEC-33 | Low | P1 | M | — (co-locate SEC-23) | Not started | Any-platform path-rooting acceptance re-opens SEC-01 on Unix; Galaxy `SnapshotCachePath` unvalidated |
| SEC-34 | Low | P2 | S | — | Not started | Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation |
| SEC-35 | Info | — | S | — | N/A (doc-only note) | Production hard-stops key on the exact `Production` environment name |
| SEC-31 | Medium | P0 | M | — | Done | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
| SEC-32 | Low | P0 | S | SEC-31 | Done | Failure-limiter LRU is flushable by junk-token spray; token prefix never validated |
| SEC-33 | Low | P1 | M | — (co-locate SEC-23) | Done | Any-platform path-rooting acceptance re-opens SEC-01 on Unix; Galaxy `SnapshotCachePath` unvalidated |
| SEC-34 | Low | P2 | S | — | Done | Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation |
| SEC-35 | Info | — | S | — | N/A (doc-only note discharged 2026-08-07) | Production hard-stops key on the exact `Production` environment name |
| SEC-36 | Low | P1 | M | cross-repo (`scadaproj/infra/glauth`) | Not started | Committed dev LDAP service-account password: remove from repo and rotate |
---
@@ -135,6 +135,8 @@ dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --fil
```
Post-run, verify no new `C:\*` file exists under any `bin/` (manual `find src -name 'C:*'`).
**Outcome (2026-08-07 — Done).** Implemented as designed. `IsRootedForAnyPlatform` deleted; `AddIfNotRooted`/`AddIfInvalidPath` promoted to a shared `GatewayConfigPathRules` internal helper (used by both validators) and now use `Path.IsPathRooted` (current OS). Both Windows literals removed from `appsettings.json`; the Galaxy `SnapshotCachePath` default is applied gateway-side via a configuration value seeded before `AddZbGalaxyRepository` (the package's `SnapshotCachePath` is **init-only**, so a `PostConfigure` mutation does not compile — deviation from the design's "PostConfigure default"; same effect). New `GalaxyRepositoryOptionsValidator` registered with `ValidateOnStart`. **Stray-file root cause:** starting the full host eagerly constructs `AuthSqliteConnectionFactory`, which creates the auth DB path; with the shipped Windows literal that path is non-rooted on macOS, so SQLite materialized `C:\ProgramData\MxGateway\gateway-auth.db` as a junk-named relative file under the test's `bin/` CWD (invisible to the hygiene test's bin/obj filter). After the literal removal the code default resolves under an unwritable `/usr/share` on macOS, so the three tests that start the real host (`GatewayApplicationTests.Build_MapsMetricsEndpoint`, `.StartAsync_InvalidGatewayConfiguration_FailsStartup`, `GatewayTlsBootstrapTests`) now pin `SqlitePath` to a temp path. No stray file remains (`find src -name 'C:*'` empty).
---
## SEC-34 — Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation `Low` · `P2`
@@ -145,7 +147,7 @@ Post-run, verify no new `C:\*` file exists under any `bin/` (manual `find src -n
**Design.**
- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.4) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.5 — bumped from 0.1.4 in `e107019`, no API change) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
- **Invalidate race (window 3): per-key generation check.** `ConcurrentDictionary<string, long> _generations`; `Invalidate(keyId)` increments the generation **before** evicting cache keys. `VerifyAsync` parses the key id from the token up front (same split the interceptor does — cheap, no store access), snapshots `g0` before calling the inner verifier, and after a success only `Set`s when the generation still equals `g0` — then re-reads the generation after the `Set` and self-evicts if it moved (bump-before-evict + set-then-recheck closes the remaining interleaving). Unparseable tokens skip caching already (`TryComputeCacheKey`).
- **CLI window (1): accept and keep documented** — cross-process invalidation is out of scope by design; the TTL is the backstop and the remarks already say so.
@@ -165,6 +167,8 @@ Post-run, verify no new `C:\*` file exists under any `bin/` (manual `find src -n
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~CachingApiKeyVerifier"
```
**Outcome (2026-08-07 — Done).** Window 3 (Invalidate race) implemented exactly as designed: per-key generation counter, bump-before-evict in `Invalidate`, snapshot-before-inner + set-then-recheck in `VerifyAsync`, key id parsed from the token up front. Covered by `Invalidate_DuringInFlightVerification_DiscardsStaleRepopulation`. **Window 2 (expiry cap) took the documented fallback**, not the cap: the design's confirmation step failed — the library verification identity (`ZB.MOM.WW.Auth.Abstractions.ApiKeys.ApiKeyIdentity`, the type on `ApiKeyVerification.Identity`) carries **no** `ExpiresUtc` (that property is on `ApiKeyRecord`, the store row, not the returned identity), so the cache cannot cap an entry at the key's expiry. Per the design's contingency, the ≤ TTL expiry window is documented in the class remarks and `docs/Authentication.md`, with a donor-library ask (surface expiry on the verification identity). Consequently the two expiry-cap tests (`CacheEntry_DoesNotOutliveKeyExpiry`, `AlreadyExpiredIdentity_IsNotCached`) are **not** added — they cannot be written against a type with no expiry field; window 1 (CLI) accepted and documented as before.
---
## SEC-35 — Production hard-stops key on the exact `Production` environment name `Info` · `—` (N/A: doc-only)
@@ -179,6 +183,8 @@ dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --fil
**Verification.** Doc-only; no build. Cross-read against `GatewayOptionsValidator.cs:23-27`.
**Outcome (2026-08-07 — discharged).** The documentation contract landed as a rider on the SEC-33/34 commit: `docs/GatewayConfiguration.md` gained a "Production hard-stops key on the exact environment name (SEC-35)" subsection stating that both hard-stops fire only on `IHostEnvironment.IsProduction()` (unset `ASPNETCORE_ENVIRONMENT` or the exact `Production` name) and that any other name keeps the permissive dev posture. No code change, as designed.
---
## SEC-36 — Committed dev LDAP service-account password: remove from repo and rotate `Low` · `P1` · cross-repo dependency
+10 -10
View File
@@ -16,17 +16,17 @@ Operating constraints carried from prior work:
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| CLI-35 | Medium | P0 | S | — | Not started | Python CLI `stream-events` crashes on a ReplayGap |
| CLI-36 | Medium | P0 | S | — | Not started | Go CLI `stream-events` silently destroys the ReplayGap signal |
| CLI-37 | Medium | P1 | M | CLI-38 | Not started | Status-array validation must branch on `category` per the proto contract (4-vs-1 divergence) |
| CLI-38 | Medium | P1 | S | — | Not started | Align .NET/Go/Java on `hresult < 0` — lands prior CLI-08 and cures the design-doc drift |
| CLI-35 | Medium | P0 | S | — | Done | Python CLI `stream-events` crashes on a ReplayGap |
| CLI-36 | Medium | P0 | S | — | Done | Go CLI `stream-events` silently destroys the ReplayGap signal |
| CLI-37 | Medium | P1 | M | CLI-38 | Done | Status-array validation must branch on `category` per the proto contract (4-vs-1 divergence) |
| CLI-38 | Medium | P1 | S | — | Done | Align .NET/Go/Java on `hresult < 0` — lands prior CLI-08 and cures the design-doc drift |
| CLI-39 | Medium | P1 | S | CLI-35..38, CLI-45 | Not started | Bump client versions off the already-published 0.1.2 before the next publish; add registry-collision guard |
| CLI-40 | Low | — | M | — | Not started | Port the exact-secret credential scrub to Rust/Java/.NET |
| CLI-41 | Low | — | M | — | Not started | Uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem |
| CLI-42 | Low | P1 | S | — | Not started | Document the vendored Rust proto layout (CLI-02's missing doc half) |
| CLI-43 | Low | — | S | — | Not started | Java style guide still prescribes "Java 21 preferred" |
| CLI-44 | Low | — | S | — | Not started | Go event goroutine can mislabel a genuine terminal error as `ErrSlowConsumer` |
| CLI-45 | Low | P1 | M | — | Not started | Standardize CLI credential env-var name and fail fast on missing/empty passwords |
| CLI-40 | Low | — | M | — | Done | Port the exact-secret credential scrub to Rust/Java/.NET |
| CLI-41 | Low | — | M | — | Done | Uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem |
| CLI-42 | Low | P1 | S | — | Done | Document the vendored Rust proto layout (CLI-02's missing doc half) |
| CLI-43 | Low | — | S | — | Done | Java style guide still prescribes "Java 21 preferred" |
| CLI-44 | Low | — | S | — | Done | Go event goroutine can mislabel a genuine terminal error as `ErrSlowConsumer` |
| CLI-45 | Low | P1 | M | — | Done | Standardize CLI credential env-var name and fail fast on missing/empty passwords |
Cross-domain dependencies: **CLI-35/CLI-36 pair with GWC-25** (gateway emits `oldest_available_sequence = 0` on an empty replay ring — the server-side half of the same reconnect story; the CLI fixes here are independently landable but the end-to-end resume walk in the smoke matrix needs both). **CLI-39 pairs with the publishing process** (`scripts/pack-clients.ps1`, `scripts/tag-go-module.ps1`, Gitea package registry).
@@ -12,9 +12,9 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior
|----|-----|------|-----|-----|--------|-------|
| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | Done | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job |
| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | Done | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs |
| TST-27 | Medium | P1 (doc batch) | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
| TST-28 | Low | P2 | S | relates IPC-02 | Not started | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
| TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
| TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
| TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
| TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
---
+3 -3
View File
@@ -181,7 +181,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
| CLI-08 | Medium | — | S | CLI-03 | Not started | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) |
| CLI-08 | Medium | — | S | CLI-03 | Done | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) — landed via 2026-07-12 [CLI-38](../2026-07-12/remediation/50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1) |
| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
@@ -236,7 +236,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification |
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification. **Gate cleared:** TST-03 CI is Done (live and green 2026-07-10; Windows/x86 tier green 2026-07-13 via the TST-25/TST-26 SSH-driven windev job), so TST-24 is unblocked — deferred by choice now, not CI-gated |
## Cross-cutting clusters
@@ -276,7 +276,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
| 2026-07-09 | P1 Wave 3 (size/backpressure topology + write ordering). IPC-02/03/04 + WRK-04/07 → `Done`. **Size negotiation (IPC-02):** added `GatewayHello.max_frame_bytes` (regen `Generated/` + `clients/proto` descriptor refresh with pinned protoc 34.1); the gateway sends its negotiated worker-frame max and the worker adopts it (`WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes`, 0 = keep default, >256 MiB rejected) instead of a hard-coded default. **Headroom (IPC-03):** the pipe frame max now sits `EnvelopeOverheadReserveBytes` (64 KiB) above the public gRPC cap (default `Worker.MaxMessageBytes` 16 MiB→16 MiB+64 KiB), cross-validated at startup; `WorkerClient` pre-checks command envelope size and fails only the offending correlation (`ResourceExhausted`) instead of `SetFaulted`ing the session. **Drain bound (IPC-04):** gateway request validator rejects `DrainEvents max_events` above 10 000; the worker caps each reply at `MaxDrainEventsPerReply` (10 000) and treats `max_events = 0` as that cap, not "drain all". **Sequence (WRK-04):** `WorkerFrameWriter` stamps the envelope `Sequence` at the point of writing under the write lock, so wire order and stamped sequence always agree under concurrent producers. **Priority (WRK-07):** the worker writer is now a cooperative priority scheduler — control frames (reply/fault/heartbeat/shutdown-ack) drain ahead of event frames; per-frame validation/size rejections fail only that frame, a stream failure fails all queued. Docs same-change (GatewayConfiguration, WorkerFrameProtocol, gateway.md). **Verified:** macOS NonWindows build clean + validator/grpc tests green; **windev** x86 worker builds clean, `Worker.Tests` 352 passed / 0 failed / 11 skipped (incl. new monotonic-sequence, control-before-event priority, negotiated-max, drain-bound tests), gateway `Tests` 799 passed / 3 failed — all 3 pre-existing windev-environmental (SelfSigned SAN + 2 `EventStreamServiceTests` timing, both pass in isolation). Commits `c8b3a22` (gateway half), `ebe6aea` (worker half), `309296f` (descriptor + default-expectation refresh). GWC-04 (event-channel decoupling) is the remaining Wave 3 item. |
| 2026-07-09 | P1 S-misc (dashboard/observability hardening). SEC-02/12/20 → `Done`. SEC-02: `DashboardAuthorizationHandler` restricts the loopback + `Authentication:Mode=Disabled` bypasses to read-only (they satisfy a Viewer-bearing requirement but never `AdminOnly`), closing the policy-layer gap where anonymous localhost was authorized for Admin surfaces. SEC-12: `DashboardSessionAdminService` now emits canonical `AuditEvent`s (`dashboard-close-session`/`dashboard-kill-worker`, category `SessionAdmin`) through `IAuditWriter` on Success/Failure/Denied, so Close/Kill land durable audit rows. SEC-20: dropped the unbounded `session_id` tag from the exported `mxgateway.heartbeats.failed` counter. Docs updated same-change (CLAUDE.md, GatewayDashboardDesign.md, Metrics.md). Server build clean (0 warnings); targeted classes 30/30 pass; broader Dashboard+Security+GatewayApplication+Metrics sweep 295/295 pass. |
| 2026-07-09 | P1 Wave 2b (security authz+hub). SEC-05/07/08/11 → `Done` (hub-token lifetime; QueryActiveAlarms scope arm; gateway-side verification cache + last-used coalescing; login rate limit + per-peer gRPC failure limiter). Full-suite checkpoint caught + fixed regressions the earlier narrow SEC-01/04/06 filter missed: cross-platform path-rooting, an `IHostEnvironment` fallback for minimal DI containers, a test-assembly `ASPNETCORE_ENVIRONMENT=Development` default, and a platform-correct default-path assertion. Suite: 747 passed / 42 failed, all 42 pre-existing macOS named-pipe-harness env failures. |
| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a; since bumped to 0.1.5 in `e107019` — 0.1.4 plus a transitive SQLitePCLRaw security pin, no API change, expiry enforcement retained). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
| 2026-07-09 | P1 Wave 1 (CI + codegen freshness + Rust buildability) via parallel agents. IPC-01/09/19/20, CLI-02 → `Done`; TST-03 → `In review` (CI pipeline authored + YAML/layout-validated, but not yet executed on a Gitea runner — proven on first push). Verified on macOS: NonWindows build clean, `ClientProtoInputTests` 5/5, `publish-client-proto-inputs.ps1 -Check` exit 0, `cargo package` (no `--no-verify`) compiles standalone. Added a vendored-Rust-proto drift guard (Check 3) to `check-codegen.ps1` closing the CLI-02 static-copy risk. IPC-09 script guards not executed end-to-end (need pinned python/JRE toolchains); logic is PATH-resolution + version assertion. |
| 2026-07-09 | WRK-01 → `Done`. Verified on Windows host (windev) via an isolated `origin/main` worktree: worker builds x86 clean, `StaRuntimeTests`+`WorkerPipeSessionTests` 33/33 pass. Fixed an `xUnit1030` build error (the new worker test used `.ConfigureAwait(false)` in `[Fact]` bodies) that the macOS tree could not surface. Also ran GWC-01's Windows-only `WorkerClientTests` on windev: 18/18 pass (incl. `ReadEventsAsync_SecondEnumerator_Throws`). All 8 P0 findings now `Done`. Not yet committed. |
| 2026-07-10 | **TST-03 → `Done`: CI is live and green** on branch `fix/ci-selfhosted-tooling`. The pipeline was authored (P1) but had never executed. Root cause it never ran: the co-located `gitea-runner` on the docker host (`10.100.0.35`) spawned job containers on an isolated network (`container.network: ""`) that could not resolve Gitea's internal clone URL `http://gitea:3000`; one-line host fix `container.network: "traefik"` + `docker restart gitea-runner`. The self-hosted `catthehacker` act image also lacks tooling GitHub-hosted runners preinstall — ci.yml now installs pwsh (dotnet global tool, for `check-codegen.ps1`) and Gradle 9.5.1 directly (act can't resolve the `gradle/actions` monorepo action; no gradle wrapper in repo). Driving to green surfaced and fixed **five real latent defects** (TST-03 doing its job): `check-codegen.ps1` `.Trim()`-on-`$null` on a clean tree; **stale vendored rust proto** (`clients/rust/protos/mxaccess_worker.proto` missing canonical `max_frame_bytes`); `OrphanWorkerTerminatorTests` hard-coded `C:\` path failing Linux `Path.GetFullPath` (0 kills); **stale java generated** `MxaccessWorker.java` (missing `max_frame_bytes`, regenerated); a sync python test building a `grpc.aio.Channel` with no current event loop on py3.12 (autouse conftest fixture). Result: `portable` **success** (NonWindows build + codegen freshness + 808/808 gateway tests + .NET/Go/Rust/Python clients) and `java` **success**. `windows` + `live-mxaccess` jobs remain `queued` pending a self-hosted **windev** runner (`10.100.0.48`) with those labels — separate follow-up, does not gate portable/java. |
+32
View File
@@ -163,6 +163,12 @@ can keep the full `MxCommandReply`, HRESULT, and status array when MXAccess
itself rejects a command. `MxAccessException.Reply` contains the raw generated
reply.
`EnsureMxAccessSuccess()` follows COM semantics: only a **negative** HRESULT is
a failure, so positive success codes such as `S_FALSE` (1) pass. A status entry
fails only when `Category` is not `MxStatusCategory.Ok``MxStatusProxy.Success`
mirrors the raw COM member for diagnostics and never decides the verdict, which
is why `IsSuccess()` branches on the category alone.
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
@@ -258,6 +264,32 @@ optionally writes a value when `--type` and `--value` are supplied, reads a
bounded event stream, and closes the session in a `finally` block. CLI error
output redacts API keys supplied through `--api-key`.
### `authenticate-user` credentials
```powershell
$env:MXGATEWAY_VERIFY_PASSWORD = "<verify-user password>"
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- authenticate-user --session-id <id> --server-handle 1 --verify-user operator --json
```
The credential comes from `--password` or, preferably, the environment variable
named by `--password-env` (default `MXGATEWAY_VERIFY_PASSWORD`) so it stays out
of shell history and the process table. It is never echoed to stdout or stderr,
and error output routes it through the same redaction seam as the API key. A
missing or empty resolved credential is a usage error naming the option and the
variable: the CLI fails before the invoke rather than authenticating with an
empty password.
`MXGATEWAY_VERIFY_PASSWORD` is the canonical variable across all five client CLIs
— see [Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
**Deprecated names.** This CLI previously used `--verify-user-password`,
`--verify-user-password-env`, and `MXGATEWAY_VERIFY_USER_PASSWORD`. All three
still resolve, for one release only, so existing scripts keep working; migrate to
the canonical names above. The full resolution order is `--password`,
`--verify-user-password`, the variable named by `--password-env` (or the
deprecated `--verify-user-password-env`, default `MXGATEWAY_VERIFY_PASSWORD`),
then `MXGATEWAY_VERIFY_USER_PASSWORD`.
## Galaxy Repository Browse
`GalaxyRepositoryClient` is a separate read-only wrapper around the
@@ -346,31 +346,78 @@ public static class MxGatewayClientCli
}
/// <summary>
/// Resolves the effective MXAccess verify-user credential from
/// <c>--verify-user-password</c> or, failing that, the
/// <c>--verify-user-password-env</c>-named environment variable (default
/// <c>MXGATEWAY_VERIFY_USER_PASSWORD</c>). The credential is never echoed;
/// this resolver exists so the error-redaction catch block can strip it
/// from any surfaced error (CLI-04), mirroring <see cref="TryResolveApiKey"/>.
/// Canonical CLI credential environment variable, shared by every official
/// client CLI (CLI-45) so one exported variable drives the same operator
/// workflow in all five languages.
/// </summary>
private const string DefaultVerifyPasswordEnvironmentName = "MXGATEWAY_VERIFY_PASSWORD";
/// <summary>
/// Pre-CLI-45 environment variable, still honoured as a deprecated fallback
/// for one release so existing scripts keep working.
/// </summary>
private const string LegacyVerifyPasswordEnvironmentName = "MXGATEWAY_VERIFY_USER_PASSWORD";
/// <summary>
/// Resolves the name of the environment variable holding the verify-user
/// credential: <c>--password-env</c>, then the deprecated
/// <c>--verify-user-password-env</c> alias, then
/// <c>MXGATEWAY_VERIFY_PASSWORD</c>.
/// </summary>
private static string ResolveVerifyPasswordEnvironmentName(CliArguments arguments)
{
string? environmentName = arguments.GetOptional("password-env");
if (!string.IsNullOrEmpty(environmentName))
{
return environmentName;
}
environmentName = arguments.GetOptional("verify-user-password-env");
return string.IsNullOrEmpty(environmentName)
? DefaultVerifyPasswordEnvironmentName
: environmentName;
}
/// <summary>
/// Resolves the effective MXAccess verify-user credential in the CLI-45
/// order: <c>--password</c>, the deprecated <c>--verify-user-password</c>
/// alias, the environment variable named by <c>--password-env</c> (default
/// <c>MXGATEWAY_VERIFY_PASSWORD</c>), then the deprecated
/// <c>MXGATEWAY_VERIFY_USER_PASSWORD</c>. An empty value from any source is
/// treated as absent. The credential is never echoed; this resolver exists so
/// the error-redaction catch block can strip it from any surfaced error
/// (CLI-04), mirroring <see cref="TryResolveApiKey" />.
/// </summary>
private static string? TryResolveVerifyUserPassword(CliArguments arguments)
{
string? password = arguments.GetOptional("verify-user-password");
string? password = arguments.GetOptional("password");
if (!string.IsNullOrEmpty(password))
{
return password;
}
string passwordEnvironmentName = arguments.GetOptional("verify-user-password-env")
?? "MXGATEWAY_VERIFY_USER_PASSWORD";
password = arguments.GetOptional("verify-user-password");
if (!string.IsNullOrEmpty(password))
{
return password;
}
return Environment.GetEnvironmentVariable(passwordEnvironmentName);
password = Environment.GetEnvironmentVariable(ResolveVerifyPasswordEnvironmentName(arguments));
if (!string.IsNullOrEmpty(password))
{
return password;
}
password = Environment.GetEnvironmentVariable(LegacyVerifyPasswordEnvironmentName);
return string.IsNullOrEmpty(password) ? null : password;
}
/// <summary>
/// Resolves the verify-user credential for <c>authenticate-user</c>, throwing
/// a redaction-safe error when neither the flag nor the env var is set. The
/// thrown message names only the option/env var, never the value.
/// a redaction-safe error when no source yields a non-empty value. Failing
/// fast keeps a misconfigured environment from becoming a real MXAccess
/// authentication attempt with an empty credential (CLI-45); the thrown
/// message names only the option/env var, never the value.
/// </summary>
private static string ResolveVerifyUserPassword(CliArguments arguments)
{
@@ -380,11 +427,10 @@ public static class MxGatewayClientCli
return password;
}
string passwordEnvironmentName = arguments.GetOptional("verify-user-password-env")
?? "MXGATEWAY_VERIFY_USER_PASSWORD";
throw new ArgumentException(
$"Verify-user password is required. Pass --verify-user-password or set {passwordEnvironmentName}.");
"Verify-user password is required. Pass --password or set "
+ $"{ResolveVerifyPasswordEnvironmentName(arguments)} (deprecated aliases: "
+ $"--verify-user-password, --verify-user-password-env, {LegacyVerifyPasswordEnvironmentName}).");
}
private static CancellationTokenSource CreateCancellation(CliArguments arguments, string command)
@@ -710,8 +756,10 @@ public static class MxGatewayClientCli
TextWriter output,
CancellationToken cancellationToken)
{
// The credential is resolved from --verify-user-password or its env var and
// is never echoed. On any surfaced error the RunCoreAsync catch block routes
// The credential is resolved from --password or its env var (default
// MXGATEWAY_VERIFY_PASSWORD) and is never echoed; a missing or empty value
// fails fast before the invoke rather than reaching the wire (CLI-45).
// On any surfaced error the RunCoreAsync catch block routes
// it through MxGatewayCliSecretRedactor so it cannot reach stderr (CLI-04).
return InvokeAndWriteAsync(
arguments,
@@ -2372,7 +2420,9 @@ public static class MxGatewayClientCli
writer.WriteLine("mxgw-dotnet activate --session-id <id> --server-handle <n> --item-handle <n> [--json]");
writer.WriteLine("mxgw-dotnet write-secured --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> --current-user-id <n> [--verifier-user-id <n>] [--json]");
writer.WriteLine("mxgw-dotnet write-secured2 --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> --current-user-id <n> [--verifier-user-id <n>] [--timestamp <iso>] [--json]");
writer.WriteLine("mxgw-dotnet authenticate-user --session-id <id> --server-handle <n> --verify-user <user> (--verify-user-password <pw> | --verify-user-password-env <ENVVAR>) [--json]");
writer.WriteLine("mxgw-dotnet authenticate-user --session-id <id> --server-handle <n> --verify-user <user> [--password <pw>] [--password-env <ENVVAR>] [--json]");
writer.WriteLine(" credential: --password, else the --password-env variable (default MXGATEWAY_VERIFY_PASSWORD); required and never empty.");
writer.WriteLine(" deprecated aliases: --verify-user-password, --verify-user-password-env, MXGATEWAY_VERIFY_USER_PASSWORD.");
writer.WriteLine("mxgw-dotnet archestra-user-to-id --session-id <id> --server-handle <n> --user-guid <guid> [--json]");
writer.WriteLine("mxgw-dotnet subscribe-bulk --session-id <id> --server-handle <n> --items <ref,ref> [--json]");
writer.WriteLine("mxgw-dotnet unsubscribe-bulk --session-id <id> --server-handle <n> --item-handles <n,n> [--json]");
@@ -32,6 +32,57 @@ public sealed class MxCommandReplyExtensionsTests
Assert.Contains("0x80040200", exception.Message);
}
/// <summary>Verifies that a non-OK status category fails even when the raw success member is set.</summary>
[Fact]
public void EnsureMxAccessSuccess_WithNonOkCategoryAndSuccessSet_Throws()
{
MxCommandReply reply = ReadReplyFixture(
"write.status-category-error-success-set.reply.json");
reply.EnsureProtocolSuccess();
MxAccessException exception = Assert.Throws<MxAccessException>(
reply.EnsureMxAccessSuccess);
Assert.Equal(1, Assert.Single(exception.Statuses).Success);
Assert.Contains("CommunicationError", exception.Message, StringComparison.Ordinal);
}
/// <summary>Verifies that an Ok status category succeeds even when the raw success member is zero.</summary>
[Fact]
public void EnsureMxAccessSuccess_WithOkCategoryAndZeroSuccess_ReturnsReply()
{
MxCommandReply reply = ReadReplyFixture(
"write.status-category-ok-success-zero.reply.json");
Assert.Equal(0, Assert.Single(reply.Statuses).Success);
Assert.Same(reply, reply.EnsureProtocolSuccess());
Assert.Same(reply, reply.EnsureMxAccessSuccess());
}
/// <summary>Verifies that a positive HResult (S_FALSE) is a COM success code, not a failure.</summary>
[Fact]
public void EnsureMxAccessSuccess_WithPositiveHResult_ReturnsReply()
{
MxCommandReply reply = ReadReplyFixture("write.hresult-s-false.reply.json");
Assert.Equal(1, reply.Hresult);
Assert.Same(reply, reply.EnsureProtocolSuccess());
Assert.Same(reply, reply.EnsureMxAccessSuccess());
}
/// <summary>Verifies that a negative HResult fails even when every status entry is Ok.</summary>
[Fact]
public void EnsureMxAccessSuccess_WithNegativeHResult_Throws()
{
MxCommandReply reply = ReadReplyFixture("write.hresult-e-fail.reply.json");
reply.EnsureProtocolSuccess();
MxAccessException exception = Assert.Throws<MxAccessException>(
reply.EnsureMxAccessSuccess);
Assert.Equal(-2147467259, exception.HResultCode);
}
/// <summary>Verifies that session-not-found protocol failures throw the correct gateway exception.</summary>
[Fact]
public void EnsureProtocolSuccess_WithSessionFailure_ThrowsSessionException()
@@ -235,7 +235,7 @@ public sealed class MxGatewayClientCliTests
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
"--verify-user-password", password,
"--password", password,
],
output,
error,
@@ -246,6 +246,235 @@ public sealed class MxGatewayClientCliTests
Assert.Contains("[redacted]", error.ToString());
}
/// <summary>
/// CLI-45: <c>--password</c> is the primary credential flag, matching the other
/// four CLIs. The credential reaches the wire but never stdout/stderr.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_AuthenticateUser_AcceptsCanonicalPasswordFlag()
{
const string password = "canonical-flag-credential";
using var output = new StringWriter();
using var error = new StringWriter();
FakeCliClient fakeClient = new();
fakeClient.InvokeReplies.Enqueue(new MxCommandReply
{
SessionId = "session-fixture",
Kind = MxCommandKind.AuthenticateUser,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
AuthenticateUser = new AuthenticateUserReply { UserId = 11 },
});
int exitCode = await MxGatewayClientCli.RunAsync(
[
"authenticate-user",
"--endpoint", "http://localhost:5000",
"--api-key", "test-api-key",
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
"--password", password,
"--json",
],
output,
error,
_ => fakeClient);
Assert.Equal(0, exitCode);
MxCommandRequest request = Assert.Single(fakeClient.InvokeRequests);
Assert.Equal(password, request.Command.AuthenticateUser.VerifyUserPassword);
Assert.DoesNotContain(password, output.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(password, error.ToString(), StringComparison.Ordinal);
}
/// <summary>
/// CLI-45: the credential is read from the environment variable named by
/// <c>--password-env</c>, whose default is the canonical
/// <c>MXGATEWAY_VERIFY_PASSWORD</c> shared by all five CLIs.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(null, "MXGATEWAY_VERIFY_PASSWORD")]
[InlineData("MXGW_TEST_CLI45_ENV", "MXGW_TEST_CLI45_ENV")]
public async Task RunAsync_AuthenticateUser_ReadsCredentialFromNamedEnvironmentVariable(
string? passwordEnvArgument,
string environmentName)
{
const string password = "env-sourced-credential";
using EnvironmentVariableScope scope = new(environmentName, password);
using var output = new StringWriter();
using var error = new StringWriter();
FakeCliClient fakeClient = new();
fakeClient.InvokeReplies.Enqueue(new MxCommandReply
{
SessionId = "session-fixture",
Kind = MxCommandKind.AuthenticateUser,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
AuthenticateUser = new AuthenticateUserReply { UserId = 12 },
});
List<string> args =
[
"authenticate-user",
"--endpoint", "http://localhost:5000",
"--api-key", "test-api-key",
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
"--json",
];
if (passwordEnvArgument is not null)
{
args.Add("--password-env");
args.Add(passwordEnvArgument);
}
int exitCode = await MxGatewayClientCli.RunAsync([.. args], output, error, _ => fakeClient);
Assert.Equal(0, exitCode);
MxCommandRequest request = Assert.Single(fakeClient.InvokeRequests);
Assert.Equal(password, request.Command.AuthenticateUser.VerifyUserPassword);
Assert.DoesNotContain(password, output.ToString(), StringComparison.Ordinal);
}
/// <summary>
/// CLI-45: the pre-rename names stay usable for one release — the deprecated
/// <c>--verify-user-password</c> flag and the deprecated
/// <c>MXGATEWAY_VERIFY_USER_PASSWORD</c> environment variable both still resolve.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_AuthenticateUser_HonoursDeprecatedAliases()
{
using var flagOutput = new StringWriter();
using var flagError = new StringWriter();
FakeCliClient flagClient = new();
flagClient.InvokeReplies.Enqueue(new MxCommandReply
{
SessionId = "session-fixture",
Kind = MxCommandKind.AuthenticateUser,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
AuthenticateUser = new AuthenticateUserReply { UserId = 13 },
});
int flagExitCode = await MxGatewayClientCli.RunAsync(
[
"authenticate-user",
"--endpoint", "http://localhost:5000",
"--api-key", "test-api-key",
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
"--verify-user-password", "legacy-flag-credential",
"--json",
],
flagOutput,
flagError,
_ => flagClient);
Assert.Equal(0, flagExitCode);
Assert.Equal(
"legacy-flag-credential",
Assert.Single(flagClient.InvokeRequests).Command.AuthenticateUser.VerifyUserPassword);
using EnvironmentVariableScope canonical = new("MXGATEWAY_VERIFY_PASSWORD", null);
using EnvironmentVariableScope legacy = new("MXGATEWAY_VERIFY_USER_PASSWORD", "legacy-env-credential");
using var envOutput = new StringWriter();
using var envError = new StringWriter();
FakeCliClient envClient = new();
envClient.InvokeReplies.Enqueue(new MxCommandReply
{
SessionId = "session-fixture",
Kind = MxCommandKind.AuthenticateUser,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
AuthenticateUser = new AuthenticateUserReply { UserId = 14 },
});
int envExitCode = await MxGatewayClientCli.RunAsync(
[
"authenticate-user",
"--endpoint", "http://localhost:5000",
"--api-key", "test-api-key",
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
"--json",
],
envOutput,
envError,
_ => envClient);
Assert.Equal(0, envExitCode);
Assert.Equal(
"legacy-env-credential",
Assert.Single(envClient.InvokeRequests).Command.AuthenticateUser.VerifyUserPassword);
}
/// <summary>
/// CLI-45: a missing or empty credential fails fast before the invoke — the CLI
/// never sends a fabricated empty password to the wire. The error names the flag
/// and the environment variable, never a value.
/// </summary>
/// <param name="explicitEmptyFlag">Whether to pass an explicit empty --password.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(false)]
[InlineData(true)]
public async Task RunAsync_AuthenticateUser_FailsFastOnMissingOrEmptyCredential(bool explicitEmptyFlag)
{
using EnvironmentVariableScope canonical = new("MXGATEWAY_VERIFY_PASSWORD", explicitEmptyFlag ? string.Empty : null);
using EnvironmentVariableScope legacy = new("MXGATEWAY_VERIFY_USER_PASSWORD", null);
using var output = new StringWriter();
using var error = new StringWriter();
FakeCliClient fakeClient = new();
List<string> args =
[
"authenticate-user",
"--endpoint", "http://localhost:5000",
"--api-key", "test-api-key",
"--session-id", "session-fixture",
"--server-handle", "12",
"--verify-user", "operator",
];
if (explicitEmptyFlag)
{
args.Add("--password");
args.Add(string.Empty);
}
int exitCode = await MxGatewayClientCli.RunAsync([.. args], output, error, _ => fakeClient);
Assert.Equal(1, exitCode);
Assert.Empty(fakeClient.InvokeRequests);
Assert.Contains("--password", error.ToString(), StringComparison.Ordinal);
Assert.Contains("MXGATEWAY_VERIFY_PASSWORD", error.ToString(), StringComparison.Ordinal);
}
/// <summary>
/// Sets an environment variable for the duration of a test and restores the
/// previous value on dispose, so credential-resolution tests do not depend on
/// (or leak into) the ambient environment.
/// </summary>
private sealed class EnvironmentVariableScope : IDisposable
{
private readonly string _name;
private readonly string? _original;
public EnvironmentVariableScope(string name, string? value)
{
_name = name;
_original = Environment.GetEnvironmentVariable(name);
Environment.SetEnvironmentVariable(name, value);
}
public void Dispose()
{
Environment.SetEnvironmentVariable(_name, _original);
}
}
/// <summary>Verifies that error output redacts sensitive API key values.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -0,0 +1,96 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Unit tests for <see cref="MxGatewaySecretRedaction"/> — the exact-substring scrub applied to
/// diagnostic text and rebuilt exceptions before they leave the client on a failure path.
/// </summary>
public sealed class MxGatewaySecretRedactionTests
{
[Fact]
public void Redact_ReplacesEveryOccurrenceOfSecret()
{
string result = MxGatewaySecretRedaction.Redact(
"pw=hunter2 retry pw=hunter2 again hunter2",
"hunter2");
Assert.DoesNotContain("hunter2", result, StringComparison.Ordinal);
Assert.Equal("pw=<redacted> retry pw=<redacted> again <redacted>", result);
}
[Fact]
public void Redact_ScrubsBothSecretsWhenOneIsSubstringOfTheOther()
{
// "secret" is a substring of "secretPassword"; both must be fully scrubbed regardless of
// supplied order — no residual leak of either verbatim value.
string result = MxGatewaySecretRedaction.Redact(
"a=secretPassword b=secret",
"secret",
"secretPassword");
Assert.DoesNotContain("secretPassword", result, StringComparison.Ordinal);
Assert.DoesNotContain("secret", result, StringComparison.Ordinal);
}
[Fact]
public void Redact_WithNullSecretsArray_ReturnsMessageUnchanged()
{
const string message = "nothing to scrub here";
string result = MxGatewaySecretRedaction.Redact(message, null!);
Assert.Equal(message, result);
}
[Fact]
public void Redact_WithEmptySecretsArray_ReturnsMessageUnchanged()
{
const string message = "nothing to scrub here";
string result = MxGatewaySecretRedaction.Redact(message);
Assert.Equal(message, result);
}
[Fact]
public void Redact_IgnoresWhitespaceOnlySecret()
{
// A whitespace-only secret must not over-redact the internal spaces of the message.
const string message = "user operator logged in";
string result = MxGatewaySecretRedaction.Redact(message, " ");
Assert.Equal(message, result);
}
[Fact]
public void Redacted_PreservesConcreteSubtypeAndDoesNotChainSecretBearingOriginal()
{
const string secret = "hunter2";
Exception transportCause = new InvalidOperationException("transport reset");
MxGatewaySessionException original = new(
$"session rejected credential '{secret}'",
"session-1",
"correlation-1",
new ProtocolStatus { Code = ProtocolStatusCode.SessionNotReady, Message = $"echoed '{secret}'" },
hResult: -1,
statuses: [new MxStatusProxy { DiagnosticText = $"denied '{secret}'" }],
innerException: transportCause);
MxGatewayException redacted = MxGatewaySecretRedaction.Redacted(original, secret);
// Concrete runtime type is preserved.
Assert.IsType<MxGatewaySessionException>(redacted);
// The secret is gone from the message and every structured accessor.
Assert.DoesNotContain(secret, redacted.Message, StringComparison.Ordinal);
Assert.DoesNotContain(secret, redacted.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(secret, redacted.ProtocolStatus!.Message, StringComparison.Ordinal);
Assert.All(redacted.Statuses, status =>
Assert.DoesNotContain(secret, status.DiagnosticText, StringComparison.Ordinal));
Assert.Contains("<redacted>", redacted.Message, StringComparison.Ordinal);
// The secret-bearing original is NOT chained; the original's transport cause is carried.
Assert.NotSame(original, redacted.InnerException);
Assert.Same(transportCause, redacted.InnerException);
}
}
@@ -0,0 +1,165 @@
using Google.Protobuf;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Tests for the credential-scrub (CLI-40) and malformed-reply (CLI-41) contracts on the
/// credential and id-returning session helpers, driven from shared behavior fixtures.
/// </summary>
public sealed class MxGatewaySessionReplyContractTests
{
/// <summary>
/// CLI-40: when MXAccess echoes the submitted credential back in its failure diagnostic,
/// the surfaced exception message must scrub it to the library redaction marker.
/// </summary>
[Fact]
public async Task AuthenticateUserAsync_RedactsEchoedCredentialInFailureMessage()
{
const string password = "sup3rSecretVerify9f3a2b";
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(ReadReplyFixture("authenticate-user.echoed-credential.reply.json"));
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
MxAccessException exception = await Assert.ThrowsAsync<MxAccessException>(
async () => await session.AuthenticateUserAsync(12, "operator", password));
Assert.DoesNotContain(password, exception.Message, StringComparison.Ordinal);
Assert.Contains("<redacted>", exception.Message, StringComparison.Ordinal);
// ToString() is what logging frameworks emit; the secret-bearing original must not be
// chained as an inner exception where it would re-surface the credential verbatim.
Assert.DoesNotContain(password, exception.ToString(), StringComparison.Ordinal);
}
/// <summary>
/// CLI-40: the redacted exception must not leak the echoed credential through any structured
/// accessor either — <see cref="MxAccessException.Reply"/> (protocol message, diagnostic
/// message, and each MXSTATUS_PROXY diagnostic text) and <see cref="MxGatewayException.Statuses"/>
/// all carry the server-echoed credential verbatim before the fix. Both the OK+negative-HRESULT
/// and the MXACCESS_FAILURE reply route to <see cref="MxAccessException"/>, so both must scrub.
/// </summary>
/// <param name="fixture">The echoed-credential reply fixture to drive.</param>
[Theory]
[InlineData("authenticate-user.echoed-credential.reply.json")]
[InlineData("authenticate-user.echoed-credential-mxaccess-failure.reply.json")]
public async Task AuthenticateUserAsync_RedactsEchoedCredentialInStructuredAccessors(string fixture)
{
const string password = "sup3rSecretVerify9f3a2b";
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(ReadReplyFixture(fixture));
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
MxAccessException exception = await Assert.ThrowsAsync<MxAccessException>(
async () => await session.AuthenticateUserAsync(12, "operator", password));
Assert.DoesNotContain(password, exception.Message, StringComparison.Ordinal);
Assert.Contains("<redacted>", exception.Message, StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.Reply.ProtocolStatus.Message, StringComparison.Ordinal);
Assert.DoesNotContain(password, exception.Reply.DiagnosticMessage, StringComparison.Ordinal);
foreach (MxStatusProxy status in exception.Reply.Statuses)
{
Assert.DoesNotContain(password, status.DiagnosticText, StringComparison.Ordinal);
}
foreach (MxStatusProxy status in exception.Statuses)
{
Assert.DoesNotContain(password, status.DiagnosticText, StringComparison.Ordinal);
}
}
/// <summary>
/// CLI-41: an OK reply that carries neither the typed AuthenticateUser payload nor an
/// int32 return_value is a malformed reply, surfaced as a typed exception rather than an NRE.
/// </summary>
[Fact]
public async Task AuthenticateUserAsync_MissingPayloadAndReturnValue_ThrowsMalformedReply()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(ReadReplyFixture("authenticate-user.missing-payload.reply.json"));
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
await Assert.ThrowsAsync<MxGatewayMalformedReplyException>(
async () => await session.AuthenticateUserAsync(12, "operator", "pw"));
}
/// <summary>
/// CLI-41: an OK reply that omits the typed payload but carries an int32 return_value
/// resolves to that return value.
/// </summary>
[Fact]
public async Task AuthenticateUserAsync_ReturnValueOnly_ResolvesReturnValue()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(ReadReplyFixture("authenticate-user.return-value-only.reply.json"));
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
int userId = await session.AuthenticateUserAsync(12, "operator", "pw");
Assert.Equal(7, userId);
}
/// <summary>
/// CLI-41: the AddBufferedItem fallback shares the malformed-reply contract — an OK reply
/// with neither a typed item handle nor an int32 return_value throws the typed exception.
/// </summary>
[Fact]
public async Task AddBufferedItemAsync_MissingPayloadAndReturnValue_ThrowsMalformedReply()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(new MxCommandReply
{
SessionId = "session-fixture",
Kind = MxCommandKind.AddBufferedItem,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
});
await using MxGatewayClient client = CreateClient(transport);
MxGatewaySession session = await client.OpenSessionAsync();
await Assert.ThrowsAsync<MxGatewayMalformedReplyException>(
async () => await session.AddBufferedItemAsync(12, "Area001.Pump001.Speed", "runtime"));
}
private static MxGatewayClient CreateClient(FakeGatewayTransport transport)
{
return new MxGatewayClient(transport.Options, transport);
}
private static FakeGatewayTransport CreateTransport()
{
return new FakeGatewayTransport(new MxGatewayClientOptions
{
Endpoint = new Uri("http://localhost:5000"),
ApiKey = "test-api-key",
});
}
private static MxCommandReply ReadReplyFixture(string fileName)
{
DirectoryInfo directory = new(AppContext.BaseDirectory);
while (directory is not null)
{
string path = Path.Combine(
directory.FullName,
"clients",
"proto",
"fixtures",
"behavior",
"command-replies",
fileName);
if (File.Exists(path))
{
return JsonParser.Default.Parse<MxCommandReply>(File.ReadAllText(path));
}
directory = directory.Parent!;
}
throw new FileNotFoundException(fileName);
}
}
@@ -19,9 +19,9 @@ public sealed class MxStatusProxyExtensionsTests
{
MxStatusProxy status = JsonParser.Default.Parse<MxStatusProxy>(
testCase.GetProperty("status").GetRawText());
int success = testCase.GetProperty("status").GetProperty("success").GetInt32();
Assert.Equal(success != 0 && status.Category is MxStatusCategory.Ok, status.IsSuccess());
bool wantSuccess = testCase.GetProperty("wantSuccess").GetBoolean();
Assert.Equal(wantSuccess, status.IsSuccess());
Assert.Equal(
testCase.GetProperty("status").GetProperty("rawCategory").GetInt32(),
status.RawCategory);
@@ -31,6 +31,22 @@ public sealed class MxStatusProxyExtensionsTests
}
}
/// <summary>Verifies that the raw success member never overrides the authoritative category.</summary>
[Theory]
[InlineData(MxStatusCategory.Ok, 0, true)]
[InlineData(MxStatusCategory.Ok, 1, true)]
[InlineData(MxStatusCategory.CommunicationError, 1, false)]
[InlineData(MxStatusCategory.Unspecified, 1, false)]
public void IsSuccess_BranchesOnCategoryOnly(
MxStatusCategory category,
int success,
bool expected)
{
MxStatusProxy status = new() { Category = category, Success = success };
Assert.Equal(expected, status.IsSuccess());
}
private static string ReadFixture(string category, string fileName)
{
DirectoryInfo directory = new(AppContext.BaseDirectory);
@@ -23,7 +23,11 @@ public static class MxCommandReplyExtensions
throw CreateProtocolException(reply, code);
}
/// <summary>Validates that the reply indicates MXAccess success (no HResult or status failures), throwing MxAccessException if not.</summary>
/// <summary>
/// Validates that the reply indicates MXAccess success, throwing MxAccessException if not.
/// Following COM semantics, only a negative HResult is a failure — positive success codes
/// such as <c>S_FALSE</c> pass — and a status entry fails only when its category is not Ok.
/// </summary>
/// <param name="reply">The command reply to check.</param>
/// <returns>The same reply, for chaining.</returns>
public static MxCommandReply EnsureMxAccessSuccess(this MxCommandReply reply)
@@ -31,7 +35,7 @@ public static class MxCommandReplyExtensions
ArgumentNullException.ThrowIfNull(reply);
bool mxAccessFailure = reply.ProtocolStatus?.Code is ProtocolStatusCode.MxaccessFailure;
bool hResultFailure = reply.HasHresult && reply.Hresult != 0;
bool hResultFailure = reply.HasHresult && reply.Hresult < 0;
bool statusFailure = reply.Statuses.Any(status => !status.IsSuccess());
if (!mxAccessFailure && !hResultFailure && !statusFailure)
@@ -0,0 +1,46 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client;
/// <summary>
/// Exception thrown when the gateway returns a protocol-OK reply that carries neither the
/// expected typed payload nor an int32 <c>return_value</c>, so the client cannot resolve the
/// operation result. This replaces the historical <see cref="NullReferenceException"/> that a
/// blind <c>reply.ReturnValue.Int32Value</c> fallback would throw.
/// </summary>
public sealed class MxGatewayMalformedReplyException : MxGatewayException
{
/// <summary>Initializes a new instance with the given message.</summary>
/// <param name="message">The error message describing the malformed reply.</param>
public MxGatewayMalformedReplyException(string message)
: base(message)
{
}
/// <summary>Initializes a new instance with full diagnostic context.</summary>
/// <param name="message">The error message describing the malformed reply.</param>
/// <param name="sessionId">The session ID, if available.</param>
/// <param name="correlationId">The correlation ID for tracing, if available.</param>
/// <param name="protocolStatus">The protocol status details, if available.</param>
/// <param name="hResult">The HResult code, if available.</param>
/// <param name="statuses">The MXAccess statuses, if available.</param>
/// <param name="innerException">The underlying exception, if any.</param>
public MxGatewayMalformedReplyException(
string message,
string? sessionId = null,
string? correlationId = null,
ProtocolStatus? protocolStatus = null,
int? hResult = null,
IReadOnlyList<MxStatusProxy>? statuses = null,
Exception? innerException = null)
: base(
message,
sessionId,
correlationId,
protocolStatus,
hResult,
statuses ?? [],
innerException)
{
}
}
@@ -0,0 +1,229 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client;
/// <summary>
/// Scrubs exact secret substrings out of diagnostic text before it leaves the client on an
/// exception path. MXAccess can echo a submitted credential or secured value back inside a
/// failure diagnostic (protocol message, MXSTATUS_PROXY diagnostic text, HRESULT description);
/// this helper replaces any such verbatim occurrence with <c>&lt;redacted&gt;</c> so the raw
/// request payload never reaches a caught exception's message. The marker matches the Go, Rust,
/// and Java clients.
/// </summary>
internal static class MxGatewaySecretRedaction
{
private const string Marker = "<redacted>";
/// <summary>
/// Replaces every usable secret in <paramref name="secrets"/> with the redaction marker
/// (ordinal comparison). Returns the message unchanged when it is null or empty, or when no
/// usable secret is supplied. A secret that is null, empty, or whitespace-only is ignored so
/// it cannot over-redact ordinary separator characters in the message.
/// </summary>
/// <param name="message">The diagnostic message to scrub.</param>
/// <param name="secrets">The secret values to remove from the message.</param>
/// <returns>The scrubbed message.</returns>
internal static string Redact(string message, params string?[] secrets)
{
if (string.IsNullOrEmpty(message) || secrets is null)
{
return message;
}
string result = message;
foreach (string? secret in secrets)
{
if (!string.IsNullOrWhiteSpace(secret))
{
result = result.Replace(secret, Marker, StringComparison.Ordinal);
}
}
return result;
}
/// <summary>
/// Returns a scrubbed clone of <paramref name="reply"/>: the protocol-status message, the
/// reply-level diagnostic message, and each MXSTATUS_PROXY diagnostic text have every verbatim
/// secret replaced with the redaction marker. The original is left untouched. MXAccess can echo
/// a submitted credential into any of these fields, so a redacted exception must carry the
/// scrubbed reply rather than the secret-bearing original.
/// </summary>
/// <param name="reply">The reply to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A scrubbed clone of the reply.</returns>
internal static MxCommandReply RedactReply(MxCommandReply reply, params string?[] secrets)
{
ArgumentNullException.ThrowIfNull(reply);
MxCommandReply clone = reply.Clone();
if (clone.ProtocolStatus is not null)
{
clone.ProtocolStatus.Message = Redact(clone.ProtocolStatus.Message, secrets);
}
clone.DiagnosticMessage = Redact(clone.DiagnosticMessage, secrets);
foreach (MxStatusProxy status in clone.Statuses)
{
status.DiagnosticText = Redact(status.DiagnosticText, secrets);
}
return clone;
}
/// <summary>
/// Returns a scrubbed clone of <paramref name="status"/> (its message with every verbatim
/// secret removed), or <see langword="null"/> when the input is null.
/// </summary>
/// <param name="status">The protocol status to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A scrubbed clone, or <see langword="null"/>.</returns>
internal static ProtocolStatus? RedactStatus(ProtocolStatus? status, params string?[] secrets)
{
if (status is null)
{
return null;
}
ProtocolStatus clone = status.Clone();
clone.Message = Redact(clone.Message, secrets);
return clone;
}
/// <summary>
/// Returns a list of scrubbed clones of <paramref name="statuses"/> — each MXSTATUS_PROXY's
/// diagnostic text has every verbatim secret removed. The originals are left untouched.
/// </summary>
/// <param name="statuses">The statuses to clone and scrub.</param>
/// <param name="secrets">The secret values to remove.</param>
/// <returns>A list of scrubbed clones.</returns>
internal static IReadOnlyList<MxStatusProxy> RedactStatuses(
IReadOnlyList<MxStatusProxy> statuses,
params string?[] secrets)
{
if (statuses is null || statuses.Count is 0)
{
return statuses ?? [];
}
MxStatusProxy[] result = new MxStatusProxy[statuses.Count];
for (int i = 0; i < statuses.Count; i++)
{
MxStatusProxy clone = statuses[i].Clone();
clone.DiagnosticText = Redact(clone.DiagnosticText, secrets);
result[i] = clone;
}
return result;
}
/// <summary>
/// Returns an exception equivalent to <paramref name="ex"/> but with any verbatim secret
/// scrubbed from its message. When nothing changes, the original exception is returned
/// unchanged; otherwise a new exception of the same concrete runtime type is built and the
/// original reply/status context is preserved. The secret-bearing original is deliberately
/// <b>not</b> chained as the inner exception — doing so would let its unredacted message
/// re-surface through <see cref="Exception.ToString"/> (which logging frameworks call). The
/// original's own inner cause (a transport error, never the request payload) is carried
/// forward instead.
/// </summary>
/// <param name="ex">The exception to redact.</param>
/// <param name="secrets">The secret values to remove from the message.</param>
/// <returns>The redacted exception, or the original when no change was needed.</returns>
internal static MxGatewayException Redacted(MxGatewayException ex, params string?[] secrets)
{
ArgumentNullException.ThrowIfNull(ex);
string redacted = Redact(ex.Message, secrets);
bool messageChanged = !string.Equals(redacted, ex.Message, StringComparison.Ordinal);
Exception? cause = ex.InnerException;
// MxAccessException derives its structured fields from the raw reply, so scrubbing must
// clone and redact that reply — the message alone changing is not enough, because the reply
// can carry the echoed secret even when the message does not.
if (ex is MxAccessException access)
{
if (!messageChanged && !ReplyContainsSecret(access.Reply, secrets))
{
return ex;
}
return new MxAccessException(redacted, RedactReply(access.Reply, secrets), cause);
}
// Other subtypes carry the secret through ProtocolStatus.Message and Statuses[].DiagnosticText.
if (!messageChanged
&& !ContainsSecret(ex.ProtocolStatus?.Message, secrets)
&& !StatusesContainSecret(ex.Statuses, secrets))
{
return ex;
}
ProtocolStatus? status = RedactStatus(ex.ProtocolStatus, secrets);
IReadOnlyList<MxStatusProxy> statuses = RedactStatuses(ex.Statuses, secrets);
return ex switch
{
MxGatewaySessionException => new MxGatewaySessionException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayWorkerException => new MxGatewayWorkerException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayAuthenticationException => new MxGatewayAuthenticationException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayAuthorizationException => new MxGatewayAuthorizationException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayMalformedReplyException => new MxGatewayMalformedReplyException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
MxGatewayCommandException => new MxGatewayCommandException(
redacted, ex.SessionId, ex.CorrelationId, status, ex.HResultCode, statuses, cause),
_ => new MxGatewayException(redacted, cause),
};
}
private static bool ContainsSecret(string? text, string?[] secrets)
{
if (string.IsNullOrEmpty(text) || secrets is null)
{
return false;
}
foreach (string? secret in secrets)
{
if (!string.IsNullOrWhiteSpace(secret) && text.Contains(secret, StringComparison.Ordinal))
{
return true;
}
}
return false;
}
private static bool StatusesContainSecret(IReadOnlyList<MxStatusProxy> statuses, string?[] secrets)
{
if (statuses is null)
{
return false;
}
foreach (MxStatusProxy status in statuses)
{
if (ContainsSecret(status.DiagnosticText, secrets))
{
return true;
}
}
return false;
}
private static bool ReplyContainsSecret(MxCommandReply reply, string?[] secrets)
{
if (reply is null)
{
return false;
}
return ContainsSecret(reply.ProtocolStatus?.Message, secrets)
|| ContainsSecret(reply.DiagnosticMessage, secrets)
|| StatusesContainSecret(reply.Statuses, secrets);
}
}
@@ -945,7 +945,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.AddBufferedItem?.ItemHandle ?? reply.ReturnValue.Int32Value;
return ResolveInt32Result(reply.AddBufferedItem?.ItemHandle, reply, "AddBufferedItem");
}
/// <summary>
@@ -1141,7 +1141,14 @@ public sealed class MxGatewaySession : IAsyncDisposable
verifierUserId,
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
try
{
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
}
catch (MxGatewayException ex)
{
throw MxGatewaySecretRedaction.Redacted(ex, ExtractSecretString(value));
}
}
/// <summary>
@@ -1215,7 +1222,14 @@ public sealed class MxGatewaySession : IAsyncDisposable
verifierUserId,
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
try
{
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
}
catch (MxGatewayException ex)
{
throw MxGatewaySecretRedaction.Redacted(ex, ExtractSecretString(value));
}
}
/// <summary>
@@ -1285,8 +1299,15 @@ public sealed class MxGatewaySession : IAsyncDisposable
verifyUserPassword,
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.AuthenticateUser?.UserId ?? reply.ReturnValue.Int32Value;
try
{
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return ResolveInt32Result(reply.AuthenticateUser?.UserId, reply, "AuthenticateUser");
}
catch (MxGatewayException ex)
{
throw MxGatewaySecretRedaction.Redacted(ex, verifyUserPassword);
}
}
/// <summary>
@@ -1337,7 +1358,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
MxCommandReply reply = await ArchestraUserToIdRawAsync(serverHandle, userIdGuid, cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.ArchestraUserToId?.UserId ?? reply.ReturnValue.Int32Value;
return ResolveInt32Result(reply.ArchestraUserToId?.UserId, reply, "ArchestrAUserToId");
}
/// <summary>
@@ -1367,6 +1388,51 @@ public sealed class MxGatewaySession : IAsyncDisposable
cancellationToken);
}
/// <summary>
/// Resolves the int32 result of an OK command reply: the typed payload value when present,
/// otherwise an int32 <c>return_value</c> when the reply carries one. A reply that provides
/// neither is malformed and surfaces as <see cref="MxGatewayMalformedReplyException"/>
/// rather than the historical <see cref="NullReferenceException"/>.
/// </summary>
/// <param name="typedValue">The typed payload value, or <see langword="null"/> when absent.</param>
/// <param name="reply">The OK command reply.</param>
/// <param name="operation">The MXAccess operation name, for the diagnostic message.</param>
/// <returns>The resolved int32 result.</returns>
private static int ResolveInt32Result(int? typedValue, MxCommandReply reply, string operation)
{
if (typedValue.HasValue)
{
return typedValue.Value;
}
if (reply.ReturnValue is not null
&& reply.ReturnValue.KindCase == MxValue.KindOneofCase.Int32Value)
{
return reply.ReturnValue.Int32Value;
}
throw new MxGatewayMalformedReplyException(
$"{operation} returned a malformed reply: OK reply carried neither the typed payload nor an int32 return_value",
reply.SessionId,
reply.CorrelationId,
reply.ProtocolStatus,
reply.HasHresult ? reply.Hresult : null,
reply.Statuses.ToArray());
}
/// <summary>
/// Extracts the raw string form of a credential-bearing <see cref="MxValue"/> for redaction,
/// or <see langword="null"/> when the value does not carry a string.
/// </summary>
/// <param name="value">The value written by a secured write.</param>
/// <returns>The string payload, or <see langword="null"/>.</returns>
private static string? ExtractSecretString(MxValue value)
{
return value.KindCase == MxValue.KindOneofCase.StringValue
? value.StringValue
: null;
}
/// <summary>
/// Invokes an MXAccess command on this session.
/// </summary>
@@ -5,15 +5,18 @@ namespace ZB.MOM.WW.MxGateway.Client;
/// <summary>Extension methods for MxStatusProxy values.</summary>
public static class MxStatusProxyExtensions
{
/// <summary>Returns whether the status indicates success (success flag set and category is Ok).</summary>
/// <summary>
/// Returns whether the status indicates success, which the wire contract defines as
/// <see cref="MxStatusCategory.Ok"/>. The raw <c>Success</c> member is a verbatim COM
/// diagnostic, not a boolean, so it never participates in the verdict.
/// </summary>
/// <param name="status">The status to check.</param>
/// <returns><see langword="true"/> if the status indicates success; otherwise <see langword="false"/>.</returns>
public static bool IsSuccess(this MxStatusProxy status)
{
ArgumentNullException.ThrowIfNull(status);
return status.Success != 0
&& status.Category is MxStatusCategory.Ok;
return status.Category is MxStatusCategory.Ok;
}
/// <summary>Returns a formatted summary of the status for diagnostic output.</summary>
@@ -27,6 +30,6 @@ public static class MxStatusProxyExtensions
? "no diagnostic text"
: status.DiagnosticText;
return $"{status.Category} by {status.DetectedBy}; detail={status.Detail}; {diagnosticText}";
return $"success={status.Success}; {status.Category} by {status.DetectedBy}; detail={status.Detail}; {diagnosticText}";
}
}
+11 -1
View File
@@ -94,6 +94,12 @@ goroutine cleanup. Raw protobuf messages remain available through the
`errors.As` for `GatewayError`, `CommandError`, and `MxAccessError`; command
errors preserve the raw reply.
`EnsureMxAccessSuccess` follows COM semantics: only a **negative** HRESULT is a
failure, so positive success codes such as `S_FALSE` (1) pass. `StatusSucceeded`
judges each `MXSTATUS_PROXY` entry by its category — an entry fails when
`Category` is not `MX_STATUS_CATEGORY_OK`, and the raw `Success` member is a
diagnostic that never decides the verdict. A nil entry is success.
### Reconnect-replay gap
Each `EventResult` carries exactly one of `Event`, `ReplayGap`, or `Err`. When
@@ -177,7 +183,11 @@ parity holds: a `WriteSecured` issued without a matching prior `AuthenticateUser
and supervisory advise fails natively, and that failure is surfaced unchanged
rather than pre-empted. The CLI exposes `authenticate-user` (credential via
`-password-env`, default `MXGATEWAY_VERIFY_PASSWORD`, or `-password`) and
`write-secured`.
`write-secured`. The credential is required: a missing or empty resolved value is
a usage error naming the flag and the variable, so the CLI fails before dialing
instead of authenticating with an empty password. `MXGATEWAY_VERIFY_PASSWORD` is
the canonical variable across all five client CLIs — see
[Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
### Array writes replace the whole array
+62 -4
View File
@@ -57,6 +57,25 @@ type commandReplyOutput struct {
Reply json.RawMessage `json:"reply"`
}
// replayGapRow is the JSON row stream-events emits for a reconnect-replay gap:
// {"replayGap":{"requestedAfterSequence":N,"oldestAvailableSequence":N}}.
//
// The cursors are typed by hand rather than marshalled with protojson on
// purpose. The proto3 JSON mapping renders 64-bit integers as JSON *strings*
// ("7"), but the Rust and Python CLIs emit JSON *numbers* (7) for this row —
// routing through protojson would silently make Go the odd one out and break
// the cross-language smoke matrix's row comparison. encoding/json renders
// uint64 as a number, which is the canonical rendering here.
type replayGapRow struct {
ReplayGap replayGapCursors `json:"replayGap"`
}
// replayGapCursors is the nested cursor object of replayGapRow.
type replayGapCursors struct {
RequestedAfterSequence uint64 `json:"requestedAfterSequence"`
OldestAvailableSequence uint64 `json:"oldestAvailableSequence"`
}
func main() {
if err := runWithIO(context.Background(), os.Args[1:], os.Stdout, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, err)
@@ -427,6 +446,11 @@ func runWriteSecured(ctx context.Context, args []string, stdout, stderr io.Write
return writeCommandOutput(stdout, *jsonOutput, "write-secured", options, reply, err)
}
// defaultVerifyPasswordEnv is the canonical CLI credential environment variable,
// shared by every official client CLI (CLI-45) so one exported variable drives
// the same operator workflow in all five languages.
const defaultVerifyPasswordEnv = "MXGATEWAY_VERIFY_PASSWORD"
func runAuthenticateUser(ctx context.Context, args []string, stdout, stderr io.Writer) error {
flags := flag.NewFlagSet("authenticate-user", flag.ContinueOnError)
flags.SetOutput(stderr)
@@ -439,7 +463,7 @@ func runAuthenticateUser(ctx context.Context, args []string, stdout, stderr io.W
// prefer the environment variable so it stays out of shell history and the
// process table. The -password flag remains for non-interactive scripting.
password := flags.String("password", "", "verify-user password (prefer -password-env)")
passwordEnv := flags.String("password-env", "MXGATEWAY_VERIFY_PASSWORD", "environment variable containing the verify-user password")
passwordEnv := flags.String("password-env", defaultVerifyPasswordEnv, "environment variable containing the verify-user password")
if err := flags.Parse(args); err != nil {
return err
@@ -452,8 +476,18 @@ func runAuthenticateUser(ctx context.Context, args []string, stdout, stderr io.W
}
resolvedPassword := *password
if resolvedPassword == "" && *passwordEnv != "" {
resolvedPassword = os.Getenv(*passwordEnv)
envName := *passwordEnv
if envName == "" {
envName = defaultVerifyPasswordEnv
}
if resolvedPassword == "" {
resolvedPassword = os.Getenv(envName)
}
// Fail fast rather than dialing: an unset or empty variable must not become a
// real MXAccess authentication attempt with an empty credential. The message
// names only the flag and the variable — never the resolved value.
if resolvedPassword == "" {
return fmt.Errorf("a password is required via -password or the %s environment variable", envName)
}
client, options, err := dialForCommand(ctx, common)
@@ -970,7 +1004,31 @@ func runStreamEvents(ctx context.Context, args []string, stdout, stderr io.Write
if result.Err != nil {
return result.Err
}
if *jsonOutput {
// A reconnect-replay gap is a typed signal, not an event: the library
// clears Event on it, so formatting Event here would print a meaningless
// zero row and discard the resume cursors the operator needs. Render it
// as its own row (matching the Rust CLI) and count it toward -limit like
// any other emitted row.
if result.IsReplayGap() {
if *jsonOutput {
row, err := json.Marshal(replayGapRow{
ReplayGap: replayGapCursors{
RequestedAfterSequence: result.ReplayGap.GetRequestedAfterSequence(),
OldestAvailableSequence: result.ReplayGap.GetOldestAvailableSequence(),
},
})
if err != nil {
return err
}
fmt.Fprintln(stdout, string(row))
} else {
fmt.Fprintf(
stdout,
"REPLAY_GAP requested_after=%d oldest_available=%d\n",
result.ReplayGap.GetRequestedAfterSequence(),
result.ReplayGap.GetOldestAvailableSequence())
}
} else if *jsonOutput {
fmt.Fprintln(stdout, string(mustMarshalProto(result.Event)))
} else {
fmt.Fprintf(stdout, "%d %s\n", result.Event.GetWorkerSequence(), result.Event.GetFamily())
+180
View File
@@ -598,6 +598,69 @@ func TestRunAuthenticateUserRequiresVerifyUser(t *testing.T) {
}
}
// TestRunAuthenticateUserRejectsEmptyPassword pins the CLI-45 fail-fast contract:
// an unresolved credential must abort before dialing rather than authenticating
// with an empty password, and the usage error must name both -password and the
// canonical environment variable without echoing any value.
func TestRunAuthenticateUserRejectsEmptyPassword(t *testing.T) {
t.Setenv("MXGATEWAY_VERIFY_PASSWORD", "")
var stdout, stderr bytes.Buffer
err := runWithIO(t.Context(), []string{
"authenticate-user",
"-session-id", "s1",
"-verify-user", "operator",
"-plaintext",
"-api-key", "test",
}, &stdout, &stderr)
if err == nil {
t.Fatal("authenticate-user without a credential must fail before dialing")
}
if !strings.Contains(err.Error(), "-password") {
t.Fatalf("error must name the -password flag: %v", err)
}
if !strings.Contains(err.Error(), "MXGATEWAY_VERIFY_PASSWORD") {
t.Fatalf("error must name the canonical environment variable: %v", err)
}
}
// TestRunAuthenticateUserReadsPasswordFromCanonicalEnv pins that the default
// -password-env is MXGATEWAY_VERIFY_PASSWORD: with it set the credential guard
// passes and the command proceeds past it to the dial, which fails against an
// unused port under a short context — proving the guard was cleared without
// needing a live gateway.
func TestRunAuthenticateUserReadsPasswordFromCanonicalEnv(t *testing.T) {
t.Setenv("MXGATEWAY_VERIFY_PASSWORD", "env-sourced-credential")
ctx, cancel := context.WithTimeout(t.Context(), 200*time.Millisecond)
defer cancel()
var stdout, stderr bytes.Buffer
err := runWithIO(ctx, []string{
"authenticate-user",
"-session-id", "s1",
"-verify-user", "operator",
"-endpoint", "127.0.0.1:1",
"-plaintext",
"-api-key", "test",
"-call-timeout", "1s",
}, &stdout, &stderr)
if err == nil {
t.Fatal("expected the dial/RPC to fail against an unused port")
}
if strings.Contains(err.Error(), "flag provided but not defined") {
t.Fatalf("test invoked an unknown flag, so it never reached the guard: %v", err)
}
if strings.Contains(err.Error(), "a password is required") {
t.Fatalf("credential guard must be satisfied from %s: %v", "MXGATEWAY_VERIFY_PASSWORD", err)
}
if strings.Contains(err.Error(), "env-sourced-credential") ||
strings.Contains(stdout.String(), "env-sourced-credential") ||
strings.Contains(stderr.String(), "env-sourced-credential") {
t.Fatal("the resolved credential must never be echoed")
}
}
// TestRunWriteBulkVariantRejectsMismatchedHandlesAndValues pins the len-mismatch
// guard so a write-bulk with unequal item-handles / values counts fails fast
// before any dial.
@@ -617,3 +680,120 @@ func TestRunWriteBulkVariantRejectsMismatchedHandlesAndValues(t *testing.T) {
t.Fatalf("write-bulk mismatched handles/values error = %v", err)
}
}
// replayGapFakeGateway streams the gateway's reconnect-replay sentinel (an MxEvent
// carrying replay_gap, family UNSPECIFIED, body unset) followed by one normal data
// event — exactly what a resume whose cursor predates the retained replay ring sees.
type replayGapFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
func (g *replayGapFakeGateway) StreamEvents(
req *pb.StreamEventsRequest,
stream grpc.ServerStreamingServer[pb.MxEvent],
) error {
sentinel := &pb.MxEvent{
SessionId: req.GetSessionId(),
Family: pb.MxEventFamily_MX_EVENT_FAMILY_UNSPECIFIED,
ReplayGap: &pb.ReplayGap{
RequestedAfterSequence: 7,
OldestAvailableSequence: 42,
},
}
if err := stream.Send(sentinel); err != nil {
return err
}
return stream.Send(&pb.MxEvent{
SessionId: req.GetSessionId(),
Family: pb.MxEventFamily_MX_EVENT_FAMILY_ON_DATA_CHANGE,
WorkerSequence: 43,
})
}
func startReplayGapGateway(t *testing.T) string {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
server := grpc.NewServer()
pb.RegisterMxAccessGatewayServer(server, &replayGapFakeGateway{})
go func() { _ = server.Serve(listener) }()
t.Cleanup(func() {
server.Stop()
_ = listener.Close()
})
return listener.Addr().String()
}
// TestRunStreamEventsPrintsReplayGap pins CLI-36: the CLI must render the typed
// ReplayGap signal in both output modes instead of formatting the library's
// cleared Event field (which printed "0 MX_EVENT_FAMILY_UNSPECIFIED" in text mode
// and an empty object in JSON mode, destroying the resume cursors).
func TestRunStreamEventsPrintsReplayGap(t *testing.T) {
endpoint := startReplayGapGateway(t)
baseArgs := []string{
"stream-events",
"-endpoint", endpoint,
"-plaintext",
"-api-key", "test",
"-session-id", "gap-session",
"-after-worker-sequence", "7",
"-limit", "2",
}
var stdout, stderr bytes.Buffer
if err := runWithIO(t.Context(), baseArgs, &stdout, &stderr); err != nil {
t.Fatalf("runWithIO() error = %v; stderr = %s", err, stderr.String())
}
text := stdout.String()
if !strings.Contains(text, "REPLAY_GAP requested_after=7 oldest_available=42") {
t.Fatalf("stream-events text output missing typed gap row: %q", text)
}
if strings.Contains(text, "0 MX_EVENT_FAMILY_UNSPECIFIED") {
t.Fatalf("stream-events text output destroyed the gap into a zero row: %q", text)
}
if !strings.Contains(text, "43 MX_EVENT_FAMILY_ON_DATA_CHANGE") {
t.Fatalf("stream-events text output dropped the normal event: %q", text)
}
stdout.Reset()
stderr.Reset()
if err := runWithIO(t.Context(), append(baseArgs, "-json"), &stdout, &stderr); err != nil {
t.Fatalf("runWithIO(-json) error = %v; stderr = %s", err, stderr.String())
}
lines := strings.Split(strings.TrimSpace(stdout.String()), "\n")
if len(lines) != 2 {
t.Fatalf("stream-events -json emitted %d rows, want 2: %q", len(lines), stdout.String())
}
// The cursors must decode as JSON numbers, not the strings the proto3 JSON
// mapping would produce for 64-bit fields: the Rust and Python CLIs emit
// numbers, and the cross-language matrix compares these rows across clients.
var gapRow struct {
ReplayGap *struct {
RequestedAfterSequence uint64 `json:"requestedAfterSequence"`
OldestAvailableSequence uint64 `json:"oldestAvailableSequence"`
} `json:"replayGap"`
}
if err := json.Unmarshal([]byte(lines[0]), &gapRow); err != nil {
t.Fatalf("parse gap row: %v\nrow: %s", err, lines[0])
}
if gapRow.ReplayGap == nil {
t.Fatalf("stream-events -json first row is not a replayGap row: %s", lines[0])
}
if gapRow.ReplayGap.RequestedAfterSequence != 7 || gapRow.ReplayGap.OldestAvailableSequence != 42 {
t.Fatalf("stream-events -json gap cursors = %+v, want 7/42", *gapRow.ReplayGap)
}
// Belt and braces on the value type: a protojson-rendered `"7"` already
// fails the decode above (encoding/json rejects a JSON string for an
// untagged uint64 field), but assert the raw bytes so a regression names
// the real problem instead of surfacing as an opaque unmarshal error.
if !strings.Contains(lines[0], `"requestedAfterSequence":7`) ||
!strings.Contains(lines[0], `"oldestAvailableSequence":42`) {
t.Fatalf("stream-events -json gap cursors must be JSON numbers, got: %s", lines[0])
}
}
+148 -9
View File
@@ -10,7 +10,9 @@ import (
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/status"
"google.golang.org/grpc/test/bufconn"
)
@@ -200,6 +202,136 @@ func TestEventsSlowConsumerYieldsErrSlowConsumerBeforeClose(t *testing.T) {
}
}
func TestEventsFullBufferTerminalErrorKeepsRootCause(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: eventBufferSize,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.EventsAfter(context.Background(), 0)
if err != nil {
t.Fatalf("EventsAfter() error = %v", err)
}
<-fake.streamStarted
// Do not drain until the stream has fully ended: the server sends exactly
// eventBufferSize events (filling the data slots) and then returns a genuine
// terminal gRPC error. The client must report that error as itself, using the
// reserved slot, rather than mislabeling it as ErrSlowConsumer.
select {
case <-fake.streamDone:
case <-time.After(2 * time.Second):
t.Fatal("event stream did not stop after terminal error")
}
// streamDone fires when the server returns; the client's producer goroutine
// still needs a moment to drain the gRPC stream, fill all data slots, and
// enqueue the terminal result. Let it settle before draining so the buffer is
// genuinely full when the terminal error is processed (which is what makes the
// mislabel bug observable).
time.Sleep(250 * time.Millisecond)
var last EventResult
gotResult := false
for {
select {
case res, ok := <-events:
if !ok {
if !gotResult {
t.Fatal("events channel closed without yielding any result")
}
var gwErr *GatewayError
if !errors.As(last.Err, &gwErr) {
t.Fatalf("final event result err is %T, want *GatewayError", last.Err)
}
if code := status.Code(last.Err); code != codes.Internal {
t.Fatalf("final event result gRPC code = %s, want %s", code, codes.Internal)
}
if errors.Is(last.Err, ErrSlowConsumer) {
t.Fatalf("final event result err = %v, must not be mislabeled as ErrSlowConsumer", last.Err)
}
return
}
last = res
gotResult = true
case <-time.After(2 * time.Second):
t.Fatal("events channel did not close after terminal error")
}
}
}
// TestSubscribeEventsFullBufferDeliversTerminalError is the CLI-44 regression for
// the never-drop Subscribe path. SubscribeEvents/SubscribeEventsAfter use
// cancelWhenResultBufferFull=false, so ordinary sends are blocking and uncapped and
// can fill every slot in the results channel — including the reserved terminal slot.
// A genuine terminal Recv error must still be delivered as the final result, never
// silently dropped. The server sends eventBufferSize+eventBufferReservedSlots events
// (filling every slot) and then returns a genuine gRPC error; with an unconditional
// non-blocking terminal send the error is dropped, so this fails red until the send
// path blocks for the never-drop mode.
func TestSubscribeEventsFullBufferDeliversTerminalError(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: eventBufferSize + eventBufferReservedSlots,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
subscription, err := session.SubscribeEvents(context.Background())
if err != nil {
t.Fatalf("SubscribeEvents() error = %v", err)
}
defer subscription.Close()
<-fake.streamStarted
// Wait for the server to finish sending every event and return the terminal
// error, so the producer goroutine has filled every buffered slot before the
// terminal result is processed. That is what makes the dropped-terminal bug
// observable: with the buffer full, an unconditional non-blocking send discards
// the terminal error.
select {
case <-fake.streamDone:
case <-time.After(2 * time.Second):
t.Fatal("event stream did not stop after terminal error")
}
time.Sleep(250 * time.Millisecond)
// Drain fully. Every data event, then the terminal gRPC error as the final
// result, must arrive; the channel must not close without yielding it.
events := subscription.Events()
var last EventResult
gotResult := false
for {
select {
case res, ok := <-events:
if !ok {
if !gotResult {
t.Fatal("events channel closed without yielding any result")
}
var gwErr *GatewayError
if !errors.As(last.Err, &gwErr) {
t.Fatalf("final event result err is %T (%v), want the terminal *GatewayError; it was dropped", last.Err, last.Err)
}
if code := status.Code(last.Err); code != codes.Internal {
t.Fatalf("final event result gRPC code = %s, want %s", code, codes.Internal)
}
return
}
last = res
gotResult = true
case <-time.After(2 * time.Second):
t.Fatal("events channel did not close after terminal error")
}
}
}
func TestEventsSurfacesReplayGapSentinelAsTypedSignal(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
@@ -694,15 +826,16 @@ func newBufconnClient(t *testing.T, fake *fakeGatewayServer) (*Client, func()) {
type fakeGatewayServer struct {
pb.UnimplementedMxAccessGatewayServer
openReply *pb.OpenSessionReply
openAuth string
streamAuth string
streamStarted chan struct{}
streamDone chan struct{}
streamEventCount int
streamReplayGap *pb.ReplayGap
invokeReply *pb.MxCommandReply
invokeRequest *pb.MxCommandRequest
openReply *pb.OpenSessionReply
openAuth string
streamAuth string
streamStarted chan struct{}
streamDone chan struct{}
streamEventCount int
streamReplayGap *pb.ReplayGap
streamTerminalErr error
invokeReply *pb.MxCommandReply
invokeRequest *pb.MxCommandRequest
}
func (s *fakeGatewayServer) OpenSession(ctx context.Context, req *pb.OpenSessionRequest) (*pb.OpenSessionReply, error) {
@@ -772,6 +905,12 @@ func (s *fakeGatewayServer) StreamEvents(req *pb.StreamEventsRequest, stream grp
return err
}
}
if s.streamTerminalErr != nil {
// Return a genuine terminal stream error immediately after sending the
// events, without waiting on the client to cancel. This exercises the
// Recv-error path while the client's result buffer is still full.
return s.streamTerminalErr
}
<-stream.Context().Done()
return io.EOF
}
@@ -0,0 +1,197 @@
package mxgateway
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/protobuf/encoding/protojson"
)
// loadCommandReplyFixture parses a shared command-reply fixture into an
// MxCommandReply so the Go client can be driven through the same wire shapes the
// other language clients exercise.
func loadCommandReplyFixture(t *testing.T, name string) *pb.MxCommandReply {
t.Helper()
path := filepath.Join("..", "..", "proto", "fixtures", "behavior", "command-replies", name)
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read fixture %s: %v", name, err)
}
var reply pb.MxCommandReply
if err := protojson.Unmarshal(data, &reply); err != nil {
t.Fatalf("parse fixture %s: %v", name, err)
}
return &reply
}
func TestAuthenticateUserMissingPayloadReturnsMalformedReplyError(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, "authenticate-user.missing-payload.reply.json"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
_, err := session.AuthenticateUser(context.Background(), 12, "operator", "secret")
var malformed *MalformedReplyError
if !errors.As(err, &malformed) {
t.Fatalf("AuthenticateUser() error = %v (%T), want *MalformedReplyError", err, err)
}
if malformed.Op != "authenticate user" {
t.Fatalf("MalformedReplyError.Op = %q, want %q", malformed.Op, "authenticate user")
}
}
func TestAuthenticateUserReturnValueOnlyUsesInt32ReturnValue(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, "authenticate-user.return-value-only.reply.json"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
userID, err := session.AuthenticateUser(context.Background(), 12, "operator", "secret")
if err != nil {
t.Fatalf("AuthenticateUser() error = %v", err)
}
if userID != 7 {
t.Fatalf("AuthenticateUser() = %d, want 7", userID)
}
}
// AddBufferedItem shares the prefer-payload / int32-return-value / malformed
// fallback code path; cover both branches for one of the siblings.
func TestAddBufferedItemFallbackHonoursReturnValueAndReportsMalformed(t *testing.T) {
t.Run("return-value-only", func(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, "authenticate-user.return-value-only.reply.json"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
itemHandle, err := session.AddBufferedItem(context.Background(), 12, "Area001.Pump001.Speed", "runtime")
if err != nil {
t.Fatalf("AddBufferedItem() error = %v", err)
}
if itemHandle != 7 {
t.Fatalf("AddBufferedItem() = %d, want 7", itemHandle)
}
})
t.Run("missing-payload", func(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, "authenticate-user.missing-payload.reply.json"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
_, err := session.AddBufferedItem(context.Background(), 12, "Area001.Pump001.Speed", "runtime")
var malformed *MalformedReplyError
if !errors.As(err, &malformed) {
t.Fatalf("AddBufferedItem() error = %v (%T), want *MalformedReplyError", err, err)
}
if malformed.Op != "add buffered item" {
t.Fatalf("MalformedReplyError.Op = %q, want %q", malformed.Op, "add buffered item")
}
})
}
// TestAuthenticateUserScrubsEchoedCredentialFromError is the CLI-40 regression:
// a gateway diagnostic that echoes the raw credential back must never reach the
// caller's surfaced error text.
func TestAuthenticateUserScrubsEchoedCredentialFromError(t *testing.T) {
const credential = "sup3rSecretVerify9f3a2b"
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, "authenticate-user.echoed-credential.reply.json"),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
_, err := session.AuthenticateUser(context.Background(), 12, "operator", credential)
if err == nil {
t.Fatal("AuthenticateUser() error = nil, want an MXAccess failure")
}
message := err.Error()
if strings.Contains(message, credential) {
t.Fatalf("surfaced error leaked the credential: %q", message)
}
if !strings.Contains(message, "<redacted>") {
t.Fatalf("surfaced error missing redaction marker: %q", message)
}
}
// TestAuthenticateUserScrubsEchoedCredentialFromStructuredReply is the CLI-40
// follow-up: redacting only the rendered Error() string is not enough. The typed
// *MxAccessError still carries the raw command reply, whose ProtocolStatus.Message,
// DiagnosticMessage, and Statuses[].DiagnosticText echo the credential verbatim. A
// logger dumping structured fields would reintroduce the leak, so the reply the
// typed error carries must be a scrubbed clone. Both the OK+negative-HRESULT and the
// MXACCESS_FAILURE fixtures route to *MxAccessError (via EnsureProtocolSuccess), so
// both must be scrubbed identically.
func TestAuthenticateUserScrubsEchoedCredentialFromStructuredReply(t *testing.T) {
const credential = "sup3rSecretVerify9f3a2b"
fixtures := []string{
"authenticate-user.echoed-credential.reply.json",
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
}
for _, fixture := range fixtures {
t.Run(fixture, func(t *testing.T) {
fake := &fakeGatewayServer{
invokeReply: loadCommandReplyFixture(t, fixture),
}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
_, err := session.AuthenticateUser(context.Background(), 12, "operator", credential)
if err == nil {
t.Fatal("AuthenticateUser() error = nil, want an MXAccess failure")
}
var mxErr *MxAccessError
if !errors.As(err, &mxErr) {
t.Fatalf("AuthenticateUser() error = %v (%T), want *MxAccessError", err, err)
}
reply := mxErr.Reply
if reply == nil {
t.Fatal("MxAccessError.Reply is nil, want the scrubbed command reply")
}
if got := reply.GetProtocolStatus().GetMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.ProtocolStatus.Message leaked the credential: %q", got)
}
if got := reply.GetDiagnosticMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.DiagnosticMessage leaked the credential: %q", got)
}
for i, status := range reply.GetStatuses() {
if got := status.GetDiagnosticText(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Reply.Statuses[%d].DiagnosticText leaked the credential: %q", i, got)
}
}
// The wrapped CommandError's status/reply must be scrubbed too.
if mxErr.Command != nil {
if got := mxErr.Command.Status.GetMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Command.Status.Message leaked the credential: %q", got)
}
if cmdReply := mxErr.Command.Reply; cmdReply != nil {
if got := cmdReply.GetDiagnosticMessage(); strings.Contains(got, credential) {
t.Fatalf("MxAccessError.Command.Reply.DiagnosticMessage leaked the credential: %q", got)
}
}
}
if got := err.Error(); strings.Contains(got, credential) {
t.Fatalf("rendered error leaked the credential: %q", got)
}
})
}
}
+5 -4
View File
@@ -51,8 +51,9 @@ func TestStatusConversionFixtures(t *testing.T) {
var fixture struct {
Cases []struct {
ID string `json:"id"`
Status json.RawMessage `json:"status"`
ID string `json:"id"`
WantSuccess bool `json:"wantSuccess"`
Status json.RawMessage `json:"status"`
} `json:"cases"`
}
if err := json.Unmarshal(data, &fixture); err != nil {
@@ -65,8 +66,8 @@ func TestStatusConversionFixtures(t *testing.T) {
if err := protojson.Unmarshal(tc.Status, &status); err != nil {
t.Fatalf("parse status: %v", err)
}
if got, want := StatusSucceeded(&status), status.GetSuccess() != 0; got != want {
t.Fatalf("StatusSucceeded() = %v, want %v", got, want)
if got := StatusSucceeded(&status); got != tc.WantSuccess {
t.Fatalf("StatusSucceeded() = %v, want %v", got, tc.WantSuccess)
}
})
}
+119 -7
View File
@@ -6,6 +6,7 @@ import (
"strings"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/protobuf/proto"
)
// redactedSecretMarker is the placeholder substituted for credential material in
@@ -49,20 +50,108 @@ func (e *secretRedactingError) Unwrap() error {
return e.err
}
// redactSecrets wraps err so any occurrence of a non-empty secret in the surfaced
// message is redacted, while errors.As / errors.Is still reach the wrapped typed
// error. It returns nil unchanged and skips wrapping when no non-empty secret is
// supplied, so non-secret-bearing calls keep their original error verbatim.
// scrubReplyStrings returns a clone of reply with every non-empty secret replaced
// by redactedSecretMarker in the free-text fields a gateway diagnostic could echo a
// credential into: ProtocolStatus.Message, DiagnosticMessage, and each
// Statuses[].DiagnosticText. It clones with proto.Clone so the caller's original
// reply is never mutated. A nil reply, or an empty/whitespace-only secret set, is a
// no-op (nil in, nil out; a clone otherwise).
func scrubReplyStrings(reply *pb.MxCommandReply, secrets []string) *pb.MxCommandReply {
if reply == nil {
return nil
}
clone, ok := proto.Clone(reply).(*pb.MxCommandReply)
if !ok {
return reply
}
for _, secret := range secrets {
if secret == "" {
continue
}
if clone.GetProtocolStatus() != nil {
clone.ProtocolStatus.Message = strings.ReplaceAll(clone.GetProtocolStatus().GetMessage(), secret, redactedSecretMarker)
}
clone.DiagnosticMessage = strings.ReplaceAll(clone.GetDiagnosticMessage(), secret, redactedSecretMarker)
for _, status := range clone.GetStatuses() {
status.DiagnosticText = strings.ReplaceAll(status.GetDiagnosticText(), secret, redactedSecretMarker)
}
}
return clone
}
// scrubProtocolStatusMessage returns a clone of status with every non-empty secret
// redacted from its Message, leaving the original untouched.
func scrubProtocolStatusMessage(status *ProtocolStatus, secrets []string) *ProtocolStatus {
if status == nil {
return nil
}
clone, ok := proto.Clone(status).(*ProtocolStatus)
if !ok {
return status
}
for _, secret := range secrets {
if secret != "" {
clone.Message = strings.ReplaceAll(clone.GetMessage(), secret, redactedSecretMarker)
}
}
return clone
}
// redactSecrets scrubs a non-empty secret set from the error it surfaces. When the
// wrapped error is a typed *MxAccessError or *CommandError it is rebuilt carrying
// scrubbed clones of its reply and protocol status, so a caller logging the typed
// error's structured fields cannot reintroduce the credential the rendered message
// hides. The rebuilt (or original, for other error types) value is then wrapped in
// secretRedactingError as a belt-and-suspenders scrub of any remaining rendered
// text. errors.As / errors.Is still reach the typed error through the wrapper. It
// returns nil unchanged and skips all work when no non-empty secret is supplied, so
// non-secret-bearing calls keep their original error verbatim.
func redactSecrets(err error, secrets ...string) error {
if err == nil {
return nil
}
hasSecret := false
for _, secret := range secrets {
if secret != "" {
return &secretRedactingError{err: err, secrets: secrets}
hasSecret = true
break
}
}
return err
if !hasSecret {
return err
}
rebuilt := rebuildScrubbedError(err, secrets)
return &secretRedactingError{err: rebuilt, secrets: secrets}
}
// rebuildScrubbedError rebuilds the typed error carrying scrubbed clones of any
// command reply / protocol status it holds, so credential text never survives in the
// error's structured fields. Non-reply-bearing error types are returned unchanged.
func rebuildScrubbedError(err error, secrets []string) error {
switch typed := err.(type) {
case *MxAccessError:
return &MxAccessError{
Command: scrubCommandError(typed.Command, secrets),
Reply: scrubReplyStrings(typed.Reply, secrets),
}
case *CommandError:
return scrubCommandError(typed, secrets)
default:
return err
}
}
// scrubCommandError rebuilds a CommandError with a scrubbed Status and Reply.
func scrubCommandError(cmd *CommandError, secrets []string) *CommandError {
if cmd == nil {
return nil
}
return &CommandError{
Op: cmd.Op,
Status: scrubProtocolStatusMessage(cmd.Status, secrets),
Reply: scrubReplyStrings(cmd.Reply, secrets),
}
}
// ErrSlowConsumer is the terminal error sent on the Events/EventsAfter
@@ -72,6 +161,25 @@ func redactSecrets(err error, secrets ...string) error {
// dropping events. Match it with errors.Is.
var ErrSlowConsumer = errors.New("mxgateway: event consumer fell behind; stream terminated")
// MalformedReplyError reports an OK command reply that carried neither the
// typed payload the operation expected nor a usable int32 return_value, so the
// client cannot produce a result. It gives every affected helper one uniform,
// inspectable failure instead of silently returning a zero value.
type MalformedReplyError struct {
// Op names the operation whose reply was malformed.
Op string
// Detail explains what the reply was missing.
Detail string
}
// Error returns the formatted malformed-reply message.
func (e *MalformedReplyError) Error() string {
if e == nil {
return ""
}
return fmt.Sprintf("mxgateway: %s returned a malformed reply: %s", e.Op, e.Detail)
}
// GatewayError wraps transport-level gRPC failures.
type GatewayError struct {
// Op names the operation that failed (for example "dial" or "invoke").
@@ -180,11 +288,15 @@ func EnsureProtocolSuccess(op string, status *ProtocolStatus, reply *MxCommandRe
// EnsureMxAccessSuccess returns a typed MxAccessError for failing HRESULTs or
// MXSTATUS_PROXY entries.
//
// Following COM semantics, only a negative HRESULT is a failure — positive
// success codes such as S_FALSE (1) pass. Status entries are judged by
// StatusSucceeded, which branches on the authoritative category.
func EnsureMxAccessSuccess(op string, reply *MxCommandReply) error {
if reply == nil {
return nil
}
if reply.Hresult != nil && reply.GetHresult() != 0 {
if reply.Hresult != nil && reply.GetHresult() < 0 {
return &MxAccessError{Reply: reply}
}
for _, status := range reply.GetStatuses() {
@@ -0,0 +1,115 @@
package mxgateway
import (
"errors"
"strings"
"testing"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
)
// TestScrubReplyStringsRedactsEveryOccurrence covers the multi-occurrence case:
// one secret appearing across ProtocolStatus.Message, DiagnosticMessage, and every
// Statuses[].DiagnosticText must be fully redacted with no residue.
func TestScrubReplyStringsRedactsEveryOccurrence(t *testing.T) {
const secret = "hunter2"
reply := &pb.MxCommandReply{
ProtocolStatus: &pb.ProtocolStatus{Message: "rejected hunter2 then hunter2 again"},
DiagnosticMessage: "echoed hunter2 back",
Statuses: []*pb.MxStatusProxy{
{DiagnosticText: "first hunter2"},
{DiagnosticText: "second hunter2 and hunter2"},
},
}
scrubbed := scrubReplyStrings(reply, []string{secret})
if strings.Contains(scrubbed.GetProtocolStatus().GetMessage(), secret) {
t.Fatalf("ProtocolStatus.Message still contains the secret: %q", scrubbed.GetProtocolStatus().GetMessage())
}
if strings.Contains(scrubbed.GetDiagnosticMessage(), secret) {
t.Fatalf("DiagnosticMessage still contains the secret: %q", scrubbed.GetDiagnosticMessage())
}
for i, status := range scrubbed.GetStatuses() {
if strings.Contains(status.GetDiagnosticText(), secret) {
t.Fatalf("Statuses[%d].DiagnosticText still contains the secret: %q", i, status.GetDiagnosticText())
}
}
if !strings.Contains(scrubbed.GetProtocolStatus().GetMessage(), redactedSecretMarker) {
t.Fatalf("ProtocolStatus.Message missing redaction marker: %q", scrubbed.GetProtocolStatus().GetMessage())
}
// The original reply must be untouched (scrubReplyStrings clones).
if !strings.Contains(reply.GetDiagnosticMessage(), secret) {
t.Fatal("scrubReplyStrings mutated the original reply instead of cloning it")
}
}
// TestScrubReplyStringsRedactsOverlappingSecrets covers two secrets where one is a
// substring of the other: both must be fully redacted, with no partial leak of the
// longer secret's non-shared remainder.
func TestScrubReplyStringsRedactsOverlappingSecrets(t *testing.T) {
const shortSecret = "pass"
const longSecret = "password123"
reply := &pb.MxCommandReply{
DiagnosticMessage: "value was password123 and also pass",
}
scrubbed := scrubReplyStrings(reply, []string{longSecret, shortSecret})
got := scrubbed.GetDiagnosticMessage()
if strings.Contains(got, shortSecret) {
t.Fatalf("scrubbed message still contains a secret substring %q: %q", shortSecret, got)
}
if strings.Contains(got, longSecret) {
t.Fatalf("scrubbed message still contains %q: %q", longSecret, got)
}
// "123" is the longer secret's remainder past the shared "pass" prefix; it must
// not survive as a partial leak.
if strings.Contains(got, "123") {
t.Fatalf("scrubbed message leaked the longer secret's remainder: %q", got)
}
}
// TestRedactSecretsEmptyOrNilLeavesErrorUnchanged confirms the no-secret paths keep
// the original typed error verbatim (no wrapping, no scrubbed clone).
func TestRedactSecretsEmptyOrNilLeavesErrorUnchanged(t *testing.T) {
base := &MxAccessError{Reply: &pb.MxCommandReply{DiagnosticMessage: "boom"}}
if got := redactSecrets(base); got != error(base) {
t.Fatalf("redactSecrets with no secrets = %v, want the original error unchanged", got)
}
if got := redactSecrets(base, ""); got != error(base) {
t.Fatalf("redactSecrets with only an empty secret = %v, want the original error unchanged", got)
}
if got := redactSecrets(nil, "secret"); got != nil {
t.Fatalf("redactSecrets(nil, ...) = %v, want nil", got)
}
}
// TestRedactSecretsRebuildsTypedCommandError confirms a *CommandError (non-MXAccess
// path) is rebuilt with a scrubbed Status and Reply, and errors.As still reaches it.
func TestRedactSecretsRebuildsTypedCommandError(t *testing.T) {
const secret = "topSecretValue"
base := &CommandError{
Op: "write secured",
Status: &pb.ProtocolStatus{Message: "rejected topSecretValue"},
Reply: &pb.MxCommandReply{DiagnosticMessage: "echoed topSecretValue"},
}
redacted := redactSecrets(base, secret)
var cmdErr *CommandError
if !errors.As(redacted, &cmdErr) {
t.Fatalf("redactSecrets result %T does not unwrap to *CommandError", redacted)
}
if strings.Contains(cmdErr.Status.GetMessage(), secret) {
t.Fatalf("CommandError.Status.Message leaked the secret: %q", cmdErr.Status.GetMessage())
}
if strings.Contains(cmdErr.Reply.GetDiagnosticMessage(), secret) {
t.Fatalf("CommandError.Reply.DiagnosticMessage leaked the secret: %q", cmdErr.Reply.GetDiagnosticMessage())
}
if strings.Contains(redacted.Error(), secret) {
t.Fatalf("rendered error leaked the secret: %q", redacted.Error())
}
}
@@ -48,6 +48,89 @@ func TestGeneratedGoldenFixturesParse(t *testing.T) {
}
}
// TestCommandReplyValidationFixtures locks the shared reply-validation rules to
// the behavior fixtures: a status entry fails iff its category is not OK (the
// raw success member is diagnostics only), and an HRESULT fails iff it is
// present and negative (S_FALSE and other positive COM success codes pass).
func TestCommandReplyValidationFixtures(t *testing.T) {
tests := []struct {
fixture string
wantFailure bool
}{
{fixture: "register.ok.reply.json", wantFailure: false},
{fixture: "write.mxaccess-failure.reply.json", wantFailure: true},
{fixture: "write.status-category-error-success-set.reply.json", wantFailure: true},
{fixture: "write.status-category-ok-success-zero.reply.json", wantFailure: false},
{fixture: "write.hresult-s-false.reply.json", wantFailure: false},
{fixture: "write.hresult-e-fail.reply.json", wantFailure: true},
}
for _, tt := range tests {
t.Run(tt.fixture, func(t *testing.T) {
data, err := os.ReadFile(filepath.Join(
"..", "..", "proto", "fixtures", "behavior", "command-replies", tt.fixture))
if err != nil {
t.Fatalf("read fixture: %v", err)
}
var reply pb.MxCommandReply
if err := protojson.Unmarshal(data, &reply); err != nil {
t.Fatalf("parse fixture: %v", err)
}
err = EnsureMxAccessSuccess("invoke", &reply)
if got := err != nil; got != tt.wantFailure {
t.Fatalf("EnsureMxAccessSuccess() failed = %v (err %v), want %v", got, err, tt.wantFailure)
}
})
}
}
// TestStatusSucceededBranchesOnCategory pins the per-entry rule directly,
// including the two edges the fixtures cannot express: a nil entry is success
// and a present entry with an unspecified category is a failure.
func TestStatusSucceededBranchesOnCategory(t *testing.T) {
tests := []struct {
name string
status *MxStatusProxy
want bool
}{
{name: "nil entry", status: nil, want: true},
{
name: "ok category with zero success",
status: &pb.MxStatusProxy{
Success: 0,
Category: pb.MxStatusCategory_MX_STATUS_CATEGORY_OK,
},
want: true,
},
{
name: "error category with success set",
status: &pb.MxStatusProxy{
Success: 1,
Category: pb.MxStatusCategory_MX_STATUS_CATEGORY_COMMUNICATION_ERROR,
},
want: false,
},
{
name: "unspecified category with success set",
status: &pb.MxStatusProxy{
Success: 1,
Category: pb.MxStatusCategory_MX_STATUS_CATEGORY_UNSPECIFIED,
},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := StatusSucceeded(tt.status); got != tt.want {
t.Fatalf("StatusSucceeded() = %v, want %v", got, tt.want)
}
})
}
}
func TestOpenSessionFixtureProtocolVersions(t *testing.T) {
data, err := os.ReadFile(filepath.Join("..", "..", "proto", "fixtures", "golden", "open-session-reply.ok.json"))
if err != nil {
+64 -14
View File
@@ -812,7 +812,13 @@ func (s *Session) AuthenticateUser(ctx context.Context, serverHandle int32, veri
if reply.GetAuthenticateUser() != nil {
return reply.GetAuthenticateUser().GetUserId(), nil
}
return reply.GetReturnValue().GetInt32Value(), nil
if x, ok := reply.GetReturnValue().GetKind().(*pb.MxValue_Int32Value); ok {
return x.Int32Value, nil
}
return 0, &MalformedReplyError{
Op: "authenticate user",
Detail: "reply carried neither an AuthenticateUser payload nor an int32 return_value",
}
}
// AuthenticateUserRaw invokes MXAccess AuthenticateUser and returns the raw
@@ -847,7 +853,13 @@ func (s *Session) ArchestrAUserToId(ctx context.Context, serverHandle int32, use
if reply.GetArchestraUserToId() != nil {
return reply.GetArchestraUserToId().GetUserId(), nil
}
return reply.GetReturnValue().GetInt32Value(), nil
if x, ok := reply.GetReturnValue().GetKind().(*pb.MxValue_Int32Value); ok {
return x.Int32Value, nil
}
return 0, &MalformedReplyError{
Op: "archestra user to id",
Detail: "reply carried neither an ArchestrAUserToId payload nor an int32 return_value",
}
}
// ArchestrAUserToIdRaw invokes MXAccess ArchestrAUserToId and returns the raw reply.
@@ -876,7 +888,13 @@ func (s *Session) AddBufferedItem(ctx context.Context, serverHandle int32, itemD
if reply.GetAddBufferedItem() != nil {
return reply.GetAddBufferedItem().GetItemHandle(), nil
}
return reply.GetReturnValue().GetInt32Value(), nil
if x, ok := reply.GetReturnValue().GetKind().(*pb.MxValue_Int32Value); ok {
return x.Int32Value, nil
}
return 0, &MalformedReplyError{
Op: "add buffered item",
Detail: "reply carried neither an AddBufferedItem payload nor an int32 return_value",
}
}
// AddBufferedItemRaw invokes MXAccess AddBufferedItem and returns the raw reply.
@@ -981,10 +999,12 @@ func stringSecrets(values ...*MxValue) []string {
// context cancellation stops Recv, or a terminal error is sent.
//
// The returned channel is buffered. If the consumer falls behind and the buffer
// overflows, the stream is terminated and a final EventResult carrying a
// GatewayError that wraps ErrSlowConsumer is delivered before the channel
// closes. Callers must match it with errors.Is(res.Err, ErrSlowConsumer) to
// distinguish a slow-consumer drop from a graceful server end. Use
// overflows with data, the stream is terminated and a final EventResult carrying
// a GatewayError that wraps ErrSlowConsumer is delivered before the channel
// closes; match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
// slow-consumer drop from a graceful server end. A genuine stream error is
// reported as itself even under overflow — it is never relabeled as
// ErrSlowConsumer, so the underlying gRPC status stays inspectable. Use
// SubscribeEvents for a blocking, backpressured stream that never drops.
func (s *Session) Events(ctx context.Context) (<-chan EventResult, error) {
return s.EventsAfter(ctx, 0)
@@ -994,7 +1014,9 @@ func (s *Session) Events(ctx context.Context) (<-chan EventResult, error) {
//
// Like Events, the returned channel is buffered and terminates with a final
// EventResult wrapping ErrSlowConsumer (matchable via errors.Is) if the consumer
// falls behind and the buffer overflows, rather than silently closing.
// falls behind and the buffer overflows with data, rather than silently closing.
// A genuine stream error is reported as itself even under overflow, never
// relabeled as ErrSlowConsumer.
func (s *Session) EventsAfter(ctx context.Context, afterWorkerSequence uint64) (<-chan EventResult, error) {
subscription, err := s.subscribeEventsAfter(ctx, afterWorkerSequence, true)
if err != nil {
@@ -1048,12 +1070,11 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
if err == io.EOF || status.Code(err) == codes.Canceled || streamCtx.Err() != nil {
return
}
sendEventResult(
streamCtx,
results,
EventResult{Err: &GatewayError{Op: "stream events", Err: err}},
cancelWhenResultBufferFull,
cancel)
// A genuine terminal stream error must be reported as itself, even
// when the data slots are full. Routing it through sendEventResult
// would let the overflow branch substitute ErrSlowConsumer and lose
// the real gRPC status, so send it directly, bypassing that branch.
sendTerminalEventResult(streamCtx, results, EventResult{Err: &GatewayError{Op: "stream events", Err: err}}, cancelWhenResultBufferFull)
return
}
}()
@@ -1072,6 +1093,35 @@ func ensureBulkSize(name string, length int) error {
return nil
}
// sendTerminalEventResult enqueues a terminal EventResult, bypassing
// sendEventResult's overflow branch so a genuine stream error is reported verbatim
// rather than relabeled as ErrSlowConsumer. How it sends depends on the mode:
//
// - cancelWhenBufferFull=true (Events/EventsAfter): ordinary data sends are capped
// at eventBufferSize, leaving eventBufferReservedSlots free, so a non-blocking
// send always lands the terminal result. Because this goroutine is the sole
// producer, at most one terminal send ever races for the reserved slot, so the
// select default only fires when the reserve is already spent — never dropping a
// first terminal error.
// - cancelWhenBufferFull=false (SubscribeEvents/SubscribeEventsAfter, never-drop):
// ordinary data sends are uncapped and blocking, so every slot including the
// reserve can hold data. A non-blocking send would then hit the full buffer and
// silently drop the terminal error, breaking the never-drop contract; instead
// block until the consumer drains a slot (or the stream context is cancelled).
func sendTerminalEventResult(ctx context.Context, results chan<- EventResult, result EventResult, cancelWhenBufferFull bool) {
if cancelWhenBufferFull {
select {
case results <- result:
default:
}
return
}
select {
case results <- result:
case <-ctx.Done():
}
}
func sendEventResult(
ctx context.Context,
results chan<- EventResult,
+12 -1
View File
@@ -1,6 +1,17 @@
package mxgateway
import (
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
)
// StatusSucceeded reports whether an MXSTATUS_PROXY entry represents success.
//
// The wire contract makes Category authoritative: an entry succeeds only when
// its category is MX_STATUS_CATEGORY_OK. The Success member mirrors the raw
// 16-bit COM value verbatim for diagnostics and is not a boolean, so it takes
// no part in the verdict. A nil entry is success (nothing was reported); a
// present entry with an unspecified category is a failure, because the worker
// always maps a category and an unmapped one is not proven OK.
func StatusSucceeded(status *MxStatusProxy) bool {
return status == nil || status.GetSuccess() != 0
return status == nil || status.GetCategory() == pb.MxStatusCategory_MX_STATUS_CATEGORY_OK
}
+13 -3
View File
@@ -139,7 +139,12 @@ commands, so you do not need to build raw `MxCommand` messages:
All of them run the same MXAccess reply validation as the bulk helpers (protocol
status plus HRESULT/`MxStatusProxy` check) via the shared `invoke` path, so an
MXAccess COM-side failure surfaces as `MxAccessException`.
MXAccess COM-side failure surfaces as `MxAccessException`. That validation
follows COM semantics: only a **negative** HRESULT is a failure, so positive
success codes such as `S_FALSE` (1) pass. `MxStatuses.succeeded` judges each
entry by its category — an entry fails when its category is not
`MX_STATUS_CATEGORY_OK`, and the raw `success` member is a diagnostic that never
decides the verdict. A `null` entry is success.
**Secret redaction.** Credentials passed to `authenticateUser` (and the
credential-sensitive values passed to `writeSecured`/`writeSecured2`) travel
@@ -167,8 +172,13 @@ session.write(serverHandle, itemHandle, value, userId);
native failure is surfaced, not papered over.
The CLI exposes `advise-supervisory`, `write-secured`, and `authenticate-user`
(credential via `--password` or `--password-env`, never echoed), and `write` /
`write2` take `--user-id`.
(credential via `--password` or the variable named by `--password-env`, default
`MXGATEWAY_VERIFY_PASSWORD`, never echoed), and `write` / `write2` take
`--user-id`. The credential is required: a missing or empty resolved value is a
picocli usage error naming the option and the variable, so the CLI fails before
connecting instead of authenticating with an empty password.
`MXGATEWAY_VERIFY_PASSWORD` is the canonical variable across all five client
CLIs — see [Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
### Array writes replace the whole array
@@ -70,6 +70,7 @@ import picocli.CommandLine.Command;
import picocli.CommandLine.Mixin;
import picocli.CommandLine.Model.CommandSpec;
import picocli.CommandLine.Option;
import picocli.CommandLine.ParameterException;
import picocli.CommandLine.Spec;
/**
@@ -182,6 +183,13 @@ public final class MxGatewayCli implements Callable<Integer> {
/** Sentinel written to stdout after every command result in batch mode. */
static final String BATCH_EOR = "__MXGW_BATCH_EOR__";
/**
* Canonical CLI credential environment variable, shared by every official
* client CLI (CLI-45) so one exported variable drives the same operator
* workflow in all five languages.
*/
static final String DEFAULT_VERIFY_PASSWORD_ENV = "MXGATEWAY_VERIFY_PASSWORD";
/** Sentinel queued by {@code stream-alarms} to mark a clean end of the alarm feed. */
private static final Object ALARM_FEED_END = new Object();
@@ -1139,7 +1147,7 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(
names = "--password-env",
defaultValue = "MXGATEWAY_VERIFY_PASSWORD",
defaultValue = DEFAULT_VERIFY_PASSWORD_ENV,
description = "Environment variable holding the password when --password is omitted.")
String passwordEnv;
@@ -1151,11 +1159,20 @@ public final class MxGatewayCli implements Callable<Integer> {
public Integer call() {
// Resolve the credential from the flag or environment. It flows only
// into the request; it is never written to output, logs, or errors.
String environmentName =
passwordEnv == null || passwordEnv.isBlank() ? DEFAULT_VERIFY_PASSWORD_ENV : passwordEnv;
String resolvedPassword = password == null || password.isBlank()
? System.getenv(passwordEnv)
? System.getenv(environmentName)
: password;
if (resolvedPassword == null) {
resolvedPassword = "";
if (resolvedPassword == null || resolvedPassword.isBlank()) {
// Fail fast instead of dialing: a misconfigured environment must not
// become a real MXAccess authentication attempt with an empty
// credential (CLI-45). The message names the option and the variable
// only never the value.
throw new ParameterException(
common.spec.commandLine(),
"a password is required via --password or the " + environmentName
+ " environment variable");
}
try (MxGatewayCliClient client = clientFactory.connect(common.resolved())) {
int userId = client.session(sessionId)
@@ -2,7 +2,10 @@ package com.zb.mom.ww.mxgateway.cli;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeTrue;
import com.zb.mom.ww.mxgateway.client.MxGatewayAlarmFeedSubscription;
import com.zb.mom.ww.mxgateway.client.MxGatewayClientOptions;
@@ -211,6 +214,73 @@ final class MxGatewayCliTests {
assertFalse(run.errors().contains("super-secret-pw"), "password must never be echoed to stderr");
}
/**
* CLI-45: an unresolved credential must abort with a picocli usage error
* before the CLI dials, instead of authenticating with an empty password.
* The message names the option and the variable, never a value.
*/
@Test
void authenticateUserRejectsMissingCredentialWithUsageError() {
FakeClientFactory factory = new FakeClientFactory();
CliRun run = execute(
factory,
"authenticate-user",
"--session-id", "session-cli",
"--server-handle", "3",
"--verify-user", "operator",
"--password-env", "MXGW_CLI45_ABSENT_PASSWORD_VAR",
"--json");
assertNotEquals(0, run.exitCode(), "a missing credential must fail");
assertTrue(run.errors().contains("--password"), run.errors());
assertTrue(run.errors().contains("MXGW_CLI45_ABSENT_PASSWORD_VAR"), run.errors());
assertNull(factory.client, "the CLI must not connect without a credential");
}
/**
* CLI-45: a blank {@code --password} is treated as missing the CLI never
* sends a fabricated empty credential to the wire.
*/
@Test
void authenticateUserRejectsBlankPasswordValue() {
FakeClientFactory factory = new FakeClientFactory();
CliRun run = execute(
factory,
"authenticate-user",
"--session-id", "session-cli",
"--server-handle", "3",
"--verify-user", "operator",
"--password", "",
"--password-env", "MXGW_CLI45_ABSENT_PASSWORD_VAR",
"--json");
assertNotEquals(0, run.exitCode(), "a blank credential must fail");
assertNull(factory.client, "the CLI must not connect without a credential");
}
/**
* CLI-45: {@code --password-env} defaults to the canonical
* {@code MXGATEWAY_VERIFY_PASSWORD}, so the usage error names it when no
* explicit variable is given. Skipped if the canonical variable happens to be
* exported in the running environment (which would satisfy the credential).
*/
@Test
void authenticateUserDefaultsToCanonicalPasswordEnvName() {
assumeTrue(System.getenv(MxGatewayCli.DEFAULT_VERIFY_PASSWORD_ENV) == null);
FakeClientFactory factory = new FakeClientFactory();
CliRun run = execute(
factory,
"authenticate-user",
"--session-id", "session-cli",
"--server-handle", "3",
"--verify-user", "operator",
"--json");
assertNotEquals(0, run.exitCode());
assertTrue(run.errors().contains("MXGATEWAY_VERIFY_PASSWORD"), run.errors());
}
// ---- ping subcommand (D4) ----
@Test
@@ -29,4 +29,18 @@ public final class MxAccessException extends MxGatewayCommandException {
public MxAccessException(String operation, MxCommandReply reply) {
super(operation, reply == null ? null : reply.getProtocolStatus(), reply);
}
/**
* Creates a new MXAccess exception with an already-built, verbatim message.
* Used to re-surface an MXAccess failure with a redacted message while
* preserving the original protocol status and reply.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status reported by the gateway
* @param reply raw command reply containing the MXAccess failure detail
* @param cause underlying error, or {@code null}
*/
public MxAccessException(String message, ProtocolStatus protocolStatus, MxCommandReply reply, Throwable cause) {
super(message, protocolStatus, reply, cause);
}
}
@@ -25,6 +25,23 @@ public class MxGatewayCommandException extends MxGatewayException {
this.reply = reply;
}
/**
* Creates a new command exception with an already-built, verbatim message.
* Used to re-surface a failure with a redacted message while preserving the
* original protocol status and reply.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status returned by the gateway
* @param reply raw command reply, or {@code null} when none was produced
* @param cause underlying error, or {@code null}
*/
protected MxGatewayCommandException(
String message, ProtocolStatus protocolStatus, MxCommandReply reply, Throwable cause) {
super(message, cause);
this.protocolStatus = protocolStatus;
this.reply = reply;
}
/**
* Returns the gateway protocol status that triggered this exception.
*
@@ -47,7 +47,9 @@ final class MxGatewayErrors {
if (reply == null) {
return;
}
if (reply.hasHresult() && reply.getHresult() != 0) {
// COM semantics: only a negative HRESULT is a failure. Positive success
// codes such as S_FALSE (1) pass.
if (reply.hasHresult() && reply.getHresult() < 0) {
throw new MxAccessException(operation, reply);
}
for (var status : reply.getStatusesList()) {
@@ -0,0 +1,32 @@
package com.zb.mom.ww.mxgateway.client;
/**
* Thrown when the gateway returns a protocol-OK command reply that carries
* neither the expected typed payload nor a usable {@code return_value}.
*
* <p>A successful reply for a value-returning command (for example
* {@code AuthenticateUser}, {@code ArchestrAUserToId}, or {@code AddBufferedItem})
* must supply either the command's typed payload or an int32 {@code return_value}.
* A reply that satisfies neither is malformed, and the client surfaces this
* distinct failure rather than silently returning a default {@code 0}.
*/
public final class MxGatewayMalformedReplyException extends MxGatewayException {
/**
* Creates a new malformed-reply exception with the supplied message.
*
* @param message human-readable description of the malformed reply
*/
public MxGatewayMalformedReplyException(String message) {
super(message);
}
/**
* Creates a new malformed-reply exception with the supplied message and cause.
*
* @param message human-readable description of the malformed reply
* @param cause underlying error that triggered the failure
*/
public MxGatewayMalformedReplyException(String message, Throwable cause) {
super(message, cause);
}
}
@@ -54,4 +54,34 @@ public final class MxGatewaySecrets {
}
return String.join(" ", parts);
}
/**
* Replaces every occurrence of each supplied secret with the redaction
* marker {@code "<redacted>"}. Unlike {@link #redactCredentials(String)},
* which scrubs by pattern, this performs an exact-substring scrub of the
* caller-known secrets used to strip a credential the gateway echoed back
* into a free-form failure message.
*
* @param message the message to scrub, may be {@code null}
* @param secrets the exact secret substrings to remove; {@code null}, empty,
* and blank (whitespace-only) entries and a {@code null} array are ignored
* @return {@code message} unchanged when it is {@code null} or no non-blank
* secret is supplied, otherwise the message with every secret occurrence
* replaced by {@code "<redacted>"}
*/
public static String redactExact(String message, String... secrets) {
if (message == null || secrets == null) {
return message;
}
String result = message;
for (String secret : secrets) {
if (secret == null || secret.isBlank()) {
// A blank "secret" would over-redact real whitespace; skip it.
continue;
}
result = result.replace(secret, "<redacted>");
}
return result;
}
}
@@ -31,6 +31,7 @@ import mxaccess_gateway.v1.MxaccessGateway.MxSparseElement;
import mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy;
import mxaccess_gateway.v1.MxaccessGateway.MxValue;
import mxaccess_gateway.v1.MxaccessGateway.OpenSessionReply;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatus;
import mxaccess_gateway.v1.MxaccessGateway.ReadBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.RegisterCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemBulkCommand;
@@ -782,15 +783,17 @@ public final class MxGatewaySession implements AutoCloseable {
*/
public MxCommandReply writeSecuredRaw(
int serverHandle, int itemHandle, int currentUserId, int verifierUserId, MxValue value) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_WRITE_SECURED)
.setWriteSecured(WriteSecuredCommand.newBuilder()
.setServerHandle(serverHandle)
.setItemHandle(itemHandle)
.setCurrentUserId(currentUserId)
.setVerifierUserId(verifierUserId)
.setValue(value))
.build());
return invokeCommandRedacted(
MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_WRITE_SECURED)
.setWriteSecured(WriteSecuredCommand.newBuilder()
.setServerHandle(serverHandle)
.setItemHandle(itemHandle)
.setCurrentUserId(currentUserId)
.setVerifierUserId(verifierUserId)
.setValue(value))
.build(),
secretStringOf(value));
}
/**
@@ -837,16 +840,18 @@ public final class MxGatewaySession implements AutoCloseable {
int verifierUserId,
MxValue value,
MxValue timestampValue) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_WRITE_SECURED2)
.setWriteSecured2(WriteSecured2Command.newBuilder()
.setServerHandle(serverHandle)
.setItemHandle(itemHandle)
.setCurrentUserId(currentUserId)
.setVerifierUserId(verifierUserId)
.setValue(value)
.setTimestampValue(timestampValue))
.build());
return invokeCommandRedacted(
MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_WRITE_SECURED2)
.setWriteSecured2(WriteSecured2Command.newBuilder()
.setServerHandle(serverHandle)
.setItemHandle(itemHandle)
.setCurrentUserId(currentUserId)
.setVerifierUserId(verifierUserId)
.setValue(value)
.setTimestampValue(timestampValue))
.build(),
secretStringOf(value));
}
/**
@@ -866,17 +871,24 @@ public final class MxGatewaySession implements AutoCloseable {
* @throws MxAccessException when MXAccess rejects the credential
*/
public int authenticateUser(int serverHandle, String verifyUser, String verifyUserPassword) {
MxCommandReply reply = invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_AUTHENTICATE_USER)
.setAuthenticateUser(AuthenticateUserCommand.newBuilder()
.setServerHandle(serverHandle)
.setVerifyUser(verifyUser)
.setVerifyUserPassword(verifyUserPassword))
.build());
MxCommandReply reply = invokeCommandRedacted(
MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_AUTHENTICATE_USER)
.setAuthenticateUser(AuthenticateUserCommand.newBuilder()
.setServerHandle(serverHandle)
.setVerifyUser(verifyUser)
.setVerifyUserPassword(verifyUserPassword))
.build(),
verifyUserPassword);
if (reply.hasAuthenticateUser()) {
return reply.getAuthenticateUser().getUserId();
}
return reply.getReturnValue().getInt32Value();
if (reply.hasReturnValue() && reply.getReturnValue().getKindCase() == MxValue.KindCase.INT32_VALUE) {
return reply.getReturnValue().getInt32Value();
}
throw new MxGatewayMalformedReplyException(
"AuthenticateUser returned a malformed reply: OK reply carried neither "
+ "the typed payload nor an int32 return_value");
}
/**
@@ -899,7 +911,12 @@ public final class MxGatewaySession implements AutoCloseable {
if (reply.hasArchestraUserToId()) {
return reply.getArchestraUserToId().getUserId();
}
return reply.getReturnValue().getInt32Value();
if (reply.hasReturnValue() && reply.getReturnValue().getKindCase() == MxValue.KindCase.INT32_VALUE) {
return reply.getReturnValue().getInt32Value();
}
throw new MxGatewayMalformedReplyException(
"ArchestrAUserToId returned a malformed reply: OK reply carried neither "
+ "the typed payload nor an int32 return_value");
}
/**
@@ -925,7 +942,12 @@ public final class MxGatewaySession implements AutoCloseable {
if (reply.hasAddBufferedItem()) {
return reply.getAddBufferedItem().getItemHandle();
}
return reply.getReturnValue().getInt32Value();
if (reply.hasReturnValue() && reply.getReturnValue().getKindCase() == MxValue.KindCase.INT32_VALUE) {
return reply.getReturnValue().getInt32Value();
}
throw new MxGatewayMalformedReplyException(
"AddBufferedItem returned a malformed reply: OK reply carried neither "
+ "the typed payload nor an int32 return_value");
}
/**
@@ -1027,6 +1049,162 @@ public final class MxGatewaySession implements AutoCloseable {
.build());
}
/**
* Invokes a credential-bearing command, scrubbing any exact secret the
* gateway may have echoed back into a surfaced failure message. The secret
* lives only in the request, but a non-parity gateway or provider can copy
* it into a diagnostic; this guarantees it never survives in the exception
* text a caller might log.
*
* <p>On failure both the exception's message <em>and</em> its structured
* context (the {@link ProtocolStatus} and {@link MxCommandReply} a caller can
* inspect and log) are scrubbed with {@link MxGatewaySecrets#redactExact}: the
* gateway echoes the credential into {@code protocolStatus.message},
* {@code reply.diagnosticMessage}, and each {@code statuses[i].diagnosticText}.
* If nothing carried the secret (the common case) the original exception is
* rethrown untouched. Otherwise it is re-thrown as the same concrete type
* carrying the redacted message and scrubbed context; the secret-bearing
* original is not chained as a cause, so it cannot leak through a printed
* stack trace.
*/
private MxCommandReply invokeCommandRedacted(MxCommand command, String... secrets) {
try {
return invokeCommand(command);
} catch (MxGatewayException ex) {
String original = ex.getMessage();
String redactedMessage = MxGatewaySecrets.redactExact(original, secrets);
boolean messageChanged = redactedMessage != null && !redactedMessage.equals(original);
ProtocolStatus status = protocolStatusOf(ex);
ProtocolStatus scrubbedStatus = scrubProtocolStatus(status, secrets);
boolean statusChanged = status != null && !status.equals(scrubbedStatus);
MxCommandReply reply = replyOf(ex);
MxCommandReply scrubbedReply = scrubReply(reply, secrets);
boolean replyChanged = reply != null && !reply.equals(scrubbedReply);
if (!messageChanged && !statusChanged && !replyChanged) {
throw ex;
}
String message = messageChanged ? redactedMessage : original;
throw rebuildRedacted(ex, message, scrubbedStatus, scrubbedReply);
}
}
/**
* Extracts the {@link ProtocolStatus} an exception carries, if any, so it can
* be scrubbed and re-attached to the rebuilt exception.
*/
private static ProtocolStatus protocolStatusOf(MxGatewayException ex) {
if (ex instanceof MxGatewayCommandException command) {
return command.protocolStatus();
}
if (ex instanceof MxGatewaySessionException session) {
return session.protocolStatus();
}
if (ex instanceof MxGatewayWorkerException worker) {
return worker.protocolStatus();
}
return null;
}
/**
* Extracts the raw {@link MxCommandReply} an exception carries, if any.
*/
private static MxCommandReply replyOf(MxGatewayException ex) {
if (ex instanceof MxGatewayCommandException command) {
return command.reply();
}
return null;
}
/**
* Rebuilds a gateway exception of the same concrete type with a redacted
* message and already-scrubbed context, mirroring the .NET client's
* type-switch. Only a truly-unknown subtype collapses to the base
* {@link MxGatewayException}. The original (secret-bearing) exception is
* deliberately not chained as a cause.
*/
private static MxGatewayException rebuildRedacted(
MxGatewayException ex, String message, ProtocolStatus status, MxCommandReply reply) {
if (ex instanceof MxAccessException) {
return new MxAccessException(message, status, reply, null);
}
if (ex instanceof MxGatewayCommandException) {
return new MxGatewayCommandException(message, status, reply, null);
}
if (ex instanceof MxGatewaySessionException) {
return new MxGatewaySessionException(message, status, null);
}
if (ex instanceof MxGatewayWorkerException) {
return new MxGatewayWorkerException(message, status, null);
}
if (ex instanceof MxGatewayMalformedReplyException) {
return new MxGatewayMalformedReplyException(message);
}
if (ex instanceof MxGatewayAuthenticationException) {
return new MxGatewayAuthenticationException(message, null);
}
if (ex instanceof MxGatewayAuthorizationException) {
return new MxGatewayAuthorizationException(message, null);
}
return new MxGatewayException(message);
}
/**
* Produces a scrubbed clone of a command reply, removing any exact secret the
* gateway echoed into {@code protocolStatus.message},
* {@code diagnosticMessage}, or a status's {@code diagnosticText}.
*
* @param reply the reply to scrub, or {@code null}
* @param secrets the exact secrets to strip
* @return {@code null} when {@code reply} is {@code null}, otherwise a clone
* with every echoed secret replaced by the redaction marker
*/
private static MxCommandReply scrubReply(MxCommandReply reply, String... secrets) {
if (reply == null) {
return null;
}
MxCommandReply.Builder builder = reply.toBuilder();
if (builder.hasProtocolStatus()) {
builder.setProtocolStatus(scrubProtocolStatus(builder.getProtocolStatus(), secrets));
}
builder.setDiagnosticMessage(MxGatewaySecrets.redactExact(builder.getDiagnosticMessage(), secrets));
for (int index = 0; index < builder.getStatusesCount(); index++) {
MxStatusProxy.Builder status = builder.getStatuses(index).toBuilder();
status.setDiagnosticText(MxGatewaySecrets.redactExact(status.getDiagnosticText(), secrets));
builder.setStatuses(index, status);
}
return builder.build();
}
/**
* Produces a scrubbed clone of a protocol status, removing any exact secret
* the gateway echoed into its free-form {@code message}.
*/
private static ProtocolStatus scrubProtocolStatus(ProtocolStatus status, String... secrets) {
if (status == null) {
return null;
}
return status.toBuilder()
.setMessage(MxGatewaySecrets.redactExact(status.getMessage(), secrets))
.build();
}
/**
* Extracts the string payload of a secured-write value so it can be scrubbed
* from an echoed failure message. Only string-kind values carry a
* credential-shaped secret worth redacting; other kinds return {@code null}
* (ignored by {@link MxGatewaySecrets#redactExact}).
*/
private static String secretStringOf(MxValue value) {
if (value != null && value.getKindCase() == MxValue.KindCase.STRING_VALUE) {
return value.getStringValue();
}
return null;
}
private static String newCorrelationId() {
byte[] bytes = new byte[16];
RANDOM.nextBytes(bytes);
@@ -20,6 +20,20 @@ public final class MxGatewaySessionException extends MxGatewayException {
this.protocolStatus = protocolStatus;
}
/**
* Creates a new session exception with an already-built, verbatim message.
* Used to re-surface a failure with a redacted message while preserving the
* (already scrubbed) protocol status.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status returned by the gateway
* @param cause underlying error, or {@code null}
*/
protected MxGatewaySessionException(String message, ProtocolStatus protocolStatus, Throwable cause) {
super(message, cause);
this.protocolStatus = protocolStatus;
}
/**
* Returns the gateway protocol status that triggered this exception.
*
@@ -20,6 +20,20 @@ public final class MxGatewayWorkerException extends MxGatewayException {
this.protocolStatus = protocolStatus;
}
/**
* Creates a new worker exception with an already-built, verbatim message.
* Used to re-surface a failure with a redacted message while preserving the
* (already scrubbed) protocol status.
*
* @param message the exact message to surface (already formatted/redacted)
* @param protocolStatus protocol status returned by the gateway
* @param cause underlying error, or {@code null}
*/
protected MxGatewayWorkerException(String message, ProtocolStatus protocolStatus, Throwable cause) {
super(message, cause);
this.protocolStatus = protocolStatus;
}
/**
* Returns the gateway protocol status that triggered this exception.
*
@@ -8,8 +8,11 @@ import mxaccess_gateway.v1.MxaccessGateway.MxStatusSource;
* Helpers for inspecting {@link MxStatusProxy} values returned by the gateway.
*
* <p>An {@code MxStatusProxy} mirrors the MXAccess COM {@code MXSTATUS_PROXY}
* struct. The success flag uses the MXAccess convention where any non-zero
* value indicates success.
* struct. Per the wire contract, {@code category} is the authoritative verdict:
* an entry succeeds only when its category is
* {@code MX_STATUS_CATEGORY_OK}. The {@code success} member carries the raw
* 16-bit COM value verbatim for diagnostics and is not a boolean, so it never
* decides success or failure.
*/
public final class MxStatuses {
private MxStatuses() {
@@ -18,12 +21,17 @@ public final class MxStatuses {
/**
* Returns whether the supplied status proxy reports success.
*
* <p>A {@code null} status is success because nothing was reported. A
* present entry whose category is {@code MX_STATUS_CATEGORY_UNSPECIFIED}
* is a failure: the worker always maps a category, so an unmapped one is
* not proven OK.
*
* @param status the status proxy, may be {@code null}
* @return {@code true} if {@code status} is {@code null} or its success
* flag is non-zero, {@code false} otherwise
* @return {@code true} if {@code status} is {@code null} or its category is
* {@code MX_STATUS_CATEGORY_OK}, {@code false} otherwise
*/
public static boolean succeeded(MxStatusProxy status) {
return status == null || status.getSuccess() != 0;
return status == null || status.getCategory() == MxStatusCategory.MX_STATUS_CATEGORY_OK;
}
/**
@@ -44,9 +52,11 @@ public final class MxStatuses {
*/
public record MxStatusView(MxStatusProxy raw) {
/**
* Returns the raw success flag (non-zero indicates success).
* Returns the raw {@code success} member exactly as MXAccess reported
* it. This is a diagnostic value, not a verdict use
* {@link MxStatuses#succeeded(MxStatusProxy)} to decide success.
*
* @return the success flag value
* @return the raw success member
*/
public int success() {
return raw.getSuccess();
@@ -701,14 +701,17 @@ final class MxGatewayClientSessionTests {
.setSessionId(request.getSessionId())
.setKind(request.getCommand().getKind())
.setProtocolStatus(ok());
// `category` is the authoritative success indicator, so the fake
// must set it a bare non-zero `success` is not a success.
var okStatus = mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy.newBuilder()
.setSuccess(1)
.setCategory(mxaccess_gateway.v1.MxaccessGateway.MxStatusCategory.MX_STATUS_CATEGORY_OK);
if (request.getCommand().getKind() == MxCommandKind.MX_COMMAND_KIND_SUSPEND) {
reply.setSuspend(mxaccess_gateway.v1.MxaccessGateway.SuspendReply.newBuilder()
.setStatus(mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy.newBuilder()
.setSuccess(1)));
.setStatus(okStatus));
} else if (request.getCommand().getKind() == MxCommandKind.MX_COMMAND_KIND_ACTIVATE) {
reply.setActivate(mxaccess_gateway.v1.MxaccessGateway.ActivateReply.newBuilder()
.setStatus(mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy.newBuilder()
.setSuccess(1)));
.setStatus(okStatus));
}
responseObserver.onNext(reply.build());
responseObserver.onCompleted();
@@ -0,0 +1,201 @@
package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.google.protobuf.util.JsonFormat;
import io.grpc.ManagedChannel;
import io.grpc.Server;
import io.grpc.inprocess.InProcessChannelBuilder;
import io.grpc.inprocess.InProcessServerBuilder;
import io.grpc.stub.StreamObserver;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.UUID;
import mxaccess_gateway.v1.MxAccessGatewayGrpc;
import mxaccess_gateway.v1.MxaccessGateway.MxCommandReply;
import mxaccess_gateway.v1.MxaccessGateway.MxCommandRequest;
import mxaccess_gateway.v1.MxaccessGateway.MxValue;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatus;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
import org.junit.jupiter.api.Test;
final class MxGatewayCredentialReplyTests {
private static final String CREDENTIAL = "sup3rSecretVerify9f3a2b";
@Test
void authenticateUserRedactsEchoedCredentialFromReplyDrivenError() throws Exception {
assertCredentialFullyRedacted(
"authenticate-user.echoed-credential.reply.json", "auth-echo-session");
}
@Test
void authenticateUserRedactsEchoedCredentialFromMxAccessFailureReply() throws Exception {
assertCredentialFullyRedacted(
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
"auth-echo-failure-session");
}
private static void assertCredentialFullyRedacted(String fixture, String sessionId) throws Exception {
MxCommandReply reply = loadReply(fixture);
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, sessionId);
MxAccessException error = assertThrows(
MxAccessException.class,
() -> session.authenticateUser(12, "operator", CREDENTIAL));
assertFalse(error.getMessage().contains(CREDENTIAL),
"credential echoed by the gateway must not survive in the surfaced message");
assertTrue(error.getMessage().contains("<redacted>"),
"the echoed credential must be replaced with the redaction marker");
// The rebuilt exception must not re-expose the credential through the
// structured reply/protocolStatus a caller can inspect and log.
MxCommandReply surfaced = error.reply();
assertNotNull(surfaced, "the redacted exception must preserve a reply for inspection");
assertFalse(surfaced.getProtocolStatus().getMessage().contains(CREDENTIAL),
"reply protocol status message must not leak the echoed credential");
assertFalse(surfaced.getDiagnosticMessage().contains(CREDENTIAL),
"reply diagnostic message must not leak the echoed credential");
for (int index = 0; index < surfaced.getStatusesCount(); index++) {
assertFalse(surfaced.getStatusesList().get(index).getDiagnosticText().contains(CREDENTIAL),
"reply status diagnostic text must not leak the echoed credential");
}
assertNotNull(error.protocolStatus(), "the redacted exception must preserve a protocol status");
assertFalse(error.protocolStatus().getMessage().contains(CREDENTIAL),
"exception protocol status must not leak the echoed credential");
}
}
@Test
void authenticateUserMissingPayloadThrowsMalformedReply() throws Exception {
MxCommandReply reply = loadReply("authenticate-user.missing-payload.reply.json");
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "auth-missing-session");
assertThrows(
MxGatewayMalformedReplyException.class,
() -> session.authenticateUser(3, "operator", "pw"));
}
}
@Test
void authenticateUserReturnValueOnlyReplyReturnsInt32Fallback() throws Exception {
MxCommandReply reply = loadReply("authenticate-user.return-value-only.reply.json");
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "auth-return-session");
assertEquals(7, session.authenticateUser(3, "operator", "pw"));
}
}
@Test
void addBufferedItemReturnValueOnlyReplyReturnsInt32Fallback() throws Exception {
MxCommandReply reply = MxCommandReply.newBuilder()
.setProtocolStatus(ok())
.setReturnValue(MxValue.newBuilder().setInt32Value(55))
.build();
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "buffered-return-session");
assertEquals(55, session.addBufferedItem(3, "Tank01.Level", "galaxy"));
}
}
@Test
void addBufferedItemMissingPayloadThrowsMalformedReply() throws Exception {
MxCommandReply reply = MxCommandReply.newBuilder().setProtocolStatus(ok()).build();
try (InProcessGateway gateway = InProcessGateway.startReturning(reply);
MxGatewayClient client = gateway.client()) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "buffered-malformed-session");
assertThrows(
MxGatewayMalformedReplyException.class,
() -> session.addBufferedItem(3, "Tank01.Level", "galaxy"));
}
}
private static ProtocolStatus ok() {
return ProtocolStatus.newBuilder()
.setCode(ProtocolStatusCode.PROTOCOL_STATUS_CODE_OK)
.build();
}
private static MxCommandReply loadReply(String fixture) throws Exception {
MxCommandReply.Builder builder = MxCommandReply.newBuilder();
JsonFormat.parser().merge(
Files.readString(fixtureRoot().resolve("command-replies/" + fixture)),
builder);
return builder.build();
}
private static Path fixtureRoot() {
Path current = Path.of(System.getProperty("user.dir")).toAbsolutePath();
for (Path path = current; path != null; path = path.getParent()) {
Path candidate = path.resolve("clients/proto/fixtures/behavior");
if (Files.exists(candidate)) {
return candidate;
}
candidate = path.resolve("../proto/fixtures/behavior").normalize();
if (Files.exists(candidate)) {
return candidate;
}
}
throw new IllegalStateException("could not locate behavior fixtures from " + current);
}
private record InProcessGateway(Server server, ManagedChannel channel) implements AutoCloseable {
static InProcessGateway startReturning(MxCommandReply reply) throws Exception {
String serverName = "mxgw-java-cred-" + UUID.randomUUID();
MxAccessGatewayGrpc.MxAccessGatewayImplBase service =
new MxAccessGatewayGrpc.MxAccessGatewayImplBase() {
@Override
public void invoke(
MxCommandRequest request, StreamObserver<MxCommandReply> responseObserver) {
responseObserver.onNext(reply);
responseObserver.onCompleted();
}
};
Server server = InProcessServerBuilder.forName(serverName)
.directExecutor()
.addService(service)
.build()
.start();
ManagedChannel channel = InProcessChannelBuilder.forName(serverName)
.directExecutor()
.build();
return new InProcessGateway(server, channel);
}
MxGatewayClient client() {
return new MxGatewayClient(
channel,
MxGatewayClientOptions.builder()
.endpoint("in-process")
.apiKey("")
.plaintext(true)
.callTimeout(Duration.ofSeconds(5))
.build());
}
@Override
public void close() {
channel.shutdownNow();
server.shutdownNow();
}
}
}
@@ -4,6 +4,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.google.gson.JsonArray;
@@ -20,6 +21,8 @@ import mxaccess_gateway.v1.MxaccessGateway.MxStatusProxy;
import mxaccess_gateway.v1.MxaccessGateway.MxValue;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
final class MxGatewayFixtureTests {
@Test
@@ -89,6 +92,50 @@ final class MxGatewayFixtureTests {
throw new AssertionError("expected MxAccessException");
}
@ParameterizedTest
@CsvSource({
"register.ok.reply.json,false",
"write.status-category-error-success-set.reply.json,true",
"write.status-category-ok-success-zero.reply.json,false",
"write.hresult-s-false.reply.json,false",
"write.hresult-e-fail.reply.json,true",
})
void replyValidationFixturesBranchOnCategoryAndNegativeHresult(String fixture, boolean expectFailure)
throws Exception {
MxCommandReply.Builder builder = MxCommandReply.newBuilder();
JsonFormat.parser().merge(
Files.readString(fixtureRoot().resolve("command-replies/" + fixture)),
builder);
MxCommandReply reply = builder.build();
if (expectFailure) {
assertThrows(MxAccessException.class, () -> MxGatewayErrors.ensureMxAccessSuccess("write", reply));
} else {
MxGatewayErrors.ensureMxAccessSuccess("write", reply);
}
}
@ParameterizedTest
@CsvSource({
"MX_STATUS_CATEGORY_OK,0,true",
"MX_STATUS_CATEGORY_OK,1,true",
"MX_STATUS_CATEGORY_COMMUNICATION_ERROR,1,false",
"MX_STATUS_CATEGORY_UNSPECIFIED,1,false",
})
void statusEntryVerdictIgnoresTheRawSuccessMember(String category, int success, boolean expectSucceeded) {
MxStatusProxy status = MxStatusProxy.newBuilder()
.setCategory(MxStatusCategory.valueOf(category))
.setSuccess(success)
.build();
assertEquals(expectSucceeded, MxStatuses.succeeded(status));
}
@Test
void absentStatusEntryIsSuccess() {
assertTrue(MxStatuses.succeeded(null));
}
@Test
void grpcAuthErrorsAreClassifiedAndRedacted() {
RuntimeException authError = MxGatewayErrors.fromGrpc(
@@ -0,0 +1,50 @@
package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import org.junit.jupiter.api.Test;
final class MxGatewaySecretsTests {
@Test
void redactExactReplacesEveryOccurrenceOfASecret() {
String message = "verify s3cr3t, retry s3cr3t, done s3cr3t";
String result = MxGatewaySecrets.redactExact(message, "s3cr3t");
assertFalse(result.contains("s3cr3t"), "no occurrence of the secret may survive");
assertEquals("verify <redacted>, retry <redacted>, done <redacted>", result);
}
@Test
void redactExactFullyRedactsOverlappingSecretsWhenOneIsASubstringOfTheOther() {
String message = "password=hunter2 token=hunter2extra";
String result = MxGatewaySecrets.redactExact(message, "hunter2extra", "hunter2");
assertFalse(result.contains("hunter2"), "both the secret and its superstring must be fully redacted");
assertEquals("password=<redacted> token=<redacted>", result);
}
@Test
void redactExactWithNoSecretsReturnsMessageUnchanged() {
String message = "nothing to scrub here";
assertEquals(message, MxGatewaySecrets.redactExact(message));
}
@Test
void redactExactToleratesNullMessage() {
assertNull(MxGatewaySecrets.redactExact(null, "secret"));
}
@Test
void redactExactIgnoresBlankSecretSoRealSpacesAreNotOverRedacted() {
String message = "keep these spaces intact";
String result = MxGatewaySecrets.redactExact(message, " ", "");
assertEquals(message, result);
}
}
@@ -0,0 +1,22 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-authenticate-echoed-mxaccess-failure",
"kind": "MX_COMMAND_KIND_AUTHENTICATE_USER",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_MXACCESS_FAILURE",
"message": "MXAccess AuthenticateUser rejected credential 'sup3rSecretVerify9f3a2b'."
},
"hresult": -2147024891,
"statuses": [
{
"success": 0,
"category": "MX_STATUS_CATEGORY_SECURITY_ERROR",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_NMX",
"detail": 5,
"rawCategory": 8,
"rawDetectedBy": 5,
"diagnosticText": "Authentication failed for password 'sup3rSecretVerify9f3a2b'."
}
],
"diagnosticMessage": "MXAccess echoed the credential 'sup3rSecretVerify9f3a2b' back in its failure diagnostic."
}
@@ -0,0 +1,22 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-authenticate-echoed",
"kind": "MX_COMMAND_KIND_AUTHENTICATE_USER",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "MXAccess AuthenticateUser rejected credential 'sup3rSecretVerify9f3a2b'."
},
"hresult": -2147024891,
"statuses": [
{
"success": 0,
"category": "MX_STATUS_CATEGORY_SECURITY_ERROR",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_NMX",
"detail": 5,
"rawCategory": 8,
"rawDetectedBy": 5,
"diagnosticText": "Authentication failed for password 'sup3rSecretVerify9f3a2b'."
}
],
"diagnosticMessage": "MXAccess echoed the credential 'sup3rSecretVerify9f3a2b' back in its failure diagnostic."
}
@@ -0,0 +1,10 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-authenticate-missing-payload",
"kind": "MX_COMMAND_KIND_AUTHENTICATE_USER",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "AuthenticateUser reached MXAccess."
},
"diagnosticMessage": "Malformed: the OK reply carried neither an AuthenticateUser payload nor a return_value."
}
@@ -0,0 +1,15 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-authenticate-return-value-only",
"kind": "MX_COMMAND_KIND_AUTHENTICATE_USER",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "AuthenticateUser reached MXAccess."
},
"returnValue": {
"dataType": "MX_DATA_TYPE_INTEGER",
"variantType": "VT_I4",
"int32Value": 7
},
"diagnosticMessage": "Legacy worker populated only return_value; the typed AuthenticateUser payload is absent."
}
@@ -0,0 +1,29 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-write-e-fail",
"kind": "MX_COMMAND_KIND_WRITE",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "Write reached MXAccess."
},
"hresult": -2147467259,
"returnValue": {
"dataType": "MX_DATA_TYPE_NO_DATA",
"variantType": "VT_EMPTY",
"isNull": true,
"rawDiagnostic": "MXAccess returned no value for the failed write.",
"rawDataType": 2
},
"statuses": [
{
"success": 1,
"category": "MX_STATUS_CATEGORY_OK",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_LMX",
"detail": 0,
"rawCategory": 0,
"rawDetectedBy": 3,
"diagnosticText": "OK"
}
],
"diagnosticMessage": "COM semantics: a negative HRESULT (E_FAIL, 0x80004005) is a failure even when every status entry is OK."
}
@@ -0,0 +1,29 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-write-s-false",
"kind": "MX_COMMAND_KIND_WRITE",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "Write completed with S_FALSE."
},
"hresult": 1,
"returnValue": {
"dataType": "MX_DATA_TYPE_NO_DATA",
"variantType": "VT_EMPTY",
"isNull": true,
"rawDiagnostic": "MXAccess returned no value for the write.",
"rawDataType": 2
},
"statuses": [
{
"success": 1,
"category": "MX_STATUS_CATEGORY_OK",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_LMX",
"detail": 0,
"rawCategory": 0,
"rawDetectedBy": 3,
"diagnosticText": "OK"
}
],
"diagnosticMessage": "COM semantics: a positive HRESULT such as S_FALSE (1) is a success code, not a failure."
}
@@ -0,0 +1,29 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-write-category-error",
"kind": "MX_COMMAND_KIND_WRITE",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "Write reached MXAccess."
},
"hresult": 0,
"returnValue": {
"dataType": "MX_DATA_TYPE_NO_DATA",
"variantType": "VT_EMPTY",
"isNull": true,
"rawDiagnostic": "MXAccess returned no value for the write.",
"rawDataType": 2
},
"statuses": [
{
"success": 1,
"category": "MX_STATUS_CATEGORY_COMMUNICATION_ERROR",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_LMX",
"detail": 77,
"rawCategory": 5,
"rawDetectedBy": 3,
"diagnosticText": "Responding LMX lost communication mid-write."
}
],
"diagnosticMessage": "Category is authoritative: a non-OK category is a failure even when the raw success member is non-zero."
}
@@ -0,0 +1,29 @@
{
"sessionId": "session-fixture",
"correlationId": "gateway-correlation-write-category-ok",
"kind": "MX_COMMAND_KIND_WRITE",
"protocolStatus": {
"code": "PROTOCOL_STATUS_CODE_OK",
"message": "Write completed."
},
"hresult": 0,
"returnValue": {
"dataType": "MX_DATA_TYPE_NO_DATA",
"variantType": "VT_EMPTY",
"isNull": true,
"rawDiagnostic": "MXAccess returned no value for the write.",
"rawDataType": 2
},
"statuses": [
{
"success": 0,
"category": "MX_STATUS_CATEGORY_OK",
"detectedBy": "MX_STATUS_SOURCE_RESPONDING_LMX",
"detail": 0,
"rawCategory": 0,
"rawDetectedBy": 3,
"diagnosticText": "OK, reported with a zero raw success member."
}
],
"diagnosticMessage": "Category is authoritative: MX_STATUS_CATEGORY_OK is success even when the raw success member is zero."
}
@@ -20,6 +20,62 @@
"path": "command-replies/write.mxaccess-failure.reply.json",
"expectation": "MXAccess failures are data-bearing replies with HRESULT and status details, not transport failures."
},
{
"id": "command-reply.write.status-category-error-success-set",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/write.status-category-error-success-set.reply.json",
"expectation": "A status entry fails when its category is not MX_STATUS_CATEGORY_OK, even though the raw success member is non-zero."
},
{
"id": "command-reply.write.status-category-ok-success-zero",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/write.status-category-ok-success-zero.reply.json",
"expectation": "A status entry succeeds when its category is MX_STATUS_CATEGORY_OK, even though the raw success member is zero."
},
{
"id": "command-reply.write.hresult-s-false",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/write.hresult-s-false.reply.json",
"expectation": "A positive HRESULT such as S_FALSE (1) is a COM success code and does not fail the reply."
},
{
"id": "command-reply.write.hresult-e-fail",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/write.hresult-e-fail.reply.json",
"expectation": "A negative HRESULT fails the reply even when every status entry reports MX_STATUS_CATEGORY_OK."
},
{
"id": "command-reply.authenticate-user.echoed-credential",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.echoed-credential.reply.json",
"expectation": "When a gateway/MXAccess diagnostic echoes the caller's credential back (OK envelope, negative HRESULT), the surfaced error redacts the exact secret from both the rendered message and the structured reply accessors."
},
{
"id": "command-reply.authenticate-user.echoed-credential-mxaccess-failure",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.echoed-credential-mxaccess-failure.reply.json",
"expectation": "The same echoed-credential redaction holds when the reply is coded PROTOCOL_STATUS_CODE_MXACCESS_FAILURE, which every client routes to its MXAccess error type."
},
{
"id": "command-reply.authenticate-user.missing-payload",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.missing-payload.reply.json",
"expectation": "An OK reply with neither the typed AuthenticateUser payload nor a return_value raises a typed malformed-reply error, never a proto3 default 0 and never an NRE."
},
{
"id": "command-reply.authenticate-user.return-value-only",
"category": "command_replies",
"messageType": "mxaccess_gateway.v1.MxCommandReply",
"path": "command-replies/authenticate-user.return-value-only.reply.json",
"expectation": "An OK reply missing the typed AuthenticateUser payload but carrying an int32 return_value falls back to the return_value (legacy-worker compatibility)."
},
{
"id": "event-stream.session-ordered",
"category": "event_streams",
@@ -3,6 +3,7 @@
"cases": [
{
"id": "ok.responding-lmx",
"wantSuccess": true,
"status": {
"success": 1,
"category": "MX_STATUS_CATEGORY_OK",
@@ -15,6 +16,7 @@
},
{
"id": "security-error.requesting-lmx",
"wantSuccess": false,
"status": {
"success": 0,
"category": "MX_STATUS_CATEGORY_SECURITY_ERROR",
@@ -27,6 +29,7 @@
},
{
"id": "raw-unknown-category",
"wantSuccess": false,
"status": {
"success": 0,
"category": "MX_STATUS_CATEGORY_UNKNOWN",
+7 -1
View File
@@ -187,7 +187,13 @@ await session.write_secured(
```
The CLI mirrors these as `authenticate-user` (credential via `--password` or,
preferably, `--password-env`) and `write-secured`.
preferably, the variable named by `--password-env`, default
`MXGATEWAY_VERIFY_PASSWORD`) and `write-secured`. The credential is required: a
missing or empty resolved value raises a `UsageError` naming the option and the
variable, so the CLI fails before connecting instead of authenticating with an
empty password. `MXGATEWAY_VERIFY_PASSWORD` is the canonical variable across all
five client CLIs — see
[Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
### Array writes replace the whole array
@@ -11,6 +11,7 @@ from .generated.galaxy_repository_pb2 import (
)
from .events import ReplayGap
from .errors import (
MalformedReplyError,
MxAccessError,
MxGatewayAuthenticationError,
MxGatewayAuthorizationError,
@@ -35,6 +36,7 @@ __all__ = [
"GalaxyRepositoryClient",
"GatewayClient",
"LazyBrowseNode",
"MalformedReplyError",
"MxAccessError",
"MxGatewayAuthenticationError",
"MxGatewayAuthorizationError",
@@ -53,6 +53,10 @@ class MxAccessError(MxGatewayCommandError):
"""MXAccess HRESULT or status failure."""
class MalformedReplyError(MxGatewayError):
"""Raised when an OK reply lacks the expected typed payload and any usable return_value fallback."""
def map_rpc_error(operation: str, error: grpc.RpcError) -> MxGatewayTransportError:
"""Map a generated gRPC exception to the client exception hierarchy."""
@@ -137,8 +141,10 @@ def ensure_mxaccess_success(operation: str, reply: pb.MxCommandReply) -> pb.MxCo
raw_reply=reply,
)
# `category` is the authoritative verdict per the wire contract; `success`
# is the raw COM member carried verbatim for diagnostics only.
for mx_status in reply.statuses:
if mx_status.success == 0:
if mx_status.category != pb.MX_STATUS_CATEGORY_OK:
raise MxAccessError(
_mxaccess_message(operation, reply),
protocol_status=status,
@@ -151,8 +157,18 @@ def ensure_mxaccess_success(operation: str, reply: pb.MxCommandReply) -> pb.MxCo
def _mxaccess_message(operation: str, reply: pb.MxCommandReply) -> str:
status_text = reply.protocol_status.message or "MXAccess command failed"
hresult = reply.hresult if reply.HasField("hresult") else None
return (
message = (
f"{operation} failed: {status_text}; "
f"session={reply.session_id}; correlation={reply.correlation_id}; "
f"hresult={hresult}; statuses={len(reply.statuses)}"
)
# Append a per-status breakdown that carries the raw `success` COM member
# verbatim for diagnostic parity with the other clients. `category` remains
# the authoritative verdict; `success` is diagnostics only.
for status in reply.statuses:
category = pb.MxStatusCategory.Name(status.category)
message += (
f" [success={status.success}, category={category}, "
f"detail={status.detail}, {status.diagnostic_text}]"
)
return message
@@ -5,7 +5,7 @@ from __future__ import annotations
from collections.abc import AsyncIterator, Sequence
from .auth import redact_secret
from .errors import MxGatewayError, ensure_mxaccess_success
from .errors import MalformedReplyError, MxGatewayError, ensure_mxaccess_success
from .events import ReplayGap
from .generated import mxaccess_gateway_pb2 as pb
from .values import MxValueInput, to_mx_value
@@ -710,7 +710,15 @@ class Session:
correlation_id=correlation_id,
secrets=[verify_user_password],
)
return reply.authenticate_user.user_id
if reply.HasField("authenticate_user"):
return reply.authenticate_user.user_id
if reply.HasField("return_value") and reply.return_value.WhichOneof("kind") == "int32_value":
return reply.return_value.int32_value
raise MalformedReplyError(
"authenticate_user returned a malformed reply: OK reply carried "
"neither the typed payload nor an int32 return_value",
raw_reply=reply,
)
async def archestra_user_to_id(
self,
@@ -730,7 +738,15 @@ class Session:
),
correlation_id=correlation_id,
)
return reply.archestra_user_to_id.user_id
if reply.HasField("archestra_user_to_id"):
return reply.archestra_user_to_id.user_id
if reply.HasField("return_value") and reply.return_value.WhichOneof("kind") == "int32_value":
return reply.return_value.int32_value
raise MalformedReplyError(
"archestra_user_to_id returned a malformed reply: OK reply carried "
"neither the typed payload nor an int32 return_value",
raw_reply=reply,
)
async def add_buffered_item(
self,
@@ -752,7 +768,15 @@ class Session:
),
correlation_id=correlation_id,
)
return reply.add_buffered_item.item_handle
if reply.HasField("add_buffered_item"):
return reply.add_buffered_item.item_handle
if reply.HasField("return_value") and reply.return_value.WhichOneof("kind") == "int32_value":
return reply.return_value.int32_value
raise MalformedReplyError(
"add_buffered_item returned a malformed reply: OK reply carried "
"neither the typed payload nor an int32 return_value",
raw_reply=reply,
)
async def set_buffered_update_interval(
self,
@@ -895,19 +919,47 @@ def _value_secrets(value: MxValueInput) -> list[str]:
def _redact_error(error: MxGatewayError, secrets: Sequence[str | None]) -> None:
"""Scrub secret substrings from a raised error's message in place.
"""Scrub secret substrings from a raised error's message and reply in place.
Rewrites ``error.args[0]`` (the message returned by ``str(error)``) through
the shared :func:`~zb_mom_ww_mxgateway.auth.redact_secret` seam so credential
text can never reach logs or be re-raised to a caller. The
``protocol_status`` / ``raw_reply`` context is left untouched those hold the
gateway's own fields, which never echo the client-supplied secret.
text can never reach logs or be re-raised to a caller.
A misbehaving MXAccess provider can echo the client-supplied credential back
verbatim in its failure diagnostics, so ``error.raw_reply`` (the protobuf
reply) can carry the secret in ``protocol_status.message``,
``diagnostic_message``, and each ``statuses[].diagnostic_text``. A logger
dumping those structured fields would reintroduce the leak the message scrub
closes. When there is a secret to scrub and a reply is attached, this rebinds
``error.raw_reply`` to a scrubbed deep copy so the raised exception carries no
credential text on any surface. The clone leaves the original reply untouched.
"""
scrubbed = [secret for secret in secrets if secret]
if not scrubbed:
return
if error.args and isinstance(error.args[0], str):
error.args = (redact_secret(error.args[0], scrubbed), *error.args[1:])
if error.raw_reply is not None:
error.raw_reply = _redact_reply(error.raw_reply, scrubbed)
def _redact_reply(reply: pb.MxCommandReply, secrets: Sequence[str]) -> pb.MxCommandReply:
"""Return a deep copy of *reply* with credential text scrubbed from diagnostics.
Operates on a clone so the caller's original reply object is never mutated.
Only the free-text diagnostic fields that can echo a client-supplied secret
are scrubbed; the structured/enum fields the gateway itself sets are left as-is.
"""
clone = type(reply)()
clone.CopyFrom(reply)
if clone.protocol_status.message:
clone.protocol_status.message = redact_secret(clone.protocol_status.message, secrets)
if clone.diagnostic_message:
clone.diagnostic_message = redact_secret(clone.diagnostic_message, secrets)
for status in clone.statuses:
if status.diagnostic_text:
status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
return clone
from .client import GatewayClient # noqa: E402
@@ -21,6 +21,7 @@ from zb_mom_ww_mxgateway import __version__
from zb_mom_ww_mxgateway.auth import redact_secret
from zb_mom_ww_mxgateway.client import GatewayClient
from zb_mom_ww_mxgateway.errors import MxGatewayError
from zb_mom_ww_mxgateway.events import ReplayGap
from zb_mom_ww_mxgateway.galaxy import GalaxyRepositoryClient
from zb_mom_ww_mxgateway.generated import galaxy_repository_pb2 as galaxy_pb
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
@@ -31,6 +32,11 @@ logger = logging.getLogger(__name__)
MAX_AGGREGATE_EVENTS = 10_000
#: Canonical CLI credential environment variable, shared by every official client
#: CLI (CLI-45) so one exported variable drives the same operator workflow in all
#: five languages.
DEFAULT_VERIFY_PASSWORD_ENV = "MXGATEWAY_VERIFY_PASSWORD"
_BATCH_EOR = "__MXGW_BATCH_EOR__"
@@ -327,7 +333,8 @@ def write_secured(**kwargs: Any) -> None:
)
@click.option(
"--password-env",
default=None,
default=DEFAULT_VERIFY_PASSWORD_ENV,
show_default=True,
help="Environment variable holding the user password.",
)
@click.option("--correlation-id", default="", help="Client correlation id.")
@@ -833,17 +840,23 @@ async def _authenticate_user(**kwargs: Any) -> dict[str, Any]:
def _resolve_password(kwargs: dict[str, Any]) -> str:
"""Resolve the authenticate-user password from --password or --password-env.
Prefers the explicit flag, then falls back to the named environment
variable. The resolved secret is never echoed; callers pass it into the
``secrets`` redaction list so it cannot leak through a surfaced error.
Prefers the explicit flag, then falls back to the environment variable named
by ``--password-env`` (default :data:`DEFAULT_VERIFY_PASSWORD_ENV`). A missing
*or empty* value from either source is a usage error (CLI-45): the CLI never
sends a fabricated empty credential to the wire. The error names the option
and the variable only the resolved secret is never echoed, and callers pass
it into the ``secrets`` redaction list so it cannot leak through a surfaced
error either.
"""
env_name = kwargs.get("password_env") or DEFAULT_VERIFY_PASSWORD_ENV
password = kwargs.get("password")
if not password:
env_name = kwargs.get("password_env")
password = os.environ.get(env_name) if env_name else None
password = os.environ.get(env_name)
if not password:
raise click.UsageError("a password is required via --password or --password-env")
raise click.UsageError(
f"a password is required via --password or the {env_name} environment variable"
)
return password
@@ -1103,7 +1116,7 @@ async def _stream_events(**kwargs: Any) -> dict[str, Any]:
max_events=kwargs["max_events"],
timeout=kwargs["timeout"],
)
return {"events": [_message_dict(event) for event in events]}
return {"events": [_event_row(event) for event in events]}
async def _stream_alarms(**kwargs: Any) -> dict[str, Any]:
@@ -1500,14 +1513,14 @@ async def _collect_events(
*,
max_events: int,
timeout: float,
) -> list[pb.MxEvent]:
) -> list[pb.MxEvent | ReplayGap]:
if max_events > MAX_AGGREGATE_EVENTS:
raise click.BadParameter(
f"must be less than or equal to {MAX_AGGREGATE_EVENTS}",
param_hint="--max-events",
)
collected: list[pb.MxEvent] = []
collected: list[pb.MxEvent | ReplayGap] = []
iterator = events.__aiter__()
try:
while len(collected) < max_events:
@@ -1630,3 +1643,26 @@ def _message_dict(message: Any) -> dict[str, Any]:
preserving_proto_field_name=False,
use_integers_for_enums=False,
)
def _event_row(item: Any) -> dict[str, Any]:
"""Render one item of an event stream as a JSON row.
``Session.stream_events`` yields ``MxEvent | ReplayGap``. ``ReplayGap`` is a
plain dataclass, so it has no protobuf descriptor and cannot go through
``MessageToDict`` it gets its own distinct row instead, matching the shape
the Rust and Go CLIs emit so the cross-language matrix can compare rows.
Keys are camelCase for the same reason ``_message_dict`` uses
``preserving_proto_field_name=False``. The gap is always rendered: never
dropped, and never re-synthesized into an event.
"""
if isinstance(item, ReplayGap):
return {
"replayGap": {
"requestedAfterSequence": item.requested_after_sequence,
"oldestAvailableSequence": item.oldest_available_sequence,
},
}
return _message_dict(item)
+140 -1
View File
@@ -752,7 +752,9 @@ def test_authenticate_user_reads_password_from_env(monkeypatch: pytest.MonkeyPat
assert fake.last_request.command.authenticate_user.verify_user_password == "env-secret-pw"
def test_authenticate_user_requires_a_password() -> None:
def test_authenticate_user_requires_a_password(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("MXGATEWAY_VERIFY_PASSWORD", raising=False)
result = CliRunner().invoke(
main,
[
@@ -770,6 +772,81 @@ def test_authenticate_user_requires_a_password() -> None:
assert result.exit_code != 0
assert "password is required" in result.output
# CLI-45: the usage error names the option and the canonical env var.
assert "--password" in result.output
assert "MXGATEWAY_VERIFY_PASSWORD" in result.output
def test_authenticate_user_reads_password_from_canonical_default_env(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""CLI-45: --password-env defaults to MXGATEWAY_VERIFY_PASSWORD.
Exporting the canonical variable alone must satisfy the credential, with no
explicit --password-env flag the same operator workflow as the other CLIs.
"""
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
reply = pb.MxCommandReply(
session_id="s1",
kind=pb.MX_COMMAND_KIND_AUTHENTICATE_USER,
protocol_status=pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK),
authenticate_user=pb.AuthenticateUserReply(user_id=11),
)
fake = _FakeInvokeClient(reply)
async def fake_connect(options, **_kwargs):
return fake
monkeypatch.setattr(commands_module.GatewayClient, "connect", fake_connect)
monkeypatch.setenv("MXGATEWAY_VERIFY_PASSWORD", "canonical-env-pw")
result = CliRunner().invoke(
main,
[
"authenticate-user",
"--plaintext",
"--session-id",
"s1",
"--server-handle",
"3",
"--verify-user",
"operator",
"--json",
],
)
assert result.exit_code == 0, result.output
assert json.loads(result.output)["userId"] == 11
assert "canonical-env-pw" not in result.output
assert fake.last_request.command.authenticate_user.verify_user_password == "canonical-env-pw"
def test_authenticate_user_rejects_empty_password_value(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""CLI-45: an empty resolved credential fails fast, never reaching the wire."""
monkeypatch.setenv("MXGATEWAY_VERIFY_PASSWORD", "")
result = CliRunner().invoke(
main,
[
"authenticate-user",
"--plaintext",
"--session-id",
"s1",
"--server-handle",
"3",
"--verify-user",
"operator",
"--password",
"",
"--json",
],
)
assert result.exit_code != 0
assert "password is required" in result.output
def test_write_secured_command_does_not_echo_value_on_failure(
@@ -817,3 +894,65 @@ def test_write_secured_command_does_not_echo_value_on_failure(
def test_write_secured_and_authenticate_user_commands_are_registered() -> None:
names = set(main.commands)
assert {"write-secured", "authenticate-user"} <= names
class _FakeReplayGapSession:
"""Session stand-in whose event stream starts with a ReplayGap sentinel.
Mirrors what ``Session.stream_events`` yields on a resume that predates the
gateway's retained replay ring: the typed gap first, then normal events.
"""
def __init__(self, gap, event) -> None:
self._gap = gap
self._event = event
def stream_events(self, **_kwargs):
async def _iterate():
yield self._gap
yield self._event
return _iterate()
def test_stream_events_renders_replay_gap(monkeypatch: pytest.MonkeyPatch) -> None:
"""CLI-35: a ReplayGap renders as its own JSON row instead of crashing the command."""
from zb_mom_ww_mxgateway.events import ReplayGap
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
gap = ReplayGap(requested_after_sequence=7, oldest_available_sequence=42)
event = pb.MxEvent(session_id="cli-test-session", worker_sequence=43)
async def fake_connect(options, **_kwargs):
return _FakeAsyncClient()
monkeypatch.setattr(commands_module.GatewayClient, "connect", fake_connect)
monkeypatch.setattr(
commands_module,
"_session",
lambda _client, _session_id: _FakeReplayGapSession(gap, event),
)
result = CliRunner().invoke(
main,
[
"stream-events",
"--plaintext",
"--session-id",
"cli-test-session",
"--after-worker-sequence",
"7",
"--max-events",
"2",
"--json",
],
)
assert result.exit_code == 0, result.output
rows = json.loads(result.output)["events"]
assert rows[0] == {
"replayGap": {"requestedAfterSequence": 7, "oldestAvailableSequence": 42},
}
# The gap is rendered, never swallowed, and the normal event still follows it.
assert "replayGap" not in rows[1]
assert rows[1]["workerSequence"] == "43"
+49
View File
@@ -32,6 +32,55 @@ def test_write_failure_fixture_preserves_raw_reply() -> None:
assert len(captured.value.raw_reply.statuses) == 2
@pytest.mark.parametrize(
("fixture", "expect_failure"),
[
("command-replies/register.ok.reply.json", False),
("command-replies/write.status-category-error-success-set.reply.json", True),
("command-replies/write.status-category-ok-success-zero.reply.json", False),
("command-replies/write.hresult-s-false.reply.json", False),
("command-replies/write.hresult-e-fail.reply.json", True),
],
)
def test_reply_validation_fixtures_branch_on_category_and_negative_hresult(
fixture: str,
expect_failure: bool,
) -> None:
reply = _load_reply(fixture)
if expect_failure:
with pytest.raises(MxAccessError):
ensure_mxaccess_success("write", reply)
else:
assert ensure_mxaccess_success("write", reply) is reply
@pytest.mark.parametrize(
("category", "success", "expect_failure"),
[
(pb.MX_STATUS_CATEGORY_OK, 0, False),
(pb.MX_STATUS_CATEGORY_OK, 1, False),
(pb.MX_STATUS_CATEGORY_COMMUNICATION_ERROR, 1, True),
(pb.MX_STATUS_CATEGORY_UNSPECIFIED, 1, True),
],
)
def test_status_entry_verdict_ignores_the_raw_success_member(
category: int,
success: int,
expect_failure: bool,
) -> None:
reply = pb.MxCommandReply(
protocol_status=pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK),
statuses=[pb.MxStatusProxy(success=success, category=category)],
)
if expect_failure:
with pytest.raises(MxAccessError):
ensure_mxaccess_success("write", reply)
else:
assert ensure_mxaccess_success("write", reply) is reply
def test_session_status_maps_to_session_error() -> None:
status = pb.ProtocolStatus(
code=pb.PROTOCOL_STATUS_CODE_SESSION_NOT_FOUND,
@@ -0,0 +1,112 @@
"""Tests for the uniform malformed-reply contract (CLI-41) and the CLI-40
credential-redaction regression, driven through the shared fixtures.
CLI-41: an OK reply that carries neither the expected typed payload nor a usable
``return_value`` int32 fallback raises :class:`MalformedReplyError`; a legacy
reply that populates only ``return_value`` falls back to that int32.
CLI-40: an OK reply whose diagnostics echo the caller's credential must never
surface that credential in the raised error message.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from google.protobuf.json_format import ParseDict
from zb_mom_ww_mxgateway import MalformedReplyError, MxAccessError
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from test_typed_command_helpers import _session_with
FIXTURE_ROOT = Path(__file__).resolve().parents[2] / "proto" / "fixtures" / "behavior"
def _load_reply(relative: str) -> pb.MxCommandReply:
path = FIXTURE_ROOT / relative
return ParseDict(json.loads(path.read_text()), pb.MxCommandReply())
@pytest.mark.asyncio
async def test_authenticate_user_missing_payload_raises_malformed_reply() -> None:
reply = _load_reply("command-replies/authenticate-user.missing-payload.reply.json")
session, _ = await _session_with([reply])
with pytest.raises(MalformedReplyError) as captured:
await session.authenticate_user(12, "operator", "any-password")
assert captured.value.raw_reply is reply
assert "malformed reply" in str(captured.value)
@pytest.mark.asyncio
async def test_authenticate_user_return_value_only_falls_back_to_int32() -> None:
reply = _load_reply("command-replies/authenticate-user.return-value-only.reply.json")
session, _ = await _session_with([reply])
user_id = await session.authenticate_user(12, "operator", "any-password")
assert user_id == 7
@pytest.mark.asyncio
async def test_add_buffered_item_falls_back_to_return_value_int32() -> None:
reply = pb.MxCommandReply(
session_id="session-1",
kind=pb.MX_COMMAND_KIND_ADD_BUFFERED_ITEM,
protocol_status=pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK),
return_value=pb.MxValue(int32_value=99),
)
session, _ = await _session_with([reply])
item_handle = await session.add_buffered_item(12, "Object.Attribute", "ctx")
assert item_handle == 99
@pytest.mark.asyncio
async def test_add_buffered_item_missing_payload_raises_malformed_reply() -> None:
reply = pb.MxCommandReply(
session_id="session-1",
kind=pb.MX_COMMAND_KIND_ADD_BUFFERED_ITEM,
protocol_status=pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK),
)
session, _ = await _session_with([reply])
with pytest.raises(MalformedReplyError) as captured:
await session.add_buffered_item(12, "Object.Attribute", "ctx")
assert captured.value.raw_reply is reply
@pytest.mark.parametrize(
"fixture",
[
"command-replies/authenticate-user.echoed-credential.reply.json",
"command-replies/authenticate-user.echoed-credential-mxaccess-failure.reply.json",
],
)
@pytest.mark.asyncio
async def test_authenticate_user_echoed_credential_is_scrubbed(fixture: str) -> None:
credential = "sup3rSecretVerify9f3a2b"
reply = _load_reply(fixture)
session, _ = await _session_with([reply])
with pytest.raises(MxAccessError) as captured:
await session.authenticate_user(12, "operator", credential)
exc = captured.value
message = str(exc)
assert credential not in message
assert "[redacted]" in message
# The credential must not survive in the structured protobuf context either:
# a logger dumping raw_reply's fields would otherwise reintroduce the leak.
assert exc.raw_reply is not None
assert credential not in exc.raw_reply.protocol_status.message
assert credential not in exc.raw_reply.diagnostic_message
for status in exc.raw_reply.statuses:
assert credential not in status.diagnostic_text
@@ -140,10 +140,19 @@ async def test_write_secured_surfaces_native_failure_without_prior_authenticate(
with pytest.raises(MxAccessError) as captured:
await session.write_secured(12, 34, secret_value, current_user_id=5, verifier_user_id=6)
# Native failure is surfaced (not "fixed") and the raw reply is preserved...
assert captured.value.raw_reply is failure
# ...but the credential-sensitive value is scrubbed from the surfaced message.
# Native failure is surfaced (not "fixed"): the raw reply's structure is
# preserved so callers still see the native verdict...
raw = captured.value.raw_reply
assert raw is not None
assert raw.kind == pb.MX_COMMAND_KIND_WRITE_SECURED
assert raw.hresult == -2147217407
assert raw.protocol_status.code == pb.PROTOCOL_STATUS_CODE_MXACCESS_FAILURE
# ...but the credential-sensitive value is scrubbed from the surfaced message
# AND from the reply's echoed diagnostics, so a logger dumping raw_reply's
# structured fields cannot reintroduce the leak.
assert secret_value not in str(captured.value)
assert secret_value not in raw.protocol_status.message
assert "[redacted]" in raw.protocol_status.message
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_WRITE_SECURED
assert command.write_secured.current_user_id == 5
+22 -4
View File
@@ -18,9 +18,22 @@ clients/rust/
crates/mxgw-cli/
```
`build.rs` reads the `.proto` files from
`../../src/ZB.MOM.WW.MxGateway.Contracts/Protos` and generates `tonic`/`prost` bindings
into Cargo build output. `src/generated.rs` declares the Rust modules that
`build.rs` resolves the `.proto` inputs repo-path-first: it prefers the
canonical protos at `../../src/ZB.MOM.WW.MxGateway.Contracts/Protos` (two
levels above `clients/rust`) so a local in-repo `.proto` edit is picked up
live without any extra step, and falls back to the vendored copies checked
into `clients/rust/protos/` only when that canonical directory is absent —
the case for a consumer building the crate unpacked from a published
tarball, where the rest of the mxaccessgw repo does not exist. The vendored
copies are shipped in the published `.crate` via `Cargo.toml`'s `include`
list, which is what makes the crate buildable standalone; they are build
inputs only, never a second source of truth. **Refresh rule:** any commit
that edits a Contracts proto (`mxaccess_gateway.proto`, `mxaccess_worker.proto`,
`galaxy_repository.proto`) must copy the changed file(s) into
`clients/rust/protos/` in that same commit — `scripts/check-codegen.ps1`
Check 3 fails the build on byte drift between the vendored copies and the
canonical Contracts protos. `tonic`/`prost` bindings are generated into
Cargo build output. `src/generated.rs` declares the Rust modules that
include those generated files. `src/generated` remains reserved for checked-in
generator output if the crate later changes to source-tree generation.
@@ -216,7 +229,12 @@ the wire — the client never logs them and never embeds them in an `Error`'s
`Display`/`Debug`; the only error text that can surface (from `tonic::Status`
messages and reply diagnostics) is scrubbed by the credential-redaction seam.
The CLI mirrors these as `authenticate-user` (password via `--password` or the
`--password-env` env var, never echoed) and `write-secured`.
variable named by `--password-env`, default `MXGATEWAY_VERIFY_PASSWORD`, never
echoed) and `write-secured`. The credential is required: a missing or empty
resolved value is a usage error naming the flag and the variable, so the CLI
fails before dialing instead of authenticating with an empty password.
`MXGATEWAY_VERIFY_PASSWORD` is the canonical variable across all five client
CLIs — see [Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
The remaining single-item command helpers round out MXAccess parity:
`unregister`, `suspend` / `activate` (each returns the operation's
+86 -11
View File
@@ -752,17 +752,7 @@ async fn dispatch(command: Command) -> Result<(), Error> {
password_env,
json,
} => {
// Resolve the credential from --password or the named env var.
// The password is passed straight to the typed helper and is never
// echoed to stdout/stderr or embedded in an error message.
let verify_user_password = password
.or_else(|| env::var(&password_env).ok())
.ok_or_else(|| Error::InvalidArgument {
name: "password".to_owned(),
detail: format!(
"supply --password or set the environment variable `{password_env}`"
),
})?;
let verify_user_password = resolve_verify_user_password(password, &password_env)?;
let session = session_for(connection, session_id).await?;
let user_id = session
.authenticate_user(server_handle, &verify_user, &verify_user_password)
@@ -1736,6 +1726,31 @@ fn print_ok(operation: &str, use_json: bool) {
}
}
/// Resolves the `authenticate-user` credential from `--password`, falling back to
/// the environment variable named by `--password-env` (default
/// `MXGATEWAY_VERIFY_PASSWORD`).
///
/// An empty value from either source counts as missing (CLI-45): the CLI fails
/// fast with a usage error rather than sending a fabricated empty credential to
/// the wire. The error names the flag and the variable only — never the value,
/// which is never echoed to stdout/stderr or embedded in an error message.
fn resolve_verify_user_password(
password: Option<String>,
password_env: &str,
) -> Result<String, Error> {
password
.filter(|value| !value.is_empty())
.or_else(|| {
env::var(password_env)
.ok()
.filter(|value| !value.is_empty())
})
.ok_or_else(|| Error::InvalidArgument {
name: "password".to_owned(),
detail: format!("supply --password or set the environment variable `{password_env}`"),
})
}
fn print_bulk_results(
operation: &str,
results: &[zb_mom_ww_mxgateway_client::generated::mxaccess_gateway::v1::SubscribeResult],
@@ -2617,6 +2632,66 @@ mod tests {
assert!(parsed.is_ok(), "parse failed: {parsed:?}");
}
/// CLI-45: `--password-env` must default to the canonical
/// `MXGATEWAY_VERIFY_PASSWORD` shared by every official client CLI.
#[test]
fn authenticate_user_password_env_defaults_to_canonical_name() {
let parsed = Cli::try_parse_from([
"mxgw",
"authenticate-user",
"--session-id",
"session-1",
"--server-handle",
"7",
"--verify-user",
"verifier",
])
.expect("parse");
match parsed.command {
Command::AuthenticateUser { password_env, .. } => {
assert_eq!(password_env, "MXGATEWAY_VERIFY_PASSWORD");
}
other => panic!("expected authenticate-user, got {other:?}"),
}
}
/// CLI-45: a credential that resolves to an empty string — whether from the
/// flag or from the named environment variable — is treated as missing, so
/// the CLI never sends a fabricated empty password to the wire. The usage
/// error names the flag and the variable, never a value.
#[test]
fn resolve_verify_user_password_rejects_missing_and_empty_values() {
const ABSENT: &str = "MXGW_CLI45_ABSENT_PASSWORD_VAR";
const EMPTY: &str = "MXGW_CLI45_EMPTY_PASSWORD_VAR";
const PRESENT: &str = "MXGW_CLI45_PRESENT_PASSWORD_VAR";
std::env::remove_var(ABSENT);
std::env::set_var(EMPTY, "");
std::env::set_var(PRESENT, "env-sourced-credential");
for (password, env_name) in [
(None, ABSENT),
(Some(String::new()), ABSENT),
(None, EMPTY),
(Some(String::new()), EMPTY),
] {
let error = super::resolve_verify_user_password(password, env_name)
.expect_err("empty or missing credential must be a usage error");
let rendered = error.to_string();
assert!(rendered.contains("--password"), "{rendered}");
assert!(rendered.contains(env_name), "{rendered}");
}
assert_eq!(
super::resolve_verify_user_password(None, PRESENT).expect("env credential"),
"env-sourced-credential"
);
assert_eq!(
super::resolve_verify_user_password(Some("flag-credential".to_owned()), EMPTY)
.expect("flag credential"),
"flag-credential"
);
}
#[test]
fn parses_write_secured_command() {
let parsed = Cli::try_parse_from([
+111 -23
View File
@@ -193,17 +193,43 @@ impl std::error::Error for CommandError {}
/// The wrapper is heap-allocated inside [`Error::MxAccess`] to keep the
/// containing enum small. Callers can recover the reply with
/// [`MxAccessError::reply`] or [`MxAccessError::into_reply`]. Its `Display`
/// summarizes the `hresult` and status entries and scrubs any credential-like
/// tokens from diagnostic text before it reaches a caller.
#[derive(Clone, Debug)]
/// summarizes the `hresult` and status entries and scrubs credentials from the
/// rendered text before it reaches a caller: credential-*shaped* tokens
/// (`mxgw_...`, `bearer`) via a pattern scrub, plus any exact caller-supplied
/// secrets registered with [`MxAccessError::with_secrets`] — the latter catches
/// a password MXAccess echoed back verbatim even though it has no token shape.
///
/// `Debug` is hand-written (not derived) so the attached exact secrets never
/// reach `{:?}` output either: it scrubs them from the reply rendering and
/// prints only the count of attached secrets, never their values.
#[derive(Clone)]
pub struct MxAccessError {
reply: MxCommandReply,
/// Exact caller-supplied secrets (e.g. an `AuthenticateUser` password or a
/// `WriteSecured` string value) scrubbed from the rendered message. Empty
/// unless a helper attaches them via [`Self::with_secrets`].
secrets: Vec<String>,
}
impl MxAccessError {
/// Wrap a reply whose MXAccess-level result reported a failure.
pub fn new(reply: MxCommandReply) -> Self {
Self { reply }
Self {
reply,
secrets: Vec::new(),
}
}
/// Register exact caller-supplied secrets to scrub from the rendered
/// message, returning the updated error.
///
/// A credential MXAccess echoes back into its diagnostic text has no
/// `mxgw_`/`bearer` shape, so the pattern scrub cannot catch it. Attaching
/// the exact secret lets `Display` replace every occurrence with
/// `<redacted>`.
pub fn with_secrets(mut self, secrets: Vec<String>) -> Self {
self.secrets = secrets;
self
}
/// Borrow the underlying reply (correlation id, hresult, statuses).
@@ -217,15 +243,43 @@ impl MxAccessError {
}
}
impl std::fmt::Debug for MxAccessError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// Render the reply, scrub any exact caller secret from it, and never
// print the raw secrets themselves — only how many are attached.
let mut reply = format!("{:?}", self.reply);
for secret in &self.secrets {
if !secret.is_empty() {
reply = reply.replace(secret.as_str(), "<redacted>");
}
}
formatter
.debug_struct("MxAccessError")
.field("reply", &format_args!("{reply}"))
.field(
"secrets",
&format_args!("[{} redacted]", self.secrets.len()),
)
.finish()
}
}
impl std::fmt::Display for MxAccessError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
use std::fmt::Write as _;
let hresult = match self.reply.hresult {
Some(value) => value.to_string(),
None => "none".to_owned(),
};
// Render the whole body first so the exact-secret scrub can sweep every
// field — including diagnostic text that already went through the
// credential-shape scrub — before any of it reaches the caller.
let mut body = String::new();
write!(
formatter,
body,
"hresult={hresult}, {} status entr{}",
self.reply.statuses.len(),
if self.reply.statuses.len() == 1 {
@@ -233,20 +287,28 @@ impl std::fmt::Display for MxAccessError {
} else {
"ies"
}
)?;
)
.expect("writing to a String is infallible");
for status in &self.reply.statuses {
let category = MxStatusCategory::try_from(status.category)
.unwrap_or(MxStatusCategory::Unspecified);
let diagnostic = redact_credentials(&status.diagnostic_text);
write!(
formatter,
body,
"; [success={}, category={category:?}, detail={}, {}]",
status.success, status.detail, diagnostic
)?;
)
.expect("writing to a String is infallible");
}
Ok(())
for secret in &self.secrets {
if !secret.is_empty() {
body = body.replace(secret.as_str(), "<redacted>");
}
}
formatter.write_str(&body)
}
}
@@ -284,10 +346,17 @@ impl From<tonic::Status> for Error {
/// Promote a non-OK protocol status carried inside an [`MxCommandReply`]
/// to an [`Error::Command`].
///
/// [`ProtocolStatusCode::MxaccessFailure`] is deliberately **not** a
/// command-level failure here: it signals an MXAccess-level rejection, so it
/// falls through to [`ensure_mxaccess_success`] and surfaces as
/// [`Error::MxAccess`] — matching the .NET, Java, Go, and Python clients. Every
/// other non-`Ok` code stays [`Error::Command`].
///
/// # Errors
///
/// Returns [`Error::Command`] when `reply.protocol_status` is missing or
/// reports any code other than [`ProtocolStatusCode::Ok`].
/// reports any code other than [`ProtocolStatusCode::Ok`] or
/// [`ProtocolStatusCode::MxaccessFailure`].
pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, Error> {
let code = reply
.protocol_status
@@ -295,7 +364,7 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
.and_then(|status| ProtocolStatusCode::try_from(status.code).ok())
.unwrap_or(ProtocolStatusCode::Unspecified);
if code == ProtocolStatusCode::Ok {
if code == ProtocolStatusCode::Ok || code == ProtocolStatusCode::MxaccessFailure {
Ok(reply)
} else {
Err(Box::new(CommandError::new(reply)).into())
@@ -306,12 +375,17 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
/// [`MxCommandReply`] to an [`Error::MxAccess`].
///
/// This is the second reply check applied to the typed command path, after
/// [`ensure_command_success`] confirms the protocol envelope is `Ok`. It
/// enforces MXAccess parity: a reply can carry an `Ok` protocol envelope while
/// MXAccess itself rejected the operation. Following COM semantics (and the
/// Python client), only a **negative** `hresult` is a failure — positive codes
/// such as `S_FALSE = 1` are success. A `MXSTATUS_PROXY` entry is treated as a
/// failure when its `success` member is `0`.
/// [`ensure_command_success`] confirms the protocol envelope is `Ok` (or a
/// [`ProtocolStatusCode::MxaccessFailure`] the first check lets fall through).
/// It enforces MXAccess parity: a reply can carry an `Ok` protocol envelope
/// while MXAccess itself rejected the operation, and a
/// [`ProtocolStatusCode::MxaccessFailure`] envelope is itself an MXAccess-level
/// failure regardless of `hresult`. Following COM semantics, only a
/// **negative** `hresult` is a failure — positive codes such as `S_FALSE = 1`
/// are success. A `MXSTATUS_PROXY` entry is treated as a failure when its
/// `category` is not [`MxStatusCategory::Ok`]; the `success` member mirrors the
/// raw COM value verbatim for diagnostics and never enters the verdict, so an
/// entry with an unspecified category fails even when `success` is non-zero.
///
/// Per-item bulk failures are reported inside each result entry
/// (`was_successful = false`) rather than in the top-level `hresult`/`statuses`
@@ -319,13 +393,24 @@ pub fn ensure_command_success(reply: MxCommandReply) -> Result<MxCommandReply, E
///
/// # Errors
///
/// Returns [`Error::MxAccess`] when `reply.hresult` is negative or any
/// `reply.statuses` entry reports a non-success `success` member.
/// Returns [`Error::MxAccess`] when the reply's protocol code is
/// [`ProtocolStatusCode::MxaccessFailure`], `reply.hresult` is negative, or any
/// `reply.statuses` entry reports a category other than
/// [`MxStatusCategory::Ok`].
pub fn ensure_mxaccess_success(reply: MxCommandReply) -> Result<MxCommandReply, Error> {
let protocol_code = reply
.protocol_status
.as_ref()
.and_then(|status| ProtocolStatusCode::try_from(status.code).ok())
.unwrap_or(ProtocolStatusCode::Unspecified);
let mxaccess_failure = protocol_code == ProtocolStatusCode::MxaccessFailure;
let hresult_failure = reply.hresult.is_some_and(|hresult| hresult < 0);
let status_failure = reply.statuses.iter().any(|status| status.success == 0);
let status_failure = reply
.statuses
.iter()
.any(|status| status.category != MxStatusCategory::Ok as i32);
if hresult_failure || status_failure {
if mxaccess_failure || hresult_failure || status_failure {
Err(Box::new(MxAccessError::new(reply)).into())
} else {
Ok(reply)
@@ -412,8 +497,10 @@ mod tests {
let mut reply = ok_reply();
// Positive hresult (e.g. S_FALSE = 1) is a success, not a failure.
reply.hresult = Some(1);
// A zero `success` member with an OK category is still a success: the
// category is authoritative and `success` is diagnostics only.
reply.statuses = vec![MxStatusProxy {
success: 1,
success: 0,
category: MxStatusCategory::Ok as i32,
..MxStatusProxy::default()
}];
@@ -424,8 +511,9 @@ mod tests {
#[test]
fn ensure_mxaccess_success_flags_failing_status_entry() {
let mut reply = ok_reply();
// A non-OK category fails even though the raw `success` member is set.
reply.statuses = vec![MxStatusProxy {
success: 0,
success: 1,
category: MxStatusCategory::CommunicationError as i32,
detail: 42,
diagnostic_text: "write rejected for mxgw_visible_secret".to_owned(),
+79 -11
View File
@@ -11,7 +11,7 @@
use std::sync::atomic::{AtomicU64, Ordering};
use crate::client::{EventStream, GatewayClient};
use crate::error::{ensure_protocol_success, Error};
use crate::error::{ensure_protocol_success, Error, MxAccessError};
use crate::generated::mxaccess_gateway::v1::mx_command::Payload;
use crate::generated::mxaccess_gateway::v1::mx_command_reply;
use crate::generated::mxaccess_gateway::v1::{
@@ -27,7 +27,7 @@ use crate::generated::mxaccess_gateway::v1::{
WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
WriteSecuredBulkCommand, WriteSecuredBulkEntry, WriteSecuredCommand,
};
use crate::value::{MxStatus, MxValue};
use crate::value::{MxStatus, MxValue, MxValueProjection};
const MAX_BULK_ITEMS: usize = 1_000;
@@ -801,6 +801,7 @@ impl Session {
verifier_user_id: i32,
value: MxValue,
) -> Result<(), Error> {
let secrets = string_secret(&value);
self.invoke(
MxCommandKind::WriteSecured,
Payload::WriteSecured(WriteSecuredCommand {
@@ -811,7 +812,8 @@ impl Session {
value: Some(value.into_proto()),
}),
)
.await?;
.await
.map_err(|error| attach_secrets(error, secrets))?;
Ok(())
}
@@ -831,6 +833,7 @@ impl Session {
value: MxValue,
timestamp_value: MxValue,
) -> Result<(), Error> {
let secrets = string_secret(&value);
self.invoke(
MxCommandKind::WriteSecured2,
Payload::WriteSecured2(WriteSecured2Command {
@@ -842,7 +845,8 @@ impl Session {
timestamp_value: Some(timestamp_value.into_proto()),
}),
)
.await?;
.await
.map_err(|error| attach_secrets(error, secrets))?;
Ok(())
}
@@ -882,7 +886,8 @@ impl Session {
verify_user_password: verify_user_password.to_owned(),
}),
)
.await?;
.await
.map_err(|error| attach_secrets(error, vec![verify_user_password.to_owned()]))?;
authenticate_user_id(&reply)
}
@@ -1074,18 +1079,81 @@ fn add_buffered_item_handle(reply: &MxCommandReply) -> Result<i32, Error> {
fn authenticate_user_id(reply: &MxCommandReply) -> Result<i32, Error> {
match reply.payload.as_ref() {
Some(mx_command_reply::Payload::AuthenticateUser(authenticate)) => Ok(authenticate.user_id),
_ => Err(Error::MalformedReply {
detail: "authenticate_user reply lacked an AuthenticateUser payload".to_owned(),
}),
_ => reply
.return_value
.as_ref()
.and_then(int32_reply_value)
.ok_or_else(|| Error::MalformedReply {
detail:
"authenticate_user reply lacked an AuthenticateUser payload or int32 return_value"
.to_owned(),
}),
}
}
fn archestra_user_id(reply: &MxCommandReply) -> Result<i32, Error> {
match reply.payload.as_ref() {
Some(mx_command_reply::Payload::ArchestraUserToId(archestra)) => Ok(archestra.user_id),
_ => Err(Error::MalformedReply {
detail: "archestra_user_to_id reply lacked an ArchestraUserToId payload".to_owned(),
}),
_ => reply
.return_value
.as_ref()
.and_then(int32_reply_value)
.ok_or_else(|| Error::MalformedReply {
detail:
"archestra_user_to_id reply lacked an ArchestraUserToId payload or int32 return_value"
.to_owned(),
}),
}
}
/// Extract an exact string secret from a credential-sensitive [`MxValue`] so a
/// failing `WriteSecured`/`WriteSecured2` can scrub it from the surfaced error.
/// Non-string values carry no scrubbable secret and yield an empty vector.
fn string_secret(value: &MxValue) -> Vec<String> {
match value.projection() {
MxValueProjection::String(text) if !text.is_empty() => vec![text.clone()],
_ => Vec::new(),
}
}
/// Attach caller-supplied exact secrets to an [`Error::MxAccess`] before it
/// propagates. This both scrubs the stored reply's caller-readable string
/// fields (so `reply()`/`into_reply()` cannot recover a credential MXAccess
/// echoed back verbatim) and keeps the secrets on the error as a
/// belt-and-suspenders for `Display`/`Debug`. Any other error variant is
/// returned unchanged.
fn attach_secrets(error: Error, secrets: Vec<String>) -> Error {
match error {
Error::MxAccess(boxed) => {
let mut reply = boxed.into_reply();
scrub_reply_strings(&mut reply, &secrets);
Error::MxAccess(Box::new(MxAccessError::new(reply).with_secrets(secrets)))
}
other => other,
}
}
/// Replace every non-empty secret occurrence with `<redacted>` in the reply's
/// caller-readable string fields — `protocol_status.message`,
/// `diagnostic_message`, and each `statuses[i].diagnostic_text`. A caller
/// reading the structured reply back off an [`Error::MxAccess`] would otherwise
/// reintroduce the leak that `Display`/`Debug` already close.
fn scrub_reply_strings(reply: &mut MxCommandReply, secrets: &[String]) {
for secret in secrets {
if secret.is_empty() {
continue;
}
if let Some(status) = reply.protocol_status.as_mut() {
status.message = status.message.replace(secret.as_str(), "<redacted>");
}
reply.diagnostic_message = reply
.diagnostic_message
.replace(secret.as_str(), "<redacted>");
for status in &mut reply.statuses {
status.diagnostic_text = status
.diagnostic_text
.replace(secret.as_str(), "<redacted>");
}
}
}
+5 -1
View File
@@ -282,7 +282,11 @@ impl MxStatus {
&self.raw
}
/// `MXSTATUS_PROXY.Success` flag (0 = error, non-zero = good/warning).
/// Raw `MXSTATUS_PROXY.Success` member, carried verbatim from COM.
///
/// This is a diagnostic value, not a verdict: the wire contract makes
/// [`Self::category`] authoritative, and `ensure_mxaccess_success` branches
/// on the category alone.
pub fn success(&self) -> i32 {
self.raw.success
}
+314 -2
View File
@@ -14,6 +14,7 @@ use tokio::sync::{mpsc, Mutex};
use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream};
use tonic::transport::Server;
use tonic::{Request, Response, Status};
use zb_mom_ww_mxgateway_client::error::ensure_mxaccess_success;
use zb_mom_ww_mxgateway_client::generated::mxaccess_gateway::v1::mx_access_gateway_server::{
MxAccessGateway, MxAccessGatewayServer,
};
@@ -83,8 +84,10 @@ async fn session_helpers_build_commands_and_preserve_command_errors() {
.write(12, 34, ClientMxValue::int32(123), 0)
.await
.unwrap_err();
let Error::Command(error) = error else {
panic!("write failure should preserve the raw command reply: {error:?}");
// A MXACCESS_FAILURE-coded reply is an MXAccess-level failure, routed to
// Error::MxAccess (matching .NET/Java/Go/Python) rather than Error::Command.
let Error::MxAccess(error) = error else {
panic!("MXACCESS_FAILURE reply should route to Error::MxAccess: {error:?}");
};
assert_eq!(
error.reply().protocol_status.as_ref().unwrap().code,
@@ -337,6 +340,57 @@ fn authentication_and_authorization_statuses_are_distinct_and_redacted() {
assert!(!auth.to_string().contains("visible_secret"));
}
#[test]
fn command_reply_validation_fixtures_branch_on_category_and_negative_hresult() {
// The shared behavior fixtures pin both reply-validation rules: a status
// entry fails iff its category is not OK (the raw `success` member is
// diagnostics only) and an HRESULT fails iff it is present and negative.
for (fixture, expect_failure) in [
("register.ok.reply.json", false),
("write.status-category-error-success-set.reply.json", true),
("write.status-category-ok-success-zero.reply.json", false),
("write.hresult-s-false.reply.json", false),
("write.hresult-e-fail.reply.json", true),
] {
let reply = command_reply_fixture(fixture);
let result = ensure_mxaccess_success(reply);
assert_eq!(
result.is_err(),
expect_failure,
"fixture {fixture} expected failure = {expect_failure}, got {result:?}"
);
}
}
#[test]
fn status_entry_verdict_ignores_the_raw_success_member() {
// Edges the fixtures cannot express: an OK category always passes and an
// unspecified category always fails, whatever `success` carries.
for (category, success, expect_failure) in [
(MxStatusCategory::Ok, 0, false),
(MxStatusCategory::Ok, 1, false),
(MxStatusCategory::CommunicationError, 1, true),
(MxStatusCategory::Unspecified, 1, true),
] {
let reply = MxCommandReply {
protocol_status: Some(ok_status("command ok")),
statuses: vec![MxStatusProxy {
success,
category: category as i32,
..MxStatusProxy::default()
}],
..MxCommandReply::default()
};
assert_eq!(
ensure_mxaccess_success(reply).is_err(),
expect_failure,
"category {category:?} with success {success} expected failure = {expect_failure}"
);
}
}
#[test]
fn command_error_display_keeps_raw_reply_accessible() {
let reply = mxaccess_failure_reply();
@@ -752,6 +806,179 @@ async fn authenticate_user_keeps_credentials_out_of_surfaced_errors() {
);
}
#[tokio::test]
async fn authenticate_user_scrubs_exact_caller_credential_echoed_in_diagnostic() {
// CLI-40: MXAccess can echo the supplied credential back inside its failure
// diagnostic (here in statuses[0].diagnostic_text). The token has no
// mxgw_/bearer shape, so the pattern scrub alone cannot catch it — the
// exact-secret scrub must replace the caller's password with <redacted>.
let credential = "sup3rSecretVerify9f3a2b";
let state = Arc::new(FakeState::default());
*state.invoke_override.lock().await = Some(InvokeOverride::CannedReply(Box::new(
command_reply_fixture("authenticate-user.echoed-credential.reply.json"),
)));
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let error = session
.authenticate_user(7, "verifier", credential)
.await
.unwrap_err();
assert!(
matches!(error, Error::MxAccess(_)),
"OK protocol + negative hresult must route to Error::MxAccess: {error:?}"
);
let rendered = error.to_string();
assert!(
!rendered.contains(credential),
"exact caller credential leaked into the surfaced error: {rendered}"
);
assert!(
rendered.contains("<redacted>"),
"credential occurrence must be replaced with <redacted>: {rendered}"
);
}
/// Drive `authenticate_user` against a canned reply that echoes the caller's
/// credential in every string field, then assert the surfaced
/// [`Error::MxAccess`] leaks it nowhere — neither through the structured reply a
/// caller can read back (`reply().protocol_status.message`,
/// `reply().diagnostic_message`, `reply().statuses[i].diagnostic_text`) nor
/// through `Display`/`Debug`.
async fn assert_authenticate_user_scrubs_structured_reply(fixture: &str) {
let credential = "sup3rSecretVerify9f3a2b";
let state = Arc::new(FakeState::default());
*state.invoke_override.lock().await = Some(InvokeOverride::CannedReply(Box::new(
command_reply_fixture(fixture),
)));
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let error = session
.authenticate_user(7, "verifier", credential)
.await
.unwrap_err();
let Error::MxAccess(mx_access) = &error else {
panic!("{fixture}: credential-echoed reply must route to Error::MxAccess, got {error:?}");
};
// The structured reply a caller can read back must be scrubbed too — the raw
// MxCommandReply otherwise reintroduces the leak Display/Debug already close.
let reply = mx_access.reply();
if let Some(status) = reply.protocol_status.as_ref() {
assert!(
!status.message.contains(credential),
"{fixture}: credential leaked via reply().protocol_status.message: {}",
status.message
);
}
assert!(
!reply.diagnostic_message.contains(credential),
"{fixture}: credential leaked via reply().diagnostic_message: {}",
reply.diagnostic_message
);
for (index, status) in reply.statuses.iter().enumerate() {
assert!(
!status.diagnostic_text.contains(credential),
"{fixture}: credential leaked via reply().statuses[{index}].diagnostic_text: {}",
status.diagnostic_text
);
}
let display = error.to_string();
let debug = format!("{error:?}");
assert!(
!display.contains(credential),
"{fixture}: credential leaked into Display: {display}"
);
assert!(
!debug.contains(credential),
"{fixture}: credential leaked into Debug: {debug}"
);
assert!(
display.contains("<redacted>"),
"{fixture}: Display must mark the redaction: {display}"
);
}
#[tokio::test]
async fn authenticate_user_scrubs_credential_from_structured_reply_ok_protocol_variant() {
// OK protocol envelope + negative hresult: already Error::MxAccess before
// ISSUE 2, but the stored reply's string fields still leaked the credential.
assert_authenticate_user_scrubs_structured_reply(
"authenticate-user.echoed-credential.reply.json",
)
.await;
}
#[tokio::test]
async fn authenticate_user_scrubs_credential_from_structured_reply_mxaccess_failure_variant() {
// PROTOCOL_STATUS_CODE_MXACCESS_FAILURE: before ISSUE 2 this landed in
// Error::Command (unscrubbed, raw Display/Debug) — the red-first case.
assert_authenticate_user_scrubs_structured_reply(
"authenticate-user.echoed-credential-mxaccess-failure.reply.json",
)
.await;
}
#[tokio::test]
async fn authenticate_user_maps_missing_payload_reply_to_malformed_reply() {
// CLI-41: an OK reply with neither a typed AuthenticateUser payload nor a
// return_value is malformed.
let state = Arc::new(FakeState::default());
*state.invoke_override.lock().await = Some(InvokeOverride::CannedReply(Box::new(
command_reply_fixture("authenticate-user.missing-payload.reply.json"),
)));
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let error = session
.authenticate_user(7, "verifier", "pw")
.await
.unwrap_err();
assert!(
matches!(error, Error::MalformedReply { .. }),
"missing payload + missing return_value must be MalformedReply, got {error:?}"
);
}
#[tokio::test]
async fn authenticate_user_falls_back_to_return_value_when_typed_payload_absent() {
// CLI-41: an OK reply that carries only a return_value (legacy worker) must
// resolve the user id from it, mirroring add_buffered_item's fallback.
let state = Arc::new(FakeState::default());
*state.invoke_override.lock().await = Some(InvokeOverride::CannedReply(Box::new(
command_reply_fixture("authenticate-user.return-value-only.reply.json"),
)));
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let user_id = session
.authenticate_user(7, "verifier", "pw")
.await
.unwrap();
assert_eq!(
user_id, 7,
"user id must resolve from the int32 return_value"
);
}
#[tokio::test]
async fn stream_alarms_emits_snapshot_then_complete_then_transition_in_order() {
let state = Arc::new(FakeState::default());
@@ -903,6 +1130,11 @@ enum InvokeOverride {
/// `AuthenticateUser` rejected by MXAccess) so the client's
/// `ensure_mxaccess_success` check is exercised on the typed helper path.
MxAccessFailure,
/// Reply with a caller-supplied canned [`MxCommandReply`]. Lets a test
/// drive a helper with a shared behavior fixture (e.g. the
/// echoed-credential / missing-payload / return-value-only
/// authenticate-user replies). Boxed to keep the enum small.
CannedReply(Box<MxCommandReply>),
}
#[derive(Clone)]
@@ -1005,6 +1237,7 @@ impl MxAccessGateway for FakeGateway {
payload: None,
..MxCommandReply::default()
})),
InvokeOverride::CannedReply(reply) => Ok(Response::new(*reply)),
InvokeOverride::WriteOk => {
// Extract and capture the WriteCommand payload so the test
// can assert on server_handle, item_handle, user_id, and value.
@@ -1358,6 +1591,85 @@ fn event(sequence: u64) -> MxEvent {
}
}
/// Load a shared command-reply fixture into an [`MxCommandReply`].
///
/// The fixtures are protobuf JSON, which prost cannot parse directly, so this
/// reads the fields the reply-validation rules actually consume (`hresult` and
/// the status `success`/`category` pair) and rebuilds the message. Enum names
/// resolve through the generated `from_str_name`, so a fixture naming a
/// category the contract does not define fails the test rather than silently
/// degrading to `Unspecified`.
fn command_reply_fixture(file_name: &str) -> MxCommandReply {
let fixture = behavior_fixture(&format!("command-replies/{file_name}"));
let statuses = fixture["statuses"]
.as_array()
.map(Vec::as_slice)
.unwrap_or_default()
.iter()
.map(|status| {
let category_name = status["category"].as_str().unwrap();
MxStatusProxy {
success: status["success"].as_i64().unwrap() as i32,
category: MxStatusCategory::from_str_name(category_name)
.unwrap_or_else(|| panic!("unknown status category {category_name}"))
as i32,
detail: status["detail"].as_i64().unwrap_or_default() as i32,
diagnostic_text: status["diagnosticText"]
.as_str()
.unwrap_or_default()
.to_owned(),
..MxStatusProxy::default()
}
})
.collect();
// The fixtures that exercise the return_value fallback path carry a typed
// `returnValue` (VT_I4). Project it so a canned reply can drive the
// helper's payload -> return_value -> MalformedReply precedence.
let return_value = fixture.get("returnValue").and_then(|value| {
value["int32Value"].as_i64().map(|int32| MxValue {
data_type: MxDataType::Integer as i32,
variant_type: value["variantType"].as_str().unwrap_or("VT_I4").to_owned(),
kind: Some(Kind::Int32Value(int32 as i32)),
..MxValue::default()
})
});
// Honor the fixture's real protocol status (code + message) so a canned
// reply can drive the MXACCESS_FAILURE routing path, not just an OK
// envelope. Falls back to an OK envelope when the fixture omits it.
let protocol_status = fixture.get("protocolStatus").map_or_else(
|| ok_status("command ok"),
|status| {
let code_name = status["code"].as_str().unwrap_or("PROTOCOL_STATUS_CODE_OK");
ProtocolStatus {
code: ProtocolStatusCode::from_str_name(code_name)
.unwrap_or_else(|| panic!("unknown protocol status code {code_name}"))
as i32,
message: status["message"].as_str().unwrap_or_default().to_owned(),
}
},
);
MxCommandReply {
session_id: fixture["sessionId"].as_str().unwrap_or_default().to_owned(),
correlation_id: fixture["correlationId"]
.as_str()
.unwrap_or_default()
.to_owned(),
protocol_status: Some(protocol_status),
hresult: fixture["hresult"].as_i64().map(|hresult| hresult as i32),
statuses,
diagnostic_message: fixture["diagnosticMessage"]
.as_str()
.unwrap_or_default()
.to_owned(),
return_value,
..MxCommandReply::default()
}
}
fn behavior_fixture(path: &str) -> Value {
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../proto/fixtures/behavior")
+42 -7
View File
@@ -99,9 +99,20 @@ library:
are skipped. Only successes are cached; failures always reach the inner verifier.
On a gateway-initiated revoke/rotate/delete the dashboard admin service calls
`IApiKeyCacheInvalidator.Invalidate(keyId)`, evicting the cached entry
immediately. The short TTL is the backstop for out-of-band mutations (a direct DB
edit, or a revoke run by the separate `apikey` CLI process, whose in-memory cache
is not the running gateway's cache).
immediately. `Invalidate` bumps a per-key generation counter **before** it evicts,
and `VerifyAsync` snapshots that generation before the inner verify and re-checks
it after writing the cache entry (set-then-recheck); a revoke that lands while a
verification is still in flight in the inner library therefore discards that
verification's repopulation instead of re-caching the just-revoked identity for a
full TTL (SEC-34). The short TTL remains the backstop for two bounded-staleness
windows it cannot close directly: (1) out-of-band mutations (a direct DB edit, or a
revoke run by the separate `apikey` CLI process, whose in-memory cache is not the
running gateway's cache); and (2) a key whose `ExpiresUtc` passes while cached keeps
authenticating until the entry's TTL elapses — expiry is enforced by the inner
library verifier, which a cache hit never reaches, and the verification identity the
library returns carries no expiry timestamp, so the cache cannot cap an entry at the
key's expiry (capping it needs the donor library to surface expiry on the
verification identity). The default 15 s TTL bounds both windows.
- **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
most one `MarkUsed` write per key per
`MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a
@@ -116,6 +127,26 @@ a dictionary lookup. Both windows are configurable and may be set to `0` to disa
the respective mechanism; see
[GatewayConfiguration](./GatewayConfiguration.md).
Failures are never cached — a wrong secret always reaches the store — so the
failure path is shielded by `ApiKeyFailureLimiter` instead, consulted before
`VerifyAsync` runs. It counts failures over one sliding
`MxGateway:Security:ApiKeyFailureWindowSeconds` window in two layers: a composite
`(transport peer, key id)` partition capped at `ApiKeyFailureLimit`, and a
per-key-id aggregate across all peers capped at `ApiKeyFailureAggregateLimit`. The
key id never partitions on its own — it is public, so an attacker-supplied one
would otherwise let any peer throttle a key it does not hold — and it joins the
partition only when the presented token is validly shaped
(`mxgw_<keyId>_<secret>`, key id at most 64 characters), with at most 32 key-id
partitions per address before the overflow collapses onto that address's fallback
partition. An over-limit state admits one probe per
`ApiKeyFailureProbeIntervalSeconds` through to the real verifier and refuses
everything else with `ResourceExhausted` before the store read, so a legitimate
holder presenting the correct secret always reaches the constant-time compare and
resets both layers; the counter's LRU eviction (`ApiKeyFailureTrackedPeers`)
prefers expired windows and will not drop an over-limit partition below a 2x
overshoot ceiling, so the memory bound cannot be turned into a way to clear an
active block. See [Authorization](./Authorization.md) for the enforcement path.
## Storage
API-key state lives in a dedicated SQLite database owned by the shared library.
@@ -128,10 +159,14 @@ is derived from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)`
(`C:\ProgramData\MxGateway\gateway-auth.db` on Windows,
`/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) so the
credential store is never written relative to the launch working directory on a
non-Windows host. The production hosts pin the explicit Windows path in
`appsettings.json`. `GatewayOptionsValidator` rejects a non-rooted (relative)
`SqlitePath` so a bad override fails fast at startup rather than scattering the store
by launch CWD (SEC-01).
non-Windows host. `appsettings.json` no longer ships an explicit path (SEC-33): the
removed Windows literal matched the Windows code default and, being non-rooted on a
Unix host, would have resolved against the CWD there; deployed hosts override it
through the NSSM environment (`MxGateway__Authentication__SqlitePath`).
`GatewayOptionsValidator` rejects a `SqlitePath` that is not rooted **on the host
running the gateway** (`Path.IsPathRooted`, current OS) — a relative filename or a
foreign-platform literal fails fast at startup rather than scattering the store by
launch CWD (SEC-01, SEC-33).
The library owns the SQLite schema and connection factory. The `api_keys` table
carries the key id, key prefix, secret-hash blob, display name, serialized scopes,
+7 -2
View File
@@ -89,9 +89,14 @@ The flow is:
The status codes are deliberately distinct: `Unauthenticated` signals "we do not know who you are," and `PermissionDenied` signals "we know who you are, but you cannot do this." Treating the two as the same code would make troubleshooting harder for client implementations.
### Rate limiting the auth surface (SEC-11)
### Rate limiting the auth surface (SEC-11, SEC-31, SEC-32)
Before the verification store read, the helper checks a cheap in-process per-peer failure counter (`ApiKeyFailureLimiter`). A peer that has accumulated more than `MxGateway:Security:ApiKeyFailureLimit` failed attempts inside the sliding `ApiKeyFailureWindowSeconds` window is short-circuited with `StatusCode.ResourceExhausted` — so online guessing of API-key secrets cannot spend a SQLite read (and, in a naive design, a cache miss) per attempt. The peer is keyed on the presented key id where the token parses, falling back to the transport peer address; keying on key id throttles a single abusive credential without penalizing co-located clients behind a shared NAT. A successful verification resets the peer's counter. The counter is a bounded LRU (`ApiKeyFailureTrackedPeers`) so it cannot grow without limit. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property.
Before the verification store read, the helper asks a cheap in-process failure counter (`ApiKeyFailureLimiter`) whether the attempt may proceed, so online guessing of API-key secrets cannot spend a SQLite read (and, in a naive design, a cache miss) per attempt. The counter has two layers over one sliding `ApiKeyFailureWindowSeconds` window:
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
+66
View File
@@ -44,6 +44,66 @@ MXAccess failures remain command replies when the gateway reached the worker and
the worker captured HRESULT or `MXSTATUS_PROXY` details. Client wrappers should
map those replies to rich command errors without discarding the raw reply.
### Reply Validation Conformance
Four command reply fixtures pin the two reply-validation rules every client
applies, because both rules have edges where a naive reading disagrees with the
wire contract:
| Fixture | Reply | Expected verdict |
|---|---|---|
| `write.status-category-error-success-set.reply.json` | one status with `success = 1`, `category = MX_STATUS_CATEGORY_COMMUNICATION_ERROR` | failure |
| `write.status-category-ok-success-zero.reply.json` | one status with `success = 0`, `category = MX_STATUS_CATEGORY_OK` | success |
| `write.hresult-s-false.reply.json` | `hresult = 1` (`S_FALSE`), statuses OK | success |
| `write.hresult-e-fail.reply.json` | `hresult = -2147467259` (`E_FAIL`), statuses OK | failure |
The rules those fixtures lock in are:
- **Status entries.** An `MxStatusProxy` entry is a failure exactly when
`category != MX_STATUS_CATEGORY_OK`. `success` mirrors the raw 16-bit COM
member and is diagnostics only, so it never participates in the verdict — the
proto contract makes `category` authoritative. An absent entry is success
(nothing was reported); a present entry with
`MX_STATUS_CATEGORY_UNSPECIFIED` is a failure, because the worker always maps
a category and an unmapped one is not proven OK.
- **HRESULT.** A reply fails on HRESULT exactly when `hresult` is present and
negative. Positive COM success codes such as `S_FALSE` pass, matching COM
semantics.
### Malformed-Reply And Credential-Redaction Conformance
Three further command reply fixtures pin the id/handle-extraction and
credential-redaction contracts for the credential-bearing helpers:
| Fixture | Reply | Expected behavior |
|---|---|---|
| `authenticate-user.echoed-credential.reply.json` | OK envelope, negative `hresult`, and the caller's credential echoed into `protocolStatus.message`, `statuses[0].diagnosticText`, and `diagnosticMessage` | the surfaced error redacts the exact secret from **both** the rendered message and the structured reply accessors (never leaks the verbatim value) |
| `authenticate-user.echoed-credential-mxaccess-failure.reply.json` | the same echo, but coded `PROTOCOL_STATUS_CODE_MXACCESS_FAILURE` | identical redaction; confirms every client routes the MXAccess-failure protocol code to its MXAccess error type and scrubs it |
| `authenticate-user.missing-payload.reply.json` | OK envelope, no `AuthenticateUser` payload, no `return_value` | a typed malformed-reply error, never a proto3 default `0` and never an NRE |
| `authenticate-user.return-value-only.reply.json` | OK envelope, `return_value.int32_value = 7`, no typed payload | the id resolves to `7` via the legacy `return_value` compatibility path |
The rules those fixtures lock in are:
- **Malformed-reply extraction (CLI-41).** Every helper that extracts a scalar
id/handle (`AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, and the
handle extractors) prefers the typed payload; when it is absent it falls back
to `return_value` **only** when `return_value` is present with the expected
int32 variant; when neither is present it raises a typed malformed-reply error.
It never surfaces a proto3 default `0` and never throws a null-reference.
- **Credential redaction (CLI-40).** The credential-bearing helpers
(`AuthenticateUser`, `WriteSecured`/`WriteSecured2`) scrub the exact secret
values they were called with from any surfaced error — both the rendered
message text **and** the structured reply the error still exposes (a
server-echoed credential lives in `protocolStatus.message` and
`statuses[].diagnosticText`, which the error's raw-reply accessor would
otherwise re-expose to a logger dumping structured fields). The redacted error
therefore carries a scrubbed clone of the reply. This is defense-in-depth on
top of the by-construction guarantee that exceptions carry reply-derived text,
not the request. The marker is `<redacted>` in the Go, Rust, and Java clients
and `[redacted]` in the Python client and the .NET CLI; each suite asserts that
neither the surfaced message nor the exposed reply still contains the
credential, and that the message contains the client's marker.
## Event Streams
Event stream fixtures live in
@@ -74,6 +134,12 @@ behavior. A language helper may expose native booleans, integers, strings,
arrays, and timestamps, but it must keep `rawDiagnostic`, raw data type fields,
and raw byte payloads accessible when conversion is incomplete.
Each status case also carries an independent `wantSuccess` boolean alongside its
`status` object. The success/failure conformance tests assert the helper's
verdict against this fixture-declared expectation rather than recomputing it from
`category` (the same formula under test), so a regression in the verdict rule
cannot hide behind a self-consistent computation.
## Auth, Timeout, And Cancel Behavior
Authentication fixtures live in `clients/proto/fixtures/behavior/auth/`. They
+27 -3
View File
@@ -115,15 +115,39 @@ remove/write family, the parity-critical single-item helpers are:
`Activate`, and `Unregister`. Each is a thin wrapper over the same raw-command
machinery the bulk helpers use — it adds no wire surface — and runs the same
MXAccess-level reply validation (HRESULT `< 0` + per-item `MxStatusProxy`) as the
rest of the client. **MXAccess parity is preserved exactly**: e.g. `WriteSecured`
rest of the client. The per-item rule is **`MxStatusProxy` failure iff
`category != MX_STATUS_CATEGORY_OK`**; `success` is the raw COM member carried
for diagnostics only and never decides the verdict, so an absent entry is
success while a present entry with an unspecified category is a failure. The
shared fixtures in `clients/proto/fixtures/behavior/command-replies/` pin both
rules across all five clients (see
[Client Behavior Fixtures](./ClientBehaviorFixtures.md)).
**MXAccess parity is preserved exactly**: e.g. `WriteSecured`
failing before a prior `AuthenticateUser` + `AdviseSupervisory` surfaces the
native failure unchanged — the helper does not pre-validate or reorder it.
**Malformed-reply extraction:** the id/handle-returning helpers
(`AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, and the handle
extractors) follow one contract across all five clients — prefer the typed
payload; fall back to `return_value` only when it is present with the expected
int32 variant; when neither is present, raise a typed malformed-reply error.
They never surface a proto3 default `0` and never throw a null-reference (CLI-41).
**Credential handling:** `AuthenticateUser` credentials and `WriteSecured`
secured payloads route through each client's secret-redaction seam so they never
reach logs, exception text, or `ToString`/`Debug`/`Display` — the value is carried
only on the wire. Each client's test suite asserts a distinctive credential is
absent from any surfaced error.
only on the wire. In addition to that by-construction guarantee (exceptions carry
reply-derived text, not the request), every client scrubs the **exact** secret
values it was called with from any surfaced error as defense-in-depth, so a
gateway or MXAccess diagnostic that echoes a credential back cannot leak it
(CLI-40). The scrub covers **both** the rendered message and the structured reply
the error still exposes (`protocolStatus.message`, `statuses[].diagnosticText`,
`diagnosticMessage`): the redacted error carries a scrubbed clone of the reply so
a logger dumping the exception's structured fields cannot reintroduce the leak.
This holds regardless of whether the reply is coded `OK` (with a failing HRESULT)
or `MXACCESS_FAILURE` — every client routes both to its MXAccess error type. Each
client's test suite asserts the distinctive credential is absent from both the
surfaced message and the exposed reply, and that the redaction marker is present.
Shipped in all five clients (.NET / Go / Rust / Python / Java).
+16
View File
@@ -117,6 +117,22 @@ The Rust workspace builds the `mxgateway-client` library crate and the `mxgw`
CLI crate. `build.rs` generates `tonic` and `prost` modules into Cargo build
output on each build that needs updated protobuf output.
`build.rs` resolves its `.proto` inputs repo-path-first, then vendored: it
prefers the canonical protos under `src/ZB.MOM.WW.MxGateway.Contracts/Protos`
so an in-repo edit is live immediately, and falls back to the copies vendored
into `clients/rust/protos/` only when the canonical directory is absent — the
case for a published crate unpacked outside this repo. The vendored copies
are declared in `Cargo.toml`'s `include` list, so `cargo package`/`cargo
publish` ship them inside the `.crate`, making the crate buildable standalone
with no access to the rest of the mxaccessgw repo. Any Contracts proto change
must refresh `clients/rust/protos/` in the same commit; `scripts/check-codegen.ps1`
Check 3 byte-compares the vendored copies against the canonical protos and
fails on drift. Because the vendored protos make a standalone build possible,
`cargo package`/`cargo publish` run **with** verification (no `--no-verify`) —
a `cargo package` that cannot build from the vendored tree alone would mean
the vendored copies are stale, and verification is what catches that before
publish.
Regenerate and compile Rust bindings:
```powershell
+90 -3
View File
@@ -34,9 +34,40 @@ When `stream-events` is resumed with an `after_worker_sequence` cursor that
predates the oldest event still in the gateway's replay ring, the gateway emits a
single `ReplayGap` sentinel at the head of the stream. Every client surfaces this
as a distinct, typed, non-terminal signal (see each client README); the resume
contract is `after_worker_sequence = oldest_available_sequence - 1`. The default
smoke sequence opens a fresh stream (no cursor) and does not exercise the gap
path; a resume-with-gap fixture case is tracked separately (TST-24).
contract is `after_worker_sequence = oldest_available_sequence - 1`, and it holds
even when the ring has been emptied entirely by age eviction — the gateway then
reports the next deliverable sequence rather than `0` (see [Sessions](Sessions.md)).
The default smoke sequence opens a fresh stream (no cursor) and does not exercise
the gap path; a resume-with-gap fixture case is tracked separately (TST-24).
The CLIs differ in how they *print* that library-level signal. Three of them consume
the typed gap and emit a dedicated row rather than a degenerate event row; the other
two hand the raw sentinel `MxEvent` straight to the formatter, so they print the
sentinel itself, whose `replayGap` field carries the same cursors:
| CLI | Text mode | JSON mode |
|-----|-----------|-----------|
| `mxgw-rs` (Rust, canonical) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-go` (Go) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line, counted toward `-limit` like any other row |
| `mxgw-py` (Python) | same JSON dump as `--json` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-dotnet` (.NET) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field | same, as one entry of the `events` array |
| `mxgw-java` (Java) | the sentinel's `worker_sequence` and `family` (`0 MX_EVENT_FAMILY_UNSPECIFIED`) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field |
Rust, Go, and Python emit the same two key names and, deliberately, the same JSON
value **types**: the cursors are JSON numbers (`7`), not strings. That is why the
Go CLI types the row by hand instead of marshalling `ReplayGap` with `protojson`
the proto3 JSON mapping renders 64-bit integers as strings (`"7"`), which is also
why the .NET and Java rows, which pass the sentinel through a protobuf JSON
formatter, carry **quoted** cursors. A matrix runner must therefore compare parsed
values, not raw bytes, and must not assume the same value type across all five
CLIs.
Two further formatting differences among the three canonical CLIs, none of them
semantic: Python sorts object keys and uses `", "` / `": "` separators
(`json.dumps(..., sort_keys=True)`), while Rust and Go emit compact,
declaration-ordered JSON; and the row sits alone on its own line for Go and for
Rust's `--jsonl`, but inside an `events` array for Python and for Rust's
aggregate `--json`.
## Integration Gate
@@ -59,6 +90,37 @@ The shared inputs are:
The commands in the matrix use `MXGATEWAY_API_KEY` through each CLI's
`api-key-env` flag. They must not embed bearer tokens or raw API keys.
### Credential contract for `authenticate-user`
Every CLI resolves the MXAccess verify-user credential the same way, so one
exported variable drives the same operator workflow in all five languages:
| Variable | Default | Purpose |
|----------|---------|---------|
| `MXGATEWAY_VERIFY_PASSWORD` | Empty | Verify-user credential read by `authenticate-user` when `--password` is omitted. |
- Flags are `--password` (Go: `-password`) for an explicit value and
`--password-env` (Go: `-password-env`) for the *name* of the environment
variable, defaulting to `MXGATEWAY_VERIFY_PASSWORD`.
- Resolution order is flag, then environment variable. Prefer the variable: the
flag puts the secret in shell history and the process table.
- A resolved credential that is **missing or empty** is a usage error. The CLI
fails fast before dialing rather than authenticating with an empty password,
and the error names only the flag and the variable — never the value. Nothing
echoes the credential to stdout, stderr, or logs.
This is CLI argument validation, not an MXAccess parity exception: the client
*libraries* still transmit whatever credential they are given. Only the operator
tools refuse to fabricate an empty one.
The .NET CLI accepted `--verify-user-password`, `--verify-user-password-env`, and
`MXGATEWAY_VERIFY_USER_PASSWORD` before this contract was unified. Those names
remain as deprecated aliases for one release; new scripts must use the canonical
names above. The full .NET resolution order is `--password`,
`--verify-user-password`, the variable named by `--password-env` (or the
deprecated `--verify-user-password-env`, default `MXGATEWAY_VERIFY_PASSWORD`),
then `MXGATEWAY_VERIFY_USER_PASSWORD`.
### TLS variant
The matrix runs over plaintext (`h2c`) by default. A TLS variant exists but stays
@@ -101,6 +163,31 @@ for quick local checks, but the full cross-language matrix uses explicit
operation commands because not every bundled smoke command streams events yet.
The explicit sequence remains the parity baseline for issue-level validation.
## Per-CLI Subcommand Coverage
The matrix sequence itself is available everywhere, but the later single-item
session commands were not added to every CLI at the same time. A runner that
reaches beyond the required sequence must branch on language, so the current
deltas are specified here rather than left to be discovered:
| Subcommand | .NET | Rust | Go | Python | Java |
|------------|------|------|----|--------|------|
| `unregister` | yes | yes | no | no | no |
| `add-buffered-item` | yes | no | no | no | no |
| `set-buffered-update-interval` | yes | no | no | no | no |
| `suspend` | yes | no | no | no | no |
| `activate` | yes | no | no | no | no |
| `write-secured` | yes | yes | yes | yes | yes |
| `write-secured2` | yes | no | no | no | no |
| `authenticate-user` | yes | yes | yes | yes | yes |
| `archestra-user-to-id` | yes | no | no | no | no |
Only .NET exposes all nine. Rust adds `unregister` and the credential pair; Go,
Python, and Java expose the credential pair only. Every gap is CLI surface only —
all five *libraries* implement all nine typed helpers, so a gap is a missing
operator command, never a missing capability. Levelling the CLIs is separate
feature work and is not tracked as a defect here.
## Validation
Run the matrix shape tests after changing the smoke matrix:
+30
View File
@@ -135,6 +135,36 @@ alarm state is gateway-wide, not session-scoped — every client wants the same
current set plus updates, and forcing each to own a worker would multiply AVEVA
polling load for no benefit.
## Session-Resilience Epic Scope
Decision (2026-07-09, archreview TST-04; migrated here 2026-08-07 from the retired
`oldtasks.md` mirror per TST-29): the session-resilience epic
(`docs/plans/2026-06-15-session-resilience.md`, 28 tasks) resolves into three per-phase
decisions rather than one open backlog.
- **Phase 3 (reconnect)** — essentially complete. Task 13 (owner re-validation) shipped
as archreview **TST-02** (P0, session attach is owner-scoped; see
[Session Reconnect](#session-reconnect) above). Task 15 (reconnect integration test)
shipped as **TST-01** (`GatewayEndToEndReconnectReplayTests`). Task 14 (client
`ReplayGap` handling) shipped as **CLI-15** for four of five clients
(.NET/Go/Rust/Python); the Java client is the only remainder.
- **Phase 4 (per-session dashboard ACL)** — scoped, not yet built. Tracked as archreview
**TST-15**. The Viewer-default decision is settled: admin-sees-all, Viewer strictly
scoped to sessions it owns or is granted — matching the gRPC owner-binding decision in
[Session Reconnect](#session-reconnect) above, for consistency between the gRPC and
dashboard surfaces.
- **Phase 5 (orphan-worker reattach)** — deferred, not planned. It would reverse the
"Gateway restart does not reattach orphan workers" invariant (see CLAUDE.md), adding a
stable gateway-instance id, an adoption-manifest SQLite store, a worker phone-home
reconnect protocol, and gateway-side adoption (re-open pipes, nonce-validate, reject
impostors). It stays deferred unless a concrete requirement appears; the invariant
stands. **`EnableOrphanReattach` does not exist and must not be referenced anywhere as
if it does** until that task actually lands.
`docs/plans/2026-06-15-session-resilience.md.tasks.json` remains the sole resume state
for the still-pending Phase 4 tasks (16-19) and the deferred Phase 5 tasks (20-28) — one
authority, no mirror.
## Authentication
Decision: API key authentication for the public gateway.
+2 -2
View File
@@ -123,9 +123,9 @@ The split uses `count: 3` because the secret portion may itself contain undersco
### Command value redaction
> **Not yet implemented.** Command-value logging is *not* wired end-to-end. There is no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob, and `RedactCommandValue` / `IsCredentialBearingCommand` have no call sites in the gateway — no command values are logged anywhere today, which is the safest posture. The helpers below exist as the intended redaction seam for a future opt-in value-logging feature; that wiring is deferred until secured-bulk command variants are covered by the redactor's credential list (the `WriteSecuredBulk` / `WriteSecured2Bulk` gap), so enabling value logging cannot leak a secured-bulk payload. Until then, treat this section as describing the planned shape, not current behavior.
> **Redaction seam wired; value logging still has no opt-in knob** (updated 2026-08-07). Since commit `47c0b64` (2026-07-27), `RedactCommandValue` **is** wired: `GatewayLogRedactorSeam` (`Diagnostics/GatewayLogRedactorSeam.cs`) calls it for any non-null `CommandValue` log property, gated on `CommandMethod`, so a command payload that reaches a log event is masked on every sink through the shared `ILogRedactor` seam. What remains true from the original note: there is still no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob — the seam exposes no opt-in, `valueLoggingEnabled` is never passed `true`, so **every** non-null command value is redacted, credential-bearing or not. In practice no log statement currently emits a `CommandValue` property, but one that does can no longer leak a payload in the clear.
The intended `RedactCommandValue` would enforce the "values are opt-in and redacted by default" rule:
`RedactCommandValue` enforces the "values are opt-in and redacted by default" rule:
```csharp
public static object? RedactCommandValue(
+39 -16
View File
@@ -1,5 +1,18 @@
# Galaxy Repository Browse
> **Adopted as a shared package (2026-06-25, commit `8e196a7`).** The Galaxy browse
> implementation described in this document no longer lives in this repository. The
> gateway consumes the shared **`ZB.MOM.WW.GalaxyRepository`** library (pinned at
> `0.2.0` from the Gitea NuGet feed; sources hosted in the `scadaproj` repo), wired
> in `GatewayApplication.cs` via `AddZbGalaxyRepository` (registration) and
> `MapZbGalaxyRepository` (endpoint mapping). The former in-repo `Server/Galaxy/`
> classes and `Grpc/GalaxyRepositoryGrpcService.cs` / `Grpc/GalaxyProtoMapper.cs`
> were deleted with the adoption. The wire contract (`galaxy_repository.v1`) is
> unchanged, so the RPC behavior, filters, paging, caching, and snapshot semantics
> below still apply as documented — but any `src/ZB.MOM.WW.MxGateway.Server/...`
> file path cited below is historical; the equivalent class now lives in the
> shared library.
The gateway exposes a read-only browse surface over the AVEVA System Platform
Galaxy Repository (the SQL Server database named `ZB`). Clients use it to
enumerate the deployed object hierarchy and each object's attributes
@@ -107,7 +120,8 @@ server and dashboard views are consistent.
## Hierarchy Cache
The gateway holds a single shared `IGalaxyHierarchyCache`
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`)every
(`GalaxyHierarchyCache`, library-side in `ZB.MOM.WW.GalaxyRepository`formerly
`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
`DiscoverHierarchy` and `GetLastDeployTime` request reads from this cache
rather than hitting SQL. Many clients can browse concurrently with at most
one SQL query in flight.
@@ -149,8 +163,13 @@ working across that gap, the cache persists its dataset to disk:
- After every successful **heavy** refresh (a deploy change), the raw
hierarchy and attribute rowsets are written to
`MxGateway:Galaxy:SnapshotCachePath`
(default `C:\ProgramData\MxGateway\galaxy-snapshot.json`). The write is
`MxGateway:Galaxy:SnapshotCachePath`. `appsettings.json` no longer ships an
explicit value (SEC-33): the gateway supplies a `CommonApplicationData`-derived
default when the bound value is blank — `C:\ProgramData\MxGateway\galaxy-snapshot.json`
on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` (or the container
equivalent) elsewhere — and `GalaxyRepositoryOptionsValidator` rejects a
non-rooted or invalid path at startup when persistence is on, so the snapshot
can never land relative to the launch working directory. The write is
atomic — a temp file plus rename — so a crash mid-write cannot corrupt the
snapshot. Cheap no-change ticks write nothing; the file is already current.
- On the **first** refresh after startup, before any SQL runs, the cache
@@ -174,7 +193,8 @@ record: deleting it only forces the next cold start to wait for live SQL.
## Deploy Notifications
`WatchDeployEvents` is a server-streaming RPC backed by
`IGalaxyDeployNotifier` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`).
`IGalaxyDeployNotifier` (`GalaxyDeployNotifier`, library-side in
`ZB.MOM.WW.GalaxyRepository`).
The notifier maintains a private bounded channel per subscriber so a slow
client cannot back-pressure other subscribers or the publisher.
@@ -322,7 +342,7 @@ fields cannot express null. Use it to distinguish "no dimension reported" from
```text
gRPC client(s)
-> GalaxyRepositoryGrpcService (src/ZB.MOM.WW.MxGateway.Server/Grpc/)
-> Galaxy repository gRPC service (library-side; mapped via MapZbGalaxyRepository)
DiscoverHierarchy, GetLastDeployTime, BrowseChildren -> IGalaxyHierarchyCache.Current
WatchDeployEvents -> IGalaxyDeployNotifier
TestConnection -> GalaxyRepository (direct SQL)
@@ -341,41 +361,44 @@ GalaxyHierarchyRefreshService (BackgroundService)
-> IGalaxyDeployNotifier.Publish (only on deploy change)
```
Component breakdown:
Component breakdown (all of these classes are **library-side**, in the shared
`ZB.MOM.WW.GalaxyRepository` package — the in-repo files formerly at the paths
below were deleted in commit `8e196a7`):
- `GalaxyRepository` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
- `GalaxyRepository` (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
the SQL. Both `HierarchySql` and `AttributesSql` walk template-derivation and
package-derivation chains via recursive CTEs and pick the most-derived
override per object. `HierarchySql` still matches the OtOpcUa original;
`AttributesSql` does not — it additionally enumerates built-in primitive
attributes (see [Built-in vs configured attributes](#built-in-vs-configured-attributes)).
- `GalaxyHierarchyCache`
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
recent immutable `GalaxyHierarchyCacheEntry` (materialized objects +
precomputed dashboard summary + counts + status). All gRPC clients share the
same entry.
- `GalaxyHierarchyRefreshService`
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
hosted `BackgroundService` that drives `RefreshAsync` on the configured
interval, with deploy-time gating to avoid unnecessary heavy queries.
- `GalaxyDeployNotifier`
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
per-subscriber-channel fan-out for streaming clients.
- `GalaxyProtoMapper`
(`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
(formerly `src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
proto messages. Used by the cache during refresh to materialize the reply
once.
- `GalaxyBrowseProjector`
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
of children out of an immutable cache entry. Memoizes the filtered child list
per cache-entry instance so repeated paging is an O(pageSize) slice rather than an
O(siblings) filter scan. The memo is keyed on the cache entry reference, so a new
entry from the background refresh makes the stale memo unreachable and it is
collected with it. `DashboardBrowseService` wraps this projector to drive the
dashboard's lazy-expand tree.
- `GalaxyRepositoryGrpcService`
(`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`) implements
the five RPCs.
- The Galaxy repository gRPC service (formerly
`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`, deleted
with the adoption) implements the five RPCs; it is now supplied by the library
and mapped via `MapZbGalaxyRepository`.
## Configuration
@@ -386,7 +409,7 @@ Bound to `MxGateway:Galaxy` via `GalaxyRepositoryOptions`.
| `MxGateway:Galaxy:ConnectionString` | `Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;` | SQL Server connection string for the Galaxy Repository. Integrated Security against `localhost` is the dev default; production deployments should override this through the standard double-underscore environment variable form, e.g. `MxGateway__Galaxy__ConnectionString`. |
| `MxGateway:Galaxy:CommandTimeoutSeconds` | `60` | Per-command SQL timeout. Applies to all three RPCs. |
| `MxGateway:Galaxy:PersistSnapshot` | `true` | Persists each successful browse dataset to disk and reloads it at startup. See [On-disk snapshot](#on-disk-snapshot). |
| `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | File path for the persisted browse snapshot. Ignored when `PersistSnapshot` is `false`. |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted browse snapshot. Ignored when `PersistSnapshot` is `false`. `appsettings.json` no longer ships an explicit value (SEC-33): the gateway seeds the `CommonApplicationData`-derived default when the bound value is blank, and `GalaxyRepositoryOptionsValidator` enforces — when `PersistSnapshot` is `true` — that the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup. |
The connection string is not treated as a secret in dev (`Integrated
Security`), but production deployments that use SQL authentication should set
+39 -19
View File
@@ -14,7 +14,6 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
"MxGateway": {
"Authentication": {
"Mode": "ApiKey",
"SqlitePath": "C:\\ProgramData\\MxGateway\\gateway-auth.db",
"PepperSecretName": "MxGateway:ApiKeyPepper",
"RunMigrationsOnStartup": true
},
@@ -71,8 +70,7 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
"ConnectionString": "Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;",
"CommandTimeoutSeconds": 60,
"DashboardRefreshIntervalSeconds": 30,
"PersistSnapshot": true,
"SnapshotCachePath": "C:\\ProgramData\\MxGateway\\galaxy-snapshot.json"
"PersistSnapshot": true
},
"Alarms": {
"Enabled": false,
@@ -93,15 +91,17 @@ Environment variables use the normal .NET double-underscore form. For example,
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Authentication:Mode` | `ApiKey` | Selects public gRPC authentication. Supported values are `ApiKey` and `Disabled`. `Disabled` bypasses API-key verification and is for local development only. |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host; the production hosts pin the explicit Windows path in `appsettings.json`, which overrides the code default. |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host. `appsettings.json` no longer ships an explicit value (SEC-33): the removed Windows literal was byte-identical to the Windows code default, and a Windows-absolute literal is **not** rooted on a Unix host, so it would have resolved against the CWD there. Deployed hosts still override the path through the NSSM environment (`MxGateway__Authentication__SqlitePath`). |
| `MxGateway:Authentication:PepperSecretName` | `MxGateway:ApiKeyPepper` | Configuration key used to read the HMAC pepper for API-key secret hashing. The dashboard effective configuration redacts this value. |
| `MxGateway:Authentication:RunMigrationsOnStartup` | `true` | Runs SQLite auth schema migrations at gateway startup when API-key authentication is enabled. |
When `Mode` is `ApiKey`, `SqlitePath` and `PepperSecretName` must be present.
`SqlitePath` must be a valid filesystem path and must be **rooted** (absolute):
the validator rejects a non-rooted path so a relative override cannot silently
resolve against the working directory and scatter the credential store by launch
CWD (SEC-01).
`SqlitePath` must be a valid filesystem path and must be **rooted** (absolute)
**on the host running the gateway**: the validator uses `Path.IsPathRooted`
(current OS), so a bare filename — or a foreign-platform literal such as a
Windows `C:\...` path on a Unix host — fails fast at startup instead of silently
resolving against the working directory and scattering the credential store by
launch CWD (SEC-01, SEC-33).
## Worker Options
@@ -149,17 +149,21 @@ All numeric session options must be greater than zero.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. |
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. Gateway-side buffering per session is at most `3 ×` this value: the bounded worker event channel plus the read loop's staging channel, which is bounded at `2 ×` it. Overflow of either bound faults the session with `ProtocolViolation` and kills its worker. Must be between `1` and `int.MaxValue / 2` so the derived `2 ×` staging bound cannot overflow when a worker client is created. |
| `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. |
| `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. |
| `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. |
| `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. |
`QueueCapacity` must be greater than zero; it bounds each per-subscriber event
channel fed by the session's single event pump. A slow subscriber overflows only
its own channel and is always disconnected with an `EventQueueOverflow` fault
rather than silently dropping MXAccess events — the pump, the session, and other
subscribers are unaffected. With `FailFast` in the single-subscriber case (the
`QueueCapacity` must be greater than zero and no greater than `int.MaxValue / 2`
(the validator rejects a larger value so the derived `2 ×` staging bound cannot
throw `OverflowException` at session creation); it bounds each per-subscriber event
channel fed by the session's single event pump, and — at `2 ×` — the worker
read loop's event staging channel, so a consumer that drains slower than its
worker produces faults the session at a fixed ceiling instead of growing gateway
memory (GWC-24). A slow subscriber overflows only its own channel and is always
disconnected with an `EventQueueOverflow` fault rather than silently dropping
MXAccess events — the pump, the session, and other subscribers are unaffected. With `FailFast` in the single-subscriber case (the
default mode), that overflow additionally faults the whole session; with multiple
subscribers `FailFast` degrades to a per-subscriber disconnect, matching
`DisconnectSubscriber`, so one slow consumer cannot fault a session shared by
@@ -182,7 +186,7 @@ events (a "gap") and must re-snapshot; whatever is still retained is replayed.
| `MxGateway:Dashboard:SnapshotIntervalMilliseconds` | `1000` | Dashboard snapshot refresh interval used by the snapshot SignalR hub and the pages that subscribe to it. |
| `MxGateway:Dashboard:RecentFaultLimit` | `100` | Maximum number of fault summaries projected into each dashboard snapshot. |
| `MxGateway:Dashboard:RecentSessionLimit` | `200` | Maximum number of session summaries projected into each dashboard snapshot. |
| `MxGateway:Dashboard:ShowTagValues` | `false` | Reserved display control for tag values. The dashboard does not show full tag values by default. |
| `MxGateway:Dashboard:ShowTagValues` | `false` | Controls whether tag values reach the dashboard's SignalR events hub mirror. `false` (default): `DashboardEventBroadcaster` blanks tag values from a deep-cloned copy of each `MxEvent` before it reaches any hub subscriber — event metadata (tag reference, quality, status, timestamps) still renders; see `docs/GatewayDashboardDesign.md`'s `EventsHub` row for the mechanism. Security-relevant because the per-session hub ACL that would scope a Viewer to specific sessions does not exist yet: with no per-session scoping, this redaction is currently the only thing standing between a low-trust Viewer and other sessions' tag values, so setting this `true` exposes every session's tag values to every authenticated dashboard viewer. The flag gates only the SignalR hub mirror — it does **not** cover the `/browse` live-value display, which remains a separate, still-open residual. |
| `MxGateway:Dashboard:GroupToRole` | _(empty)_ | LDAP group → dashboard role mapping. Keys are LDAP group names (short CN or full DN — leading-RDN match). Values must be `Admin` (read/write, API-key CRUD) or `Viewer` (read-only). A user whose LDAP groups don't intersect this map cannot sign in; with no mapping at all, only the loopback bypass admits anyone. |
| `MxGateway:Dashboard:DisableLogin` | `false` | Dev/test only. When `true`, replaces the cookie authentication handler with `DashboardAutoLoginAuthenticationHandler`, which auto-authenticates every dashboard request — including requests from remote browsers, not just loopback — as `AutoLoginUser` holding both `Administrator` and `Viewer` roles. No login form, LDAP bind, or cookie is involved. A loud one-time startup warning is logged. Differs from `AllowAnonymousLocalhost`: `DisableLogin` mints a real authenticated principal (so role-gated write affordances appear), whereas `AllowAnonymousLocalhost` satisfies the authorization requirement on loopback only without minting a principal (write affordances stay hidden). Never enable in production. **Production hard-stop (SEC-04):** when the host runs in the `Production` environment and `DisableLogin` is `true`, startup validation fails and the process aborts — the flag is only accepted outside Production, where the one-time startup warning still fires. |
| `MxGateway:Dashboard:AutoLoginUser` | `(null)` | Username stamped on the synthetic principal when `DisableLogin` is `true`. Default `(null)` — a null or blank value falls back to `multi-role`. Has no effect when `DisableLogin` is `false`. |
@@ -255,6 +259,20 @@ When LDAP is enabled, `Server`, `SearchBase`, `ServiceAccountDn`,
must be in range. See `glauth.md` for the shared dev instance and the
dev→production hardening posture.
### Production hard-stops key on the exact environment name (SEC-35)
Both production hard-stops above — `Dashboard:DisableLogin` and the plaintext
`Ldap:Transport=None` guard — fire only when `IHostEnvironment.IsProduction()` is
true, i.e. `ASPNETCORE_ENVIRONMENT` is unset (it defaults to `Production`, which
covers the NSSM-deployed hosts) or is set to the exact string `Production`. This
is ASP.NET Core's environment-name convention. A host launched under any other
name — `Staging`, `Prod`, or a custom label — keeps the permissive dev posture
and these guards do **not** fire, by design (inverting to "anything but
Development is production-like" would refuse to boot a legitimate permissive
staging rig, e.g. one pointed at the plaintext shared GLAuth). A production-like
deployment must therefore run with the literal `Production` environment name for
the hard-stops to apply.
## Secrets Master Key
`${secret:...}` tokens in configuration — currently just
@@ -361,9 +379,11 @@ model requires otherwise.
| `MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` | `60` | Coalescing window, in seconds, for the `last_used_utc` write. The library verifier writes `last_used` on every successful verification; this bounds the write to at most one per key per window, so a hammered key does not churn the WAL. `0` forwards every write. Must be zero or greater. |
| `MxGateway:Security:LoginRateLimitPermitLimit` | `10` | Maximum `POST /auth/login` attempts permitted per remote IP within `LoginRateLimitWindowSeconds` before requests are rejected with HTTP 429. Throttles LDAP credential stuffing before the bind is relayed to the directory. Must be greater than zero. |
| `MxGateway:Security:LoginRateLimitWindowSeconds` | `60` | Fixed-window length, in seconds, for the per-IP login rate limit. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureLimit` | `10` | Failed API-key verifications, per peer, within `ApiKeyFailureWindowSeconds` that trip the in-process short-circuit. Once tripped, the gRPC auth path rejects further attempts with `ResourceExhausted` **before** the store read; a successful verification resets the peer's counter. The peer is keyed on the presented key id (falling back to the transport address) so a single abusive credential behind a shared NAT is throttled without locking out co-located clients. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureWindowSeconds` | `60` | Sliding-window length, in seconds, over which API-key verification failures are counted per peer. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct peers tracked by the failure counter (a bounded LRU) so a spray of unique peer keys cannot grow memory without limit. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureLimit` | `10` | Failed API-key verifications, per `(transport peer, key id)` partition, within `ApiKeyFailureWindowSeconds` that trip the in-process short-circuit. Once tripped, the gRPC auth path rejects further attempts from that partition with `ResourceExhausted` **before** the store read — except for the probe admitted every `ApiKeyFailureProbeIntervalSeconds` — and a successful verification resets the partition. The partition always includes the sender's transport address: key ids are public (they ride in every token and are listed on the dashboard), so keying on the key id alone let any peer deny a key to its legitimate holder. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureWindowSeconds` | `60` | Sliding-window length, in seconds, over which API-key verification failures are counted, for both the per-partition and the per-key-id aggregate layer. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureAggregateLimit` | `30` | Failed verifications for one key id counted across **all** transport peers within `ApiKeyFailureWindowSeconds` before that key id enters probe mode. This second layer bounds a distributed or source-rotating sprayer that never trips any single `(peer, key id)` partition. `0` disables the aggregate layer, leaving only per-partition counting. Must be zero or greater. |
| `MxGateway:Security:ApiKeyFailureProbeIntervalSeconds` | `5` | Minimum interval, in seconds, between probe admissions for an over-limit partition or key-id aggregate. An over-limit state is a valve rather than a wall: one request per interval reaches the real verifier — exactly one, even when a burst arrives together at the interval boundary — so the holder of the correct secret always gets through and clears the state, while everything else is still refused before the store read. `0` blocks absolutely instead — **not recommended**, because an unauthenticated peer can then deny the key to its holder for the whole window. Must be zero or greater. |
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
## Galaxy Options
@@ -373,7 +393,7 @@ model requires otherwise.
| `MxGateway:Galaxy:CommandTimeoutSeconds` | `60` | Per-command SQL timeout for all Galaxy browse RPCs. |
| `MxGateway:Galaxy:DashboardRefreshIntervalSeconds` | `30` | Interval between background refreshes of the dashboard Galaxy summary cache. SQL is hit at most once per interval regardless of dashboard render rate. |
| `MxGateway:Galaxy:PersistSnapshot` | `true` | Persists the latest successful Galaxy browse dataset to disk. When `true`, the cache reloads that snapshot at startup so clients can still browse last-known data while the Galaxy database is unreachable. The restored data is served with `Stale` status until a live query confirms it. |
| `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). Set an **absolute** path — this option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package (not by `GatewayOptions`), so the gateway validator does not enforce rooting on it; a relative value would resolve against the launch working directory (SEC-01). |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). `appsettings.json` no longer ships an explicit value (SEC-33): the option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package, so the gateway supplies the `CommonApplicationData`-derived default when the bound value is blank and registers `GalaxyRepositoryOptionsValidator` to enforce that — when `PersistSnapshot` is `true` — the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup instead of resolving against the launch working directory (SEC-01, SEC-33). |
See [Galaxy Repository Browse](./GalaxyRepository.md) for the RPC surface and
behavior.
+29 -9
View File
@@ -483,7 +483,7 @@ Internally it owns:
- write loop,
- event write loop,
- outbound command/control channel serialized by the write loop,
- unbounded event staging channel drained by the event write loop,
- bounded event staging channel drained by the event write loop,
- bounded inbound event channel,
- pending command dictionary keyed by correlation id,
- heartbeat monitor,
@@ -513,16 +513,36 @@ If the pipe closes while the session is not closing, fault the session.
The read loop never awaits event enqueue. Events are staged to the event write
loop with a non-blocking write, so a full inbound event channel (a slow or
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
applied by the event write loop, not the read loop.
delay a command reply or heartbeat (GWC-04). The timed backpressure window
(`EventChannelFullModeTimeout`) is applied by the event write loop, not the read
loop; the read loop's only event-path fault is the staging-bound rejection
described below, which uses the non-blocking `SetFaulted`.
### Event write loop
### Event write loop and the two overflow faults
The event write loop drains the staging channel and performs the timed write
into the bounded inbound event channel. When the inbound channel stays full past
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
same overflow contract as before, moved off the read loop.
The staging channel is bounded at `2 ×` the inbound event channel capacity, so
gateway-side buffering per session is at most `3 × MxGateway:Events:QueueCapacity`
— the fault fires as soon as *staging* is full, which is anywhere between `2 ×`
and `3 ×` depending on how much the event writer has already drained.
An unbounded staging channel would let a consumer that drains slower than the
worker produces grow gateway memory without limit and without any fault or
metric, because each individual timed write still completed inside the window
(GWC-24). Two distinct faults now bound the event path, both `ProtocolViolation`
and both killing the worker:
- **Full stall** — the event write loop's timed write into the bounded inbound
channel stays blocked past `EventChannelFullModeTimeout`. Recorded as
`QueueOverflow("worker-events")`. Catches a consumer that stopped entirely,
earlier than the staging bound would.
- **Sustained slow drain** — the read loop's staging `TryWrite` is rejected
because staging is full at its `2 ×` bound, meaning the writer has been
saturated for as long as the worker took to emit that many further events.
Recorded as `QueueOverflow("worker-event-staging")`. A rejected `TryWrite`
during shutdown (the staging channel is completed) stays a silent drop.
The worker event queue-depth gauge (`mxgateway.events.worker_queue.depth`) is
incremented at staging and decremented at consumer read, so it reports total
undelivered events across both channels rather than only the inbound channel.
### Write loop
+3 -1
View File
@@ -82,6 +82,8 @@ return mapper.MapCommandReply(workerReply);
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
### `StreamEvents`
`StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
@@ -247,7 +249,7 @@ StatusCode statusCode = exception.ErrorCode switch
};
```
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` becomes `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, and unmapped codes fall through to `Unavailable`.
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` becomes `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`.
## Event Streaming Model
+3 -2
View File
@@ -72,7 +72,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
|------------|--------------|-------------|
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Last reported depth of the worker-side event queue. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24). Incremented when the read loop stages an event, decremented when the consumer reads it, so a backlog stuck in the staging channel is visible rather than invisible. |
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
## Snapshot Shape
@@ -146,8 +146,9 @@ _metrics.RemoveSessionEvents(session.SessionId);
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
- `HeartbeatFailed(SessionId)` per missed heartbeat.
- `SetWorkerEventQueueDepth(queueDepth)` after each event ingest.
- `SetWorkerEventQueueDepth(queueDepth)` when the read loop stages an event and when the consumer reads one, so the gauge tracks staged + queued events.
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.
- `QueueOverflow("worker-events")` when the timed write into the bounded consumer channel exceeds `EventChannelFullModeTimeout`, and `QueueOverflow("worker-event-staging")` when the staging channel is full at its `2 × EventChannelCapacity` bound. The two labels distinguish a stalled consumer from one that merely drains too slowly; both fault the session with `ProtocolViolation`.
### Worker process launcher
+105 -17
View File
@@ -378,6 +378,20 @@ If event conversion throws, catch it inside the event handler, record a
structured `WorkerFault`, and keep the worker alive only if the fault policy
allows it.
The event drain loop streams queued events as `WorkerEvent` frames. A single
event whose envelope exceeds the negotiated frame maximum is **undeliverable end
to end** — the pipe maximum sits only the envelope-overhead reserve above the
public gRPC cap, so a frame the pipe rejects would also be rejected on the
client-facing stream. The session therefore faults on it rather than dropping it
(a silent drop makes the event stream unfaithful, and a synthesized placeholder
is barred by the no-synthesized-events rule), but the death is structured: the
worker logs the event's identity — family, handles, worker sequence, and sizes,
never the value — writes a `WorkerFault` with category `ProtocolViolation` and
command method `EventDrain` carrying the same identity, and only then exits.
Operator remediation is configuration: raise `MxGateway:Worker:MaxMessageBytes`
for that workload. Other per-frame rejection codes keep their previous behavior
because they indicate worker bugs, not workload size.
## Command Queue
The pipe reader converts `WorkerCommand` messages into `StaCommand` entries.
@@ -440,6 +454,29 @@ Diagnostics:
- `DrainEvents`
- `ShutdownWorker`
`DrainEvents` is answered on the message-loop thread, not the STA, and its reply
is bounded on **two** axes because no diagnostics command may be session-fatal:
- **Count**`GatewayContractInfo.MaxDrainEventsPerCommand` (10,000) is the
single home of the ceiling, shared by the gateway's request validator (which
rejects a larger `max_events` at the public boundary) and this worker clamp
(which also interprets `max_events = 0`, "as many as available").
- **Bytes** — the count cap alone is not sufficient: byte-heavy events (large
string or array `MxValue`s) overshoot the negotiated frame maximum long before
10,000 events. The drain is therefore byte-budgeted against the negotiated
maximum less a 64 KiB envelope/reply-wrapper reserve, and the size decision
happens inside the event queue's lock, so an event is dequeued only once it is
known to fit. An event that does not fit stays at the head of the queue and is
never lost.
Truncation is reported in the reply's existing `DiagnosticMessage`
("N events returned, M remain; repeat DrainEvents for the rest") rather than in a
new field, so the contract is unchanged and callers drain iteratively until a
reply comes back empty. In the degenerate case where the head event alone exceeds
the budget, the reply returns whatever fit before it (possibly nothing) and names
the blocked event's worker sequence so an operator can find the offending tag;
that event needs a larger `MxGateway:Worker:MaxMessageBytes` to move at all.
Implement method-specific dispatch instead of a generic string method invoker.
Parity tests need stable command-specific request and reply shapes.
@@ -601,27 +638,61 @@ Rules:
## Outbound Queues
The worker should use bounded outbound queues for replies, events, heartbeats,
and faults.
`WorkerFrameWriter` is a two-class cooperative priority scheduler, not the
five-level queue an earlier design draft called for. Every outbound frame is
enqueued at one of two `WorkerFrameWritePriority` values:
Priority order when writing:
- **Control** — hello, ready, command replies, faults, heartbeats, shutdown
acknowledgements.
- **Event** — MXAccess events drained from the queue.
1. faults,
2. command replies,
3. shutdown acknowledgements,
4. heartbeats,
5. events.
A caller enqueues its frame under a lock, then contends for a single write
lock; whichever caller wins drains **every** currently-queued frame before
releasing the lock, so a reply, fault, or heartbeat enqueued while a large
event backlog is draining still gets written on the very next drain pass
rather than waiting behind it. Within a class the queue is strict FIFO — a
fault does not jump ahead of an already-queued heartbeat or reply. This
collapses the originally-specified five-level order (faults > replies >
shutdown acks > heartbeats > events) into two classes; the decision is
deliberate, not a shortfall: the control queue is shallow in practice (faults
and shutdown acks are rare, replies and heartbeats are the steady traffic), so
the FIFO delay within it is bounded, and a two-class scheduler is simpler to
reason about and test than a five-level one for that same bound. See
`docs/WorkerFrameProtocol.md`'s "Write scheduling and sequencing" section for
the wire-level detail (sequence stamping, per-frame vs. stream-failure
semantics, flush coalescing).
Event overflow policy defaults to fail-fast for parity testing. If the event
queue fills:
Event overflow policy is fail-fast, not "stop accepting new commands and let
the gateway close or kill the worker." When `MxAccessEventQueue.Enqueue` finds
the queue full, it throws and the queue self-records a `WorkerFault` with
category `QueueOverflow`. The event drain loop's next pass observes the fault
through `DrainFault()`, writes the fault frame — a Control-priority frame, so
it is not stuck behind a queued event backlog — and then throws to unwind
`RunAsync`: the worker process exits rather than continuing in a state where
events are being silently lost. The exit currently uses the generic
`WorkerExitCode.UnexpectedFailure` rather than a dedicated overflow code; a
distinct exit code remains open (tracked separately). Do not drop or coalesce
events to avoid this exit — that is explicitly out of scope for v1. Production
coalescing may be added later, but it must be explicit and tested.
1. Capture overflow metrics.
2. Send `WorkerFault` if possible.
3. Stop accepting new commands.
4. Let the gateway close or kill the worker.
Production coalescing may be added later, but it must be explicit and tested.
Do not drop or coalesce events in v1.
The gateway side of the event path is bounded to match. `WorkerClient` buffers
inbound events in a bounded consumer channel plus a staging channel bounded at
`2 ×` that capacity, so a session holds at most three times
`MxGateway:Events:QueueCapacity` undelivered events before it faults with
`ProtocolViolation` and kills this worker. Two faults cover the two failure
shapes — a consumer that stops entirely (the timed write past
`EventChannelFullModeTimeout`, metric `QueueOverflow("worker-events")`) and a
consumer that merely drains slower than this worker produces (the staging bound,
metric `QueueOverflow("worker-event-staging")`). See
`docs/GatewayProcessDesign.md`. A worker that outruns its consumer therefore
dies loudly rather than growing gateway memory silently.
No control reply is session-fatal on size. Reply builders size their payloads
against the negotiated frame maximum, and the two reply-write seams (the
control-command path and the STA command path) additionally catch a
`MessageTooLarge` per-frame rejection and answer that correlation with a small
`InvalidRequest` reply instead of unwinding the session. Oversized *event*
frames keep the opposite policy — see Event Sink — because an event above the
frame maximum cannot be delivered to the client at all.
## Heartbeat And Watchdog
@@ -676,6 +747,23 @@ heartbeat fields until dedicated thresholds own those warnings. The worker
reports stale STA activity, but the gateway owns the final kill decision
through its existing heartbeat and worker lifecycle policy.
The alarm poll runs outside the command dispatcher — `RunAlarmPollLoopAsync`
invokes `PollOnce` directly on the STA rather than through
`StaCommandDispatcher`, so it does not inflate `PendingCommandCount` or perturb
dispatch ordering for real gateway commands. Because it is not a dispatched
command it has no `CurrentCommandCorrelationId`, so a healthy-but-slow poll (a
large `GetXmlCurrentAlarms2` against a busy provider) blocking the STA past
`HeartbeatGrace` would otherwise fault a healthy session at 15 s while a
dispatched command gets the 75 s ceiling. To close that asymmetry (WRK-27) the
poll advertises itself on the heartbeat snapshot's `StaCallInProgress` flag —
set on the STA thread for exactly the span of the COM call — and the watchdog
suppression honors that flag alongside `CurrentCommandCorrelationId`. The poll
therefore receives the same grace-to-ceiling treatment as a dispatched command:
suppressed up to `HeartbeatStuckCeiling`, faulted past it (a poll that blocks
the STA more than 75 s without pumping *should* fault — that is the ceiling's
contract). The flag is named generically so any future non-dispatcher STA work
reuses it.
The in-flight-command suppression itself is bounded by
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
`HeartbeatGrace`). The motivating case for the suppression is a legitimately
+10 -2
View File
@@ -197,7 +197,13 @@ Event streaming uses `AttachEventSubscriber` which returns a disposable lease. W
`FailFast` event backpressure faults the whole session only in single-subscriber mode; in multi-subscriber mode it degrades to a per-subscriber disconnect so one slow consumer never faults a session shared by others. The session passes its mode to the `SessionEventDistributor` at construction, so this decision is made on the fixed mode rather than a live subscriber-count snapshot.
The single worker event channel has exactly one direct reader: the `SessionEventDistributor` pump (`MapWorkerEventsAsync`). Both gateway-owned internal consumers — the dashboard mirror and the central alarm monitor — attach as distributor subscribers rather than draining the worker channel themselves. `GatewaySession.AttachInternalEventSubscriber` mirrors the dashboard-mirror lease (`isInternal: true`): the alarm monitor's `SessionManager.ReadAlarmEventsAsync` registers one so it consumes the same mapped `MxEvent`s the pump fans to every subscriber, without counting against `MaxEventSubscribersPerSession` and without a slow reconcile faulting the session. This is what keeps the alarm feed and the dashboard from splitting the stream between two raw drains (which would silently lose Acknowledge and provider-mode transitions); the worker channel is single-reader and a second `WorkerClient.ReadEventsAsync` consumer throws so a regression fails loudly.
The single worker event channel has exactly one direct reader: the `SessionEventDistributor` pump (`MapWorkerEventsAsync`). Both gateway-owned internal consumers — the dashboard mirror and the central alarm monitor — attach as distributor subscribers rather than draining the worker channel themselves. `GatewaySession.AttachInternalEventSubscriber` mirrors the dashboard-mirror lease (`isInternal: true`): the alarm monitor calls it directly on its session so it consumes the same mapped `MxEvent`s the pump fans to every subscriber, without counting against `MaxEventSubscribersPerSession` and without a slow reconcile faulting the session. This is what keeps the alarm feed and the dashboard from splitting the stream between two raw drains (which would silently lose Acknowledge and provider-mode transitions); the worker channel is single-reader and a second `WorkerClient.ReadEventsAsync` consumer throws so a regression fails loudly.
The monitor takes that lease **before** it issues `SubscribeAlarms`, so no transition window is missed: the pump has been running since `MarkReady` started the dashboard mirror, and the distributor only fans to subscribers registered at fan-out time, so a lease taken after the subscribe + first-reconcile round trips would lose every transition raised inside that window. Transitions arriving while the monitor subscribes and reconciles buffer in the lease's bounded channel and are applied after the reconcile, which is order-safe because a live transition can update an alarm the reconciled snapshot already holds.
The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above.
`AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
@@ -225,12 +231,14 @@ The handoff is sealed by a watermark. `RegisterWithReplay` returns `LiveResumeSe
Emit order on a resumed stream:
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the oldest still-retained sequence). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the resume anchor described below). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
2. **Retained replay batch.** The still-retained events newer than the requested watermark, in ascending `worker_sequence` order.
3. **Live events**, resuming strictly after `LiveResumeSequence`.
When `after_worker_sequence` is inside the retained window (nothing was evicted), step 1 is skipped: the stream replays the retained tail then resumes live with no sentinel.
**`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted.
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count) and `ReplayRetentionSeconds` (age); see [Configuration](GatewayConfiguration.md).
### Close
+105
View File
@@ -29,6 +29,40 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
means "use the worker's built-in default". This keeps both ends framing to the
same limit rather than depending on matched compile-time constants.
The worker accepts a negotiated value in the closed range [1024, 256 MiB]
(`MinNegotiableFrameBytes` .. `MaxNegotiableFrameBytes`); 0 keeps the default.
A value outside that range is rejected at the handshake with a fault frame
rather than adopted, because a nonsensical maximum — a gateway bug or a
foreign/old peer — would otherwise leave a session that handshakes cleanly and
then fails every subsequent frame with per-frame size errors, the worst
diagnostic shape for an operator. The 1024-byte floor matches the gateway's own
`GatewayOptionsValidator.MinimumMaxMessageBytes`, so the worker never rejects a
value the gateway's validator accepts as legal configuration, and 1024 still
guarantees hellos, heartbeats, acks, and faults fit.
Every worker-to-gateway frame must serialize within this limit, control replies
included, so reply builders truncate to fit rather than emit a frame the writer
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
truncation in the reply's `DiagnosticMessage`; the caller contract is to repeat
`DrainEvents` until it returns an empty reply. Should a reply still overshoot,
`MessageTooLarge` at the reply-write seam is answered with a small
`InvalidRequest` reply for that correlation, not with session teardown — no
diagnostics command may kill a session.
An oversized *event* frame is the deliberate exception. Such an event is
undeliverable end to end (the pipe maximum sits only the envelope-overhead
reserve above the public gRPC cap), so the session faults: the worker logs the
event's identity and sizes — never its value — writes a `WorkerFault` with
category `ProtocolViolation` and command method `EventDrain`, then exits.
Remediation is raising `MxGateway:Worker:MaxMessageBytes` for that workload.
A per-frame rejection does not consume an envelope `sequence`. The writer stamps
a candidate sequence, runs the empty-payload and size checks against the stamped
envelope, and commits the counter only immediately before the stream write, so
the sequences observed on the wire stay contiguous across rejections and an
operator reading a pipe capture never sees a phantom gap.
## Envelope Validation
`WorkerFrameReader` and `WorkerFrameWriter` validate each envelope against the
@@ -42,6 +76,77 @@ Protocol violations throw `WorkerFrameProtocolException` with a
`WorkerFrameProtocolErrorCode` so callers can distinguish malformed frames,
oversized frames, protocol version mismatches, and session mismatches.
## Write Scheduling And Sequencing
This section covers write scheduling (priority classes, enqueue-then-contend,
flush coalescing) and sequencing (write-time stamping) together, because both
are properties of the same single write lock.
`WorkerFrameWriter` is a two-class cooperative priority scheduler
(`WorkerFrameWritePriority.Control` and `.Event`), not a strict per-kind
priority order. A caller enqueues its frame into the control or event queue
under a lock, then contends for a single write lock; whichever caller wins
drains every frame queued at that moment, control frames first and each class
in FIFO order, so a command reply, fault, heartbeat, or shutdown
acknowledgement is never delayed behind a backlog of queued events. Priority
only reorders *which frame writes next* — it does not affect the sequence
value a frame receives (see below), so a caller cannot infer priority class
from the wire sequence.
The envelope `Sequence` is stamped by the draining lock-holder at the actual
moment of writing, not when the frame is enqueued, so the on-wire order and
the stamped sequence always agree regardless of caller concurrency or
priority reordering. Stamping uses peek-stamp-commit: a candidate sequence is
assigned and the frame is validated (size, non-empty payload) against that
stamped value, but the counter is committed only immediately before the
stream write. A per-frame rejection therefore leaves the counter untouched —
the next accepted frame reuses the candidate number, so the wire sequence
stays contiguous across rejections and an operator reading a pipe capture
never sees a phantom gap from a rejected frame.
Two failure shapes are distinguished during a drain pass:
- **Per-frame rejection** (`InvalidEnvelope`, `MessageTooLarge`,
`ProtocolVersionMismatch`, `SessionMismatch`) is specific to the one frame
that failed validation or sizing. Nothing was written for it, so it fails
only that frame's completion and draining continues with the next queued
frame.
- **Stream failure** (anything else — a broken pipe, an I/O error) means the
underlying stream itself is no longer trustworthy. It fails the frame that
triggered it, every frame already written this batch but not yet flushed,
and every frame still queued, then stops draining entirely so no caller
waits forever on a stream that will not recover.
Flushes are coalesced across a drained batch: each frame in the batch is
written to the stream without an individual flush, then one `FlushAsync`
runs after the whole batch, and only then does every successfully-written
frame's completion resolve — so a caller's `WriteAsync` still does not
complete until its bytes are both written *and* flushed, but a batch that
happened to contain several queued frames pays one flush instead of one per
frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
every frame under one `_gate` acquisition, takes the write lock once, and
drains them together, so a burst of N events costs one flush rather than N —
the coalescing the batch machinery was built for now engages on the event hot
path, not only when independent producers happen to queue behind a blocked
write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
concurrently queued control frame is still drained ahead of the batch. A
per-frame rejection inside a batch (for example one oversized event) surfaces
from the batch's awaited completions as that frame's
`WorkerFrameProtocolException`; the remaining completions are still observed
so none faults unobserved.
Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
for the write lock when its token fires tombstones the queued frame: the
cancelled caller marks its frame under `_gate`, and the draining lock-holder's
`DequeueNext` skips any tombstoned frame, so a cancelled call is guaranteed
never to reach the wire — *unless* a lock-holder has already claimed the frame
to write it. Claiming and cancelling are interlocked under `_gate`, so exactly
one wins; a frame already claimed is mid-write and can no longer be recalled,
so the caller observes `OperationCanceledException` while that one frame still
reaches the wire. That residual window is by design: blocking the canceller
behind the very write it is abandoning would defeat the point of cancellation.
## Verification
The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`,
+3 -1
View File
@@ -5,7 +5,9 @@ library, CLI, and tests.
## Baseline
- Target the Java version defined by the client build, with Java 21 preferred.
- Target Java 17 (the Ignition 8.3 baseline; the client build enforces
`options.release = 17` with a Gradle toolchain 17). Code must compile and
run on 17; newer JDKs may host the build.
- Use Gradle unless the repository standardizes on Maven.
- Apply a formatter such as Spotless or Google Java Format when configured.
- Keep generated protobuf code separate from handwritten wrappers.
+38 -7
View File
@@ -154,6 +154,29 @@ session. The worker event channel is single-reader and asserts it (a second
`WorkerClient.ReadEventsAsync` consumer throws), so a regression cannot silently
split the event stream between two drains.
The monitor takes that internal lease **before** it sends `SubscribeAlarms`, and
drains it after the first reconcile. The pump is already running by then (the
dashboard mirror starts it at `MarkReady`) and the distributor fans only to
subscribers registered at fan-out time, so attaching after the subscribe +
reconcile round trips would drop every transition raised in that window —
including an `Acknowledge`, which the presence-only reconcile deltas would never
repair. Transitions arriving during the window buffer in the lease's bounded
channel instead. As a backstop for any window this ordering cannot cover (worker
restart, internal-subscriber overflow disconnect), a reconcile that finds a known
alarm now reported `ActiveAcked` broadcasts an `Acknowledge` transition on the
alarm feed. That is a feed-level repair rebuilt from the worker's own snapshot on
the `StreamAlarms` surface — it is not an `MxEvent` and never reaches
`StreamEvents`, so the "never synthesize events" rule is untouched.
**Feed repair transitions are at-least-once, not exactly-once.** A reconcile reads
the worker's current state while the corresponding live transition may still be
buffered in the monitor's lease, so both can broadcast and the two are
indistinguishable on the feed. This applies to the acked-state delta and equally
to the older Raise/Clear presence repair: nothing serializes a reconcile pass
against the in-flight live stream. Alarm-feed consumers (`StreamAlarms` clients
and the dashboard alarm hub) must apply transitions idempotently — treat one as
"set this alarm to this state", never as an increment or a toggle.
### Alarm providers and failover
The alarm feed has two providers, both implemented worker-side:
@@ -325,9 +348,13 @@ messages, tagged from 10 upward:
Rules:
- `sequence` is a monotonic per-sender counter used as a diagnostic aid; it is
not validated for gaps or ordering on receive (the named pipe already
guarantees FIFO delivery).
- `sequence` is a monotonic per-sender counter used as a diagnostic aid. Both
sides stamp it at the point of writing, inside their single write path — the
gateway on its outbound-channel write loop, the worker on its own writer — so
the numbers stay monotonic in wire order no matter how concurrent callers
interleave while building envelopes. It is not validated for gaps or ordering
on receive (the named pipe already guarantees FIFO delivery); if inbound
enforcement is ever added, this is the property it will rely on.
- `correlation_id` links a command to its reply; it is authoritative on the
envelope, and the inner `MxCommandReply.correlation_id` echoes it for
MXAccess parity.
@@ -447,10 +474,14 @@ Optional diagnostics:
- `Ping`
- `GetSessionState`
- `GetWorkerInfo`
- `DrainEvents` — diagnostic; `max_events` is bounded (the gateway rejects requests
above a public ceiling, and the worker caps each reply at its own per-reply limit,
treating `max_events = 0` as "the default cap") so one drain cannot pack an
unbounded, session-killing reply frame.
- `DrainEvents` — diagnostic; the reply is bounded on two axes so one drain cannot
pack an unbounded, session-killing reply frame. By **count**: the gateway rejects
requests above the shared ceiling `GatewayContractInfo.MaxDrainEventsPerCommand`
and the worker clamps to the same value, treating `max_events = 0` as "the default
cap". By **bytes**: the worker sizes the reply while draining, against the
negotiated worker-frame maximum, so a byte-heavy queue truncates instead of
overshooting and events that do not fit stay queued. Truncation is reported in the
reply's `DiagnosticMessage`; callers drain iteratively until an empty reply.
- `ShutdownWorker`
Do not compress MXAccess semantics into generic verbs too early. A command enum
-95
View File
@@ -1,95 +0,0 @@
# Saved Task List — Session Resilience Epic
> Snapshot taken 2026-06-16, before switching to the dashboard disable-login feature.
> This is the in-flight epic from `docs/plans/2026-06-15-session-resilience.md`.
## How to resume
```
/superpowers-extended-cc:executing-plans docs/plans/2026-06-15-session-resilience.md
```
The authoritative resume state lives in
`docs/plans/2026-06-15-session-resilience.md.tasks.json` (tasks 112 completed,
1328 pending). This file is just a human-readable mirror.
## Status
**Governance update 2026-07-09 (archreview TST-04).** The epic is resolved into three
decisions rather than one open backlog:
- **Phase 3 (reconnect) — finishing now, essentially complete.** Task 13 (owner
re-validation) shipped as archreview **TST-02** (P0, session attach is owner-scoped,
see CLAUDE.md Authentication). Task 15 (reconnect integration test) shipped as
**TST-01** (`GatewayEndToEndReconnectReplayTests`). Task 14 (client `ReplayGap`
handling) shipped as **CLI-15** for four of five clients (.NET/Go/Rust/Python);
the Java client is the only remainder, batched to the windev build host.
- **Phase 4 (per-session dashboard ACL) — scoped, not yet built.** Tracked as archreview
**TST-15**. The previously-open Viewer-default decision is **settled**: admin-sees-all,
Viewer strictly scoped to sessions it owns/is granted — matching TST-02's gRPC owner
binding for consistency.
- **Phase 5 (orphan-worker reattach) — DEFERRED, not planned.** It reverses the CLAUDE.md
invariant "Gateway restart does not reattach orphan workers" and adds an adoption manifest
store + worker phone-home protocol. It stays deferred unless a concrete requirement
appears. **The `EnableOrphanReattach` flag (Task 26) does not exist and must not be
referenced anywhere as if it does** until that task actually lands.
Original snapshot (historical): **12 of 28 tasks complete** (Phases 12 + reconnect core of
Phase 3), merged to `main` (commit `c446bef`).
### Completed — Phase 1 (Foundation)
- ✅ Task 1 (#108): Add OwnerKeyId to the session
- ✅ Task 2 (#109): SessionEventDistributor skeleton
- ✅ Task 3 (#110): Bounded replay ring buffer
- ✅ Task 4 (#111): Rewire AttachEventSubscriber + EventStreamService onto distributor
- ✅ Task 5 (#112): Per-subscriber backpressure isolation
- ✅ Task 6 (#113): Dashboard broadcaster becomes a distributor subscriber
### Completed — Phase 2 (Multi-subscriber fan-out)
- ✅ Task 7 (#114): Remove validator block + add subscriber cap option
- ✅ Task 8 (#115): Subscriber-lease collection + cap enforcement
- ✅ Task 9 (#116): Multi-subscriber end-to-end test (FakeWorkerHarness)
### Completed — Phase 3 (Reconnect core)
- ✅ Task 10 (#117): Proto — ReplayGap signal
- ✅ Task 11 (#118): Detach-grace session retention
- ✅ Task 12 (#119): Replay-on-reconnect + emit ReplayGap
### Phase 3 finish — DONE (via archreview P0/P2)
- ✅ Task 13 (#120): Owner re-validation on reconnect — shipped as **TST-02** (P0).
- 🔄 Task 14 (#121): Client ReplayGap handling — shipped as **CLI-15** for 4/5 clients
(.NET/Go/Rust/Python); Java pending (windev batch). Per-language presence-check idiom
for `optional` message fields carried in each client's surface.
- ✅ Task 15 (#122): Reconnect integration test (fake worker) — shipped as **TST-01**.
### Phase 4 (Per-session dashboard ACL) — SCOPED, tracked as archreview TST-15
- ⏳ Task 16 (#123): gRPC session-owner gate + all-sessions admin scope — blockedBy 9, 1
- Note: the gRPC owner gate itself already exists (TST-02); Phase 4 adds the admin
all-sessions scope + the dashboard-side twin.
- ⏳ Task 17 (#124): Session Tag + dashboard group-to-tag config — blockedBy 9
- ⏳ Task 18 (#125): EventsHub per-session ACL + hub-token tag claim — blockedBy 17
- Decision SETTLED: admin-sees-all, Viewer strictly scoped to owned/granted sessions
(matches TST-02 gRPC owner binding).
- ⏳ Task 19 (#126): ACL tests incl. live LDAP users — blockedBy 18
### Phase 5 (Orphan-worker reattach) — DEFERRED, NOT PLANNED
Deferred unless a concrete requirement appears. It reverses the CLAUDE.md invariant
"Gateway restart does not reattach orphan workers" and adds an adoption manifest store +
worker phone-home protocol. **`EnableOrphanReattach` (Task 26) does not exist** — do not
reference it as if it does until the task lands.
- 🚫 Task 20 (#127): Stable gateway-instance id + stable pipe naming
- 🚫 Task 21 (#128): Adoption manifest store (SQLite)
- 🚫 Task 22 (#129): Proto — worker adopt/reconnect frame
- 🚫 Task 23 (#130): Worker phone-home reconnect loop + self-terminate (net48/x86, windev)
- 🚫 Task 24 (#131): Gateway adoption — re-open pipes, nonce-validate, reject impostors
- 🚫 Task 25 (#132): Resync adopted worker + ReplayGap to subscribers
- 🚫 Task 26 (#133): EnableOrphanReattach flag (default off) + terminator fallback
- 🚫 Task 27 (#134): Gateway-restart reattach round-trip (WINDEV + live worker)
- 🚫 Task 28 (#135): Documented-rule reversals + stillpending refresh
## Notes
- Phase 5 was designed to reverse the "Gateway restart does not reattach orphan workers"
rule (CLAUDE.md), but is now **deferred, not planned** (TST-04) — the invariant stands.
- Two deferred follow-ups noted earlier: dashboard visibility of `DetachedAtUtc` on
`DashboardSessionSummary`.
- Worker (net48/x86) tasks build/test on windev; everything else builds on macOS.
@@ -17,6 +17,21 @@ public static class GatewayContractInfo
/// <summary>Default backend name identifying the MXAccess worker process type.</summary>
public const string DefaultBackendName = "mxaccess-worker";
/// <summary>
/// Ceiling on how many events one <c>DrainEvents</c> command may move in a single reply.
/// Shared so the gateway's request-validation ceiling
/// (<c>MxAccessGrpcRequestValidator</c>, which rejects a larger <c>max_events</c> loudly at
/// the public boundary) and the worker's per-reply clamp
/// (<c>WorkerPipeSession.CreateDrainEventsReply</c>, the backstop that also interprets
/// <c>max_events = 0</c>) cannot drift apart. A count cap alone is necessary but not
/// sufficient: the worker additionally caps the reply by serialized bytes against the
/// negotiated frame maximum (WRK-21), so a reply may carry fewer events than this ceiling
/// and fewer than are queued. Callers drain iteratively until an empty reply.
/// This is a documented behavioral bound, not wire schema — it is deliberately a C#
/// constant and not a <c>.proto</c> field.
/// </summary>
public const uint MaxDrainEventsPerCommand = 10_000;
/// <summary>
/// Environment variable name that opts an xUnit suite into running live
/// MXAccess COM tests. Single source of truth shared by both
@@ -210,6 +210,17 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
try
{
// Attach the internal distributor subscriber BEFORE subscribing (GWC-26). The pump
// has been running since MarkReady started the dashboard mirror, and the distributor
// only fans to subscribers registered at the time of the fan-out, so a subscriber
// taken after SubscribeAlarms + the first reconcile would silently lose every
// transition raised inside that two-round-trip window — and a missed Acknowledge is
// never repaired by the presence-only reconcile deltas. Transitions arriving while we
// subscribe and reconcile simply buffer in this lease's bounded channel; if it ever
// overflowed, the internal subscriber is disconnected (it never faults the session),
// the enumeration below ends, and the supervisor loop restarts the lifecycle.
using IEventSubscriberLease alarmLease = session.AttachInternalEventSubscriber();
await SubscribeAlarmsAsync(session.SessionId, subscription, stoppingToken).ConfigureAwait(false);
await ReconcileAsync(session.SessionId, stoppingToken).ConfigureAwait(false);
@@ -228,9 +239,11 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
// Consume mapped MxEvents through the session's single distributor pump (as an
// internal, non-counted subscriber) rather than opening a second raw drain of the
// worker event channel — a second drain would split events with the dashboard
// mirror pump and silently lose Acknowledge/mode-change transitions.
await foreach (MxEvent mxEvent in _sessionManager
.ReadAlarmEventsAsync(session.SessionId, linked.Token)
// mirror pump and silently lose Acknowledge/mode-change transitions. The lease was
// taken above, before SubscribeAlarms; draining it only now is order-safe because
// ApplyTransition handles alarms the reconcile already placed in the cache.
await foreach (MxEvent mxEvent in alarmLease.Reader
.ReadAllAsync(linked.Token)
.ConfigureAwait(false))
{
if (mxEvent is { BodyCase: MxEvent.BodyOneofCase.OnAlarmTransition }
@@ -511,6 +524,20 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
// Replaces the cache with the worker's authoritative snapshot, broadcasting
// a synthetic transition for any alarm the live stream missed.
//
// Repair scope (GWC-26): presence deltas (Clear/Raise) plus acked-state deltas. These are
// ALARM FEED transitions (AlarmFeedMessage on the StreamAlarms/dashboard surface), rebuilt
// from the worker's own authoritative snapshot to repair what the live feed missed. They are
// not MxEvents and never reach the gRPC StreamEvents path, so this feed-level repair does not
// breach the "never synthesize events" rule, which governs MxEvent emission.
//
// Delivery semantics: feed repair transitions are AT-LEAST-ONCE, not exactly-once. A reconcile
// reads the worker's current state while the corresponding live transition may still be
// buffered in the alarm lease's channel; both then broadcast, and the two are indistinguishable
// on the feed. This is inherent to the reconcile design and pre-dates the acked-state delta
// (the Raise/Clear repair has always had it), since nothing serializes a reconcile against the
// in-flight live stream. Consumers must therefore treat alarm state idempotently — apply a
// transition as "set the alarm to this state", never as an increment or a toggle.
private void ApplyReconcile(IEnumerable<ActiveAlarmSnapshot> snapshots)
{
Dictionary<string, ActiveAlarmSnapshot> next = new(StringComparer.Ordinal);
@@ -536,12 +563,23 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
foreach (KeyValuePair<string, ActiveAlarmSnapshot> incoming in next)
{
if (!_alarms.ContainsKey(incoming.Key))
if (!_alarms.TryGetValue(incoming.Key, out ActiveAlarmSnapshot? existing))
{
Broadcast(
new AlarmFeedMessage { Transition = TransitionFromSnapshot(incoming.Value, AlarmTransitionKind.Raise) },
incoming.Key);
}
else if (existing.CurrentState != incoming.Value.CurrentState
&& incoming.Value.CurrentState == AlarmConditionState.ActiveAcked)
{
// The alarm was already known but the worker now reports it acknowledged: the
// live Acknowledge transition never reached the feed. Without this the acked
// state is absorbed silently by the snapshot replace below and subscribers show
// the alarm unacked until it clears.
Broadcast(
new AlarmFeedMessage { Transition = TransitionFromSnapshot(incoming.Value, AlarmTransitionKind.Acknowledge) },
incoming.Key);
}
}
_alarms.Clear();
@@ -0,0 +1,41 @@
using ZB.MOM.WW.Configuration;
using ZB.MOM.WW.GalaxyRepository;
namespace ZB.MOM.WW.MxGateway.Server.Configuration;
/// <summary>
/// Gateway-side startup validation for the shared <see cref="GalaxyRepositoryOptions"/>. The
/// <c>ZB.MOM.WW.GalaxyRepository</c> package binds the options but deliberately ships no validator
/// (see <c>A2-galaxyrepository-adoption-handoff.md</c>); the gateway owns the rule because it is the
/// process that writes the snapshot. When persistence is on, the snapshot path must be a valid,
/// rooted path on the running host for the same reason the auth DB path must be (SEC-33): a
/// non-rooted value silently resolves against the launch working directory.
/// </summary>
public sealed class GalaxyRepositoryOptionsValidator : OptionsValidatorBase<GalaxyRepositoryOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GalaxyRepositoryOptions options)
{
if (!options.PersistSnapshot)
{
// Persistence disabled: the snapshot path is never used, so nothing to validate.
return;
}
if (string.IsNullOrWhiteSpace(options.SnapshotCachePath))
{
builder.Add(
"MxGateway:Galaxy:SnapshotCachePath is required when MxGateway:Galaxy:PersistSnapshot is true.");
return;
}
GatewayConfigPathRules.AddIfInvalidPath(
options.SnapshotCachePath,
"MxGateway:Galaxy:SnapshotCachePath must be a valid filesystem path.",
builder);
GatewayConfigPathRules.AddIfNotRooted(
options.SnapshotCachePath,
"MxGateway:Galaxy:SnapshotCachePath must be an absolute (rooted) path so the Galaxy snapshot never lands in the launch working directory.",
builder);
}
}
@@ -0,0 +1,77 @@
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Configuration;
/// <summary>
/// Shared filesystem-path validation primitives used by more than one options validator
/// (<see cref="GatewayOptionsValidator"/> and <see cref="GalaxyRepositoryOptionsValidator"/>).
/// Both the auth credential store and the Galaxy snapshot are written by the running gateway
/// process, so both must reject paths the host cannot use — the rules live here once so the two
/// validators cannot drift.
/// </summary>
internal static class GatewayConfigPathRules
{
/// <summary>
/// Fails validation when <paramref name="value"/> is not an absolute (rooted) path <em>on the
/// host running the validator</em>. Security-sensitive paths (the auth DB, the self-signed
/// private key, the Galaxy snapshot) must be absolute: a non-rooted value silently resolves
/// against the launch working directory, so the store moves with the CWD and can leak into the
/// source tree. Rooting is checked with <see cref="Path.IsPathRooted(string)"/> — the current
/// OS — so a Windows drive/UNC literal on a Unix host fails fast at startup rather than being
/// blessed and then written as a junk-named relative file (the SEC-01/SEC-33 mechanism). Reject
/// rather than auto-root; silent relocation of a credential store is worse than a boot error.
/// Blank is handled by the caller's required-field check and is not treated as non-rooted here.
/// </summary>
/// <param name="value">The configured path value.</param>
/// <param name="message">The failure message to record when the value is not rooted.</param>
/// <param name="builder">The validation builder accumulating failures.</param>
public static void AddIfNotRooted(string? value, string message, ValidationBuilder builder)
{
if (string.IsNullOrWhiteSpace(value))
{
return;
}
if (!Path.IsPathRooted(value))
{
builder.Add(message);
}
}
/// <summary>
/// Fails validation when <paramref name="value"/> is non-blank but not a syntactically valid
/// filesystem path (as judged by <see cref="Path.GetFullPath(string)"/>). Blank values are the
/// caller's required-field concern and pass here.
/// </summary>
/// <param name="value">The configured path value.</param>
/// <param name="message">The failure message to record when the value is not a valid path.</param>
/// <param name="builder">The validation builder accumulating failures.</param>
public static void AddIfInvalidPath(string? value, string message, ValidationBuilder builder)
{
if (string.IsNullOrWhiteSpace(value))
{
return;
}
try
{
_ = Path.GetFullPath(value);
}
catch (ArgumentException)
{
builder.Add(message);
}
catch (NotSupportedException)
{
builder.Add(message);
}
catch (PathTooLongException)
{
builder.Add(message);
}
catch (IOException)
{
builder.Add(message);
}
}
}
@@ -87,6 +87,18 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
options.ApiKeyFailureTrackedPeers,
"MxGateway:Security:ApiKeyFailureTrackedPeers must be greater than zero.",
builder);
// The two-layer limiter knobs (SEC-31) accept 0 as "disable this layer": a zero aggregate
// limit turns off cross-peer counting, and a zero probe interval restores absolute blocking.
// Negatives express no intent.
AddIfNegative(
options.ApiKeyFailureAggregateLimit,
"MxGateway:Security:ApiKeyFailureAggregateLimit must be greater than or equal to zero (0 disables the per-key aggregate layer).",
builder);
AddIfNegative(
options.ApiKeyFailureProbeIntervalSeconds,
"MxGateway:Security:ApiKeyFailureProbeIntervalSeconds must be greater than or equal to zero (0 blocks absolutely instead of admitting probes).",
builder);
}
private static void ValidateAuthentication(AuthenticationOptions options, ValidationBuilder builder)
@@ -280,10 +292,23 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
// knob, not incorrect behavior.
}
// QueueCapacity flows into WorkerClient as EventChannelCapacity, where the staging channel is
// sized checked(2 * EventChannelCapacity) (GWC-24). Cap it at int.MaxValue/2 so that doubling
// cannot overflow and throw OverflowException at session creation; mirrors the MaxSparseArrayLength
// upper-bound pattern.
private const int MaximumEventQueueCapacity = int.MaxValue / 2;
private static void ValidateEvents(EventOptions options, ValidationBuilder builder)
{
AddIfNotPositive(options.QueueCapacity, "MxGateway:Events:QueueCapacity must be greater than zero.", builder);
if (options.QueueCapacity > MaximumEventQueueCapacity)
{
builder.Add(
$"MxGateway:Events:QueueCapacity must be less than or equal to {MaximumEventQueueCapacity} "
+ "so the derived worker event-staging channel (2 x QueueCapacity) cannot overflow.");
}
if (!Enum.IsDefined(options.BackpressurePolicy))
{
builder.Add("MxGateway:Events:BackpressurePolicy must be a supported backpressure policy.");
@@ -512,79 +537,13 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
builder.RequireThat(value >= 0, message);
}
// Path rooting/validity rules are shared with GalaxyRepositoryOptionsValidator (both write a
// host file) so the two validators cannot drift; see GatewayConfigPathRules. Rooting is checked
// against the running OS via Path.IsPathRooted — a Windows drive/UNC literal on a Unix host now
// fails fast instead of being blessed and written as a junk-named relative file (SEC-33).
private static void AddIfNotRooted(string? value, string message, ValidationBuilder builder)
{
// Security-sensitive paths (the auth DB, the self-signed private key) must be absolute:
// a non-rooted value silently resolves against the launch working directory, so the store
// moves with the CWD and can leak into the source tree. Reject rather than auto-root —
// silent relocation of a credential store is worse than a boot error. Blank is handled by
// AddIfBlank; an empty value is not treated as non-rooted here.
if (string.IsNullOrWhiteSpace(value))
{
return;
}
if (!IsRootedForAnyPlatform(value))
{
builder.Add(message);
}
}
/// <summary>
/// Determines whether <paramref name="value"/> is an absolute path for <em>any</em> platform,
/// not just the host running the validator. This matters on the macOS dev box, where the
/// production <c>appsettings.json</c> ships Windows-absolute paths (<c>C:\ProgramData\...</c>)
/// that <see cref="Path.IsPathRooted(string)"/> reports as non-rooted on Unix. The intent of the
/// rooting check is to reject bare filenames that resolve against the launch working directory,
/// so a valid Windows drive-qualified or UNC path must pass regardless of the current OS.
/// </summary>
private static bool IsRootedForAnyPlatform(string value)
{
// Rooted on the current OS (Unix "/...", or a Windows drive/UNC path when on Windows).
if (Path.IsPathRooted(value))
{
return true;
}
// Windows drive-qualified path ("C:\..." or "C:/...") checked on a non-Windows host.
if (value.Length >= 3
&& char.IsLetter(value[0])
&& value[1] == ':'
&& (value[2] == '\\' || value[2] == '/'))
{
return true;
}
// Windows UNC path ("\\server\share") checked on a non-Windows host.
return value.StartsWith(@"\\", StringComparison.Ordinal);
}
=> GatewayConfigPathRules.AddIfNotRooted(value, message, builder);
private static void AddIfInvalidPath(string? value, string message, ValidationBuilder builder)
{
if (string.IsNullOrWhiteSpace(value))
{
return;
}
try
{
_ = Path.GetFullPath(value);
}
catch (ArgumentException)
{
builder.Add(message);
}
catch (NotSupportedException)
{
builder.Add(message);
}
catch (PathTooLongException)
{
builder.Add(message);
}
catch (IOException)
{
builder.Add(message);
}
}
=> GatewayConfigPathRules.AddIfInvalidPath(value, message, builder);
}

Some files were not shown because too many files have changed in this diff Show More