Compare commits

..

169 Commits

Author SHA1 Message Date
Joseph Doherty 61c00d59a9 feat(gateway): readiness check for the worker executable (#206)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m27s
ci / java (push) Successful in 2m13s
ci / portable (push) Failing after 4m25s
Readiness covered only the auth store, so a gateway whose worker exe was
missing or of the wrong architecture reported ready while every OpenSession
failed. The new ready-tier check worker-executable applies the launcher's own
rule (path resolution, .exe, presence, PE architecture), now shared as
WorkerExecutableValidator.ResolveAndValidate, and reports only the launch
error code (the endpoint is anonymous). The session check stays on the
active tier, so zero sessions keeps readiness green.

The bounded Galaxy SQL check is not in this change: its tier, failure status
and probe cadence need a decision (see the issue).

Tests: WorkerExecutableHealthCheckTests (healthy, missing, wrong architecture,
and the ready-tier registration, which fails without the wiring).
2026-11-06 16:13:09 -05:00
Joseph Doherty 574369b666 feat(mxgw-go): add -require-certificate-validation for strict TLS (#199)
Over TLS without -ca-cert the CLI was always skip-verify, with no way to ask
for verification against the system roots even though the library supports
it. The new common flag is wired to Options.RequireCertificateValidation in
both the gateway and Galaxy dialers and echoed in the redacted JSON options.

Test: tls_flag_test.go starts a self-signed TLS gRPC gateway; the default
open-session succeeds (positive control) and the same call with the flag
fails to dial (fails on the old code with 'flag provided but not defined').
2026-11-06 16:10:39 -05:00
Joseph Doherty 9351ec1d2c feat(client-go): typed Unauthenticated vs PermissionDenied errors (#198)
Every RPC failure surfaced as a GatewayError with no way to tell a rejected
key from a missing scope, though docs/ClientLibrariesDesign.md requires the
two be treated distinctly. GatewayError now matches the new sentinels
ErrUnauthenticated and ErrPermissionDenied via errors.Is (by gRPC status code
only, so it works through the secret-redaction wrapper) and exposes Code().

Tests: errors_auth_test.go — sentinel matching per code, Code(), and a bufconn
AcknowledgeAlarm round trip proving the match survives the wire and the
password stays redacted (both behavioural tests fail without GatewayError.Is).
2026-11-06 16:08:51 -05:00
Joseph Doherty b9096f7c57 fix(client-dotnet): size the safe-unary retry budget for every attempt (#197)
ExecuteSafeUnaryAsync capped the whole retry pipeline at one
DefaultCallTimeout while each attempt carried the same deadline, so a first
attempt ending in DeadlineExceeded spent the entire budget and the retry
never ran. The outer cap is now MaxAttempts x DefaultCallTimeout plus MaxDelay
per backoff (MxGatewayClientRetryPolicy.SafeUnaryBudget), in both
MxGatewayClient and GalaxyRepositoryClient; a budget too large for a
cancellation timer leaves only the per-attempt deadlines.

Tests: MxGatewayClientWireTests.SafeUnaryRetry_AfterDeadlineExceededFirstAttempt_RunsTheRetry
(real gRPC loopback server stalls the first Invoke; fails with Cancelled on
the old code), plus two SafeUnaryBudget unit tests.
2026-11-06 16:07:35 -05:00
Joseph Doherty ff397b27be fix(gateway): remove the worker startup-probe retry options that could never retry (#188)
WorkerProcessStartedProbe only checks HasExited and throws a non-retryable
WorkerProcessLaunchException, so StartupProbeRetryAttempts and
StartupProbeRetryDelayMilliseconds configured a Polly retry that no shipped
probe could trigger. Real readiness is WorkerClient's WorkerReady handshake
under the same StartupTimeoutSeconds budget, so the probe now runs once and
the two options, their validation, the Polly pipeline (and the server's now
unused Polly.Core reference) and their docs are removed. A stale value in
configuration is ignored by the binder.

Tests: WorkerProcessLauncherTests.LaunchAsync_WhenStartupProbeFails_DoesNotRetryTheProbe,
WorkerOptions_HasNoStartupProbeRetryOptions (both fail on the old code).
2026-11-06 16:05:42 -05:00
Joseph Doherty 3a3d9e8e06 fix(worker): reply WorkerUnavailable to commands that race shutdown (#182)
TryStartCommandTask returned silently when command intake had stopped, so a
command refused during shutdown got no reply and no log and the gateway's
correlation wait ran out its own timeout. The refusal now writes a
WorkerUnavailable reply for the correlation and logs
WorkerCommandRejectedAfterShutdown. StopAcceptingCommands is internal so the
test can reach the guard (the serial read loop cannot race it today).

Test: WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable.
2026-11-06 16:03:47 -05:00
Joseph Doherty e5e7f1854f chore(clients/dotnet): bump Contracts + Client 0.2.1 -> 0.3.0 for publish (#214)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 4m35s
0.2.1 on the feed predates the 488649c AcknowledgeAlarmRequest
operator_password/operator_domain fields; 0.3.0 ships them so OtOpcUa can
adopt operator-authenticated alarm ack (lmxopcua#749).
2026-11-06 16:00:21 -05:00
Joseph Doherty 777fef9cad fix(sessions): drop the deny-NETWORK pipe ACE (#187)
ci / windows-x86 (push) Successful in 1m26s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m30s
ci / portable (push) Failing after 4m47s
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
2026-11-06 15:52:46 -05:00
Joseph Doherty 34a7453ac3 fix(sessions): restrict the worker named pipe to the gateway's account (#187)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m20s
ci / portable (push) Failing after 4m46s
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
2026-11-06 15:45:50 -05:00
Joseph Doherty 9354d69bf9 fix(diagnostics): redact WriteSecuredBulk/WriteSecured2Bulk payloads (#193)
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
2026-11-06 15:45:50 -05:00
Joseph Doherty 5a235776b6 docs: point markdown backlogs at Gitea issues (status lives in Gitea)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m28s
ci / java (push) Successful in 2m29s
ci / portable (push) Failing after 4m58s
2026-11-06 15:32:27 -05:00
Joseph Doherty 8c0a51dc90 Merge branch 'fix/review-2026-10-21' into main (2026-10-21 system review remediation) 2026-10-27 09:56:50 -04:00
Joseph Doherty b2bce9c58b fix(go-client): go vet copylocks in types_alias_test (declare pointer types)
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / portable (push) Failing after 4m8s
ci / windows-x86 (push) Successful in 6m28s
2026-10-22 10:15:58 -04:00
Joseph Doherty c9597f7adc docs(archreview): record stream M final-review fixes 2026-10-22 08:48:09 -04:00
Joseph Doherty abd135a220 Merge stream M (cross-stream review fixes: terminal-worker attach, fault double-count, cancelled/stopping opens, docs) 2026-10-22 08:47:59 -04:00
Joseph Doherty dd623356f4 docs(clients): ReadBulk timeout is one shared deadline for the command, not per tag
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
2026-10-22 08:46:29 -04:00
Joseph Doherty 3ffba83723 docs: dashboard live-data session reaping, subscriber-refusal code, distributor-based StreamEvents model
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
  unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
  EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
  replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
  clients see while the gateway is stopping.
2026-10-22 08:43:41 -04:00
Joseph Doherty a0b5e91227 fix(grpc): count a worker fault once, not once per StreamEvents subscriber
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
2026-10-22 08:42:35 -04:00
Joseph Doherty a178d68b60 fix(sessions): refuse a dead worker at attach, fault on pump readiness failure, keep cancel/stopping out of OpenFailed
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
  worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
  fails, so a worker that dies between attach and the first pump read faults the session
  and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
  (gRPC Cancelled); an attach refused because the drain closed the session reports
  GatewayStopping.
- docs/Sessions.md updated.
2026-10-22 08:41:02 -04:00
Joseph Doherty 2375df5df3 chore(java-client): regenerate protos for stream L comment-only proto edits 2026-10-22 08:30:34 -04:00
Joseph Doherty 4962fa57cb docs(archreview): mark stream E/L rows done after merge; defer MxStatusDetailText test 2026-10-22 08:29:14 -04:00
Joseph Doherty ad67c486e8 Merge stream K (2026-10-21 tracker, cycle-1 re-triage, 2026-08-19 header fix) 2026-10-22 08:29:03 -04:00
Joseph Doherty 20135734b3 Merge stream E (worker: cache eviction, ReadBulk shared deadline + dedupe, alarm poll threshold, STA loop fault, unhandled-exception hook, live control-command facts) 2026-10-22 08:28:55 -04:00
Joseph Doherty ac4edc6e0a docs(archreview): 2026-10-21 remediation tracker, cycle-1 re-triage, fix 2026-08-19 header
Add archreview/2026-10-21 (overall summary + tracker with R21-* rows).
Re-triage the 68 cycle-1 Not-started rows in place: 21 Done, 13 Won't fix,
34 Open (promoted as R21-O1..O34). Correct the 2026-08-19 tracker header
so CLI-47..50, IPC-36/37, TST-35 read Done like their rows.
2026-10-22 08:28:44 -04:00
Joseph Doherty 2fad124bf8 Merge stream L (proto reserved/propagation comments + regen, stale docs, gateway test gaps) 2026-10-22 08:25:25 -04:00
Joseph Doherty 908cf7ccca test(server): cover GatewayConfigPathRules, DashboardRedactor, EventSubscriberReplayAttachment 2026-10-22 08:25:11 -04:00
Joseph Doherty dd3ea056b0 docs: note operator_full_name not propagated; refresh stillpending and deferred-followups status 2026-10-22 08:24:04 -04:00
Joseph Doherty 7f665bf25b docs(contracts): document unpropagated operator_full_name, reserved enum values, correlation echo 2026-10-22 08:23:26 -04:00
Joseph Doherty 38fbbde191 fix(worker): review follow-up — ghost cache entries, ReadBulk dedupe/pumping, fault hygiene
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m19s
ci / windows-x86 (push) Successful in 6m16s
- AddItem/AddItem2/AddBufferedItem drop stale cache entries for a reused handle id
- ReadBulk: dedupe handles within a batch (advise/teardown once, fan out result);
  pump every 16 items in add/advise and teardown phases
- alarm poll stops without recording a fault once StaRuntime.LoopFault is set
- StaRuntime skips the loop-fault handler during a requested shutdown
- StaHung loop fault diagnostic carries type + HRESULT only
- gateway.md documents ReadBulk's shared deadline
2026-10-22 08:23:05 -04:00
Joseph Doherty 7f0720f956 Merge stream A (session lifecycle: startup orphan, sweep metrics, stopping gate, kill-fallback Closed, unattached grace, worker-exit faults session) 2026-10-22 08:21:47 -04:00
Joseph Doherty cc1e895474 fix(sessions): fault a Ready session whose worker reached a terminal state on its own
Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.

IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.

FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
2026-10-22 06:48:31 -04:00
Joseph Doherty 3f9448f619 Merge stream C (gRPC surface: audit aggregation, streaming auth, alarm overflow/snapshot, write cancellation, correlation echo, disconnect reasons, verifier single-flight, opt-in /metrics auth) 2026-10-22 06:47:49 -04:00
Joseph Doherty 82a856b61e fix(metrics-auth): DisableLogin no longer opens /metrics; log correlation pair; host-level tests
- The metrics scheme skips its dashboard-cookie fallback while
  Dashboard:DisableLogin is on, so the dev auto-login handler cannot
  satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
  loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
  non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
  vs gRPC per connection, so only the per-key aggregate is shared.
2026-10-22 06:47:36 -04:00
Joseph Doherty ec9f26400a fix(sessions): review follow-up — never orphan a late worker, exempt internal subscribers from unattached grace, tolerate lost close races
- Open path: the attach gate now waits on CancellationToken.None and the
  manager discards (kills + disposes) the late worker client on ANY
  exception out of the attach step, including the caller's own cancellation
  (checked first, so an abandoned open never yields a session). Previously
  an OperationCanceledException from the gate wait escaped before the
  discard and orphaned the x86 worker. The gate Release is guarded against
  ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
  the gateway's own alarm-monitor session (internal subscriber only) is
  never reaped; the idle clock starts at Ready (MarkReady stamps
  _readyAtUtc) instead of at open, so a slow worker start does not eat the
  window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
  the close gate on a session that is no longer registered means a
  concurrent close already removed, accounted and disposed it; both now
  return AlreadyClosed with no fault, no second gauge decrement and no sweep
  failure. The general kill catch only decrements the gauge when the
  session is still registered. The sweep no longer counts AlreadyClosed
  results among the sessions it closed.
2026-10-22 06:43:32 -04:00
Joseph Doherty 4ea3672b1e test(integration): ShutdownWorker live fact pins reply-then-exit; log gateway session state
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m10s
ci / portable (push) Failing after 4m15s
ci / windows-x86 (push) Successful in 6m31s
2026-10-22 06:37:55 -04:00
Joseph Doherty 65bca804bb feat(security): opt-in authentication for /metrics (SEC-14)
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.

MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
2026-10-22 06:37:31 -04:00
Joseph Doherty ed3a2c9c8b fix(auth): single-flight API-key verification misses; prune key-id index on eviction
CachingApiKeyVerifier's key-id index was only pruned by Invalidate, so it
grew with every token ever verified. A post-eviction callback now drops the
slot (under the same lock as Set + index, so a late callback cannot
unindex a re-cached entry). Concurrent misses for the same token and key
generation now share one inner verification, so a TTL expiry under load no
longer stampedes SQLite; keying the flight on the generation keeps the
revoke guarantee for joiners.
2026-10-22 06:33:18 -04:00
Joseph Doherty 8212b0e550 Merge stream D (dashboard: bounded hub fan-out, alarm reset marker, browse debounce, snapshot config once, health status, per-viewer LRU, circuit retention, doc rewrite) 2026-10-22 06:32:17 -04:00
Joseph Doherty bd7abf4e35 fix(dashboard): re-check hub viewers under lock before starting a pump; document alarm send-timeout tradeoff 2026-10-22 06:32:17 -04:00
Joseph Doherty 285ef64160 feat(sessions): add UnattachedSessionGraceSeconds to reap never-subscribed idle sessions
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.

MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.

GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
2026-10-22 06:31:39 -04:00
Joseph Doherty 069972aff0 test(integration): attach event stream in ShutdownWorker live fact so the gateway observes the worker exit
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 4m16s
ci / windows-x86 (push) Successful in 6m30s
2026-10-22 06:31:36 -04:00
Joseph Doherty 68d6b5fd6f perf(metrics): cache MxEventFamily names for per-event metric tags
WorkerClient.StageWorkerEvent and StreamEvents called Enum.ToString per
event (per subscriber on the stream path). MxEventFamilyNames serves the
contract-fixed names from a frozen lookup, falling back to ToString for
undefined values so exported tag values are unchanged.
2026-10-22 06:30:57 -04:00
Joseph Doherty 6ae5ed905f fix(grpc): record the real StreamEvents disconnect reason
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
2026-10-22 06:29:31 -04:00
Joseph Doherty 0b4a1682c8 fix(grpc): typed alarm-feed overflow, memoized alarm snapshot, cancellable stream writes, uniform reply correlation id
- StreamAlarms: slow subscribers are completed with
  AlarmFeedSubscriberOverflowException and surface as ResourceExhausted
  (like a StreamEvents overflow) instead of warning-with-stack + Unavailable.
- GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection
  instead of cloning the whole set under the broadcast lock per attach; the
  prefix filter runs outside the lock.
- StreamEvents/StreamAlarms pass the call token to response writes; a write
  failure after client cancel maps to Cancelled without a warning.
- Every MxCommandReply.correlation_id now echoes the request's
  client_correlation_id (previously only constraint-denied replies did;
  normal replies leaked the internal envelope id).
2026-10-22 06:27:36 -04:00
Joseph Doherty 2c963b7230 docs(dashboard): describe the pages as shipped and the bounded hub/feed behaviour
Rewrite the Pages 'Show:' lists to what each page actually renders (no
per-session subscriber/pending-command/queue counters, no worker memory/CPU,
no per-family command counters, counters not rates). Document the per-session
EventsHub pump, alarm-hub send bound and AlarmFeedReset marker, configuration
sent once per snapshot-hub connection, health-derived gateway status, Browse
search debounce/index, bounded disposal, 30s circuit retention with
connection-gated polling, per-viewer advise-set share, and AlarmsPage memo.
2026-10-22 06:27:14 -04:00
Joseph Doherty 2b165cd3a4 fix(sessions): report a close whose kill fallback succeeded as Closed, not CloseFailed
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.

A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
2026-10-22 06:26:33 -04:00
Joseph Doherty db7833609d fix(dashboard): debounced off-dispatcher Browse search, bounded disposal, circuit-aware polling
- Search: the oninput setter scanned the whole Galaxy synchronously per
  keystroke. It now debounces 150ms, runs on the thread pool with
  cancellation (newer keystroke supersedes), and reads a shared singleton
  DashboardBrowseSearchIndex of pre-lowercased references keyed on the
  Galaxy cache Sequence.
- Disposal: DisposeAsync awaited both loops unbounded and the loops caught
  only OCE. Loops now handle faults per iteration (poll faults surface in
  _readError, deploy-notifier faults in a retrying warning), mutate the tree
  on the dispatcher, drain concurrently with a 5s bound, and the CTSs are
  disposed in finally.
- Circuits: DisconnectedCircuitRetentionPeriod is 30s (was the 3 min
  default) and a scoped DashboardCircuitConnectionTracker (CircuitHandler)
  lets the 2s ReadBulk poll skip ticks while the browser is disconnected.
- Reads pass a per-page viewer id to the live-data per-viewer share.
2026-10-22 06:25:46 -04:00
Joseph Doherty 37ab629d70 fix(sessions): refuse OpenSession while the gateway is stopping; drain until the registry is empty
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.

SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
2026-10-22 06:23:40 -04:00
Joseph Doherty 5d4aefc99c fix(dashboard): per-viewer share of the live-data advise set
The 256-tag LRU was global, so two Browse circuits on different areas evicted
each other's whole sets every poll. Tags now record the viewer that read them
last; each active viewer (read within 30s) owns MaxSubscribedTags/active
(floor 32). Eviction takes other viewers' excess over their share first
(inactive viewers' share is zero), then the reader's own untouched tags, and
never strips a neighbour within its share (soft cap). Single-viewer behaviour
is unchanged. IDashboardLiveDataService gains a viewerId ReadAsync overload.
2026-10-22 06:22:17 -04:00
Joseph Doherty 6c2d238e20 test(worker): add TestSupport using to StaRuntimeTests
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m16s
ci / java (push) Successful in 2m32s
ci / portable (push) Failing after 4m45s
2026-10-22 06:21:34 -04:00
Joseph Doherty 2a7d6facce perf(dashboard): memoize AlarmsPage filtered rows and area list
FilteredAlarms() ran the full filter+sort twice per render (Showing card and
table) and Areas() once more, including every search keystroke render. Both
now go through DashboardAlarmView, recomputed only when the alarm generation
(bumped per poll) or the filter criteria change.
2026-10-22 06:19:34 -04:00
Joseph Doherty 7618cafb6c fix(worker): review 2026-10-21 stream E — cache eviction, phased ReadBulk, poll/STA fault handling
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m3s
ci / java (push) Successful in 2m17s
ci / portable (push) Failing after 4m42s
- RemoveItem/Unregister evict value + write-completion caches (leak under churn)
- ReadBulk snapshot path: add/advise all, one shared deadline, then tear down
- alarm poll faults only after 3 consecutive failures; transient ones logged
- AlarmCommandHandler disposes primary when standby build throws
- StaRuntime pumps between queued work items
- WRK-02: post-start STA loop death surfaces as StaHung WorkerFault
- WRK-09: AppDomain unhandled-exception hook, exit code 7
- RETRIGGER documented as reserved, never synthesized
- TST-05: live facts for Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker
2026-10-22 06:19:32 -04:00
Joseph Doherty 09666cc719 fix(sessions): account SessionClosed for sweep-driven closes
TryBeginCloseIfExpired set _closeStarted when it reserved a session, so the
sweep's own CloseAsync reported AlreadyClosed: true and CloseSessionCoreAsync
skipped SessionClosed: lease-expired, faulted-reaped and detach-grace-expired
closes never left the sessions.open gauge and under-counted sessions.closed.
The reservation now writes Closing only; the first CloseAsync to take the
close lock does the accounting exactly once.
2026-10-22 06:19:20 -04:00
Joseph Doherty 6733519ae5 fix(dashboard): derive gateway status from health checks; send config once per hub connection
SEC-27: DashboardSnapshot.GatewayStatus was hardcoded Healthy. It is now the
aggregate HealthCheckService status, evaluated once per snapshot tick (2s
bound, zero-wait gate) and cached for GetSnapshot; Unknown before the first
evaluation, on failure, or without a health service.

The startup-static EffectiveGatewayConfiguration is JsonIgnore'd on the
snapshot, so /hubs/snapshot ticks carry only volatile state. The hub sends it
once per connection (ConfigurationUpdated) and serves GetConfiguration().
Snapshot broadcasts are time-bounded (1s) so one slow client cannot delay
ticks. StatusBadge maps Unhealthy to Bad.
2026-10-22 06:18:02 -04:00
Joseph Doherty 87f23d9712 fix(sessions): refuse late worker attach after close during startup; route open rollback through removal path
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.

AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.

The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
2026-10-22 06:16:53 -04:00
Joseph Doherty 3688fa19e6 fix(dashboard): time-bound alarm hub sends and mark each resubscribe
AlarmsHubPublisher awaited every group send unbounded inside the StreamAsync
loop, so one slow client stalled the drain until the monitor dropped the
subscriber and the resubscribe replayed the full snapshot to everyone. Each
broadcast is now bounded (2s) and abandoned on timeout, and every
(re)subscription is preceded by an AlarmFeedReset frame so clients discard
state before the replayed snapshot.
2026-10-22 06:14:53 -04:00
Joseph Doherty 19cc74f21a fix(auth): authenticate client- and duplex-streaming gRPC calls
GatewayGrpcAuthorizationInterceptor only overrode the unary and
server-streaming handlers; the base Interceptor passes client- and
duplex-streaming calls through unauthenticated. Both now run the same
API-key pipeline and fail closed to the admin scope, since there is no
single request message to resolve a scope from.
2026-10-22 06:14:33 -04:00
Joseph Doherty f1db002b87 fix(dashboard): bound EventsHub delivery with one drop-oldest pump per session
Publish fired a group SendAsync per event and returned, so a slow remote
viewer accumulated unbounded pending sends plus cloned MxEvents. Hub delivery
now goes through a 256-deep DropOldest channel per hub-watched session with a
single drain task and time-bounded sends. The registry counts hub viewers
separately and raises HubViewersReleased so the pump is torn down with the
last remote viewer; in-process-only sessions start no pump.
2026-10-22 06:13:05 -04:00
Joseph Doherty ca0cf19a1f fix(auth): aggregate bulk constraint denials into one audit event; ValueTask constraint checks
A bulk command with N denied items wrote N audit events into the bounded,
drop-on-full audit channel, so a low-privilege key could evict every other
pending audit record. Bulk filters now accumulate a ConstraintDenialSummary
(count, constraint names, <=20 sample targets) and write one aggregated
constraint-denied event per command.

The per-item constraint checks are synchronous, so IConstraintEnforcer's
Check* methods now return ValueTask instead of allocating a Task per item.
2026-10-22 06:12:02 -04:00
Joseph Doherty 81d4cd8644 Merge stream B (bound WorkerClient.ShutdownAsync enqueue by shutdown timeout) 2026-10-22 06:11:17 -04:00
Joseph Doherty b0100e738a fix(worker): bound ShutdownAsync's envelope enqueue by the shutdown timeout
WorkerClient.ShutdownAsync marked the client Closing and then queued the
WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS
was only created afterwards. The outbound channel is bounded, and a worker that
has stopped reading the pipe parks the write loop in a pipe write it can only
leave through _stopCts while the channel behind it fills with commands and
their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing
cancelled _stopCts and the enqueue waited for the caller's token - the lease
sweep's stopping token, which fires at host stop and never before - with
GatewaySession.CloseAsync holding its close lock throughout.

Create the linked timeout CTS before the enqueue and run the enqueue, the loop
waits and the process-exit wait on it, so the whole call is bounded by the
shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout
fault, which cancels the loops, kills the worker process tree and hands
GatewaySession its kill fallback.

Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker
stalls the gateway-side stream (a worker that stopped reading), fills the
outbound channel with abandoned commands and their cancels, and asserts that
ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake
worker. It hung past the bound before the fix.
2026-10-22 06:10:52 -04:00
Joseph Doherty b2a7fc525d Merge stream I (Rust client: async CA read, typed diagnostics) 2026-10-22 06:10:07 -04:00
Joseph Doherty 3a1ee28729 Merge stream F (.NET client: no ack retry, shared channel, safe dispose, typed diagnostics, AttachSession) 2026-10-22 06:09:55 -04:00
Joseph Doherty c18566bcf2 Merge stream G (Java client: configurable event queue, overflow retention, interrupt cancel, typed diagnostics) 2026-10-22 06:09:55 -04:00
Joseph Doherty 0345726a79 fix(rust-client): async CA file read; typed ping/session-state/worker-info/drain-events 2026-10-22 06:09:51 -04:00
Joseph Doherty 6f0b408833 fix(dotnet-client): no ack retry, caller-owned channels, safe session dispose, typed diagnostics, AttachSession 2026-10-22 06:09:25 -04:00
Joseph Doherty 48c9323d4a fix(java-client): configurable event queue, keep events on overflow, cancel on interrupt, typed diagnostics
- eventQueueCapacity option (default 1024, was hard-coded 16)
- overflow retains buffered events and delivers the error after them
- interrupt cancels the gRPC call and restores the interrupt flag
- ping/getSessionState/getWorkerInfo/drainEvents on MxGatewaySession
2026-10-22 06:09:25 -04:00
Joseph Doherty 7636573ebb Merge stream H (Go client: event buffer option, grpc.NewClient, typed diagnostics) 2026-10-22 06:08:48 -04:00
Joseph Doherty a3a7bd2170 feat(go-client): configurable event buffer, grpc.NewClient dial, typed diagnostics
- Options.EventBufferSize (default 1024) replaces the fixed 16-slot Events buffer
- Dial/DialGalaxy use grpc.NewClient + explicit READY wait (replaces DialContext+WithBlock)
- Session.Ping/GetSessionState/GetWorkerInfo/DrainEvents typed diagnostics
2026-10-22 06:08:31 -04:00
Joseph Doherty 29d4e51c7b Merge stream J (Python typed diagnostics) into fix/review-2026-10-21 2026-10-22 06:07:17 -04:00
Joseph Doherty d716dec160 docs(python): trim diagnostics note 2026-10-22 06:06:56 -04:00
Joseph Doherty 99347d3f00 feat(python): typed ping/get_session_state/get_worker_info/drain_events session helpers 2026-10-22 06:06:54 -04:00
Joseph Doherty 488649ce99 Merge branch 'feat/ackmsg-as-operator' into main (PR #162)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m22s
ci / java (push) Successful in 2m7s
ci / portable (push) Failing after 4m10s
AcknowledgeAlarm acks by writing the comment to <alarm>.AckMsg authenticated
as the operator, so the Historian records the real operator. Fixes #164,
fixes #165, fixes #166.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 08:52:19 -04:00
Joseph Doherty 02228b2717 Merge branch 'main' into feat/ackmsg-as-operator
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m7s
ci / java (pull_request) Successful in 2m18s
ci / nightly-windev (pull_request) Has been skipped
ci / portable (push) Failing after 4m47s
ci / windows-x86 (pull_request) Successful in 2m40s
ci / portable (pull_request) Failing after 4m43s
ci / windows-x86 (push) Successful in 5m38s
Picks up the System.Security.Cryptography.Pkcs 10.0.13 pin (NU1903).

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:28:45 -04:00
Joseph Doherty fb20997dc9 Merge branch 'fix/pkcs-cve-pin' into main
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m47s
ci / java (push) Successful in 2m6s
ci / portable (push) Successful in 8m57s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:24:10 -04:00
Joseph Doherty 463748e395 fix(worker): classify ack interop failures at the COM call; ignore late completions
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m24s
ci / java (push) Successful in 2m32s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m38s
ci / portable (push) Failing after 2m52s
ci / portable (pull_request) Failing after 2m53s
ci / windows-x86 (pull_request) Successful in 8m8s
Address review of the #164-#166 fixes:

- The HRESULT check could not tell interop from programming errors
  (a managed ArgumentException already defaults to 0x80070057). Wrap each
  MXAccess call (AuthenticateUser, AddItem, Advise, Write) in a narrow
  filter that converts an interop failure to a private marker exception,
  and catch only that marker, so exceptions from the writer's own code
  (argument checks, caching, the pump) propagate.
- A late OnWriteComplete for a dropped handle recreated the cache entry
  that DropItem removed. The writer now records completions only for item
  handles it holds on its live registration.
- The item-handle-reuse probe now asserts that handles within one
  registration strictly increase, so a rerun is a regression check.

Refs #164, #165, #166

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:18:02 -04:00
Joseph Doherty bd82036d5f docs(worker): record live item-handle-reuse probe for ack writer
ci / windows-x86 (push) Successful in 1m29s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m47s
ci / nightly-windev (pull_request) Has been skipped
ci / portable (push) Failing after 3m1s
ci / portable (pull_request) Failing after 1m46s
ci / windows-x86 (pull_request) Successful in 1m24s
ci / java (pull_request) Successful in 2m36s
Add the Skip-gated ItemHandleReuseProbeTests probe and its windev result
(2026-10-20): within one MXAccess registration, item handles increase
monotonically and are never reused across remove/re-add, so the ack
writer's Unconfirmed drop-and-re-add mitigation holds and no per-handle
generation counter is needed. Numbering restarts only on a new
registration, which teardown already isolates. Document the guarantee at
the Dirty re-add and in AlarmProbeFindings.md.

Fixes #166

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:13:31 -04:00
Joseph Doherty f13950e453 fix(worker): forget write-completion entries for dropped ack item handles
Add MxAccessWriteCompletionCache.Remove and RemoveServer and call them
from AlarmAckWriter when it drops an item (dirty re-add, LRU eviction,
failed AddItem/Advise/Write) and when it tears down its registration, so
the cache no longer keeps one entry per dropped handle for the life of
the worker.

Fixes #165

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:11:37 -04:00
Joseph Doherty 219611c821 fix(worker): narrow AlarmAckWriter ArgumentException interop filter
Resolve the operator user, password and comment before the interop try
blocks, and match ArgumentException as an MXAccess failure only when it
carries E_INVALIDARG and is not ArgumentNullException or
ArgumentOutOfRangeException. A programming error inside the try blocks
now propagates instead of being reported as an authentication or write
failure.

Fixes #164

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:11:37 -04:00
Joseph Doherty a09fa0a4d6 fix: pin System.Security.Cryptography.Pkcs 10.0.13 (NU1903)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 10m44s
Microsoft.Data.SqlClient 6.0.2 pulls System.Security.Cryptography.Pkcs 9.0.4,
which carries GHSA-846f-f764-76mx (high) and GHSA-2fmv-rv2c-59x7 (moderate).
With TreatWarningsAsErrors the NU1902/NU1903 audit warnings failed restore.
10.0.13 patches both.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:07:59 -04:00
Joseph Doherty 3e988ede88 docs: record gateway-level live verification of AckMsg-as-operator acks
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m21s
ci / nightly-windev (pull_request) Has been skipped
ci / java (push) Successful in 2m3s
ci / portable (push) Failing after 2m19s
ci / windows-x86 (pull_request) Successful in 1m19s
ci / java (pull_request) Successful in 2m2s
ci / portable (pull_request) Failing after 2m17s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 20:07:36 -04:00
Joseph Doherty e7f56622c1 fix: address final ackmsg-as-operator review findings
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m0s
ci / windows-x86 (push) Successful in 2m11s
ci / portable (push) Failing after 2m21s
- AlarmAckWriter: classify AuthenticateUser and AddItem/Advise/Write
  failures by HRESULT for any interop exception (ArgumentException for
  E_INVALIDARG, UnauthorizedAccessException, COMException); server-gone
  HRESULTs at AuthenticateUser tear down the registration and report
  WriteRejected.
- Bound the ack item cache with a 256-entry LRU (eviction best-effort
  UnAdvise+RemoveItem).
- After an Unconfirmed ack, drop and re-add the item before the next
  write so a late OnWriteComplete on the old handle is not credited.
- Pass StaRuntime.PumpPendingMessages to the ack writer through a
  pump-aware alarm handler factory so long completion waits refresh the
  STA watchdog heartbeat.
- Rust CLI: --operator is required for acknowledge-alarm.
- Docs: full ack status table, credential-oracle trade-off, upgrade note.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:58:24 -04:00
Joseph Doherty 1998ee3e20 docs: address Task 12 review (supersede stale ack steps, outcome rows)
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m10s
ci / windows-x86 (push) Successful in 2m11s
ci / portable (push) Failing after 2m18s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:45:43 -04:00
Joseph Doherty 027cc7393d fix(python-client): redact operator password from AcknowledgeAlarmReply errors
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:45:02 -04:00
Joseph Doherty 7b81cb7f72 feat(python-client): operator credentials on acknowledge-alarm
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:45:02 -04:00
Joseph Doherty 0143591858 docs(dotnet-client): keep XML docs with their CLI members
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:56 -04:00
Joseph Doherty e8ace39418 feat(dotnet-client): operator credentials on acknowledge-alarm
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:56 -04:00
Joseph Doherty 9907a97cb0 fix(java-client): redact operator password on async acknowledge-alarm failures
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:32 -04:00
Joseph Doherty 84e04b6552 feat(java-client): operator credentials on acknowledge-alarm
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:32 -04:00
Joseph Doherty 417a47bdb5 feat(rust-client): operator credentials on acknowledge-alarm
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:32 -04:00
Joseph Doherty cfe9f817a3 feat(go-client): operator credentials on acknowledge-alarm
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:32 -04:00
Joseph Doherty 9409488008 docs: alarm acknowledgement writes AckMsg as the authenticated operator
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:21 -04:00
Joseph Doherty 2858e42e80 feat(gateway): forward operator credentials on alarm acknowledgement
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:40:23 -04:00
Joseph Doherty d6db0ff40d feat(worker): route alarm acks through the operator-authenticated AckMsg writer
ci / nightly-windev (push) Has been skipped
ci / java (push) Failing after 2m5s
ci / windows-x86 (push) Successful in 2m10s
ci / portable (push) Failing after 2m17s
AlarmCommandHandler -> AlarmDispatcher -> IAlarmAckWriter: the dispatcher resolves the ack item through the consumer and the writer authenticates the operator and writes the comment. The executor builds AlarmAckOperator from operator_user/domain/password, maps AlarmAckOutcome to the reply (Applied/Unconfirmed Ok, AlarmNotFound/NotAckable InvalidRequest, AuthenticationFailed/WriteRejected MxaccessFailure) and reports escaping exceptions by type and HRESULT only, so operator_password cannot leak. Removes the inert managed/WNAL alarm-client ack path.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:35:55 -04:00
Joseph Doherty 0d4b2aea2d refactor(worker): alarm consumers resolve the AckMsg item instead of acking
IMxAccessAlarmConsumer.ResolveAckTarget/ResolveAckTargetByName replace AcknowledgeByGuid/AcknowledgeByName. wnwrap maps Galaxy-provider alarms to <tag>.AckMsg and drops its inert parallel ack consumer; subtag resolves to the target's AckCommentSubtag and no longer advises or writes it; ISubtagAlarmSource.Write is removed. Does not build alone: AlarmDispatcher moves to the new API in the next commit.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:35:49 -04:00
Joseph Doherty 8199553734 fix(worker): handle AckMsg add/advise/write COM failures and server death
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m55s
ci / java (push) Failing after 2m2s
ci / portable (push) Failing after 2m17s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:25:43 -04:00
Joseph Doherty 7900df3dce fix(worker): disambiguate MxStatusCategory in AlarmAckWriter
ci / nightly-windev (push) Has been skipped
ci / java (push) Failing after 2m25s
ci / portable (push) Failing after 2m26s
ci / windows-x86 (push) Successful in 2m27s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:22:27 -04:00
Joseph Doherty 76db822df3 feat(worker): acknowledge alarms via authenticated AckMsg write
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 56s
ci / java (push) Failing after 2m17s
ci / portable (push) Failing after 2m23s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:21:55 -04:00
Joseph Doherty 64dc1215fd feat(worker): attribute alarm ack transitions from wnwrap ACK fields
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m7s
ci / java (push) Failing after 2m17s
ci / portable (push) Failing after 2m23s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:18:24 -04:00
Joseph Doherty a94b502ce6 feat(contracts): carry operator credentials on alarm acknowledgement
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:16:39 -04:00
Joseph Doherty bf4b9c6c09 docs(plan): AckMsg-as-operator alarm acknowledgement plan
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:15:20 -04:00
Joseph Doherty 91bbdbf154 Merge branch 'docs/managed-alarm-ack-live-result' into main
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m14s
ci / windows-x86 (push) Successful in 2m23s
ci / portable (push) Failing after 2m32s
2026-10-18 15:15:58 -04:00
Joseph Doherty 8c095b9a56 Record the managed GUID alarm-ack live result
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m25s
ci / java (push) Successful in 2m18s
ci / portable (push) Failing after 2m39s
The 2026-10-18 live test on the dev rig showed the opt-in managed ack path
cannot work as built: AlarmClient.RegisterConsumer returns -65 alongside the
worker's wnwrap reader (the worker then silently falls back to wnwrap), the
managed client does not recognise the wnwrap GUID even standalone, and the
gateway's GUID arm sends empty identity fields. Replace the docs' claim that
the path works with the measured results and tell operators to leave
MXGATEWAY_USE_MANAGED_ALARM_ACK unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:11:11 -04:00
Joseph Doherty 6c82de55c6 Merge pull request 'Add opt-in managed GUID alarm-ack path' (#160) from feat/managed-guid-alarm-ack into main
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m59s
ci / portable (push) Failing after 2m21s
ci / java (push) Successful in 2m7s
2026-10-18 13:12:12 -04:00
Joseph Doherty 6f4c7c3bf1 Keep subtag acks and alarm reads safe on the managed ack path
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (pull_request) Successful in 2m0s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m16s
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 2m31s
ci / portable (pull_request) Failing after 2m36s
ci / windows-x86 (push) Successful in 3m11s
Review fixes for the opt-in managed GUID ack:

- Subtag-mode GUID acks (forced subtag, or failover on the standby) stay on
  the consumer; their GUIDs are synthetic and only the subtag consumer can
  resolve them. Forced-subtag sessions no longer build a managed client.
- A failed managed Initialize/Register/Subscribe drops back to the wnwrap
  ack path instead of failing the alarm subscription.
- Drop stale crefs to a nonexistent Unverified member in
  WnalNativeAckConsumer.
- Unit tests for routing, fallback, and disposal; document
  MXGATEWAY_USE_MANAGED_ALARM_ACK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 13:11:01 -04:00
Joseph Doherty 6affbd57ea feat(worker): add opt-in managed GUID alarm-ack path
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 2m23s
ci / nightly-windev (pull_request) Has been skipped
ci / windows-x86 (pull_request) Successful in 1m18s
ci / java (pull_request) Successful in 2m3s
ci / portable (pull_request) Failing after 2m17s
Adds a managed acknowledge path over AVEVA's modern alarm client so GUID
acks carry full operator identity (domain + full name) and work on builds
where the wnwrap COM AlarmAckByGUID is E_NOTIMPL. Default off; opt in with
MXGATEWAY_USE_MANAGED_ALARM_ACK=1 (wnwrap behavior unchanged otherwise).

- AaManagedAlarmAckConsumer: reflection wrapper over
  aaAlarmManagedClient.AlarmClient (InitializeConsumer/RegisterConsumer/
  Subscribe/AlarmAckByGUID); EnvGatedFactory degrades to null (wnwrap) if the
  DLL can't load, so alarm reads never break.
- IManagedAlarmAckClient seam. AlarmDispatcher takes an optional client and
  routes GUID acks to it (subscribing + disposing it alongside the reader).
- AlarmCommandHandler gains an optional managedAckFactory; WorkerPipeSession
  wires AaManagedAlarmAckConsumer.EnvGatedFactory at the production sites.
- WnalNativeAckConsumer: confirmed native alarmlst.dll WNAL submit layer
  (WNALalarmReqSetText / WNALUserRequest / WNALUserRequestEx + lpAlmAck
  layout). List acquisition is intentionally not hand-rolled: the alarm-record
  layout and selection semantics are provider-defined and not recoverable from
  alarmlst.dll, so inferring them risks acking the wrong alarm.
- AlarmManagedGuidAckLiveSmokeTests: skip-gated live smoke that pulls a real
  (non-degraded) alarm GUID and asserts the ack APPLIES (Acknowledge
  transition / acked snapshot), not merely rc=0.

By-name acks stay on wnwrap (AlarmClient has no AlarmAckByName). Grounded in
static RE of alarmlst.dll, aaAlarmManagedClient.dll and alarmmgr.exe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-18 12:42:07 -04:00
dohertj2 6fc423f608 Merge pull request 'Retain fake worker fault diagnostics in test fixtures' (#155) from fix/fake-worker-fault-diagnostics into main
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m3s
ci / portable (push) Successful in 8m23s
2026-10-13 10:46:14 -04:00
Joseph Doherty 83aa76c33d Retain fake worker fault diagnostics in test fixtures
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m20s
ci / java (push) Successful in 2m19s
ci / windows-x86 (pull_request) Successful in 1m20s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m23s
ci / portable (push) Successful in 9m10s
ci / portable (pull_request) Successful in 9m2s
2026-10-13 10:35:28 -04:00
dohertj2 d779daf6af Merge pull request 'Make framed worker and readiness test gates deterministic' (#153) from fix/framed-worker-test-gates into main
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m22s
ci / java (push) Successful in 2m2s
ci / portable (push) Successful in 8m26s
2026-10-13 09:59:45 -04:00
Joseph Doherty 42c120aa83 Preserve framed reads across fake worker emission gates
ci / windows-x86 (push) Successful in 1m20s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / windows-x86 (pull_request) Successful in 1m25s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m27s
ci / portable (push) Successful in 9m43s
ci / portable (pull_request) Successful in 9m47s
2026-10-13 09:42:09 -04:00
dohertj2 19920b13da Merge pull request 'Allow redundant must_use only in generated Rust service traits' (#152) from fix/generated-rust-must-use into main
ci / windows-x86 (push) Successful in 1m25s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m15s
ci / portable (push) Successful in 9m6s
2026-10-13 09:37:53 -04:00
Joseph Doherty a99070005f Fix Rust 1.99 lint for generated service traits
ci / windows-x86 (push) Successful in 1m27s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m45s
ci / java (pull_request) Successful in 2m55s
ci / windows-x86 (pull_request) Successful in 1m20s
ci / nightly-windev (pull_request) Has been skipped
ci / portable (push) Successful in 10m40s
ci / portable (pull_request) Successful in 10m18s
2026-10-13 09:24:07 -04:00
Joseph Doherty f40409a688 docs: strip task-tracking IDs from code comments and fill missing XML docs
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m16s
ci / java (push) Successful in 2m5s
ci / portable (push) Failing after 5m9s
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
2026-10-07 15:34:32 -04:00
Joseph Doherty f2e0861709 docs: note that psbridge and winrm remote access to the wonder boxes are disabled (2026-09-11)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m25s
ci / java (push) Successful in 3m34s
ci / portable (push) Successful in 11m28s
Claude-Session: https://claude.ai/code/session_01PpXxRCFznFqE65ay6wAJwe
2026-09-11 09:05:35 -04:00
Joseph Doherty 4f931e4b7e chore(deps): bump ZB.MOM.WW.Auth 0.2.1->0.3.0
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m20s
ci / java (push) Successful in 2m20s
ci / portable (push) Successful in 10m43s
Bumps ZB.MOM.WW.Auth.Abstractions, .ApiKeys, .AspNetCore, and .Ldap in
ZB.MOM.WW.MxGateway.Server.csproj and ZB.MOM.WW.MxGateway.IntegrationTests.csproj.
0.3.0 is additive (new IApiKeyStoreMigrator, SqliteAuthStoreMigrator implements it,
old ctors retained) — no source changes needed. Restore, Release build (0 warnings/
0 errors), and dotnet test on the NonWindows solution are unchanged versus the
pre-bump baseline: 1198 passed, 0 failed, 19 skipped (live-only tests, opt-in via
env vars).

Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
2026-09-03 10:57:10 -04:00
Joseph Doherty d16ae4e267 docs(claude): note the umbrella index's short-row convention for cross-repo propagation
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m34s
ci / portable (push) Successful in 9m14s
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
2026-09-03 10:47:48 -04:00
Joseph Doherty 74ba889dea docs: record cycle-3 client release
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m21s
ci / java (push) Successful in 2m29s
ci / portable (push) Successful in 9m46s
2026-08-25 12:47:54 -04:00
Joseph Doherty 5776021970 ci(rust): allow generated result size lint
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m24s
ci / portable (push) Successful in 9m31s
2026-08-25 12:28:12 -04:00
Joseph Doherty a11786cbea docs: record batch N portable gates
ci / portable (push) Failing after 17m19s
ci / java (push) Successful in 2m29s
ci / windows-x86 (push) Successful in 5m55s
ci / nightly-windev (push) Has been skipped
2026-08-25 12:09:40 -04:00
Joseph Doherty aea2c74fdf docs: start batch N release gate
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m11s
ci / portable (push) Failing after 8m43s
2026-08-25 12:06:34 -04:00
Joseph Doherty 8b147a32a2 docs: record batch M verification
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m26s
ci / portable (push) Failing after 5m27s
2026-08-25 12:02:19 -04:00
Joseph Doherty cc143b52cf ci(TST-35): run gateway suite on windev nightly
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m44s
ci / windows-x86 (push) Successful in 5m56s
ci / portable (push) Failing after 9m21s
2026-08-25 11:56:35 -04:00
Joseph Doherty ad79238c84 docs: record batch L verification
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 10m12s
2026-08-25 11:50:48 -04:00
Joseph Doherty 986c6c0ce4 fix(IPC-36): import obsolete attribute namespace
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m4s
ci / java (push) Successful in 2m22s
ci / portable (push) Failing after 9m44s
2026-08-25 11:49:02 -04:00
Joseph Doherty 484028e721 feat(GWC-37,WRK-32): refresh live alarm watch lists
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 51s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 1h1m54s
2026-08-25 11:46:07 -04:00
Joseph Doherty 1b11e89f29 docs: record batch K verification 2026-08-25 11:26:12 -04:00
Joseph Doherty 5cdb3da8c0 fix(CLI-47,CLI-48,CLI-49,CLI-50): align alarm feed clients 2026-08-25 11:25:47 -04:00
Joseph Doherty feadcf5740 docs: record batch J verification 2026-08-25 11:14:33 -04:00
Joseph Doherty d6c5a11ae6 fix(IPC-36,IPC-37): guard secondary contracts 2026-08-25 11:14:14 -04:00
Joseph Doherty f3827c71c9 docs: record batch I verification 2026-08-25 11:01:38 -04:00
Joseph Doherty 1875426e4b fix(SEC-38,SEC-39): preserve aggregate and fallback throttles 2026-08-25 11:00:29 -04:00
Joseph Doherty 780720165a fix(GWC-35): repair re-raised alarm snapshots 2026-08-25 10:58:21 -04:00
Joseph Doherty b0156c685e fix(GWC-33): unregister terminal worker queue gauge 2026-08-25 10:56:23 -04:00
Joseph Doherty f2090fe1b5 docs: record batch H verification 2026-08-25 10:53:03 -04:00
Joseph Doherty 92aae28f07 chore(GWC-36,IPC-38,TST-33,TST-34,TST-36): close cycle-3 currency gaps 2026-08-25 10:52:39 -04:00
Joseph Doherty 4614080e24 docs: normalize continuation design formatting 2026-08-25 10:41:22 -04:00
Joseph Doherty 8e30ec5be9 docs: plan and track cycle-3 P2 continuation 2026-08-25 10:40:55 -04:00
Joseph Doherty 30548da71c docs: design cycle-3 P2 continuation 2026-08-25 10:37:30 -04:00
Joseph Doherty 914495a9f3 chore: ignore project-local worktrees 2026-08-20 12:14:23 -04:00
Joseph Doherty fa364ce139 chore(archreview): cycle-3 remediation integration gate recorded (gateway 1183/1183, windev worker 524/524)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m27s
ci / portable (push) Successful in 9m25s
2026-08-20 09:41:49 -04:00
Joseph Doherty f2ab9cc879 Merge fix/c3-tracker-hygiene: GWC-31 perf-pass comments relabeled PERF-nn, archreview prefixes reserved in REVIEW-PROCESS.md, cycle-1/2 tracker housekeeping (GWC-16 Won't fix, IPC-08/IPC-31 Done, TST-06/17 refreshed, WRK-29..31 reserved)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m57s
ci / java (push) Successful in 2m11s
ci / portable (push) Successful in 9m23s
2026-08-20 09:40:05 -04:00
Joseph Doherty cb820bb666 chore(archreview): tracker row G → Done with outcome notes 2026-08-20 09:39:59 -04:00
Joseph Doherty c6650c156b Merge fix/c3-session-lifecycle: GWC-32 WorkerClient dispose is terminal, GWC-34 attach-after-dispose throws SessionNotReady, GWC-37 subtag watch-list coverage graded on provider status + re-resolved at failover edge 2026-08-20 09:39:53 -04:00
Joseph Doherty 193221773d fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently
The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.

ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:

- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
  SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
  and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
  nothing", so the Auto case where alarmmgr is healthy and the standby is empty
  is visible. DashboardAlarmProviderStatus renders that combination as the amber
  "Alarm Manager (no subtag standby)" badge rather than the failover label, which
  would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
  monitor re-resolves BEFORE publishing the status and reports any difference as
  SubtagWatchListStale with both counts.

The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.

Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.

Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
2026-08-20 09:38:46 -04:00
Joseph Doherty 06cd8b844c chore(archreview): tracker rows D/E → Done with outcome notes 2026-08-20 09:33:59 -04:00
Joseph Doherty cfbc99581b Merge fix/c3-ipc-boundaries: IPC-33 drain budget bounded under the gRPC send cap + validator upper coupling, IPC-39 pre-check sequence allowance, IPC-34 WorkerCancel on caller cancellation + CallerCancelled code, IPC-35 handshake WorkerFault surfaces category 2026-08-20 09:33:55 -04:00
Joseph Doherty ab41362a5c fix(IPC-35): surface a handshake-phase WorkerFault with its category and diagnostic
The worker reports startup failures as structured WorkerFault frames --
MxAccessCreationException carries the COM HRESULT/ProgID context and a
WorkerFaultCategory -- so the gateway can tell "MXAccess is not installed / not
registered / access denied" apart from a pipe-level protocol problem.
ReadHandshakeEnvelopeAsync did not look inside: any body other than the expected
one became a ProtocolViolation naming only the body type, and that text is what
SessionManager puts in the operator log and the client's OpenFailed detail. The
information was lost exactly in the window where it matters most -- first
session open on a freshly deployed host, where the answer is almost always an
MXAccess installation or permission problem.

Special-case BodyCase == WorkerFault and throw WorkerClientException(
WorkerFaulted, CreateWorkerFaultMessage(...)) -- the same formatter the
steady-state fault path already uses, so both windows read alike. Every other
unexpected body still gets ProtocolViolation.

Test: StartAsync_WhenWorkerFaultsDuringHandshake_SurfacesFaultCategoryAndDiagnostic
has the fake worker answer the hello and then write a WorkerFault
{MxaccessCreationFailed} in place of WorkerReady; the existing
FakeWorkerHarness.EmitFaultAsync covered it, so the harness needed no change.
2026-08-20 09:32:01 -04:00
Joseph Doherty b27035b3a6 fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.

- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
  takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
  which sends the cancel under the same `removed` ownership gate as the timeout
  arm (so a reply that won the race is never chased) and skips it once _stopCts
  has fired, where the outbound channel is being completed and every pending
  command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
  those arms used, so metrics and logs stop recording a shutdown that did not
  happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
  to, so nothing changes for clients.

Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.

Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
2026-08-20 09:31:29 -04:00
Joseph Doherty 9f0c3fd9fa Merge fix/c3-dashboard-acl: SEC-41 ACL re-resolves tag grants from group claims, SEC-40 alarms hub/page behind non-bypassable authenticated-role policy, GWC-38 browse scope fails closed without identity under ApiKey mode 2026-08-20 09:31:21 -04:00
Joseph Doherty a71663a2a2 fix(IPC-39): make the CommandTooLarge pre-check allow for the sequence stamp
The pre-check measures the command envelope while Sequence is still 0, and an
unset protobuf field costs nothing to encode -- so the measured size is a lower
bound on the frame WriteLoopAsync emits after GWC-28 stamps the sequence. Being
a lower bound is the right kind of conservative for the knownSize question the
comment answers, and the wrong kind for the pre-check itself: it admits an
envelope in the last eleven bytes of the range that the writer then rejects with
a session-fatal MessageTooLarge.

Compare against MaxMessageBytes - MaxSequenceEncodingBytes (1 tag byte + a
10-byte varint, the widest 64-bit field encoding) and say in the comment that
the pre-check is a lower bound relying on that allowance.

Not reachable under the shipped validator (caller-controlled commands arrive
through gRPC capped at least 64 KiB below the pipe max, gateway-authored
commands are tiny), so this closes the argument rather than a live defect. The
existing CommandTooLarge test still passes unchanged.
2026-08-20 09:30:56 -04:00
Joseph Doherty 8257a0333e docs(SEC-40): correct the hub-mapping comment now that AlarmsHub uses its own policy 2026-08-20 09:30:45 -04:00
Joseph Doherty 2ef7b93761 fix(SEC-41, SEC-40): re-resolve dashboard tag grants; gate the alarm feed on a real role
Committed together because the two changes share the same doc paragraphs (the ACL
section of GatewayDashboardDesign.md, the GroupToTag / AllowAnonymousLocalhost rows
of GatewayConfiguration.md, and CLAUDE.md's authentication paragraph), and splitting
them would leave one of the two describing behaviour that does not yet exist.

SEC-41 — tag grants no longer trusted from login-stamped claims
---------------------------------------------------------------
DashboardSessionAcl read the zb:dashboardtag claims stamped into the cookie at
login. GatewayOptions binds once at startup, so applying a GroupToTag edit needs a
restart — but the restart does not clear the auth cookie (data-protection keys
persist), so a browser returning afterwards re-authenticated with tag claims
resolved under the OLD map. A revoked grant therefore survived on the cookie seam
for the sliding 8-hour idle life, renewable indefinitely by activity, while the
hub-token seam had no such gap because its mint re-resolves.

CanViewSession now re-resolves the grant on every subscribe from the caller's
mxgateway:ldap_group claims through DashboardGroupTagMapping.MapGroupsToTags
against the currently bound map — the same one call HubTokenService.Issue makes.

Fallback (documented on the class and in the design doc): a principal carrying NO
group claims falls back to its stamped tag claims. That arm exists for the
hub-token principal — HubTokenService.Validate rehydrates roles and tags but not
groups, so re-resolution already happened at mint, one lifetime ago at worst. It
cannot re-open the cookie hole: DashboardAuthenticator stamps group and tag claims
from one group list, so a cookie principal holding a tag claim always holds the
groups it came from, and a login yielding no groups yields no tags. A principal
with neither kind of claim stays an empty-grant Viewer.

SEC-40 — the alarm feed requires a real authenticated dashboard role
--------------------------------------------------------------------
AlarmsHub sat behind HubClientsPolicy, which the read-only environment bypasses
satisfy, so an anonymous-localhost caller or a Mode=Disabled deployment received
galaxy-wide alarm metadata — alarm reference (a full tag path), state, severity,
category, area, acknowledging operator — for the whole subscription expression.

Decided: tighten the gate, keep the scope. New AuthenticatedDashboardRoleRequirement
+ its handler, and a MxGateway.Dashboard.Alarms policy built from it. Because
DashboardAuthorizationHandler is typed to DashboardAuthorizationRequirement, it is
never invoked for the new type, so neither Authentication:Mode=Disabled nor
anonymous localhost can satisfy the policy — the same structural device the mounted
Secrets UI already relies on. The policy keeps both schemes so external hub clients
can still authenticate with a bearer token, and DisableLogin auto-login is
unaffected (it mints a genuinely authenticated both-roles principal).

The /alarms page carries the same policy, not just the hub: it reads the same
central monitor in process, so gating only the hub would have left the disclosure
one navigation away. The side rail hides the Alarms link for callers the policy
denies, matching the Secrets link precedent. AlarmsHubPublisher is unchanged and
stays unfiltered by design — audience is decided at hub admission, and the monitor
carries no per-session provenance to narrow by.

Alarm metadata deliberately remains galaxy-wide for authenticated Viewers: an
AlarmFeedMessage has no session id or owning key for the SEC-25 tag intersection to
work against, and equating a visibility tag with an alarm-reference prefix would
turn GroupToTag into the data-access constraint it explicitly is not.

Tests
-----
DashboardSessionAclTests: the SEC-41 regression — a cookie-style principal whose
stale tag claim is no longer granted by any of its groups under the current map is
denied; plus the allowing direction (including with no stamped tag at all, which
proves the decision comes from the groups), the ignore-case corner, group claims
that map to nothing, and the hub-token fallback asserted against an EMPTY map so
dropping the fallback would fail loudly.

AlarmFeedAuthorizationTests (new): both handlers run together in the most permissive
environment (Mode=Disabled + loopback + AllowAnonymousLocalhost) do not satisfy the
alarm requirement, with the bypassable requirement as the control so the negative
cannot pass because the bypass was misconfigured; authenticated Viewer/Admin do
satisfy it; an unauthenticated role claim does not; the registered policy carries
the non-bypassable requirement and both schemes; and both readers — the hub type's
[Authorize] and the /alarms endpoint metadata — carry it.

SecretsNavRenderTests: registers the new policy (an unregistered policy name throws
rather than denying in an AuthorizeView) and gains the Alarms-link negative/control
pair.

Verified: dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx (zero warnings);
filtered run of the touched classes — 92 passed, 0 failed.
2026-08-20 09:29:44 -04:00
Joseph Doherty b177636d7a fix(GWC-38): deny Galaxy browse when the ambient identity is absent under API-key auth
GatewayBrowseScopeProvider resolved `identityAccessor.Current?.EffectiveConstraints
?? ApiKeyConstraints.Empty`, so an absent ambient identity produced an empty
BrowseSubtrees list — which the shared GalaxyRepository library reads as "no
scoping". The control's fail state was therefore open: any future path reaching the
library's galaxy service without GatewayGrpcAuthorizationInterceptor having pushed
the identity would silently hand a browse-scoped key the full hierarchy, with
nothing failing and nothing logged. A constraint-narrowing control's absence is
invisible in normal operation, which is what made the honest "safe only because the
global interceptor runs first" comment insufficient on its own.

The provider now distinguishes the two states the old expression collapsed:

- authenticated key with no BrowseSubtrees constraint -> empty list, unscoped
  (unchanged, and now asserted in its own test rather than implied);
- no ambient identity with Authentication:Mode=ApiKey -> log a warning naming the
  broken ordering invariant and throw RpcException(PermissionDenied);
- no ambient identity with Authentication:Mode=Disabled -> unscoped, because
  nothing authenticates there so no key and no constraints exist to scope by.

The registration-order comment at GatewayApplication.cs now says the ordering is
still required but no longer silently trusted.

Tests: GatewayBrowseScopeProviderTests gains the PermissionDenied case and the
Disabled-mode case alongside the unconstrained-identity one.
GalaxyRepositoryHostWiringTests phase 1 now pushes an identity carrying no
BrowseSubtrees instead of pushing none — that, not an absent identity, is
production's unscoped case.

Docs: docs/Authorization.md's constraint section states the two-state distinction
and both fail behaviours under browse_subtrees.
2026-08-20 09:28:54 -04:00
Joseph Doherty 282a0e70ad fix(GWC-34): fail an attach that races session disposal instead of hanging it
EnsureDistributorCreated read a null _eventDistributor as "not created yet",
which is true before MarkReady and false after DisposeAsync nulled it. An
attach that passed its readiness gate while the session was still Ready and
then lost the CPU across the whole close + dispose therefore built a
REPLACEMENT distributor: startNow came back false because the pump-start flag
is a one-way latch, so the subscriber registered on a distributor whose pump
never starts and whose channel is never completed, and EventStreamService
blocked in WaitToReadAsync until the client's own deadline — no error, no log,
no metric, the backlog gauge reporting 0 forever. This is the disposal-side
twin of the premature-attach poison GWC-27 closed.

DisposeAsync now sets _eventDistributorDisposed under _syncRoot in the same
section that nulls the distributor, and EnsureDistributorCreated throws
SessionManagerException(SessionNotReady) when it is set. The attach paths'
existing catch/rollback releases the subscriber slot, so a second attempt
still reports SessionNotReady rather than EventSubscriberAlreadyActive.

StartDashboardMirror is the one caller that is not an attach: it abandons the
mirror (releasing its loop token source) instead of propagating, so MarkReady
stays total. Losing that race previously left a started orphan distributor
behind, so this path strictly improves too.

Test: AttachAfterDisposeThrowsSessionNotReadyInsteadOfRegisteringOnANeverPumped-
Distributor covers all three attach entry points plus the rollback, and fails
on the pre-fix code. docs/Sessions.md records the disposal end of the
one-distributor-per-session rule.
2026-08-20 09:25:21 -04:00
Joseph Doherty 3619152b9f chore(GWC-31): relabel perf-pass GWC-3x comments to PERF-nn; tracker housekeeping
Sweep code/doc comments citing unregistered GWC-30/GWC-31 (2026-08-15 perf
pass) to PERF-02/PERF-07/PERF-22, keyed to task numbers in
docs/plans/2026-08-15-perf-review-remediation.md; add a PERF-ID register
table to that plan. Genuine cycle-2 GWC-30 citations (gateway-side frame-
reader 4-byte prefix fix in Server/Workers/WorkerFrameReader.cs and its
test) are left untouched.

Add a line to REVIEW-PROCESS.md reserving the GWC/WRK/IPC/SEC/CLI/TST
prefixes for the archreview domain registers.

Tracker housekeeping per archreview/2026-08-19/00-overall.md:
- GWC-16 -> Won't fix (rationale: PERF(followup) comment documents the
  double session lookup as measured-negligible, deliberately not collapsed)
- IPC-08 -> Done (closed by 7c1ea12/a1a38b5; residual tracked as IPC-34)
- IPC-31 -> Done (superseded by GWC-28's write-time sequence stamping)
- TST-06, TST-17 rows refreshed per cycle-3 60-testing-docs-gaps.md
- WRK-31 (and defensively WRK-29/30) registered as reserved/do-not-reuse
  interim IDs burned by e913dab, in both the cycle-2 tracker and the
  cycle-2 worker remediation doc

Flip Batch C to Done in archreview/2026-08-19/remediation/00-tracking.md.

Worker-side files (Worker/Ipc/WorkerFrameReader.cs, WorkerFrameWriter.cs)
are comment-only edits, unverified here (net48 x86, windev-only build).
2026-08-20 09:23:36 -04:00
Joseph Doherty ee68cc3f72 Merge fix/c3-client-versions: CLI-46 client versions bumped past the published train (0.2.1 / Java 0.2.2), pypi 0.2.0 broken-connect() note, check-client-versions.ps1 drift guard in CI 2026-08-20 09:22:48 -04:00
Joseph Doherty 7200023bcc fix(CLI-46): bump client versions past the already-published train + add PR-time drift guard
.NET/Go/Python/Rust clients (plus Contracts, which releases in lockstep
with the .NET Client) move 0.2.0 -> 0.2.1; Java moves 0.2.1 -> 0.2.2. This
starts the next release train the finding recommended, since post-publish
source changes (including the Python connect() event-loop fix, a8f86b5)
had been sitting at already-published version numbers for nine days with
nothing recording the divergence.

- Note the pypi 0.2.0 connect() defect and its supersession by 0.2.1 in
  clients/python/README.md and docs/ClientPackaging.md.
- Add scripts/check-client-versions.ps1: a PR-time guard that finds each
  client's latest publish tag (clients/go/vX.Y.Z for Go, the only client
  with a git tag; the other four publish straight to a registry with no
  tag, so the check reports "no tags found" and passes for them) and fails
  if clients/<lang> changed since that tag while the version constant
  still equals the tag's version. Wired into .gitea/workflows/ci.yml's
  portable job next to the codegen freshness checks (needs fetch-depth: 0
  for tag history).
- Mark remediation batch A Done in
  archreview/2026-08-19/remediation/00-tracking.md.

Verified on macOS: dotnet build clients/dotnet/...slnx (pass), cargo check
--workspace (pass), python -m pytest (168 passed, 1 skipped), gradle test
with JDK 17 (pass, no generated-tree churn to revert this time), gofmt -l
+ go build ./... + go test ./... (clean), and
scripts/check-client-versions.ps1 via pwsh (pass; also verified it fails
correctly when a version constant is rolled back).
2026-08-20 09:22:28 -04:00
Joseph Doherty 058d8d529f fix(GWC-32): make WorkerClient disposal a terminal transition
DisposeAsync is the third path that completes the outbound, staging and event
channels, but unlike MarkClosed and SetFaulted it left _state untouched. A
client disposed while still Ready therefore kept reporting
IsTerminalState() == false, so a read loop already inside DispatchEnvelope when
the staging channel was completed took StageWorkerEvent's overflow arm: a
spurious ProtocolViolation fault, a worker-event-staging overflow metric, a
redundant kill and a WorkerStopped("ProtocolViolation") metric during an
intentional teardown — on exactly the path an operator inspects after a failed
SessionManager.OpenSessionAsync.

DisposeAsync now takes _syncRoot once, claims _disposed there, and transitions
to Closed only when the state is not already Closed or Faulted, so a
dispose-after-fault keeps Faulted and a double dispose is a no-op. Because the
transition is now real, the dispose path records the worker stop through the
existing RecordWorkerStoppedOnce latch, which also stops the running-worker
gauge leaking when a Ready client is disposed without a prior close or fault.
MarkClosed is deliberately not reused: it would overwrite Faulted with Closed
and would re-run a teardown sequence DisposeAsync performs itself.

The StageWorkerEvent XML doc now names all three completion paths.

Test: DisposeAsync_WhileWorkerIsEmittingEvents_ClosesWithoutFaultingOrOverflowing
disposes a Ready client mid-event-stream and asserts Closed state, zero faults,
zero queue overflows, zero kills and exactly one recorded worker exit. It fails
on the pre-fix code.
2026-08-20 09:22:17 -04:00
Joseph Doherty aa07497cba Merge fix/c3-worker-residuals: WRK-33 observed command-task wrapper, WRK-32 pre-baseline pump + recency contract documented, WRK-34 Watchdog CommandMethod label 2026-08-20 09:21:42 -04:00
Joseph Doherty f1353dda86 chore(plans): mark batch F (WRK-32, WRK-33, WRK-34) done-pending-windev 2026-08-20 09:21:16 -04:00
Joseph Doherty 3679e59eed fix(WRK-34): stop writing the watchdog correlation id into CommandMethod
ReportWatchdogFaultIfNeededAsync built the StaHung fault with
CommandMethod = snapshot.CurrentCommandCorrelationId, so the wire
fault's command_method field carried an opaque correlation id (or
empty) instead of a method name -- a consumer reading CommandMethod as
a method name got a confusing value. Use a stable "Watchdog" label and
fold the stuck command's correlation id (when present) into the
diagnostic message instead. No test asserts CommandMethod for the
StaHung fault; the existing assertions only check Category and that
DiagnosticMessage contains "STA activity is stale", both unchanged.
2026-08-20 09:20:34 -04:00
Joseph Doherty f59c34a544 Merge fix/c3-glauth-doc: TST-31 glauth.md reconciled against config.toml, live WriteSecured credential re-derived, auth outcome logged 2026-08-20 09:20:28 -04:00
Joseph Doherty 20a3eefb1e fix(WRK-32): pre-baseline pump for write-completion correlation + honest contract docs
Write-completion correlation is recency-based per (server_handle,
item_handle): OnWriteComplete carries no per-call token, so the executor
attributes the first completion whose version exceeds a pre-call baseline
to that write. A stale completion from an earlier timed-out write to the
same handle could ride the next write's reply if it was already queued
when the baseline was captured.

Add CaptureCompletionBaseline (Write, Write2, WriteSecured,
WriteSecured2) to run one pumpStep() immediately before snapshotting
CurrentVersion, so an already-queued stale completion folds into the
baseline instead of being available to the next wait. This narrows, but
per the finding cannot close, the correlation gap -- a completion queued
after the pump step and before the COM call returns (or during the wait)
is still indistinguishable from the new write's own completion; closing
it needs an MXAccess-side correlation token that does not exist.

Rewrite the AwaitWriteCompletion doc comment to state the recency-based
contract honestly instead of implying exact per-call matching, and add
the same contract to docs/MxAccessWorkerInstanceDesign.md's
write-completion section.

Deferred: the reply-field comment in
src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto is not
edited here. A comment-only proto change still requires regenerating
every generated artifact (C#, Go, Python, Java, Rust, protoset) in the
same commit per repo convention, and this macOS box cannot verify all
five client generator pins together for a Low-severity doc-only change.
The contract is documented in MxAccessWorkerInstanceDesign.md instead;
the existing proto comment ("concurrent writes... can swap rows") is not
wrong, just less precise than the new doc section.
2026-08-20 09:20:28 -04:00
Joseph Doherty 22d5d52c14 fix(TST-31): reconcile glauth.md with the shared GLAuth directory, fix propagated live-test credential
glauth.md's OPC-perm group table and pre-provisioned-users table described a
directory that no longer exists: all five OPC-perm gids were off by 100
(5501-5505 vs actual 5601-5605), and the user table listed
readonly/writeop/writetune/writeconfig/alarmack accounts that scadaproj's
shared GLAuth (config.toml) never provisioned. The real per-role testers are
opc-readonly/opc-writeop/opc-writetune/opc-writeconfig/opc-alarmack (renamed
to avoid a cn= case-collision with their group), and every dev user's
password is "password" per config.toml's own comment and passsha256 (not
admin123). admin itself is no longer a member of any OPC-perm group in the
shared directory (only GwAdmin + OtOpcUa-Admins) - multi-role is the actual
"covers every gw capability" account now.

The wrong admin123 default had propagated into
MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD (docs/GatewayTesting.md) and
the WorkerLiveMxAccessSmokeTests.cs literal/comment attributing the rig's
GLAuth-bridged ArchestrA identity - both corrected to password. Also made
the WriteSecured parity test log (never assert) whether AuthenticateUser
succeeded or was rejected: MXAccess parity means a bad credential is a
legitimate failure reply, so it must not fail the test, but a stale rig
credential was silently degrading the nightly live smoke's WriteSecured leg
from an authenticated round-trip to a bare failure-reply-shape check with no
visible signal.

Verified: dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx succeeds
(IntegrationTests builds cleanly on macOS, net10.0, no COM/x86 dependency).

Flips Batch B (TST-31) to Done in the cycle-3 remediation tracker.
2026-08-20 09:19:59 -04:00
Joseph Doherty 93edb1b74d fix(WRK-33): observe the discarded command-task wrapper's fault
TryStartCommandTask discarded ObserveCommandTaskAsync's task without a
fault-observing continuation -- the one gap in the NEXT-04 discipline
applied everywhere else in this file. ObserveCommandTaskAsync only
swallows OperationCanceledException, so any other exception escaping
ProcessCommandAsync's own catch path would fault the discarded wrapper
and surface only as a finalizer-time unobserved-task exception. Wrap it
with the existing ObserveFaultWhenever helper, same as every other
discarded task in this file.
2026-08-20 09:17:21 -04:00
Joseph Doherty ddfd41b06d docs(archreview): cycle-3 remediation plan + tracker (batches A-G, model assignments) 2026-08-20 09:16:52 -04:00
Joseph Doherty 7c5eb062e3 docs(archreview): cycle-3 remediation plan + tracker (batches A-G, model assignments) 2026-08-20 09:16:14 -04:00
Joseph Doherty b75f8b1704 docs(archreview): cycle-3 remediation plan + tracker (batches A-G, model assignments) 2026-08-20 09:15:50 -04:00
Joseph Doherty 6c784e3db7 docs(archreview): cycle-3 remediation plan + tracker (batches A-G, model assignments) 2026-08-20 09:14:40 -04:00
357 changed files with 29035 additions and 4871 deletions
+23 -6
View File
@@ -4,16 +4,16 @@
# .NET Framework 4.8 / x86 and need MXAccess COM installed, so they build ONLY on a Windows host.
# They are out of scope for this Linux `portable` job — the `windows-x86` job below builds and
# tests them on windev (10.100.0.48) over SSH (see scripts/ci/run-windev-ci.sh), and the nightly
# `nightly-windev` job additionally runs the live-MXAccess smoke, which is scheduled-only so it
# never gates a push.
# `nightly-windev` job additionally runs the full solution build, gateway suite, and live-MXAccess
# smoke, which are scheduled-only so they never gate a push.
name: ci
on:
push:
pull_request:
schedule:
# 06:00 UTC nightly: the `nightly-windev` job re-runs the x86 Worker build + full Worker.Tests on
# windev, then the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full-slnx build.
# 06:00 UTC nightly: x86 Worker build + Worker.Tests, full-slnx build, gateway tests, then the
# live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1), serialized on windev's host lock.
# Push/PR runs skip it — the job is gated `if: github.event_name == 'schedule'`.
- cron: '0 6 * * *'
@@ -23,7 +23,12 @@ jobs:
# tests, the codegen/descriptor freshness guards, and the clients that build on Linux.
runs-on: ubuntu-latest
steps:
# fetch-depth: 0 (full history + tags): the CLI-46 client-version-freshness check below
# diffs each client's source against its last publish tag (e.g. clients/go/vX.Y.Z), which
# a shallow, tagless checkout cannot resolve.
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-dotnet@v4
with:
@@ -77,6 +82,18 @@ jobs:
shell: pwsh
run: ./scripts/check-codegen.ps1
# CLI-46: catches a client whose source changed since its last publish
# tag while the version constant still reads that tag's already-shipped
# number — the drift the pack-clients.ps1/tag-go-module.ps1 collision
# guards only catch at actual publish time. Needs tag history, hence
# `fetch-depth: 0` on this job's checkout above.
- name: Client version freshness
shell: pwsh
run: ./scripts/check-client-versions.ps1
- name: Code-reviews README consistency
run: python code-reviews/regen-readme.py --check
- name: Gateway fake-worker tests
run: dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj -c Release --no-build
@@ -165,7 +182,7 @@ jobs:
run: ./scripts/ci/run-windev-ci.sh test
nightly-windev:
# TST-25: scheduled deep run — full Worker.Tests + live-MXAccess smoke + full-slnx build on windev.
# TST-35: scheduled deep run — Worker, full-slnx, gateway, and live-smoke tiers on windev.
# Gated to the schedule event via `if:` (not an unsatisfiable `runs-on`), so push/PR runs mark it
# skipped, not queued. Never gates a push. On failure it opens a Gitea issue so a red nightly is
# visible even though nobody watches the Actions page.
@@ -181,7 +198,7 @@ jobs:
PUBLIC_SERVER_URL: https://gitea.dohertylan.com
steps:
- uses: actions/checkout@v4
- name: Full Worker.Tests + live-MXAccess smoke on windev
- name: Worker + gateway tests + live-MXAccess smoke on windev
env:
# WINDEV_SSH_KEY is stored base64-encoded (single line) so Gitea's line-oriented secret
# masker redacts it in the step env echo; run-windev-ci.sh decodes it. A raw multiline PEM
+1
View File
@@ -155,3 +155,4 @@ generated-scratch/
# Agent worktrees (subagent isolation) — never commit
.claude/worktrees/
.worktrees/
+22 -4
View File
@@ -49,7 +49,18 @@ mxaccessgw changes here — remote/push status, the `.proto` contracts (`mxacces
`mxaccess_worker.proto`, `galaxy_repository.proto`), the two-process architecture, shared-lib
consumption, or per-project commands — update the **MxAccessGateway entry in
[`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index
never drifts from this repo. (Mirrors the same rule in the peer repos.)
never drifts from this repo. (Mirrors the same rule in the peer repos.) **Keep the index row to
a few lines** — since 2026-09-03 the umbrella keeps only short summary rows and parks long
per-project narrative in `../scadaproj/.claude/skills/sister-project-history/<project>.md`
(create `mxaccessgateway.md` there if this gateway ever needs one) and shared-library history
in `../scadaproj/<lib>/CLAUDE.md`.
> ⛔ **Wonder box remote access (2026-09-11): no `psbridge`, no `winrm`.** Both skills are retired/disabled —
> the WinRM remote-exec channel set off the security software on the wonder boxes (vd03/vq03/vq04). Do not
> run remote commands (service control, publish/swap, PowerShell) on those boxes and do not stand up a
> replacement channel until the user clears one. Still allowed: UNC/SMB file copies and logging directly
> into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella
> [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md) for the current hold.
## Build, Test, Run
@@ -115,11 +126,18 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
- **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `docs/DesignDecisions.md` (Session-Resilience Epic Scope, session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`.
- **Gateway restart does not reattach orphan workers.** The first version terminates orphaned workers on startup; do not design code paths that assume reattachment.
- **No Blazor UI component libraries.** Dashboard uses local Bootstrap CSS/JS only — do not introduce MudBlazor, Radzen, FluentUI, etc.
- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, and `AuthenticateUser` credentials must never reach logs. Value logging is opt-in and redacted.
- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, `AuthenticateUser` credentials, and `AcknowledgeAlarm` `operator_password` must never reach logs. Value logging is opt-in and redacted.
- **Generated code** under `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`, `clients/*/generated*/`, `clients/python/src/zb_mom_ww_mxgateway/generated/`, etc., is build output. Don't hand-edit. To regenerate, build the contracts project (`dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj`) or run the per-client generation step in that client's README.
- **Documentation style** (`StyleGuide.md`): PascalCase filenames, no marketing language, present tense, explain *why* not *what*.
- **Update docs in the same change as the source.** When public APIs, contracts, configuration, build steps, security behavior, event shapes, value conversion, status mapping, or lifecycle rules change, the affected docs (`gateway.md`, `docs/`, client READMEs, design docs) must change in the same commit. Don't leave stale prose describing old behavior.
## Backlog / open work
- **Open work lives in Gitea issues** on `gitea.dohertylan.com/dohertj2/mxaccessgw` (`tea issues list --repo dohertj2/mxaccessgw`). The 2026-11-06 migration filed the open markdown items as issues labelled `migrated` (#169–#213).
- **Markdown backlogs are historical records with `→ #N` pointers**, not status registers: the archreview trackers (`archreview/remediation/00-tracking.md`, `archreview/2026-07-12/remediation/00-tracking.md` + `90-candidate-findings-next-cycle.md`, `archreview/2026-08-19/remediation/00-tracking.md`, `archreview/2026-10-21/remediation/00-tracking.md`), `stillpending.md`, `docs/plans/2026-06-14-deferred-followups.md`, and `code-reviews/*/findings.md`. Do not update status there.
- **File new follow-ups, deferrals and review findings as issues**, using the family labels `type:{bug,feature,followup,review-finding,deploy-gate,debt,docs,test,infra}`, `priority:p0`..`p3`, `blocked`, `cross-repo`.
- Reference issues from plans and docs as `#N` (cross-repo: `lmxopcua#N`, `ScadaBridge#N`).
## Source Update Workflow
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
@@ -157,9 +175,9 @@ External analysis sources referenced by design docs:
Gateway gRPC clients authenticate with an API key in metadata: `authorization: Bearer mxgw_<key-id>_<secret>`. Keys are stored hashed (with a peppered SHA) in a gateway-owned SQLite DB (default `C:\ProgramData\MxGateway\gateway-auth.db`). Scopes (`session:open`, `session:close`, `invoke:read`, `invoke:write`, `invoke:secure`, `events:read`, `metadata:read`, `admin`) gate specific RPCs; missing → `Unauthenticated`, insufficient → `PermissionDenied`. The `apikey` subcommand on the server exe manages keys; see `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/`.
Session event streaming is **owner-scoped**: the API key that opened a session is recorded on the session, and every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless the caller's key id matches the owner. Possessing the `event` scope and knowing a session id is not sufficient — this closes the reconnect/fan-out trust boundary (detach-grace and replay retention are on by default) so an `event`-scoped key cannot attach to another key's retained session.
Session event streaming is **owner-scoped**: the API key that opened a session is recorded on the session, and every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless the caller's key id matches the owner. Possessing the `events:read` scope and knowing a session id is not sufficient — this closes the reconnect/fan-out trust boundary (detach-grace and replay retention are on by default) so an `events:read`-scoped key cannot attach to another key's retained session.
Dashboard auth is LDAP-backed (separate from the gRPC API-key model). `/login` binds against `MxGateway:Ldap` and maps the user's LDAP groups to `Admin` or `Viewer` via `MxGateway:Dashboard:GroupToRole`, then issues an HTTP-only secure cookie named `__Host-MxGatewayDashboard` when `Dashboard:RequireHttpsCookie` is true (default) and no `Dashboard:CookieName` override is set, else the plain `MxGatewayDashboard` (the `__Host-` prefix requires a Secure cookie). SignalR hubs at `/hubs/{snapshot,alarms,events}` accept either the cookie or a 5-minute bearer minted at `/hubs/token`. Dashboard event visibility is **tag-scoped per session** (`IDashboardSessionAcl`, gating both the events hub and the session-details page's in-process subscribe): an Administrator sees every session, while any other caller sees a session only when its tags — inherited from the owning API key's `apikey --dashboard-tags`, never from the client's request — intersect the tags their LDAP groups grant via `Dashboard:GroupToTag`; untagged sessions follow `Dashboard:UntaggedSessionVisibility` (default `AdminOnly`), and a principal with no tag claims (anonymous localhost included) is an empty-grant Viewer. `Dashboard:AllowAnonymousLocalhost` (default `true`) grants **read-only** loopback access — it satisfies the Viewer requirement but never the Admin-only requirement, so anonymous localhost can view the dashboard but not reach API-key CRUD or session Close/Kill (`Authentication:Mode=Disabled` is scoped the same way). `Dashboard:DisableLogin` (default `false`) auto-authenticates every dashboard request — including remote browsers — as `Dashboard:AutoLoginUser` (default `multi-role`) with both Admin and Viewer roles; dev/test only, never enable in production.
Dashboard auth is LDAP-backed (separate from the gRPC API-key model). `/login` binds against `MxGateway:Ldap` and maps the user's LDAP groups to `Admin` or `Viewer` via `MxGateway:Dashboard:GroupToRole`, then issues an HTTP-only secure cookie named `__Host-MxGatewayDashboard` when `Dashboard:RequireHttpsCookie` is true (default) and no `Dashboard:CookieName` override is set, else the plain `MxGatewayDashboard` (the `__Host-` prefix requires a Secure cookie). SignalR hubs at `/hubs/{snapshot,alarms,events}` accept either the cookie or a 5-minute bearer minted at `/hubs/token`; `/hubs/alarms` and the `/alarms` page additionally require a **real authenticated dashboard role** (`MxGateway.Dashboard.Alarms`, a requirement type the environment-bypass handler does not handle), because the alarm feed is gateway-central with no per-session provenance and so cannot be tag-scoped — its metadata stays galaxy-wide for authenticated Viewers, but anonymous localhost and `Authentication:Mode=Disabled` do not receive it at all. Dashboard event visibility is **tag-scoped per session** (`IDashboardSessionAcl`, gating both the events hub and the session-details page's in-process subscribe): an Administrator sees every session, while any other caller sees a session only when its tags — inherited from the owning API key's `apikey --dashboard-tags`, never from the client's request — intersect the tags their LDAP groups grant via `Dashboard:GroupToTag`. That grant is **re-resolved from the caller's LDAP group claims against current options on every subscribe** (not read from the tag claims stamped at login, which would outlive a config-revoked grant for the cookie's 8-hour sliding life); login-stamped tag claims are the fallback only for a principal carrying no group claims — i.e. the hub-token principal, whose grant the mint already re-resolved. Untagged sessions follow `Dashboard:UntaggedSessionVisibility` (default `AdminOnly`), and a principal with neither claim kind (anonymous localhost included) is an empty-grant Viewer. `Dashboard:AllowAnonymousLocalhost` (default `true`) grants **read-only** loopback access — it satisfies the Viewer requirement but never the Admin-only requirement, so anonymous localhost can view the dashboard but not reach API-key CRUD, session Close/Kill, or the alarm feed (`Authentication:Mode=Disabled` is scoped the same way). `Dashboard:DisableLogin` (default `false`) auto-authenticates every dashboard request — including remote browsers — as `Dashboard:AutoLoginUser` (default `multi-role`) with both Admin and Viewer roles; dev/test only, never enable in production.
## Process / Platform Notes
+9 -2
View File
@@ -79,6 +79,12 @@ Add one entry per finding to the `## Findings` section of the module's
- **Finding ID** — `<Module>-NNN`, numbered sequentially within the module and
never reused (e.g. `Worker-001`). IDs are permanent even after resolution.
The `GWC`/`WRK`/`IPC`/`SEC`/`CLI`/`TST` prefixes are reserved for the
`archreview/` domain registers (a separate, whole-repo review series with its
own tracking documents) — never assign one of these prefixes to a
`code-reviews/` finding, and any other work stream (perf passes, ad hoc
remediation plans, etc.) that wants a stable per-item ID must mint its own
distinct prefix rather than borrowing one of these six.
- **Severity:**
- **Critical** — data loss, security breach, crash/deadlock, or outage.
- **High** — incorrect behaviour with significant impact; no safe workaround.
@@ -127,8 +133,9 @@ finding carries an unrecognised Status value. The PowerShell wrapper
`scripts/check-code-reviews-readme.ps1` runs that check and is the intended hook
for CI or a pre-commit step.
> The repo's installed `python` is the real interpreter; the bare `python3`
> alias resolves to the Windows Store stub and fails. Use `python`.
> On Windows, the installed interpreter may be available only as `python`
> because `python3` can resolve to the Windows Store stub. macOS and Linux
> normally use `python3`; CI uses the configured `python` executable.
The per-module `findings.md` files are the source of truth; `README.md` is the
aggregated index and must always agree with them — which the script guarantees.
@@ -1,5 +1,10 @@
# MxAccessGateway — Remediation Tracking (2026-07-12 review)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
>
> Rows still open in this cycle's tracker were re-triaged and carried into the [2026-10-21 tracker](../../2026-10-21/remediation/00-tracking.md) (`R21-*` rows; cycle-1 rows promoted as `R21-O1`..`R21-O34`), whose `→ #N` pointers name the Gitea issues.
Master progress tracker for the 2026-07-12 follow-up architecture review. Generated 2026-07-13.
Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation designs are linked below and hold the full **Finding / Impact / Design / Implementation / Verification** for every entry here. The first-cycle tracker ([`../../remediation/00-tracking.md`](../../remediation/00-tracking.md)) remains the record for the original 153 findings; this document tracks only the 47 new IDs (GWC-24+, WRK-21+, IPC-23+, SEC-31+, CLI-35+, TST-25+) plus the old-tracker actions listed at the end.
@@ -79,6 +84,9 @@ Full design + implementation for each row lives in the linked domain doc under i
| WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |
| WRK-29 | — | — | — | — | Reserved (do not reuse) | Interim ID space possibly burned alongside WRK-31 by `e913dab`'s ad hoc numbering; not confirmed in the repo or its history, held back defensively |
| WRK-30 | — | — | — | — | Reserved (do not reuse) | Interim ID space possibly burned alongside WRK-31 by `e913dab`'s ad hoc numbering; not confirmed in the repo or its history, held back defensively |
| WRK-31 | — | — | — | — | Reserved (do not reuse) | Interim ID burned by `e913dab` (session-owned transport disposal), cited in code at `Ipc/WorkerPipeSession.cs:45,177` and `Ipc/WorkerPipeSessionTests.cs` outside the register; not a finding — registered per cycle-3 `20-worker.md` (~line 9) so it is never reassigned. Cycle-3 new findings start at WRK-32. |
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
@@ -92,7 +100,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| IPC-28 | Low | — | S | — | Done | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
| IPC-29 | Low | — | S | WRK-26 (discharged by) | Done (discharged by WRK-26) | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
| IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
| IPC-31 | Info | — | — | — | Done (superseded by GWC-28) | Gateway creation-time sequence stamping — resolved when GWC-28 moved gateway stamping to write time; the two sides now implement the same semantics |
| IPC-32 | Info | — | S | IPC-25 (folded in) | Done | check-codegen banner relabel 1/4…4/4 |
### Security & dashboard — [40-security-dashboard.md](40-security-dashboard.md)
@@ -152,6 +160,14 @@ Sequence these together rather than piecemeal — several are one change set spa
- ~~When CLI-38 lands, close old **CLI-08** with a pointer here.~~ Done 2026-08-07: CLI-38 landed and old CLI-08 is now `Done` in the first-cycle tracker, pointing at [CLI-38](50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1).
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). ~~When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.~~ TST-05 revisited 2026-08-10: `Partially done` in the first-cycle tracker — the `nightly-windev` job closes the scheduled-cadence half, but the finding's coverage-audit half stays open (the live suite reaches all six late-added COM commands and none of the five control commands). TST-24 revisited in the same change and closed `Done`: Go/Rust/Java already had real-server wire tests, and the two genuine gaps (.NET, Python) now have them plus a CI step.
## 2026-08-25 documentation correction
The 2026-08-07 IPC-23 change-log entry's `docs/Grpc.md` deviation rationale is inaccurate:
`DrainEvents` is not private worker IPC. It is a public `MxCommandKind` accepted by the
`Invoke` RPC. Cycle-3 IPC-38 adds the missing public count-cap, byte-cap,
drain-until-empty, and destructive-stream interaction contract to `docs/Grpc.md` while
preserving the historical entry below.
## Change log
| Date | Change |
@@ -24,6 +24,7 @@ members, no positional records). The worker builds and tests only on the Windows
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
| WRK-31 | — | — | — | — | Reserved (do not reuse) | Not a finding. `e913dab` (session-owned transport disposal) coined "WRK-31" ad hoc in code comments (`Ipc/WorkerPipeSession.cs:45,177`) and in `Ipc/WorkerPipeSessionTests.cs`, before the register assigned it. WRK-29/WRK-30 appear nowhere in the repo or its history but may have been burned by the same interim numbering, so they are held back defensively too — see the tracker's WRK-29/30/31 rows. Registered here per cycle-3 `archreview/2026-08-19/20-worker.md` (~line 9, "Finding-ID note"); that review's new findings start at **WRK-32** to guarantee no collision. |
---
@@ -231,6 +231,8 @@ The published-side collection (`CollectPublishedSymbols`, `:61-79`) grows matchi
**Disposition: N/A — no action.** `gateway.md:328-330` already declares sequence a per-sender diagnostic aid, never validated; both validators ignore it; the divergence is harmless and now explicitly documented by the IPC-29 doc section (which states the two sides' stamping points). If sequence ever becomes load-bearing, stamp in the gateway write loop (`WorkerClient.cs:390-397`) — recorded here so the next editor finds the decision.
**Resolution: Done — superseded by GWC-28 (2026-08-20, per cycle-3 verification).** GWC-28 moved the gateway-side stamp from creation to the point of writing: `WriteLoopAsync` now stamps `Sequence` immediately before `_writer.WriteAsync` (`Server/Workers/WorkerClient.cs:465-471`), and `CreateEnvelope` leaves it unset (`:1229-1236`); `gateway.md:425-431` was rewritten to "both sides stamp at the point of writing", closing the divergence this finding recorded. Regression test `WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequencesOnTheWire` covers it. Per `archreview/2026-08-19/30-contracts-ipc.md` (IPC-31 row), tracker flipped from `N/A` to `Done (superseded by GWC-28)`. See cycle-3 **IPC-39** for the one interaction the change created with the IPC-03 pre-check.
---
## IPC-32 — `check-codegen.ps1` check labels are miscounted `Info` · `—` (rides IPC-25)
@@ -1,5 +1,10 @@
# Candidate Findings for the Next Review Cycle (surfaced during 2026-07-12 remediation)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
>
> Rows still open in this cycle's tracker were re-triaged and carried into the [2026-10-21 tracker](../../2026-10-21/remediation/00-tracking.md) (`R21-*` rows; cycle-1 rows promoted as `R21-O1`..`R21-O34`), whose `→ #N` pointers name the Gitea issues.
These were discovered while remediating the 2026-07-12 backlog but were **out of scope** for it — each is either pre-existing, by-design residual, or a new observation. They are recorded here so the next review cycle can triage them. None blocks the 2026-07-12 cycle, which is complete. Rows struck through have since been fixed ahead of that cycle; the original finding text is kept so the triage record stays readable.
| ID (proposed) | Area | Severity (est.) | Summary |
@@ -11,7 +16,7 @@ These were discovered while remediating the 2026-07-12 backlog but were **out of
| ~~NEXT-05~~ | Worker frame writer | Info | **Resolved 2026-08-10 as a documented decision** — the lazy `DequeueNext` purge stays: any subsequent write drains both queues to empty and the heartbeat loop bounds tombstone residency to one interval, while eager `Queue<T>` rebuilds under `_gate` would add ordering-invariant surface next to the WRK-22 interlock for no real gain. Rationale recorded in `docs/WorkerFrameProtocol.md`. Original finding: A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. |
| ~~NEXT-06~~ | Testing / live LDAP | Medium | **Resolved 2026-08-07** — fixtures realigned to the shared directory (`admin`/`password` for the GwAdmin success path, `gw-viewer`/`password` for the bind-succeeds-but-no-role path); verified `Failed: 0, Passed: 5` live against the shared GLAuth at `10.100.0.35:3893`, so the success-path assertion (GwAdmin group claim + Admin role claim) now fails if the service-account credential is wrong. Original finding: `DashboardLdapLiveTests` fixtures have drifted from the shared GLAuth directory, leaving the suite with **no positive-proof coverage of the service-account bind**. Its only success-path test, `AuthenticateAsync_AdminInGwAdminGroup_Succeeds`, binds `admin`/`admin123`, but the directory's `admin` user carries the standard dev password (`scadaproj/infra/glauth/config.toml`), so that assertion cannot pass. `AuthenticateAsync_ReadOnlyUserMissingGwAdminGroup_Fails` binds fixture user `readonly`, which **does not exist** in the GLAuth config at all — it passes for the wrong reason (user-not-found rather than the group-missing branch it names; the `readonly` name is in fact barred by the README's user/group case-collision rule). The three remaining tests are negative assertions that pass whether or not the service account can bind. Net effect: a green `DashboardLdapLiveTests` run proves nothing about the bind credential — surfaced during SEC-36, where the suite was considered as a substitute for the deferred dashboard-login check and rejected. Fix: realign the fixtures to real directory users (e.g. `multi-role`/`gw-viewer`) or add the missing users to the GLAuth config, and add one test that fails when the service-account credential is wrong. |
| ~~NEXT-07~~ | Deployment / windev | High | **Resolved 2026-08-07** — a fresh portable framework-dependent publish of `origin/main` (`a346d51`) was built in a clean clone at `C:\build\mxgw-redeploy`, deployed to `C:\publish\mxaccessgw\Server-20260807`, and the `MxAccessGw` NSSM service repointed at it; the service now holds a stable PID with both `5120`/`5130` listening, a worker spawned, the Galaxy snapshot restored (129 objects / 56,731 attributes) and a clean event log. Root cause confirmed as the version skew this row predicted: the deployed 2026-06-25 build carried `ZB.MOM.WW.Auth.ApiKeys` 0.1.2.0, which supports auth-DB schema 2, against a `gateway-auth.db` stamped at schema 3 on 2026-07-15 by an ephemeral run of newer code — schema 3 is the current shared-lib version (`SqliteAuthSchema.CurrentVersion=3` in Auth 0.1.5), so the redeploy is the forward fix and the DB was left alone. Rollback artifacts kept: `C:\ProgramData\MxGateway\gateway-auth.db.bak-next07` (with `-wal`/`-shm`) and the previous `C:\publish\mxaccessgw\Server` directory. Two side effects worth recording: the old deploy's `appsettings.json` held the LDAP bind password in **plaintext on disk**, while the new one keeps the repo's `${secret:ldap/mxgateway/bind}` token with the NSSM environment supplying the value, so no plaintext LDAP secret remains on that host; and the redeploy tripped the SEC-06 `Ldap:Transport=None` production hard-stop (`GatewayOptionsValidator.cs:178`), resolved by relabelling the host — windev runs `Dashboard:DisableLogin=true`, which this repo's own docs mark dev/test-only, so its `Production` label contradicted its configuration and `DOTNET_ENVIRONMENT` was changed to `Staging` (that one NSSM environment entry only; the other nine preserved byte-identical). SEC-06 is untouched for genuinely production hosts — see NEXT-08 for the posture problem that relabelling defers. Original finding: The `10.100.0.48` (windev) gateway deployment is **stale and crash-looping**, and has been since at least 2026-08-06 (~10k Hosting-failed events/day). The deployed Server binary dates to 2026-06-25 and predates the auth-DB migration of 2026-07-15: it opens a schema-version-3 `gateway-auth.db` that it supports only at version 2 and aborts at startup, so the `MxAccessGw` service never reaches a listening state. Not a code defect in the current tree — a deploy-drift/operations gap — but it means the repo's only deployed host has been dark for over a day and any host-level verification (including SEC-36's dashboard-login check) is blocked until it is repaired. Fix: deploy a current Server build to windev, or restore/downgrade the auth DB to schema 2 if the old binary must stand. Worth asking separately why a service in a permanent restart loop raised no alert. Discovered during SEC-36. |
| NEXT-08 | Security / LDAP posture | Medium | **The shared GLAuth offers no TLS, so SEC-06 makes it undeployable from a `Production`-labelled host.** `GatewayOptionsValidator` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:178`) refuses to start when `Ldap:Transport=None` in the `Production` environment, and `docs/GatewayConfiguration.md`'s `Transport` row states "Deployed hosts must set `Ldaps` or `StartTls`" — but the shared instance at `10.100.0.35:3893` has `[ldaps] enabled=false`, port `3894` closed, and answers StartTLS with `protocolError`, so neither value can work against it. That instruction is currently unsatisfiable for every host that authenticates there. windev sidestepped it on 2026-08-07 by moving to the `Staging` environment name (NEXT-07), which is honest for a dev/test rig but is not available to a real production host. Resolution needs either LDAPS/StartTLS on the shared GLAuth (certificate plus a trust story on each gateway host) or an explicit written posture decision that production gateways bind a different, TLS-capable directory. Surfaced during the NEXT-07 redeploy. |
| NEXT-08 → #180 | Security / LDAP posture | Medium | **The shared GLAuth offers no TLS, so SEC-06 makes it undeployable from a `Production`-labelled host.** `GatewayOptionsValidator` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:178`) refuses to start when `Ldap:Transport=None` in the `Production` environment, and `docs/GatewayConfiguration.md`'s `Transport` row states "Deployed hosts must set `Ldaps` or `StartTls`" — but the shared instance at `10.100.0.35:3893` has `[ldaps] enabled=false`, port `3894` closed, and answers StartTLS with `protocolError`, so neither value can work against it. That instruction is currently unsatisfiable for every host that authenticates there. windev sidestepped it on 2026-08-07 by moving to the `Staging` environment name (NEXT-07), which is honest for a dev/test rig but is not available to a real production host. Resolution needs either LDAPS/StartTLS on the shared GLAuth (certificate plus a trust story on each gateway host) or an explicit written posture decision that production gateways bind a different, TLS-capable directory. Surfaced during the NEXT-07 redeploy. |
| ~~NEXT-09~~ | Build / versioning | Low | **Resolved 2026-08-10** — the Exec path now appends `.` so the trailing backslash can no longer escape the closing quote, and `SourceRevisionId` is additionally gated on a short-SHA regex so no future git failure text can be stamped either. macOS stamp verified unchanged; Windows stamp verified on windev with this batch. Original finding: **Windows builds stamp git's error text into `InformationalVersion`.** `src/Directory.Build.props:29` runs `git -C "$(MSBuildThisFileDirectory)" …`; MSBuild's directory property ends in a backslash, which escapes the closing quote, so the command is malformed on Windows. The target carries `ContinueOnError`, so the failure is silent and git's stderr is captured as the revision — an observed stamp reads `0.1.2+fatal: cannot change to …`. Any Windows build without a preset `SourceRevisionId` therefore ships a binary that cannot be correlated back to a commit, defeating the point of TST-11. Not reproducible on macOS/Linux, where the separator is `/`. Fix sketch: append `.` to the path or trim the trailing separator before quoting. Surfaced while identifying the deployed binary during NEXT-07. |
| NEXT-10 | Docs / glauth | Medium | **`glauth.md`'s "Pre-provisioned users" table contradicts both the directory and the rest of its own file.** It documents `readonly`/`readonly123` and `admin`/`admin123`, neither of which matches `scadaproj/infra/glauth/config.toml` (`readonly` does not exist there; `admin` carries the standard dev password), and lists the `ReadOnly` gid as `5501` against an actual `5601`. Its dashboard section, by contrast, is correct — so the file is internally inconsistent and a reader cannot tell which half to trust. This table was the **root cause of the NEXT-06 fixture drift**, and it has propagated further: `docs/GatewayTesting.md`'s `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` default and the matching literal in `WorkerLiveMxAccessSmokeTests` both take `admin123` from it. Deliberately **not** fixed in the 2026-08-07 pass: the table is entangled with the OPC-UA group taxonomy (gids, role mapping, and the sister-repo consumers of the same directory), so reconciling it means sweeping that taxonomy as one unit rather than patching two rows. |
@@ -21,5 +26,5 @@ These are **live-infrastructure actions the operator must execute** — the repo
- ~~**SEC-36** — rotate the dev LDAP service-account credential per `docs/runbooks/SEC-36-ldap-credential-rotation.md` (generate new secret in `scadaproj/infra/glauth`, pre-stage the NSSM env var on deployed hosts, rotate GLAuth on `10.100.0.35`, verify dashboard login). The committed literal is gone from the working tree but remains recoverable from git history until rotation completes — **rotation is the load-bearing half.**~~ **Executed 2026-08-07**: the `serviceaccount` `passsha256` was replaced in `scadaproj/infra/glauth/config.toml` (commit `aada53b`) and the shared GLAuth recreated on `10.100.0.35`, so the literal recoverable from this repo's history no longer binds `dc=zb,dc=local`. The new value lives only in the GLAuth hash, windev's NSSM environment, and dev user-secrets. `wonder-app-vd03` was out of scope (it binds a different, `dc=scadalink`/`dc=scadabridge` directory). **Caveat, since closed:** windev's dashboard `/login` verification was deferred while that host's gateway was crash-looping on the unrelated stale-deployment fault filed as NEXT-07, so the bind was verified directly by `ldapsearch` as `cn=serviceaccount,dc=zb,dc=local` instead. After the 2026-08-07 redeploy the real check ran on windev (login as `multi-role` → 302 + dashboard cookie, anonymous control → `/login`), so the rotated credential is now proven through the `DashboardAuthenticator` path itself; see `docs/runbooks/SEC-36-ldap-credential-rotation.md` Correction 3. SEC-36 is fully `Done`.
- ~~**TST-30** — register a second Gitea `act_runner` on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` to relieve the single-shared-runner bottleneck.~~ **Executed 2026-08-07**: `gitea-runner-2` (id 5, capacity 2) is online on `10.100.0.35` via the `/opt/gitea` compose stack, same `container.network: traefik`, token from a `0600` file mount; the existing runner (id 1, capacity 4) was untouched. Concurrency verified — jobs from three runs ran simultaneously across both runners, and a `gitea-runner-2` job cloned successfully from `http://gitea:3000`. TST-30 is now fully `Done`.
- **TST-30 follow-up — reset the Gitea instance runner registration token.** Runner-1's compose block was moved to the same `0600` file-mount pattern as runner-2 on 2026-08-07 (compose and both backups now `0600 root:root`, runner-1 recreated with its identity intact), but hygiene alone does not retire the token: both runners share **one instance-scope registration token** that was world-readable for roughly five months and is still live — a probe registered runner id 6 with it, then deleted it. Gitea 1.26.4 exposes no rotation via CLI or API (both paths are get-or-create and hand back the same value), so the reset must be done in the admin web UI ("Reset registration token"). Afterwards, refresh `/opt/gitea/runner_token` on `10.100.0.35` and shred the two token-bearing compose backups — they are the last copies of the old value. See `docs/runbooks/TST-30-second-ci-runner.md`.
- **TST-30 follow-up — reset the Gitea instance runner registration token.** → #181 Runner-1's compose block was moved to the same `0600` file-mount pattern as runner-2 on 2026-08-07 (compose and both backups now `0600 root:root`, runner-1 recreated with its identity intact), but hygiene alone does not retire the token: both runners share **one instance-scope registration token** that was world-readable for roughly five months and is still live — a probe registered runner id 6 with it, then deleted it. Gitea 1.26.4 exposes no rotation via CLI or API (both paths are get-or-create and hand back the same value), so the reset must be done in the admin web UI ("Reset registration token"). Afterwards, refresh `/opt/gitea/runner_token` on `10.100.0.35` and shred the two token-bearing compose backups — they are the last copies of the old value. See `docs/runbooks/TST-30-second-ci-runner.md`.
- **TST-25 follow-ups** — old **TST-05** (scheduled live-MXAccess smoke) is now covered by the `nightly-windev` job; old **TST-24** (client wire tests in CI) is unblocked by the working Windows tier.
@@ -0,0 +1,264 @@
# MxAccessGateway — Remediation Plan & Tracking (2026-08-19 review)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
>
> Rows still open in this cycle's tracker were re-triaged and carried into the [2026-10-21 tracker](../../2026-10-21/remediation/00-tracking.md) (`R21-*` rows; cycle-1 rows promoted as `R21-O1`..`R21-O34`), whose `→ #N` pointers name the Gitea issues.
Source review: [`../00-overall.md`](../00-overall.md) (HEAD `8b3e3b3`, committed as `f4919ba`). Generated 2026-08-20.
Tracks the 34 cycle-3 IDs (GWC-31+, WRK-32+, IPC-33+, SEC-37+, CLI-46+, TST-31+) plus the tracker-housekeeping actions the overall asked for. Cycle-1/2 trackers remain the record for their own IDs.
**Status legend:** `Not started` · `In progress` · `Done` · `Blocked (external)` · `Deferred (P2)` · `N/A`.
## Execution model
This plan is executed as **seven parallel batches**, each by one subagent working in an isolated git worktree on its own `fix/c3-<batch>` branch, merged back into `main` by the coordinator in the order listed (lowest conflict surface first). Every batch must: build the affected component(s) on macOS (`NonWindows.slnx` — the x86 worker cannot build here), run only the filtered tests it touches, update docs in the same commit, and cite finding IDs in commit messages (`fix(IPC-34): …`). Worker-side changes (batches E, F) are verified afterwards on windev via `windev-worker-ci.ps1 -Mode test`.
**Model assignment rationale:** `opus` for batches that change concurrency/authorization logic or require reasoning across several interacting files with regression tests; `sonnet` for contained, well-specified edits with a clear recipe (version bumps, doc reconciliation, a two-line worker fix); `haiku` for none — every batch either touches code invariants or needs judgement about stale prose, where the cost of a wrong mechanical sweep exceeds the saving.
| Batch | Branch | Findings | Model | Why this model | Status |
|---|---|---|---|---|---|
| A | `fix/c3-client-versions` | CLI-46 | sonnet | Mechanical five-client bump + README note + one script guard; recipe is fully specified | Done |
| B | `fix/c3-glauth-doc` | TST-31 | sonnet | Reconcile a table against one TOML source of truth, re-derive one default, change an assert to a log | Done |
| C | `fix/c3-tracker-hygiene` | GWC-31 + housekeeping (close GWC-16/IPC-08/IPC-31, refresh TST-06/TST-17, register WRK-31) | sonnet | Cross-file comment/ID sweep needs care about what each ID *means*, but no logic changes | Done (`3619152`) |
| D | `fix/c3-dashboard-acl` | SEC-40, SEC-41, GWC-38 | opus | Authorization semantics, fail-closed decisions, claims re-resolution; each needs a regression test | **Done** (`b177636` GWC-38, `2ef7b93` SEC-41+SEC-40, `8257a03` comment fixup; branch not merged/pushed) |
| E | `fix/c3-ipc-boundaries` | IPC-33, IPC-39, IPC-34, IPC-35 | opus | Changes the WorkerClient command lifecycle, the options validator, and the worker drain budget; race-sensitive | **Done** (`577a0db` IPC-33, `a71663a` IPC-39, `b27035b` IPC-34, `ab41362` IPC-35; branch `fix/c3-ipc-boundaries`, not merged/pushed) — worker half verified on windev (`f2ab9cc`, Worker.Tests 524/524) |
| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Done (windev x86 build + Worker.Tests 524/524 green on `f2ab9cc`) |
| G | `fix/c3-session-lifecycle` | GWC-32, GWC-34, GWC-37 | opus | Distributor/dispose lifecycle races and an alarm-monitor re-resolve path; needs lock-discipline reasoning | Done (`058d8d5`, `282a0e7`, `1932217`) |
**Out of scope for this execution (recorded, not dropped):**
| ID | Status | Why |
|---|---|---|
| TST-32 | Blocked (external) → #181 | Gitea runner registration-token reset is a web-UI action on the Gitea admin page; no repo change. Operator to-do. |
| SEC-37 | Blocked (external) → #180 | Family-wide: TLS on the shared GLAuth lives in `scadaproj/infra/glauth/` (separate repo + docker host). Plan separately; after it lands, move the deployed service to `Production`. |
| CLI-47/48/49/50, IPC-36/37, TST-35 | Done | Activated by the 2026-08-25 P2 continuation below and closed there: CLI-47/48/49/50 in batch K (`5cdb3da`), IPC-36/37 in batch J (`d6c5a11`), TST-35 in batch M (`cc143b5`). |
| GWC-33/35, SEC-38/39 | Done | Closed by continuation batch I; regression counts and commits are recorded below. |
| GWC-36, IPC-38, TST-33/34/36 | Done | Closed by continuation batch H; verification and commit are recorded below. |
| WRK-34 | Done | Completed opportunistically in batch F (`3679e59`): watchdog faults use `CommandMethod=Watchdog` and carry the stuck correlation ID in the diagnostic message. Tracker correction recorded by the continuation audit. |
## Batch specifications
### A — CLI-46 client release-train bump (sonnet)
- Bump .NET/Go/Python/Rust clients `0.2.0 → 0.2.1`, Java `0.2.1 → 0.2.2` in every version-bearing file (`pyproject.toml` + `version.py`, `Cargo.toml` workspace + any `version =` consumers, `Directory.Build.props`/csproj `<Version>`, Java `build.gradle`/`gradle.properties`, Go module is tag-driven — update any `Version` const if one exists). Follow `docs/ClientPackaging.md` for the authoritative file list.
- `clients/python/README.md` + `docs/ClientPackaging.md`: note that published pypi 0.2.0 has a broken `connect()` (event-loop binding, fixed by `a8f86b5`) and is superseded by 0.2.1.
- Add a PR-time check to `scripts/check-codegen.ps1` (or a new `scripts/check-client-versions.ps1` wired where the other checks run): fail if any `clients/<lang>` source differs from its last publish tag while the version constant still equals that tag's version. Document in `docs/ClientPackaging.md`.
- Verify: `dotnet build clients/dotnet/...slnx`; `cargo check --workspace`; `python -m pytest` (clients/python); `JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` then **revert** spurious generated-proto churn; `go build ./... && go test ./...`.
### B — TST-31 glauth.md + live-test credential (sonnet)
- Reconcile `glauth.md` against `~/Desktop/scadaproj/infra/glauth/config.toml`: ReadOnly gid, the pre-provisioned users table (drop `readonly/readonly123`, fix `admin` password), the worked example at `:124`. Keep the OPC-UA taxonomy note.
- Re-derive `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` default in `docs/GatewayTesting.md` and the `?? "admin123"` literal in `WorkerLiveMxAccessSmokeTests.cs` from the config; make the test **log** (not assert) whether `AuthenticateUser` succeeded.
- Verify: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (IntegrationTests compiles on mac? if excluded, note it).
### C — GWC-31 + tracker housekeeping (sonnet)
- Sweep code comments citing unregistered `GWC-30`/`GWC-31` (perf pass) in `WorkerClient.cs`, `GatewayMetrics.cs`, `WorkerClientTests.cs`, worker `WorkerFrameReader/Writer.cs` to `PERF-nn` keyed to task numbers in `docs/plans/2026-08-15-perf-review-remediation.md`; add a PERF-ID table to that plan. Do not touch the genuine GWC-30 (frame-reader prefix) citations if any exist.
- `archreview/REVIEW-PROCESS.md` (or wherever the ID rule lives): state that GWC/WRK/IPC/SEC/CLI/TST prefixes are reserved for archreview registers.
- Cycle-2 tracker (`archreview/2026-07-12/remediation/00-tracking.md` + domain docs): GWC-16 → Won't fix (rationale from cycle-3 10-gateway-core.md), IPC-08 → Done (`7c1ea12`, residual IPC-34), IPC-31 → Done (superseded by GWC-28), refresh TST-06/TST-17 rows, register WRK-31 (interim ID burned by `e913dab`) so it is never reused.
- Verify: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`; `dotnet test --filter FullyQualifiedName~WorkerClientTests`.
### D — SEC-40, SEC-41, GWC-38 (opus)
- SEC-41: `DashboardSessionAcl` re-resolves tags from `LdapGroupClaimType` claims via `DashboardGroupTagMapping.MapGroupsToTags` against current options (as `HubTokenService` does) instead of trusting login-stamped tag claims; keep stamped claims only as fallback when no group claims exist (document). Test: principal with stale tag claim + group whose grant was removed → denied.
- SEC-40: decide **tighten**: `AlarmsHub` gets a dedicated policy requiring a real role claim (anonymous-loopback/Disabled bypass excluded) — and document in `docs/GatewayDashboardDesign.md` ACL section + `docs/GatewayConfiguration.md` GroupToTag row that alarm metadata remains galaxy-wide for authenticated Viewers (central monitor, no per-session provenance). Test for the policy.
- GWC-38: `GatewayBrowseScopeProvider` throws `RpcException(PermissionDenied)` (and logs) when `Current` is null **and** API-key auth is enabled; unchanged when auth mode is Disabled. Test both branches.
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~DashboardSessionAcl|FullyQualifiedName~AlarmsHub|FullyQualifiedName~BrowseScope"` (adjust to real class names).
**Outcome (done, branch `fix/c3-dashboard-acl`, commits `b177636`, `2ef7b93`, `8257a03`; not merged, not pushed).** Decisions worth carrying forward:
- **SEC-41 fallback:** the ACL re-resolves from `mxgateway:ldap_group` claims when the principal has any, and falls back to the stamped `zb:dashboardtag` claims only when it has none. That is exactly the hub-token principal — `HubTokenService.Validate` rehydrates roles and tags but **not** groups, so re-resolution already happened at mint. The fallback cannot re-open the cookie gap, because `DashboardAuthenticator` stamps both claim kinds from one group list.
- **SEC-40 requirement type:** new `AuthenticatedDashboardRoleRequirement` + `AuthenticatedDashboardRoleHandler` and a `MxGateway.Dashboard.Alarms` policy. `DashboardAuthorizationHandler` is `AuthorizationHandler<DashboardAuthorizationRequirement>`, so it is never invoked for the new type — the same structural device the Secrets UI's `secrets:manage` already relies on. Both hub schemes are retained on the policy.
- **SEC-40 scope answer:** the `/alarms` Blazor page carries the SAME policy (it reads the central monitor in process — gating only the hub left the disclosure one navigation away), and the side-rail Alarms link is wrapped in an `AuthorizeView` on that policy. `AlarmsHubPublisher` is unchanged and stays unfiltered by design: audience is decided at hub admission and the feed carries no per-session provenance.
- **GWC-38:** `PermissionDenied` + warning only when `Authentication:Mode=ApiKey`; `Mode=Disabled` still returns unscoped. `GalaxyRepositoryHostWiringTests` phase 1 was changed to push an *unconstrained identity* rather than none — that is production's real unscoped case.
- New/changed tests: `AlarmFeedAuthorizationTests` (new, 7), `DashboardSessionAclTests` (+5), `GatewayBrowseScopeProviderTests` (+2), `SecretsNavRenderTests` (+2, and it must register the new policy — an unregistered policy name *throws* inside `AuthorizeView` rather than denying). Filtered run over the touched classes: 92 passed / 0 failed; `NonWindows.slnx` builds zero-warning.
### E — IPC-33, IPC-39, IPC-34, IPC-35 (opus)
- IPC-33 option (a): `GatewayOptionsValidator` adds the upper coupling `Worker:MaxMessageBytes ≤ Protocol:MaxGrpcMessageBytes + 2×reserve`; worker `ResolveDrainReplyByteBudget` becomes `MaxMessageBytes − 2×DrainReplyFrameHeadroomBytes` (floor `MaxMessageBytes/2`); update `docs/GatewayConfiguration.md`, `docs/WorkerFrameProtocol.md`. Test: `GatewayOptionsTests` case for the new bound.
- IPC-39: pre-check compares against `MaxMessageBytes − MaxSequenceEncodingBytes (11)`; fix the comment direction.
- IPC-34: caller-cancellation arms call `TrySendCancelForAbandonedCommand` (renamed) gated on `removed`, skipped when `_stopCts` is the source; new `WorkerClientErrorCode.CallerCancelled` replacing the `GatewayShutdown` mislabel, mapped to `Cancelled` in the gRPC layer; update `docs/Grpc.md` mapping line, `gateway.md:913-919`, `docs/GatewayProcessDesign.md:596-602`. Test: twin of the timeout test driven by the caller token.
- IPC-35: `ReadHandshakeEnvelopeAsync` special-cases `WorkerFault` → `WorkerClientException(WorkerFaulted, CreateWorkerFaultMessage(...))`. Test: fake writes `WorkerFault{MxaccessCreationFailed}` instead of `WorkerReady`, category text surfaces.
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayOptions"`. Worker half → windev afterwards.
### F — WRK-32, WRK-33 (sonnet)
- WRK-33: `ObserveFaultWhenever(ObserveCommandTaskAsync(commandTask));` at `WorkerPipeSession.cs:1235` (or catch-all-and-log in the observer).
- WRK-32: one `pumpStep()` before capturing the `CurrentVersion` baseline in `MxAccessCommandExecutor.cs`; rewrite the executor doc comment (`:996-1003`) to state the recency-based contract; add the same to `docs/MxAccessWorkerInstanceDesign.md` write-completion section and the reply-field comment in `mxaccess_gateway.proto` (**comment-only proto edit → regenerate all generated artifacts in the same commit**, per the repo rule; if regen tooling is unavailable on macOS, skip the proto comment and say so).
- WRK-34 if trivial: `CreateFault(StaHung, "Watchdog", …)` with the correlation id moved into the message.
- Cannot build on macOS: keep diff minimal, net48-safe (no init-only/positional records). Verified on windev afterwards.
### G — GWC-32, GWC-34, GWC-37 (opus)
- GWC-32: `DisposeAsync` transitions `_state` to `Closed` under `_syncRoot` before completing channels (or `if (_disposed || IsTerminalState()) return;` in `StageWorkerEvent`); update the XML doc to name all three completion paths. Test: dispose a Ready client mid-dispatch → no `ProtocolViolation` fault/metric.
- GWC-34: `_eventDistributorDisposed` flag set under `_syncRoot` when nulling; `EnsureDistributorCreated` throws `SessionManagerException(SessionNotReady)` when set. Test: attach after dispose → throws, no silent hang.
- GWC-37: on `ApplyProviderModeChangeAsync(toMode == Subtag)` re-resolve the watch list and re-subscribe; additionally surface "zero targets resolved while mode can need them" on the provider status (`Degraded` + reason). Update `docs/` alarm-subtag section. Test with the fake worker: GR discovery fails at start, succeeds at failover → watch list non-empty after failover.
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewaySession|FullyQualifiedName~AlarmMonitor|FullyQualifiedName~AlarmWatchList"`.
- **Outcome (Done):** `058d8d5` GWC-32, `282a0e7` GWC-34, `1932217` GWC-37 on `fix/c3-session-lifecycle`. Filtered run 176/176 green; `NonWindows.slnx` clean.
- GWC-37 **does not re-send** the watch-list on the failover edge. The worker's `AlarmCommandHandler.Subscribe` rejects a second `SubscribeAlarms` while subscribed, and the only expressible replacement (`UnsubscribeAlarms` + `SubscribeAlarms`) rebuilds `FailoverAlarmConsumer` — restarting it on the primary that just failed and discarding the warm standby — so it would flap the feed rather than repair it. The re-resolve happens and any drift is reported on the same status frame as the failover; a monitor restart is the supported way to apply a new list. Adding a watch-list-update worker command is a possible future item, not done here.
- GWC-37 changes `IAlarmWatchListResolver.ResolveAsync` to return `AlarmWatchListResolution`, and gives `Degraded && Mode==ALARMMGR` a real meaning (empty/undiscoverable standby) with a new dashboard label `Alarm Manager (no subtag standby)`.
- GWC-32's dispose path now also records `WorkerStopped("dispose")` via the existing once-latch, closing a `workers_running` gauge leak on the dispose-a-Ready-client path. GWC-12's double-dispose interlock got a natural home (the `_disposed` claim moved under `_syncRoot`) but is **not** claimed as fixed.
## Merge order
C is merged **last** (its comment sweep touches the same files E and G edit). Order: A, B, F, D, G, E, C. After merge: full `NonWindows` build + one full gateway test run; then windev worker CI for E+F.
## Integration gate (2026-08-20)
All seven branches merged into `main` (A `ee68cc3`, B `f59c34a`, C `f2ab9cc`, D `9f0c3fd`, E `cfbc995`, F `aa07497`, G `c6650c1`). Merged tree: `NonWindows.slnx` 0 warnings; full gateway suite **1183/1183** passed; windev `windev-worker-ci.ps1 -Sha f2ab9cc -Mode test`: x86 Worker build succeeded, `Worker.Tests` **524 passed / 11 skipped / 0 failed**. Follow-ups surfaced by the batches: a worker command to refresh a live alarm subscription's watch-list (would let GWC-37's re-resolve be applied, not just reported); WRK-32's proto reply-field comment deferred until the next proto regen wave; the five clients are bumped but not yet published.
## P2 continuation (planned 2026-08-25)
Current-state audit: refreshed `origin/main` and re-checked every deferred finding and integration-gate follow-up. Sixteen of the seventeen P2 findings remain open; WRK-34 was already completed by `3679e59`. All three follow-ups remain open. Exact target client coordinates are still absent from Gitea (HTTP 404), and `clients/go/v0.2.1` does not exist.
Approved design: [`../../../docs/plans/2026-08-25-cycle-3-p2-followups-design.md`](../../../docs/plans/2026-08-25-cycle-3-p2-followups-design.md) (`30548da`).
Execution plan: [`../../../docs/plans/2026-08-25-cycle-3-p2-followups.md`](../../../docs/plans/2026-08-25-cycle-3-p2-followups.md).
The continuation runs serially in the isolated `.worktrees/c3-p2-followups` worktree on `codex/c3-p2-followups`, with a tracker update at every batch boundary. Codex equivalents are selected from the implementation risk: `gpt-5.6-terra`/medium for contained documentation, CI, and client work; `gpt-5.6-sol`/high for concurrency, authorization, protocol, and live alarm lifecycle changes.
| Batch | Findings / follow-ups | Codex model | Status | Exit gate |
|---|---|---|---|---|
| H — tracker and currency | WRK-34 record, GWC-36, IPC-38, TST-33/34/36 | `gpt-5.6-terra` medium | Done | Currency greps clean; README check in portable CI; NonWindows build clean |
| I — gateway correctness/security | GWC-33/35, SEC-38/39 | `gpt-5.6-sol` high | Done | Focused regression tests green; NonWindows build clean |
| J — secondary contracts | IPC-36/37 | `gpt-5.6-sol` high | Done | Shared launch constants and descriptor parity tests green |
| K — client alarm parity | CLI-47/48/49/50 | `gpt-5.6-terra` high | Done | All five client suites green |
| L — live watch-list/proto wave | watch-list refresh follow-up + WRK-32 proto comment | `gpt-5.6-sol` high | Done | Codegen clean; portable alarm/contract tests green; windev Worker.Tests green |
| M — Windows gateway CI | TST-35 | `gpt-5.6-terra` medium | Done | windev `live` mode proves Worker, gateway, full-slnx, and live smoke tiers |
| N — integration and release | publish five clients + Contracts | `gpt-5.6-sol` high | Done | Full local/windev gates; merged `main`; registry coordinates and Go tag verified |
### Continuation tasklist
#### H — tracker and currency
- [x] Correct WRK-34 from Deferred to Done with commit `3679e59`.
- [x] GWC-36: correct `EventSubscriberReplayAttachment.OldestAvailableSequence` semantics.
- [x] IPC-38: add public `DrainEvents` behavior to `docs/Grpc.md` and correct the cycle-2 omission rationale.
- [x] TST-33: reconcile the five stale/self-contradictory `stillpending.md` claims and mark its live-status boundary.
- [x] TST-34: run `code-reviews/regen-readme.py --check` in portable CI and qualify the interpreter note.
- [x] TST-36: replace prose ``event`` scope references with canonical ``events:read``.
- [x] Record batch-H commit and verification evidence here.
Batch-H verification (2026-08-25): `python3 code-reviews/regen-readme.py --check`
reported the generated README current and consistent; the TST-33 and live-document
TST-36 stale-pattern greps returned no matches; `docs/Grpc.md` contains the public
`DrainEvents` contract; and `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`
succeeded with **0 warnings / 0 errors** in 1.59 s. Batch commit: `92aae28`.
#### I — gateway correctness and security
- [x] GWC-33: unregister terminal clients from the event-queue-depth gauge without shortening fault retention.
- [x] GWC-35: repair acknowledged-to-active snapshot deltas with a Raise transition and instance identity.
- [x] SEC-38: feed/consult the aggregate for valid key IDs that overflow the peer partition cap.
- [x] SEC-39: prove a null effective key ID cannot reset the shared fallback partition.
- [x] Run focused tests and clean NonWindows build; record counts and commits here.
Batch-I verification (2026-08-25): the combined `WorkerClientTests`,
`GatewayAlarmMonitorAttachOrderTests`, and `ApiKeyFailureLimiterTests` filter passed
**55/55** tests in 12 s; `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`
succeeded with **0 warnings / 0 errors** in 3.42 s. Finding commits: GWC-33
`b0156c6`, GWC-35 `7807201`, and SEC-38/SEC-39 `1875426`.
#### J — secondary contracts
- [x] IPC-36: centralize all six worker-launch environment names in Contracts and consume them on both sides.
- [x] IPC-36: document variable/config/default mapping and test launcher completeness.
- [x] IPC-37: add normalized retained-vs-served Galaxy descriptor parity test.
- [x] Run focused tests and clean NonWindows build; record counts and commits here.
Batch-J verification (2026-08-25): the combined `WorkerProcessLauncherTests` and
`GalaxyContractParityTests` filter passed **10/10** tests in 1 s; the descriptor
mutation proof identified the changed protobuf message; and `dotnet build
src/ZB.MOM.WW.MxGateway.NonWindows.slnx` succeeded with **0 warnings / 0
errors** in 2.20 s. IPC-36/IPC-37 commit: `d6c5a11`.
#### K — client alarm-feed parity
- [x] CLI-47: make .NET text status flags explicit and document deliberate JSON differences.
- [x] CLI-48: test provider/snapshot status arms in .NET, Go, and Java; add Go stream-alarm scaffold.
- [x] CLI-49: export Go aliases for `AlarmProviderStatus` and `AlarmSnapshotStatus`; fix five-arm documentation.
- [x] CLI-50: stream Python alarm rows incrementally with an unbounded default and bounded opt-in.
- [x] Run .NET, Go, Python, Rust, and Java client suites; record counts and commits here.
Batch-K verification (2026-08-25): `dotnet test
clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` passed **134/134** portable
tests (1 live test skipped); `go test ./...` passed both tested packages;
`python -m pytest` passed **169/169** portable tests (1 opt-in TLS test
skipped); `cargo test --workspace` passed **108/108** tests; and
`JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` passed **132/132** tests.
CLI-47/CLI-48/CLI-49/CLI-50 commit: `5cdb3da`.
#### L — live watch-list refresh and protobuf wave
- [x] Add explicit fallback-enabled subscription semantics for an initially empty discovered list.
- [x] Add and round-trip `UpdateAlarmWatchListCommand` and its `MxCommandKind`.
- [x] Route the command through the worker executor and alarm handler on the STA.
- [x] Build/subscribe/attach a replacement standby before swap; preserve the old standby on failure.
- [x] Preserve current primary/subtag mode during a successful replacement.
- [x] Re-resolve on the existing gateway reconciliation cadence and send only material changes/recovery.
- [x] Refresh immediately at failover edge, update acknowledged fingerprint only after worker OK, and reconcile after success.
- [x] Expand WRK-32's public reply-field comment to the sequential stale-callback/recency case.
- [x] Replace IPC-36's internal env-var name in the proto comment with the public configuration key.
- [x] Regenerate Contracts and all client generated trees once; run codegen and all affected suites.
- [x] Verify x86 Worker build and Worker.Tests on windev; record SHA/counts here.
Batch-L verification (2026-08-25): `pwsh scripts/check-codegen.ps1` passed all
four descriptor/C#/Rust/Go/Python freshness checks, and Java
`checkGeneratedClean` passed. The exact portable alarm/contract/descriptor
filter passed **77/77**; `NonWindows.slnx` built with **0 warnings / 0 errors**;
the .NET, Go, Python, Rust, and Java client suites were green (respectively
**134 passed / 1 live skip**, both Go packages, **169 passed / 1 opt-in TLS
skip**, **108 passed**, and **132 passed**). Windev `test` mode at
`986c6c0ce4f02c62b4f59ec44780de5af6d291b8` built the x86/net48 Worker with
**0 warnings / 0 errors** and passed Worker.Tests **529 passed / 11 opt-in
skips / 0 failed**. Feature/proto commit: `484028e`; Batch-J x86 compile
follow-up: `986c6c0`.
#### M — Windows gateway CI
- [x] TST-35: add the serialized net10 gateway suite to windev `live` mode.
- [x] Update `docs/GatewayTesting.md` and `scripts/ci/README.md` with order/failure semantics.
- [x] Execute `live` mode on windev and record Worker, gateway, full-slnx, and live-smoke evidence here.
Batch-M verification (2026-08-25): windev `live` mode at exact commit
`cc143b52cf04be0b2f08350922a987b2e250bc1d` completed the serialized sequence in
**4m45s** with remote exit code 0. The x86/net48 Worker build succeeded with **0
warnings / 0 errors** in 22.59 s; Worker.Tests passed **529**, skipped **11**
opt-in probes, and failed **0**; the full Windows `slnx` build succeeded with **0
warnings / 0 errors** in 43.24 s; the net10 gateway suite passed **1193/1193**
in 53 s; and the live-MXAccess smoke passed **8/8** in 1m46s. TST-35 commit:
`cc143b5`.
#### N — integration and release
- [x] Run `git diff --check`, codegen, client-version, README, NonWindows build, and full gateway suite gates.
- [x] Run all five client suites on the final candidate.
- [x] Run final windev `live` gate at the exact candidate SHA.
- [x] Merge/push the verified continuation to `main`; wait for green CI.
- [x] Re-query all package coordinates and Go tag immediately before publish; abort on collision.
- [x] Publish .NET Client + Contracts 0.2.1, Python 0.2.1, Rust 0.2.1, Java 0.2.2.
- [x] Create/push `clients/go/v0.2.1` at the exact release commit.
- [x] Verify every registry coordinate and Go module/tag, update `docs/ClientPackaging.md`, and record immutable release evidence here.
Batch-N portable verification (2026-08-25): `git diff --check`, codegen freshness,
client-version freshness, and the generated README check all passed without changing
the worktree. `NonWindows.slnx` built with **0 warnings / 0 errors**; the gateway
suite passed **1193/1193**, and the portable integration suite passed **5** with
**19** live-only skips. The final .NET, Go, Python, Rust, and Java client gates were
green (respectively **134 passed / 1 live skip**, all Go packages built/tested with
clean `gofmt`, **169 passed / 1 opt-in TLS skip** from a clean virtual environment,
**108 passed** plus clean `cargo fmt`/Clippy, and **132 passed / 0 failed** with the
generated Java tree unchanged).
Batch-N integration/release verification (2026-08-25): `main` was integrated and
released at exact commit `5776021970cbbe59813ba57579abe00d9fc61a46`; Gitea CI
run **783** concluded success. Windev `live` mode at that SHA built the x86/net48
Worker with **0 warnings / 0 errors**, passed Worker.Tests **529 passed / 11
opt-in skips / 0 failed**, built the full Windows solution with **0 warnings / 0
errors**, passed the Windows gateway suite **1193/1193**, and passed the live
MXAccess smoke **8/8**. One preceding full-suite attempt hit a transient fake-worker
disconnect in the reconnect/replay test; the test then passed **50/50** isolated
macOS runs and **20/20** isolated windev runs before the clean full-gate pass.
Immediately before publication, all five package coordinates returned HTTP 404
and `clients/go/v0.2.1` was absent locally/remotely. The published coordinates are
NuGet `ZB.MOM.WW.MxGateway.Client` **0.2.1** and
`ZB.MOM.WW.MxGateway.Contracts` **0.2.1**, PyPI
`zb-mom-ww-mxaccess-gateway-client` **0.2.1**, Cargo
`zb-mom-ww-mxgateway-client` **0.2.1**, and Maven
`com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client` **0.2.2**. Each exact Gitea
package API coordinate returned HTTP 200 after publication. Annotated Go tag
`clients/go/v0.2.1` resolves locally, remotely, and through `go list -m` to
`5776021970cbbe59813ba57579abe00d9fc61a46`.
+27
View File
@@ -0,0 +1,27 @@
# MxAccessGateway — System Review (2026-10-21)
Summary of the fourth review cycle. Per-finding status is in [`remediation/00-tracking.md`](remediation/00-tracking.md); the per-domain finding text is not kept in this directory, so the tracker rows carry the description.
## Dimensions
Six reviewers each took one dimension of the system.
| Dimension | Reviewed | Outcome |
|---|---|---|
| Sessions | Session lifecycle, sweep, shutdown, lease, metrics accounting (`GatewaySession`, `SessionManager`) | One High: a kill or shutdown during worker startup orphaned the worker. Nine findings, all fixed in stream A. Replay-ring byte cap remains open (low) |
| Worker | STA runtime, alarm polling, `ReadBulk`, caches, process exit paths (`ZB.MOM.WW.MxGateway.Worker`) | Eight findings fixed on `fix/review-2026-10-21-e`, not yet merged; four deferred (one needs a parity probe, one is a security tradeoff) |
| gRPC | Interceptor coverage, audit, alarm feed, correlation ids, `/metrics` | Eleven findings fixed. `/metrics` authentication is opt-in |
| Dashboard | Hub fan-out, browse, snapshot feed, health status, design doc | Ten findings fixed; three low races left open |
| Clients | Five language clients: ack retry, buffers, dispose, typed diagnostics | All fixed. The Go `go vet` copylocks error in a test predates the review and remains |
| Docs vs code | Proto comments, `stillpending.md`, tracker headers, untested helpers | In progress in stream L |
## Cross-cutting observations
- The two-process lifecycle is the most failure-prone area: most of the nine session findings are races between open, close, kill and shutdown. Each fix added a regression test; the pattern to keep is that every exit path accounts for the session exactly once.
- Silent failure was the recurring theme in the worker (STA death, alarm-poll failure, missing unhandled-exception hook). Each now ends in a typed fault or a distinct exit code.
- Bounded fan-out was the recurring theme in the dashboard: every hub send now has a bound, a timeout or a drop policy.
- Three external items are outside this repository: shared GLAuth TLS, the Gitea runner token, and an OtOpcUa session leak.
## Cycle-1 backlog
The review also re-triaged 68 first-cycle findings that no later tracker had re-listed: 21 are done, 13 are closed as won't fix with a reason, and 34 remain open and are promoted as `R21-O1` to `R21-O34`. The highest-weight open items are the missing reply to a command that races worker shutdown, the worker pipe without `CurrentUserOnly`, the no-op startup probe with retry options that cannot retry, the redactor gap for `WriteSecuredBulk`/`WriteSecured2Bulk`, the .NET retry budget that a `DeadlineExceeded` consumes, and the Go client's missing typed auth errors and strict-TLS flag.
@@ -0,0 +1,227 @@
# MxAccessGateway — Remediation Tracking (2026-10-21 system review)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
Source review: [`../00-overall.md`](../00-overall.md) (summary of the six review dimensions). Generated 2026-10-22 from the review outcomes and the integration branch `fix/review-2026-10-21`.
Tracks the findings of the 2026-10-21 system review and the cycle-1 findings that no later tracker had re-listed. The earlier trackers remain the record for their own IDs: [cycle 1](../../remediation/00-tracking.md), [cycle 2](../../2026-07-12/remediation/00-tracking.md), [cycle 3](../../2026-08-19/remediation/00-tracking.md).
## How to use this document
- IDs in this cycle use the prefix `R21-` so they cannot collide with the `GWC`/`WRK`/`IPC`/`SEC`/`CLI`/`TST` registers of earlier cycles. `R21-S` is sessions, `R21-W` is the worker client, `R21-G` is the gRPC surface, `R21-D` is the dashboard, `R21-K` is the Windows worker, `R21-C*` are the client libraries, `R21-L` is docs and test gaps, `R21-X` is external, and `R21-O` is a cycle-1 finding re-triaged as still open.
- Where a finding is the same as an earlier-cycle ID, the description names it (for example `R21-G11` is `SEC-14`).
- The review's own severity table is not in the repository. Only `R21-S1` carries a severity from the review (High); every other severity here is the triage estimate from the finding's impact and is open to correction.
- The **Status** column is the single source of truth. Update it in the same change that lands the fix, and cite the finding ID in commit messages. *(Superseded 2026-11-06: status now lives in the Gitea issue named by each `→ #N` pointer; the Status column is frozen as of the migration.)*
**Status legend:** `Done` · `Done (merge 2013573)` · `Done (merge 2fad124)` · `Deferred` · `Won't fix` · `Open` · `Blocked (external)`.
## Execution model
The review was remediated by ten parallel streams on branches `fix/review-2026-10-21-{a..j}`, merged into the integration branch `fix/review-2026-10-21`, plus stream K (this tracker triage) and stream L (docs and test gaps).
| Stream | Branch | Scope | Status |
|---|---|---|---|
| A | `fix/review-2026-10-21-a` | Sessions (`R21-S*`) | Done, merged (`7f0720f`) |
| B | `fix/review-2026-10-21-b` | WorkerClient (`R21-W1`) | Done, merged (`81d4cd8`) |
| C | `fix/review-2026-10-21-c` | gRPC surface (`R21-G*`) | Done, merged (`3f9448f`) |
| D | `fix/review-2026-10-21-d` | Dashboard (`R21-D*`) | Done, merged (`8212b0e`) |
| E | `fix/review-2026-10-21-e` | Worker (`R21-K*`) | Done (merge 2013573) — `7618caf`, follow-up `38fbbde` |
| F | `fix/review-2026-10-21-f` | .NET client | Done, merged (`3a1ee28`) |
| G | `fix/review-2026-10-21-g` | Java client | Done, merged (`c18566b`) |
| H | `fix/review-2026-10-21-h` | Go client | Done, merged (`7636573`) |
| I | `fix/review-2026-10-21-i` | Rust client | Done, merged (`b2a7fc5`) |
| J | `fix/review-2026-10-21-j` | Python client | Done, merged (`29d4e51`) |
| K | `fix/review-2026-10-21-k` | Tracker triage (this document) | Done |
| L | (stream L branch) | Proto comments, docs, test gaps | Done (merge 2fad124) |
Worker changes (stream E) need the windev x86 build and `Worker.Tests` run after merge; stream E's own live facts passed on windev. → #169 (merge to origin `main` + windev Worker gate)
## Sessions (stream A)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-S1 | High | Sessions | A kill or shutdown during worker startup orphaned the worker | Done | A: `87f23d9`, `ec9f264` | Gated attach, discard on any exception, `OpenRecorded`; a late worker is never orphaned |
| R21-S2 | Medium | Sessions | Sweep-driven closes skipped `SessionClosed()`, so the open-sessions gauge drifted | Done → #168 | A: `09666cc` | |
| R21-S3 | Medium | Sessions | `OpenSession` was still accepted while the host was stopping | Done | A: `37ab629` | `GatewayStopping` returns `Unavailable`; `ShutdownAsync` re-snapshots until the registry is empty |
| R21-S4 | Low | Sessions | A shutdown timeout whose kill succeeded was reported as `CloseFailed` | Done | A: `2b165cd` | Now `Closed`, with `WorkerShutdownFailure` recorded |
| R21-S5 | Low | Sessions | A failed open left a metrics entry behind | Done | A: `87f23d9` | Open rollback goes through the removal path |
| R21-S6 | Medium | Sessions | Sessions that never stream pinned a worker for the whole lease | Done → #167 | A: `285ef64`, `ec9f264` | Config-gated `UnattachedSessionGraceSeconds` (default 0 = off); `ExtendLease` deleted; internal subscribers count as attached; idle clock starts at Ready |
| R21-S7 | Low | Sessions | A kill racing a close double-decremented the gauge | Done | A: `ec9f264` | Found in review of stream A |
| R21-S8 | Low | Sessions | A sweep that lost the race to a close was reported as a failure | Done | A: `ec9f264` | Found in review of stream A |
| R21-S9 | Medium | Sessions | After `ShutdownWorker` the session stayed `Ready` | Done | A: `cc1e895` | The session faults and the sweep reaps it. Found live |
| R21-S10 | Low | Sessions | The replay ring has a count cap but no byte cap | Open → #172 | — | Documented as bounded by count. Next step: add a byte cap if large-array events make the count cap insufficient |
## WorkerClient (stream B)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-W1 | Medium | WorkerClient | `ShutdownAsync` could block past `ShutdownTimeout` | Done | B: `b0100e7` | The timeout CTS now covers the envelope enqueue |
## gRPC surface (stream C)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-G1 | Medium | gRPC | Denied bulk commands flooded the audit channel | Done | C: `ca0cf19` | One aggregated event per bulk call |
| R21-G2 | Medium | gRPC | The authorization interceptor did not cover client-streaming or duplex calls (cycle-1 `SEC-17`) | Done | C: `19cc74f` | Both fail closed to the admin scope |
| R21-G3 | Low | gRPC | `StreamAlarms` overflow returned `Unavailable` | Done | C: `0b4a168` | Now `ResourceExhausted` |
| R21-G4 | Low | gRPC | `StreamAlarms` cloned the snapshot under the lock on every attach | Done | C: `0b4a168` | Memoized projection |
| R21-G5 | Low | gRPC | `WriteAsync` was not given the call's cancellation token | Done | C: `0b4a168` | |
| R21-G6 | Low | gRPC | `correlation_id` handling was inconsistent across replies | Done | C: `0b4a168`, `82a856b` | `client_correlation_id` is echoed; a Debug log records the client/envelope pair |
| R21-G7 | Low | gRPC | The `StreamDisconnected` reason was a constant (cycle-1 `GWC-22`) | Done | C: `6ae5ed9` | |
| R21-G8 | Low | Security | `CachingApiKeyVerifier` index pruning and single-flight misses | Done | C: `ed3a2c9` | |
| R21-G9 | Low | gRPC | `ConstraintEnforcer` allocated a `Task` per item | Done | C: `ca0cf19` | `ValueTask` |
| R21-G10 | Low | Metrics | `Enum.ToString` was called per event | Done | C: `68d6b5f` | `MxEventFamilyNames` cache |
| R21-G11 | Medium | Security | `/metrics` was unauthenticated (cycle-1 `SEC-14`) | Done | C: `65bca80`, `82a856b` | Opt-in `Metrics:RequireAuthentication` (admin key, or dashboard Admin/Viewer cookie); `DisableLogin` does not bypass it; `/health` stays open by design |
| R21-G12 | Low | Metrics | The `/metrics` per-peer limiter is keyed by IP while the gRPC limiter is keyed by connection | Won't fix | — | Documented difference; the two surfaces have different peer identities |
| R21-G13 | Low | Security | The verifier generation guard depends on key-id case | Open → #173 | — | Latent only. Next step: normalize the key id (ordinal-ignore-case) at the guard and add a test |
## Dashboard (stream D)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-D1 | Medium | Dashboard | Unbounded events-hub sends | Done | D: `f1db002`, `bd7abf4` | One drop-oldest pump per session; a lock re-check race was fixed in review |
| R21-D2 | Medium | Dashboard | One slow client stalled the alarms hub | Done | D: `3688fa1` | 2 s send timeout plus an `AlarmFeedReset` marker. Tradeoff documented: an abandoned send is not followed by a reset |
| R21-D3 | Low | Dashboard | Browse search ran on every keystroke | Done | D: `db78336` | Debounce, off-dispatcher, indexed |
| R21-D4 | Medium | Dashboard | The full snapshot plus Configuration went out every second | Done | D: `6733519` | Configuration is sent once per connection; sends have a timeout |
| R21-D5 | Low | Dashboard | `AlarmsPage` re-sorted twice per render | Done | D: `2a7d6fa` | `DashboardAlarmView` memo |
| R21-D6 | Low | Dashboard | `BrowsePage` disposal was unbounded | Done | D: `db78336` | |
| R21-D7 | Low | Dashboard | Two viewers thrashed the advise LRU | Done | D: `5d4aefc` | Per-viewer share |
| R21-D8 | Low | Dashboard | Disconnected circuits kept polling | Done | D: `db78336` | 30 s retention plus a connection tracker |
| R21-D9 | Low | Dashboard | `GatewayStatus` was hardcoded (cycle-1 `SEC-27`) | Done | D: `6733519` | Derived from health checks |
| R21-D10 | Low | Docs | `docs/GatewayDashboardDesign.md` described counters that do not exist | Done | D: `2c963b7` | Doc rewritten to match the shipped pages |
| R21-D11 | Low | Dashboard | `BrowsePage` search CTS cancel/dispose race | Open → #174 | — | Only a caught `ObjectDisposedException`. Next step: swap-and-dispose under a lock when the page is next touched |
| R21-D12 | Low | Dashboard | Cached health status is stale while the publisher is idle | Open → #174 | — | Next step: refresh on first subscribe |
| R21-D13 | Low | Dashboard | Hub pumps are not awaited on broadcaster dispose | Open → #174 | — | Next step: await the pumps with a bounded timeout in `DisposeAsync` |
## Worker (stream E)
All rows are on `fix/review-2026-10-21-e` (`7618caf`, follow-up `38fbbde`), not yet merged. After merge, run `windev-worker-ci.ps1 -Mode test` and update these rows to plain `Done`. → #169
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-K1 | Medium | Worker | The write-completion cache and value cache were never evicted | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Ghost-entry eviction on `AddItem` added in the follow-up |
| R21-K2 | Medium | Worker | `ReadBulk` was serial (N x timeout) | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Shared deadline; dedupe and periodic message pumping in the follow-up |
| R21-K3 | Medium | Worker | One alarm-poll failure faulted the session (cycle-1 `WRK-05`) | Done (merge 2013573) | E: `7618caf` | Consecutive-failure threshold of 3 |
| R21-K4 | Low | Worker | The primary alarm consumer leaked when the standby build failed | Done (merge 2013573) | E: `7618caf` | |
| R21-K5 | Low | Worker | `StaRuntime` drained the whole queue before pumping messages | Done (merge 2013573) | E: `7618caf` | |
| R21-K6 | Medium | Worker | STA thread death after start was silent (cycle-1 `WRK-02`) | Done (merge 2013573) | E: `7618caf` | `LoopFault` becomes a `StaHung` WorkerFault |
| R21-K7 | Low | Worker | No unhandled-exception hook (cycle-1 `WRK-09`) | Done (merge 2013573) | E: `7618caf` | Exit code 7 |
| R21-K8 | Low | Worker | `RETRIGGER` handling was undocumented | Done (merge 2fad124) | stream L | Documented as reserved, never synthesized; the proto comment is stream L's |
| R21-K9 | Medium | Tests | Live control-command facts for the worker (cycle-1 `TST-05` residual) | Done (merge 2013573) | E: `069972a`, `4ea3672` | Facts added and passing on windev |
| R21-K10 | Low | Worker | Unregister-first shutdown cleanup | Deferred → #175 | — | Needs a parity probe against the installed MXAccess before it can change cleanup order |
| R21-K11 | Medium | Security | A per-operator `AuthenticateUser` cache | Deferred → #176 | — | Security tradeoff (caching credentials' outcome); decide with the security owner before building |
| R21-K12 | Low | Worker | Batch task-completion-source allocation rework | Deferred → #177 | — | Low value |
| R21-K13 | Low | Worker | The alarm poll builds a full-BSTR `XmlDocument` every 500 ms | Deferred → #177 | — | Steady GC cost, not a correctness issue; revisit if profiling flags it |
## Clients (streams F to J)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-CN1 | Medium | .NET client | Acknowledge retried on transient failure | Done | F: `6f0b408` | No ack retry |
| R21-CN2 | Medium | .NET client | Client always owned its channel | Done | F: `6f0b408` | Caller-owned channel constructors |
| R21-CN3 | Medium | .NET client | `DisposeAsync` could throw or block (cycle-1 `CLI-06`) | Done | F: `6f0b408` | Safe, best-effort dispose |
| R21-CN4 | Low | .NET client | No typed Ping/GetSessionState/GetWorkerInfo/DrainEvents | Done | F: `6f0b408` | |
| R21-CN5 | Medium | .NET client | A session could not be re-attached by id (cycle-1 `CLI-05`) | Done | F: `6f0b408` | `AttachSession` |
| R21-CJ1 | Medium | Java client | Event queue was fixed at 16 | Done | G: `48c9323` | Configurable, default 1024 |
| R21-CJ2 | Medium | Java client | Overflow dropped buffered events | Done | G: `48c9323` | Overflow keeps buffered events |
| R21-CJ3 | Low | Java client | Interrupt did not cancel the call | Done | G: `48c9323` | |
| R21-CJ4 | Low | Java client | No typed diagnostics | Done | G: `48c9323` | |
| R21-CG1 | Medium | Go client | Event buffer fixed at 16 | Done | H: `a3a7bd2` | `EventBufferSize` option, default 1024 |
| R21-CG2 | Medium | Go client | Deprecated `DialContext` + `WithBlock` (cycle-1 `CLI-10`) | Done | H: `a3a7bd2` | `grpc.NewClient` |
| R21-CG3 | Low | Go client | No typed diagnostics | Done | H: `a3a7bd2` | |
| R21-CG4 | Low | Go client | `go vet` copylocks error in `types_alias_test.go` | Done | see commit "fix(go-client): go vet copylocks" | The test declares pointer types; `go vet ./...` is clean |
| R21-CR1 | Medium | Rust client | Blocking file read on an async path (CA file) | Done | I: `0345726` | Async read |
| R21-CR2 | Low | Rust client | No typed diagnostics | Done | I: `0345726` | |
| R21-CP1 | Low | Python client | No typed diagnostics | Done | J: `99347d3`, `d716dec` | The client already had no ack retry, no event buffer, and cancels on exit |
## Docs and test gaps (stream L)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-L1 | Low | Docs | `operator_full_name` is silently dropped; proto comment and docs do not say so | Done (merge 2fad124) | stream L | |
| R21-L2 | Low | Contracts | `RETRIGGER` and the `WorkerFaultCategory` values `INVALID_ARGUMENTS` and `GATEWAY_AUTHENTICATION_FAILED` are reserved but not labeled so | Done (merge 2fad124) | stream L | Proto comments |
| R21-L3 | Low | Docs | Stale notes in `stillpending.md` (sections 1.4, 3.4, 7.1) and the deferred-followups header | Done (merge 2fad124) | stream L | |
| R21-L4 | Low | Docs | The 2026-08-19 tracker header says CLI-47..50, IPC-36/37, TST-35 are `Not started` while the rows say `Done` | Done | K | Header row corrected in this stream |
| R21-L5 | Low | Tests | No tests for `GatewayConfigPathRules` | Done (merge 2fad124) | stream L | |
| R21-L6 | Low | Tests | No tests for `DashboardRedactor` | Done (merge 2fad124) | stream L | |
| R21-L7 | Low | Tests | No tests for `EventSubscriberReplayAttachment` | Done (merge 2fad124) | stream L | |
| R21-L8 | Low | Tests | No tests for the `MxStatusDetailText` table | Deferred → #178 | — | Worker-side test; needs a windev run, not done in stream L |
## External / blocked
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-X1 | Medium | Security | TLS on the shared GLAuth (cycle-3 `SEC-37`) | Blocked (external) → #180 | — | Lives in `scadaproj/infra/glauth/` (separate repo and docker host); after it lands, move the deployed service to `Production` |
| R21-X2 | Low | CI | Gitea runner registration-token reset (cycle-3 `TST-32`) | Blocked (external) → #181 | — | Web-UI action on the Gitea admin page |
| R21-X3 | Medium | OtOpcUa | `GalaxyDriver.ShutdownAsync` leaks sessions | Blocked (external) | — | Defect is in the OtOpcUa repo; the gateway's lease expiry and detach grace eventually reap a leaked session |
## Cycle-1 rows re-triaged as still open (promoted)
These are cycle-1 findings that no later tracker re-listed. Each was checked against the integration branch, and the cycle-1 row now points here. The `Medium` rows carry correctness, security or operability weight (`R21-O1`, `R21-O8`, `R21-O9`, `R21-O16`, `R21-O21` to `R21-O23`, `R21-O30`, `R21-O31`); the `Low` rows are hygiene to batch with the next change that touches the file.
| ID | Cycle-1 ID | Sev | Area | Description | Status | Next step |
|---|---|---|---|---|---|---|
| R21-O1 | WRK-03 | Medium | Worker | `TryStartCommandTask` returns silently when `_acceptingCommands` is false, so a command racing shutdown gets no reply and no log (`Ipc/WorkerPipeSession.cs:1240`) | Open → #182 | Reply `WorkerUnavailable` (at minimum log `WorkerCommandResultDropped`) and add `CommandAfterShutdown_RepliesWorkerUnavailable`; verify on windev |
| R21-O2 | WRK-08 | Low | Worker | Residual events queued after `WorkerShutdown` are discarded, and the shutdown section of `docs/MxAccessWorkerInstanceDesign.md` does not say so | Open → #183 | Document the discard as v1 behavior (docs-only) |
| R21-O3 | WRK-10 | Low | Worker | `WorkerApplication.Run` catch blocks log the exception type but never the message | Open → #184 | Add `exception_message` through `WorkerLogRedactor` in the three catch blocks; add a `WorkerApplicationTests` assertion |
| R21-O4 | WRK-16 | Low | Worker | Seven near-identical `CreateEnvelope` overload pairs and eight `WorkerPipeClient` constructors | Open → #185 | Refactor when `WorkerPipeSession` is next touched; existing IPC tests are the guard |
| R21-O5 | WRK-17 | Low | Worker | Gateway death surfaces as `EndOfStream` and exits 6 (`ProtocolViolation`) instead of 5 | Open → #186 | Map `EndOfStream` to `PipeConnectionFailed`; add a `WorkerApplicationTests` case |
| R21-O6 | WRK-18 | Low | Worker | Event-queue overflow exits as generic `UnexpectedFailure` (1) | Open → #186 | Add a dedicated exit code (stream E already uses 7 for unhandled exceptions, so use 8) and update the overflow paragraph of `docs/MxAccessWorkerInstanceDesign.md` |
| R21-O7 | WRK-19 | Low | Worker | The documented per-command start/end log with correlation id and latency does not exist | Open → #184 | Add level-gated `WorkerCommandStarted`/`WorkerCommandCompleted` in `StaCommandDispatcher.ExecuteQueuedCommandAsync` |
| R21-O8 | GWC-05 | Medium | Gateway | Worker pipe is created with no ACL and no `CurrentUserOnly` (`SessionWorkerClientFactory.CreatePipe`); only the unguessable name and `maxNumberOfServerInstances: 1` limit access | Open → #187 | Add `PipeOptions.CurrentUserOnly` (or a `PipeSecurity` for the worker account); confirm the worker runs as the same user on windev; add a launch test |
| R21-O9 | GWC-09 | Medium | Gateway | `WorkerProcessStartedProbe` only checks `HasExited`, so `StartupProbeRetryAttempts` and `StartupProbeRetryDelayMilliseconds` can never retry anything | Open → #188 | Either make the probe wait for a real readiness signal, or remove the two options and their docs so config does not promise a retry that cannot happen |
| R21-O10 | GWC-13 | Low | Gateway | Worker-ready wait is a 25 ms poll loop (`GatewaySession`, `pollIntervalMs = 25`) | Open → #189 | Replace with a `TaskCompletionSource` signalled on state change |
| R21-O11 | GWC-17 | Low | Gateway | `SparseArrayExpander` throws `Grpc.Core.RpcException` from the Sessions layer | Open → #190 | Throw a domain exception and map it to `InvalidArgument` in the gRPC layer |
| R21-O12 | GWC-18 | Low | Gateway | `GatewaySession` exposes `DisposeAsync` without implementing `IAsyncDisposable` | Open → #191 | Declare the interface; check `await using` call sites |
| R21-O13 | GWC-19 | Low | Gateway | `GatewaySession.KillWorker(string)` has no production callers (only test comments mention it) | Open → #191 | Delete it and fix the comments, or document it as the synchronous kill primitive |
| R21-O14 | GWC-20 | Low | Gateway | `HeartbeatCheckInterval` is fed from `HeartbeatIntervalSeconds`, conflating the send and check intervals | Open → #192 | Split the option or rename it, and update `docs/GatewayConfiguration.md` |
| R21-O15 | GWC-21 | Low | Gateway | `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` are compile-time constants (`WorkerClientOptions`) | Open → #192 | Expose both under `MxGateway:Worker:*` with validation, or record them as intentionally fixed |
| R21-O16 | SEC-13 | Medium | Security | `GatewayLogRedactor.SensitiveCommandMethods` lacks `WriteSecuredBulk` and `WriteSecured2Bulk`. Latent in cycle 1, but `GatewayLogRedactorSeam` now calls `RedactCommandValue`, so a secured-bulk payload would be logged unredacted if value logging is enabled | Open → #193 | Add both bulk kinds plus a redactor test; cross-check every command kind that carries a password |
| R21-O17 | SEC-16 | Low | Security | `apikey --pepper` is accepted on the command line (`ApiKeyAdminCommandLineParser`) and lands in shell history and the process list | Open → #194 | Prefer an env var or stdin prompt; keep the flag only with a warning |
| R21-O18 | SEC-18 | Low | Security | `DashboardApiKeyManagementService` detects a missing pepper by matching `"pepper unavailable"` in an exception message | Open → #195 | Catch a typed exception from `ZB.MOM.WW.Auth.ApiKeys`, or check the pepper before the call |
| R21-O19 | SEC-23 | Low | Security | The options validator does not reject a `__Host-` `CookieName` combined with `RequireHttpsCookie=false`, a cookie browsers silently drop | Open → #196 | Add the cross-field check to `ValidateDashboard` |
| R21-O20 | SEC-24 | Low | Security | `EffectiveDashboardConfiguration` omits `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie` and `CookieName`, so the Settings page cannot show that login is disabled | Open → #196 | Project the four fields in `GatewayConfigurationProvider` |
| R21-O21 | CLI-07 | Medium | .NET client | `ExecuteSafeUnaryAsync` wraps the retry pipeline in `CancelAfter(DefaultCallTimeout)` while each attempt has the same deadline, so a `DeadlineExceeded` first attempt consumes the whole budget and retries never run | Open → #197 | Make the outer budget `MaxAttempts x DefaultCallTimeout` (or drop the outer cap); add a wire test with a slow first attempt |
| R21-O22 | CLI-09 | Medium | Go client | Go wraps every RPC failure in `GatewayError`; there is no typed `Unauthenticated` vs `PermissionDenied` mapping, though `ClientLibrariesDesign.md` requires one | Open → #198 | Add sentinel or typed errors usable with `errors.Is`; keep credential redaction |
| R21-O23 | CLI-11 | Medium | Go client | `mxgw-go` has no flag for strict TLS validation, so `--tls` without a CA is always skip-verify | Open → #199 | Add `--require-certificate-validation` wired to `Options.RequireCertificateValidation` |
| R21-O24 | CLI-19 | Low | .NET client | `InternalsVisibleTo` is declared in both `Properties/AssemblyInfo.cs` and the csproj | Open → #200 | Delete one (cosmetic) |
| R21-O25 | CLI-22 | Low | Go client | `newCorrelationID` returns an empty string if `crypto/rand` fails | Open → #201 | Fall back to a counter-based id |
| R21-O26 | CLI-27 | Low | Python client | `Session.close()` has no lock around `_closed`, and a repeat close returns a synthesized reply | Open → #202 | Guard with an `asyncio.Lock` and cache the real reply, as .NET and Go do |
| R21-O27 | CLI-31 | Low | Rust client | The Rust CLI `main.rs` is now 3,023 lines | Open → #203 | Split by command group when next touched |
| R21-O28 | CLI-33 | Low | Clients | Slow-consumer behavior is documented in the Go README but not in the Java README or the cross-language `docs/ClientBehaviorFixtures.md` | Open → #204 | Add one comparison table now that this cycle changed the Go and Java buffers |
| R21-O29 | TST-07 | Low | Tests | Real-clock sleeps with negative assertions remain (for example `WorkerClientTests.cs:975`, 150 ms) | Open → #205 | Convert to a fake time provider where one exists; otherwise leave |
| R21-O30 | TST-09 | Medium | Ops | Health checks cover the auth store and sessions only; Galaxy SQL, LDAP and worker-executable presence are unchecked (`GatewayApplication.cs:118`) | Open → #206 | Add a Ready-tag worker-exe presence check first, then a bounded Galaxy SQL ping; zero sessions must stay Healthy because the Overview dashboard polls the Active tier |
| R21-O31 | TST-10 | Medium | Ops | No deploy or upgrade script or runbook for the NSSM-wrapped service | Open → #207 | Write a deployment doc from the windev/wonder deploy notes; respect the current ban on remote execution on the wonder boxes |
| R21-O32 | TST-18 | Low | Tests | `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` timer/startup wiring has no direct tests | Open → #205 | Add timer and startup tests with a fake time provider |
| R21-O33 | TST-21 | Low | Ops | Serilog file sink rolls daily with no `retainedFileCountLimit` or size cap | Open → #208 | Set `retainedFileCountLimit` and `fileSizeLimitBytes` in `appsettings.json`; document |
| R21-O34 | TST-22 | Low | Docs | The Configuration Shape JSON block in `docs/GatewayConfiguration.md` omits `DisableLogin`, `AutoLoginUser`, `CookieName`, `Tls` and `Alarms:Fallback` | Open → #209 | Add the keys to the block (docs-only) |
## Cycle-1 triage summary
The 68 cycle-1 rows that were `Not started` with no later re-listing resolved as follows.
| Verdict | Count | Rows |
|---|:-:|---|
| Done | 21 | WRK-02, WRK-05, WRK-09, WRK-13, WRK-20, GWC-10, GWC-12, GWC-22, IPC-10, IPC-12, SEC-14, SEC-17, SEC-19, SEC-21, SEC-26, SEC-27, CLI-05, CLI-06, CLI-10, CLI-13, TST-14 |
| Won't fix | 13 | GWC-11, IPC-11, SEC-15, CLI-14, CLI-17, CLI-20, CLI-23, CLI-25, CLI-28, CLI-32, TST-19, TST-20, WRK-14 |
| Open (promoted) | 34 | `R21-O1` through `R21-O34` above |
`WRK-02`, `WRK-05` and `WRK-09` are closed by stream E and become plain `Done` when `fix/review-2026-10-21-e` merges.
The seven rows singled out for assessment: `WRK-03` Open (`R21-O1`: the worker still returns silently from `TryStartCommandTask` after shutdown starts); `WRK-05` Done (stream E); `CLI-06` Done (stream F); `CLI-07` Open (`R21-O21`: the outer `CancelAfter` still equals the per-attempt deadline); `CLI-09` Open (`R21-O22`: no typed auth errors in Go); `CLI-11` Open (`R21-O23`: no strict-TLS flag on `mxgw-go`); `CLI-13` Done (stream G).
## Change log
| Date | Change |
|---|---|
| 2026-10-22 | Initial tracker from the 2026-10-21 review outcomes. Stream E is recorded as done but unmerged; stream L items are in progress. 68 cycle-1 rows re-triaged and the 34 still-open ones promoted. |
## Final integration review (stream M)
| ID | Severity | Area | Finding | Status | Commit | Notes |
|---|---|---|---|---|---|---|
| R21-M1 | Medium | Sessions | A worker that dies right after its handshake left the session `Ready` with a dead worker until the lease expired | Done | M: `a178d68` | Attach refuses a terminal worker; the pump observes terminal state on its first read |
| R21-M2 | Low | Metrics | One worker fault was counted 1+K times (K = number of `StreamEvents` subscribers) | Done | M: `a0b5e91` | |
| R21-M3 | Low | Sessions | A cancelled open, or one drained at shutdown, was reported as `OpenFailed` | Done | M: `a178d68` | Now `Cancelled` and `GatewayStopping` respectively |
| R21-M4 | Low | Docs | Unattached-grace exemptions, subscriber-limit status code, stale event-stream description in Grpc.md | Done | M: `3ffba83` | |
| R21-M5 | Low | Docs | Client docs described the ReadBulk timeout as per-tag | Done | M: `dd62335` | Worker-side comments (`MxAccessCommandExecutor.cs:15`, `MxAccessValueCache.cs:219`, `MxAccessValueCacheTests.cs:246`) still say per-tag: Open → #179 |
+79 -73
View File
@@ -1,5 +1,10 @@
# MxAccessGateway — Remediation Tracking
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
>
> Rows still open in this cycle's tracker were re-triaged and carried into the [2026-10-21 tracker](../2026-10-21/remediation/00-tracking.md) (`R21-*` rows; cycle-1 rows promoted as `R21-O1`..`R21-O34`), whose `→ #N` pointers name the Gitea issues.
Master progress tracker for the 2026-07-08 architecture review. Generated 2026-07-09.
Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation designs are linked below and hold the full **Finding / Impact / Design / Implementation / Verification** for every entry here.
@@ -11,7 +16,7 @@ Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation
- Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` (depends-on) column within a tier.
- When a fix lands: flip Status to `Done`, and per the repo rule, update the affected docs in the same commit.
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action).
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action) · `Open` (re-triaged 2026-10-21: still present, promoted to the [2026-10-21 tracker](../2026-10-21/remediation/00-tracking.md)).
## Severity roll-up
@@ -63,24 +68,24 @@ Full design + implementation for each row lives in the linked domain doc under i
| GWC-02 | High | P0 | M | — | Done | Faulted sessions are never swept |
| GWC-03 | High | P0 | S | — | Done | Documented sparse-array max-length bound is unimplemented |
| GWC-04 | Medium | P1 | M | — | Done | Full event channel stalls the worker read loop behind command replies |
| GWC-05 | Medium | — | S | — | Not started | Worker pipe created with no ACL / no CurrentUserOnly |
| GWC-05 | Medium | — | S | — | Open | Worker pipe created with no ACL / no CurrentUserOnly — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O8 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-06 | Medium | P2 | S | GWC-07,08 | Done | Stopwatch allocated per streamed event |
| GWC-07 | Medium | P2 | S | GWC-06,08 | Done | Every mapped event is deep-cloned |
| GWC-08 | Medium | P2 | M | GWC-06,07 | Done | Pipe framing allocates per frame and writes twice |
| GWC-09 | Medium | — | M | — | Not started | Startup probe is a no-op; its retry pipeline can never retry |
| GWC-10 | Medium | — | S | — | Not started | Envelope `sequence` monotonicity specified but not enforced |
| GWC-11 | Low | — | S | — | Not started | `_workerClient` written under lock but read lock-free |
| GWC-12 | Low | — | S | — | Not started | `WorkerClient.DisposeAsync` not safe against double-dispose |
| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
| GWC-09 | Medium | — | M | — | Open | Startup probe is a no-op; its retry pipeline can never retry — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O9 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-10 | Medium | — | S | — | Done | Envelope `sequence` monotonicity specified but not enforced — **2026-10-21 triage:** resolved by spec change, not enforcement: `gateway.md:440` states `sequence` is a diagnostic aid, stamped in each side's single write path and not validated on receive |
| GWC-11 | Low | — | S | — | Won't fix | `_workerClient` written under lock but read lock-free — **2026-10-21 triage:** reference reads are atomic and every ready/state decision reads `_workerClient` under `_syncRoot` (`EvaluateReadyUnderLock`); only diagnostic properties read it lock-free |
| GWC-12 | Low | — | S | — | Done | `WorkerClient.DisposeAsync` not safe against double-dispose — **2026-10-21 triage:** closed by cycle-3 GWC-32 (`DisposeAsync` claims `_disposed` under `_syncRoot`; see the 2026-08-19 tracker) |
| GWC-13 | Low | — | M | — | Open | Worker-ready wait is a 25 ms poll loop — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O10 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-14 | Low | P2 | M | — | Done | Replay ring is a `LinkedList` with a node alloc per event |
| GWC-15 | Low | P2 | S | — | Done | Per-event gauge reads `ChannelReader.Count` every event |
| GWC-16 | Low | — | S | — | Not started | `Invoke` resolves the session twice per command |
| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
| GWC-20 | Low | — | S | — | Not started | Heartbeat config semantics conflated (send vs check interval) |
| GWC-21 | Low | — | S | GWC-04 | Not started | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable |
| GWC-22 | Low | — | S | — | Not started | `StreamDisconnected` always labeled `"Detached"` |
| GWC-16 | Low | — | S | — | Won't fix | `Invoke` resolves the session twice per command — measured negligible, deliberately not collapsed (see domain doc) |
| GWC-17 | Low | — | M | — | Open | Sessions layer throws `Grpc.Core.RpcException` (layering leak) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O11 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-18 | Low | — | S | — | Open | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O12 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-19 | Low | — | S | — | Open | Dead `GatewaySession.KillWorker(string)` + stale doc — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O13 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-20 | Low | — | S | — | Open | Heartbeat config semantics conflated (send vs check interval) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O14 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-21 | Low | — | S | GWC-04 | Open | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O15 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-22 | Low | — | S | — | Done | `StreamDisconnected` always labeled `"Detached"` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G7), `6ae5ed9`: `StreamDisconnected` records the real reason |
| GWC-23 | Info | — | — | — | N/A | `MaxEventSubscribersPerSession` dead in default single-subscriber mode |
### Worker — [20-worker.md](20-worker.md)
@@ -88,25 +93,25 @@ Full design + implementation for each row lives in the linked domain doc under i
| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| WRK-01 | High | P0 | M | — | Done | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
| WRK-02 | Medium | — | M | — | Not started | STA thread death after startup is silent; future work hangs forever |
| WRK-03 | Medium | — | S | WRK-02 | Not started | Commands after shutdown starts are dropped with no reply |
| WRK-02 | Medium | — | M | — | Done | STA thread death after startup is silent; future work hangs forever — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K6): an STA loop fault raises a `StaHung` WorkerFault; stream E `7618caf`, merged |
| WRK-03 | Medium | — | S | WRK-02 | Open | Commands after shutdown starts are dropped with no reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O1 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-04 | Medium | — | M | — | Done | Envelope `sequence` can appear out of order on the wire |
| WRK-05 | Medium | — | M | — | Not started | One transient alarm-poll failure kills the whole session |
| WRK-05 | Medium | — | M | — | Done | One transient alarm-poll failure kills the whole session — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K3): consecutive alarm-poll failure threshold of 3 in `MxAccessStaSession`; stream E `7618caf`, merged |
| WRK-06 | Medium | P2 | S | — | Done | `MXSTATUS_PROXY` conversion reflects per field, per event |
| WRK-07 | Medium | P1 | M | WRK-04 | Done | Documented outbound write priority not implemented |
| WRK-08 | Low | — | S | — | Not started | Residual event queue discarded at graceful shutdown, undocumented |
| WRK-09 | Low | — | S | — | Not started | No `AppDomain.UnhandledException` hook |
| WRK-10 | Low | — | S | — | Not started | Top-level catch logs exception type but never message |
| WRK-08 | Low | — | S | — | Open | Residual event queue discarded at graceful shutdown, undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O2 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-09 | Low | — | S | — | Done | No `AppDomain.UnhandledException` hook — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K7): `AppDomain.UnhandledException` hook, exit code 7; stream E `7618caf`, merged |
| WRK-10 | Low | — | S | — | Open | Top-level catch logs exception type but never message — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O3 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-11 | Low | P2 | S | — | Done | Every accepted event is defensively cloned on enqueue |
| WRK-12 | Low | P2 | M | WRK-04 | Done | No event batching per envelope; one flush per event; 25 ms poll |
| WRK-13 | Low | — | S | — | Not started | Frame writer allocates a fresh buffer per frame; reader pools |
| WRK-14 | Low | — | M | — | Not started | Private-field naming split `_camelCase` vs `camelCase` |
| WRK-13 | Low | — | S | — | Done | Frame writer allocates a fresh buffer per frame; reader pools — **2026-10-21 triage:** worker `WorkerFrameWriter` rents the frame from `ArrayPool` (`Ipc/WorkerFrameWriter.cs:675`); the reader was already pooled |
| WRK-14 | Low | — | M | — | Won't fix | Private-field naming split `_camelCase` vs `camelCase` — **2026-10-21 triage:** cosmetic; the style guide permits both `_camelCase` and bare `camelCase` where established, and a mass rename is churn without behavior change |
| WRK-15 | Low | P2 | S | — | Done | Doc drift: STA thread name and heartbeat-counter note |
| WRK-16 | Low | — | S | — | Not started | Boilerplate duplication in IPC envelope/ctor overloads |
| WRK-17 | Low | — | S | — | Not started | Gateway death exits with wrong exit code (6 not 5) |
| WRK-18 | Low | — | S | — | Not started | Event-queue overflow exits as generic `UnexpectedFailure` |
| WRK-19 | Low | — | S | — | Not started | Command start/end logging with correlation id absent |
| WRK-20 | Low | — | M | WRK-01..04 | Not started | Test-coverage gaps for the failure modes above |
| WRK-16 | Low | — | S | — | Open | Boilerplate duplication in IPC envelope/ctor overloads — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O4 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-17 | Low | — | S | — | Open | Gateway death exits with wrong exit code (6 not 5) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O5 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-18 | Low | — | S | — | Open | Event-queue overflow exits as generic `UnexpectedFailure` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O6 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-19 | Low | — | S | — | Open | Command start/end logging with correlation id absent — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O7 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-20 | Low | — | M | WRK-01..04 | Done | Test-coverage gaps for the failure modes above — **2026-10-21 triage:** STA-death, sequence-monotonic, long-ReadBulk and late-reply tests exist; the one residual (command-after-shutdown reply) is carried by WRK-03 as R21-O1 |
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
@@ -119,11 +124,11 @@ Full design + implementation for each row lives in the linked domain doc under i
| IPC-05 | Medium | P2 | S | — | Done | Redundant deep copies on command and event hot paths |
| IPC-06 | Medium | P2 | S | — | Done | `gateway.md` Worker Envelope sketch no longer matches the contract |
| IPC-07 | Medium | P2 | S | — | Done | `docs/Grpc.md` says six RPCs; there are seven |
| IPC-08 | Medium | — | M | — | Not started | `WorkerCancel` defined and handled but never sent |
| IPC-08 | Medium | — | M | — | Done | `WorkerCancel` defined and handled but never sent — resolved by `7c1ea12`/`a1a38b5`; residual (caller-cancellation leg) tracked as cycle-3 IPC-34 |
| IPC-09 | Medium | P1 | M | IPC-01 | Done | Known codegen fragilities have no in-repo guards |
| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
| IPC-12 | Low | — | S | — | Not started | Gateway frame writer has no write lock; integrity rests on undocumented invariant |
| IPC-10 | Low | — | S | — | Done | Envelope `sequence` is write-only; monotonicity never validated — **2026-10-21 triage:** same resolution as GWC-10: `gateway.md:440` documents `sequence` as a diagnostic aid |
| IPC-11 | Low | — | S | — | Won't fix | No protocol version negotiation despite `supported_protocol_version` name — **2026-10-21 triage:** the review itself recommended no change; exact-match protocol version is correct for a lockstep-deployed pair, and the next bump should introduce a min/max range |
| IPC-12 | Low | — | S | — | Done | Gateway frame writer has no write lock; integrity rests on undocumented invariant — **2026-10-21 triage:** IPC-13 collapsed the gateway write to one pooled-buffer write, and `WorkerClient.WriteLoopAsync` is its only caller |
| IPC-13 | Low | P2 | S | IPC-05 | Done | Gateway writer serializes twice and issues two stream writes |
| IPC-14 | Low | P2 | S | IPC-05 | Done | Gateway reader allocates fresh array per frame; worker rents from `ArrayPool` |
| IPC-15 | Low | P2 | M | — | Done | Worker event delivery 25 ms poll with per-event write+flush, no batching |
@@ -151,21 +156,21 @@ Full design + implementation for each row lives in the linked domain doc under i
| SEC-10 | Medium | P1 | L | — | Done | No API-key expiry |
| SEC-11 | Medium | P1 | M | SEC-08 | Done | No rate limiting or lockout on either auth surface |
| SEC-12 | Medium | P1 | M | — | Done | Dashboard Close/Kill bypass the canonical audit store |
| SEC-13 | Low | — | S | SEC-30 | Not started | Redactor's credential-command list omits secured-bulk variants |
| SEC-14 | Low | — | S | SEC-02,20 | Not started | `/metrics` + `/health` unauthenticated; a metric leaks session ids |
| SEC-15 | Low | — | S | — | Not started | GET `/logout` skips antiforgery |
| SEC-16 | Low | — | S | — | Not started | CLI accepts the pepper as a command-line argument |
| SEC-17 | Low | — | S | — | Not started | Interceptor does not override client-streaming/duplex handlers |
| SEC-18 | Low | — | S | — | Not started | Pepper-unavailable detection matches library message text |
| SEC-19 | Low | — | S | — | Not started | Canonical audit store re-issues `CREATE TABLE` per write/read |
| SEC-13 | Low | — | S | SEC-30 | Open | Redactor's credential-command list omits secured-bulk variants — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O16 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-14 | Low | — | S | SEC-02,20 | Done | `/metrics` + `/health` unauthenticated; a metric leaks session ids — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G11), `65bca80` + `82a856b`: opt-in `Metrics:RequireAuthentication`; `/health` stays open by design |
| SEC-15 | Low | — | S | — | Won't fix | GET `/logout` skips antiforgery — **2026-10-21 triage:** accepted nuisance: GET `/logout` skips antiforgery deliberately (comment in `DashboardEndpointRouteBuilderExtensions.cs`); the POST path validates antiforgery |
| SEC-16 | Low | — | S | — | Open | CLI accepts the pepper as a command-line argument — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O17 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-17 | Low | — | S | — | Done | Interceptor does not override client-streaming/duplex handlers — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G2), `19cc74f`: client- and duplex-streaming calls authenticate and fail closed to admin |
| SEC-18 | Low | — | S | — | Open | Pepper-unavailable detection matches library message text — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O18 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-19 | Low | — | S | — | Done | Canonical audit store re-issues `CREATE TABLE` per write/read — **2026-10-21 triage:** `SqliteCanonicalAuditStore` latches schema creation, so `CREATE TABLE IF NOT EXISTS` runs once (a failure leaves the latch open) |
| SEC-20 | Low | P1 | S | — | Done | `session_id` metric tag is unbounded cardinality |
| SEC-21 | Low | — | M | — | Not started | Snapshot publisher works every second regardless of audience |
| SEC-21 | Low | — | M | — | Done | Snapshot publisher works every second regardless of audience — **2026-10-21 triage:** `DashboardSnapshotFeed` is idle-gated (no viewers, no timer), API-key summaries refresh every 15 s, and configuration is sent once per connection (R21-D4, `6733519`) |
| SEC-22 | Low | P2 | M | — | Done | `docs/Authentication.md` documents types no longer in this repo |
| SEC-23 | Low | — | S | SEC-03,04 | Not started | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` |
| SEC-24 | Low | — | S | SEC-04 | Not started | Effective-config view omits the riskiest dashboard flags |
| SEC-23 | Low | — | S | SEC-03,04 | Open | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O19 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-24 | Low | — | S | SEC-04 | Open | Effective-config view omits the riskiest dashboard flags — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O20 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-25 | Low | P2 | M | SEC-02 | Done | Per-session EventsHub ACL is an acknowledged TODO |
| SEC-26 | Low | — | M | — | Not started | Audit trail has no retention or pruning |
| SEC-27 | Low | — | S | — | Not started | Dashboard `GatewayStatus` is hardcoded Healthy |
| SEC-26 | Low | — | M | — | Done | Audit trail has no retention or pruning — **2026-10-21 triage:** `Security/Audit/AuditDrainService.cs` prunes by `AuditRetentionDays` (validated in `GatewayOptionsValidator`) on an hourly sweep |
| SEC-27 | Low | — | S | — | Done | Dashboard `GatewayStatus` is hardcoded Healthy — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-D9), `6733519`: `GatewayStatus` derives from health checks |
| SEC-28 | Info | — | S | — | N/A | Positive security observations (preserve under change) |
| SEC-29 | Info | — | S | — | N/A | UI-stack rule verified compliant |
| SEC-30 | Info | P2 | S | SEC-13 | Done | Value-logging feature is unwired end-to-end |
@@ -178,35 +183,35 @@ Full design + implementation for each row lives in the linked domain doc under i
| CLI-02 | High | P1 | M | — | Done | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) |
| CLI-03 | High | P0 | M | — | Done | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
| CLI-04 | High | P2 | L | CLI-15 | Done | Typed-command parity gap across all five clients |
| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
| CLI-05 | Medium | — | S | — | Done | .NET session cannot be re-attached to an existing session id — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN5), `6f0b408`: `MxGatewayClient.AttachSession` |
| CLI-06 | Medium | — | S | — | Done | .NET `DisposeAsync` blocks/throws on unreachable gateway — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN3), `6f0b408`: `MxGatewaySession.DisposeAsync` is best-effort and never throws |
| CLI-07 | Medium | — | S | — | Open | .NET retry budget self-defeats on `DeadlineExceeded` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O21 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-08 | Medium | — | S | CLI-03 | Done | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) — landed via 2026-07-12 [CLI-38](../2026-07-12/remediation/50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1) |
| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
| CLI-09 | Medium | — | M | — | Open | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O22 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-10 | Medium | — | M | — | Done | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CG2), `a3a7bd2`: `grpc.NewClient` replaces `DialContext` + `WithBlock` |
| CLI-11 | Medium | — | S | — | Open | Go CLI cannot opt into strict TLS validation — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O23 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-12 | Medium | P2 | S | — | Done | Java docs still say "Java 21" after the JDK 17 retarget |
| CLI-13 | Medium | — | M | — | Not started | Java event-stream buffer hardcoded 16, cancel-on-overflow |
| CLI-14 | Medium | — | S | — | Not started | Python default TLS is blocking TOFU pin with silent `localhost` SNI |
| CLI-13 | Medium | — | M | — | Done | Java event-stream buffer hardcoded 16, cancel-on-overflow — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CJ1/CJ2), `48c9323`: event queue is configurable (default 1024) and overflow keeps buffered events |
| CLI-14 | Medium | — | S | — | Won't fix | Python default TLS is blocking TOFU pin with silent `localhost` SNI — **2026-10-21 triage:** documented and intentional: the Python README describes the trust-on-first-use default and `--require-certificate-validation`; the `localhost` SNI default matches the gateway's self-signed SAN (`options.py:160`) |
| CLI-15 | Medium | P2 | M | — | Done | No client handles `ReplayGap` or offers a reconnect helper |
| CLI-16 | Medium | P2 | S | CLI-12 | Done | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` |
| CLI-17 | Medium | — | M | CLI-14 | Not started | TLS default posture inconsistent across the five clients |
| CLI-17 | Medium | — | M | CLI-14 | Won't fix | TLS default posture inconsistent across the five clients — **2026-10-21 triage:** documented divergence (`docs/CrossLanguageSmokeMatrix.md`); a strict default would break every deployment on the gateway's self-signed certificate. Strict opt-in exists per client; the Go CLI gap is R21-O23 |
| CLI-18 | Low | P2 | S | — | Done | .NET csproj records no `<Version>` |
| CLI-19 | Low | — | S | — | Not started | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) |
| CLI-20 | Low | — | S | CLI-17 | Not started | .NET `--tls` without CA installs accept-all callback |
| CLI-19 | Low | — | S | — | Open | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O24 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-20 | Low | — | S | CLI-17 | Won't fix | .NET `--tls` without CA installs accept-all callback — **2026-10-21 triage:** same decision as CLI-17: documented, intentional default; `RequireCertificateValidation` opts in |
| CLI-21 | Low | P2 | S | — | Done | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases |
| CLI-22 | Low | — | S | — | Not started | Go `newCorrelationID` swallows `crypto/rand` error → empty id |
| CLI-23 | Low | — | S | — | Not started | Go nil-vs-empty bulk short-circuit asymmetry |
| CLI-22 | Low | — | S | — | Open | Go `newCorrelationID` swallows `crypto/rand` error → empty id — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O25 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-23 | Low | — | S | — | Won't fix | Go nil-vs-empty bulk short-circuit asymmetry — **2026-10-21 triage:** cosmetic; an empty bulk is a local no-op in some helpers and a wire call in others |
| CLI-24 | Low | — | S | — | Done | Java `MxEventStream` single-consumer constraint undocumented (closed 2026-08-07 per 2026-07-12 review old-tracker action; documented at MxEventStream.java:25 "Single consumer") |
| CLI-25 | Low | — | S | — | Not started | Java `close()` does not await channel termination |
| CLI-25 | Low | — | S | — | Won't fix | Java `close()` does not await channel termination — **2026-10-21 triage:** `closeAndAwaitTermination()` is the documented awaiting path; the leak window is bounded by JVM exit |
| CLI-26 | Low | P2 | S | — | Done | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) |
| CLI-27 | Low | — | S | — | Not started | Python `Session.close()` not concurrency-safe; synthesizes reply |
| CLI-28 | Low | — | S | — | Not started | Python circular-import workaround at end of `session.py` |
| CLI-27 | Low | — | S | — | Open | Python `Session.close()` not concurrency-safe; synthesizes reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O26 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-28 | Low | — | S | — | Won't fix | Python circular-import workaround at end of `session.py` — **2026-10-21 triage:** the import cycle is deliberate, marked `# noqa: E402`, and covered by the Python test suite |
| CLI-29 | Low | P2 | S | — | Done | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) |
| CLI-30 | Low | — | S | CLI-04 | Done | Rust has no `unregister` typed helper |
| CLI-31 | Low | — | M | — | Not started | Rust CLI is a single 2,699-line `main.rs` |
| CLI-32 | Low | — | S | — | Not started | Client-side bulk caps differ (.NET/Java unbounded) |
| CLI-33 | Low | — | S | CLI-01,13 | Not started | Per-language event backpressure semantics undocumented |
| CLI-31 | Low | — | M | — | Open | Rust CLI is a single 2,699-line `main.rs` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O27 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-32 | Low | — | S | — | Won't fix | Client-side bulk caps differ (.NET/Java unbounded) — **2026-10-21 triage:** the gateway is the authority for bulk caps; the divergence affects only the error type for an oversized call |
| CLI-33 | Low | — | S | CLI-01,13 | Open | Per-language event backpressure semantics undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O28 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-34 | Low | — | S | — | Done | Python `build/`/`.pytest_cache/` present on disk (untracked) (closed 2026-08-07 per 2026-07-12 review old-tracker action; both gitignored in clients/python/.gitignore) |
### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)
@@ -218,23 +223,23 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-03 | High | P1 | M | — | Done | No CI exists |
| TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
| TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *` → `scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
| TST-06 | Medium | — | M | — | Materially narrowed | Dashboard live-data path untested — `DashboardLiveDataServiceTests`, `EventsHubTests`, `EventsHubViewerRegistryTests`, `AlarmsHubPublisherTests`, `SessionDetailsPageEventAclTests` now exist (landed with SEC-25/TST-16); needs a re-audit-and-close pass, not re-implementation |
| TST-07 | Medium | — | S | — | Open | Real-clock sleeps with negative assertions are latent flakes — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O29 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
| TST-10 | Medium | — | M | — | Not started | Deployment/upgrade undocumented and hand-rolled |
| TST-09 | Medium | — | M | — | Open | Health checks cover only the auth store — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O30 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-10 | Medium | — | M | — | Open | Deployment/upgrade undocumented and hand-rolled — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O31 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-11 | Medium | P2 | M | — | Done | No version discipline on server; client versions drift |
| TST-12 | Medium | P0 | S | — | Done | CLAUDE.md misstates default retention behaviour |
| TST-13 | Medium | P2 | S | — | Done | gateway.md carries stale design-era sketches |
| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
| TST-14 | Medium | P2 | S | — | Done | Repo-root working artifacts need triage — **2026-10-21 triage:** root `oldtasks.md` and the `*-docs-*.md` working files are no longer in the tree; `stillpending.md` remains as the backlog and stream L is refreshing it |
| TST-15 | Medium | P2 | M | TST-04 | Done | Dashboard EventsHub has no per-session ACL |
| TST-16 | Medium | P1 | S | — | Done | `Dashboard:ShowTagValues` is a dead flag |
| TST-17 | Medium | — | S | — | Not started | Vendor-gated alarm parity residuals silently lossy |
| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
| TST-17 | Medium | — | S | — | Materially changed | Vendor-gated alarm parity residuals silently lossy — `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName`; the vendor 8-arg leg is recorded "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, `d1ae43d`, `540a118`); "silent drop, unchanged" is no longer accurate — needs a worker-domain refresh pass |
| TST-18 | Low | — | S | — | Open | Hosted-service wrappers untested — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O32 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-19 | Low | — | S | — | Won't fix | Keep FakeWorkerHarness canned replies in lockstep — **2026-10-21 triage:** ongoing discipline rather than a work item; the harness shares production framing and the nightly live tier (`nightly-windev`) catches canned-reply drift |
| TST-20 | Low | — | S | — | Won't fix | E2E script papers over a real advise-without-consumer sharp edge — **2026-10-21 triage:** documented sharp edge in `docs/GatewayTesting.md`; the e2e script's interleaved drains are the intended workaround |
| TST-21 | Low | — | S | — | Open | Log rotation configured but minimal — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O33 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-22 | Low | — | S | — | Open | Config-shape JSON block omits documented keys — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O34 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
| TST-24 | Low | P2 | M | TST-03 | Done | Client wire behaviour has no automated verification — closed 2026-08-10. Go/Rust/Java already had real-server wire tests (the finding's premise was stale); the genuine gaps were .NET (transport-interface fake everywhere, no server package) and Python (stub monkeypatch everywhere but one opt-in TLS test). Added `MxGatewayClientWireTests` + `WireFakeGatewayServer` (Kestrel h2c) and `tests/test_wire_fake_gateway.py` (`grpc.aio` loopback), plus a `dotnet test` step for the .NET client in the `portable` CI job. Caught a real bug: Python `connect()` built the `grpc.aio` channel inside `asyncio.to_thread` and failed for every non-stub connection |
@@ -253,6 +258,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
| Date | Change |
|---|---|
| 2026-10-22 | **Cycle-1 `Not started` rows re-triaged against the 2026-10-21 integration branch** (stream K). All 68 rows that no later tracker had re-listed now carry a verdict: 21 `Done` (closed by earlier work or by the 2026-10-21 review), 13 `Won't fix` (reason on the row), 34 `Open` (promoted as `R21-O1`..`R21-O34` in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) with a next step). Rows closed by stream E are marked pending merge of that stream. |
| 2026-08-17 | **TST-16 → `Done`: `Dashboard:ShowTagValues` now gates every dashboard value surface** (branch `feat/followup-closeout`). The flag was live on one seam only (the events-hub mirror, shipped with SEC-25); the two recorded residuals are closed, so it is no longer a partly-dead flag. **Alarms hub:** `AlarmsHubPublisher` took `IOptions<GatewayOptions>` and now clears `current_value`/`limit_value` from a **deep clone** of each `AlarmFeedMessage` on both value-bearing payload arms (`transition`, `active_alarm`) before the `/hubs/alarms` broadcast. Clone-only, matching `DashboardEventBroadcaster.RedactValues`: the source message fans out from one feed to gRPC `StreamAlarms` subscribers and the alarms page, none of which this dashboard-display flag governs. The redaction switches on the value-bearing arms and passes everything else through untouched and uncloned, so a payload arm added later (the `snapshot_status` arm landing alongside this change) is forwarded unbroken rather than silently dropped. **`/browse`:** the gate lives at the service boundary per TST-16's own recommendation — `DashboardLiveDataService` substitutes `DashboardTagValue.RedactedValueText` (`[redacted]`) for the formatted value text, so the value never enters the render tree and `BrowsePage.razor` needed no change; data type, quality, source timestamp, and read errors stay real, keeping the panel diagnostic. A visible marker, not a blank, so a suppressed value is distinguishable from a tag that read back empty. Tests: new `AlarmsHubPublisherTests` (4 — both arms redacted with metadata intact, source not mutated, `true` passes values through as the same instance, valueless arms forwarded as the same instance) and two `DashboardLiveDataServiceTests` cases (`false` → `[redacted]`, `true` → formatted); `DashboardEventBroadcasterTests` re-run as regression. Docs: `docs/GatewayConfiguration.md`'s `ShowTagValues` row rewritten from "gates only the events hub mirror, two still-open residuals" to the three seams it now covers, with the "no gRPC client is affected" scope statement; `docs/GatewayDashboardDesign.md` gained the redaction note on the `AlarmsHub` row and had its Browse-panel paragraph corrected (it claimed the panel "always shows values regardless of `Dashboard:ShowTagValues`"). |
| 2026-08-17 | **Alarm-snapshot truncation now has a structural degraded-status signal** (branch `feat/deferred-closeout`, commits `693a78d` + `b9fb0dd`). No review ID — this is branch work outside the 153-finding register, recorded here so the tracker is not silent on a shipped change to the alarm surface. Before it, a capped `GetXmlCurrentAlarms2` fetch suppressed absence-implies-Clear inference and said so only in a rate-limited worker stderr warning, so no client and no operator could tell a complete active set from a capped one. Two additive proto3 booleans carry the verdict out — `QueryActiveAlarmsReplyPayload.snapshot_truncated = 2` and `ActiveAlarmSnapshot.from_truncated_snapshot = 16` (per record, because `QueryActiveAlarms` is a bare `stream ActiveAlarmSnapshot` with no envelope; the reply payload states it too, since a prefix filter can leave zero records and a capped fetch with nothing to report still has to say so). Flow: `WnWrapAlarmConsumer` → `AlarmDispatcher` / `IAlarmCommandHandler` → `MxAccessCommandExecutor` reply → `GatewayAlarmMonitor` → `IGatewayAlarmService.SnapshotTruncated` → `AlarmsPage` banner. `b9fb0dd` then made the pairing structural after review: `IMxAccessAlarmConsumer` and `IAlarmCommandHandler` expose one accessor (`SnapshotActiveAlarms(out bool truncated)` / `QueryActive(..., out bool snapshotTruncated)`) satisfied from a single lock acquisition, so the snapshot and its verdict can no longer be read across a poll; the separate `LastSnapshotTruncated` property is gone from every layer. Detection is deliberately unchanged (`fetchedRecordCount >= maxAlarmsPerFetch`); switching to `ALARM_RECORDS/@COUNT` stays blocked on probe evidence (`docs/AlarmProbeFindings.md`). Not latched, and dropped with the cache generation by `ClearCache`. Additive gateway metadata about our fetch mechanics, not MXAccess behaviour — no synthesized event, so not a parity deviation. Docs: `gateway.md` alarm surface, `docs/DesignDecisions.md`. |
| 2026-08-17 | **TST-15 → `Done` (discharges the ACL half of SEC-25): per-session dashboard event ACL shipped** (branch `feat/deferred-closeout`, commits `693a78d` + `7ec0b35`). Implements `docs/plans/2026-07-10-dashboard-session-acl-tst15.md`, whose header is now `Implemented` with as-built notes in its §12. `IDashboardSessionAcl.CanViewSession` is the single decision **both** subscribe seams consult — `EventsHub.SubscribeSession` (denial is a `HubException`; the caller is neither joined to the group nor registered in `EventsHubViewerRegistry`, so the mirror stays off) and `SessionDetailsPage`'s in-process subscribe (inline denial, no subscription) — so neither path is the weaker one and the `TODO(per-session-acl)` is gone. Decision order, fail-closed on every branch: authenticated Administrator → allow (evaluated **before** the registry lookup, so Admin × unknown-session allows — pinned by a test because reordering the two checks is a plausible refactor); session not found → deny; untagged session → `Dashboard:UntaggedSessionVisibility` (`AdminOnly` default); else allow iff `session.Tags ∩ zb:dashboardtag` claims, ordinal-ignore-case. Session tags are inherited from the owning API key's `dashboard_tags` constraint (`apikey --dashboard-tags`, already in the `ApiKeyConstraints` JSON blob — no SQLite migration) and never from the client's wire request. Viewer grants come from `Dashboard:GroupToTag` over the user's LDAP groups, stamped at cookie login (`DashboardAuthenticator.CreatePrincipal`) and **re-resolved, not copied**, at hub-token mint (`HubTokenService.Issue`), so the 5-minute token lifetime bounds a stale grant. Anonymous localhost is an empty-grant Viewer; `Dashboard:DisableLogin` auto-login carries both roles and so takes the admin bypass unchanged. Tests: `DashboardSessionAclTests` (decision table, every branch asserted in its denying direction too), `EventsHubTests`, `DashboardAuthenticatorTests`, `HubTokenServiceTests`, a `GatewayOptionsTests` case proving `Dashboard:GroupToTag` keeps its ordinal-ignore-case lookup through configuration binding, and two `[LiveLdapFact]`s in `DashboardLdapLiveTests` that drive a real bind against the shared GLAuth (`gw-viewer` → `team-a` grant admits the `team-a` session and refuses the `team-b` one; `multi-role` bypasses on the sessions its own grant does not cover). The live pair needed **no GLAuth change** — the tag layer is config-side, keyed on the existing `GwAdmin`/`GwReader` groups (recorded in `glauth.md`). Docs: `docs/Sessions.md` (session-tag model), `gateway.md` + CLAUDE.md dashboard-auth paragraphs, `docs/GatewayDashboardDesign.md` (three passages that described the ACL as outstanding), `docs/GatewayConfiguration.md` (`ShowTagValues` row: redaction is now the second of two layers, not the only one), `docs/Authentication.md` (`--dashboard-tags` is the only *constraint* flag that splits on commas). |
@@ -330,6 +330,8 @@ Open question the review cannot settle: whether a future probe (COM-init readine
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"`.
**Resolution: Won't fix (2026-08-20, closed by cycle-3 verification).** `Grpc/MxAccessGatewayService.cs:111-115` carries a `PERF(followup)` comment documenting the double session lookup as measured-negligible and deliberately not collapsed — the fix would duplicate `InvokeAsync`'s fault mapping at a second entry point, trading a negligible per-command cost for a maintenance liability. Per the cycle-3 gateway-core review (`archreview/2026-08-19/10-gateway-core.md`, GWC-16 row), the rationale is sound; tracker flipped from `Not started` to `Won't fix`.
---
## GWC-17 — Sessions layer throws `Grpc.Core.RpcException` `Low` · `—`
@@ -175,6 +175,8 @@ The frame protocol and proto evolution hygiene are sound; the remediable risk cl
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` + `dotnet test ...Tests --filter FullyQualifiedName~WorkerClient`; worker side `-p:Platform=x86`.
**Resolution: Done (2026-08-20, closed by cycle-3 verification, residual filed).** Wired up substantially as designed: `InvokeAsync` forwards a `WorkerCancel` when the gateway's own command timeout fires and it owned the removal (`Server/Workers/WorkerClient.cs:269-291`), `TrySendCancelForTimedOutCommand` is structurally no-throw (`:835-870`), the worker routes it to `CancelCommand` (`Worker/Ipc/WorkerPipeSession.cs:755-756` → `MxAccess/MxAccessStaSession.cs:529-532` → `Sta/StaCommandDispatcher.CancelQueuedCommand:151`), regression test `WorkerClientTests.InvokeAsync_WhenCommandTimesOut_SendsWorkerCancelForThatCorrelation` exists, and `docs/GatewayProcessDesign.md:596-602` documents it. Landed in `7c1ea12`/`a1a38b5`. **Residual, not covered by this closure:** the caller-cancellation leg (context deadline / client disconnect, as opposed to the gateway's own `CommandTimeout`) sends nothing — filed as cycle-3 **IPC-34**. Per `archreview/2026-08-19/30-contracts-ipc.md` (IPC-08 row), tracker flipped from `Not started` to `Done` with the residual cross-referenced.
---
## IPC-09 — Known codegen fragilities have no in-repo guards `Medium` · `P1`
@@ -191,6 +191,8 @@ This finding is the umbrella; TST-01/02/15 are its actionable slices. The remedi
**Verification.** `dotnet test ... --filter FullyQualifiedName~DashboardLiveDataServiceTests`.
**Status refresh (2026-08-20, per cycle-3 verification).** Materially narrowed — the tracker row was stale. `Tests/Gateway/Dashboard/DashboardLiveDataServiceTests.cs`, `EventsHubTests.cs`, `EventsHubViewerRegistryTests.cs`, `AlarmsHubPublisherTests.cs`, and `SessionDetailsPageEventAclTests.cs` all exist now (mostly landed with the SEC-25/TST-16 closeouts and the in-process dashboard-feed work). Per `archreview/2026-08-19/60-testing-docs-gaps.md` (TST-06 row), this needs a re-audit-and-close pass against the current coverage rather than fresh implementation — left open pending that audit, tracker status changed from `Not started` to `Materially narrowed` to reflect the actual state.
---
## TST-07 — Real-clock sleeps with negative assertions are latent flakes `Medium` · `—`
@@ -371,6 +373,8 @@ If TST-02's interim mitigation (flip retention off) is chosen instead of impleme
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86`; `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/... -p:Platform=x86` (Windows).
**Status refresh (2026-08-20, per cycle-3 verification).** Materially changed — the tracker row needs a refresh from the worker domain. `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName` into the COM call (the `_ = ackOperatorDomain` discard is gone), and the 2026-08 probe cycle recorded the vendor 8-arg leg as "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, commits `d1ae43d`, `540a118`). Per `archreview/2026-08-19/60-testing-docs-gaps.md` (TST-17 row), "silent drop, unchanged" is no longer an accurate description of the row; whether a caller-visible degrade signal now exists is the worker reviewer's call to close out. Tracker status changed from `Not started` to `Materially changed`.
---
## TST-18 — Hosted-service wrappers untested `Low` · `—`
+5
View File
@@ -122,6 +122,11 @@ The .NET client applies a bounded Polly retry policy only to idempotent calls:
streams, writes, secured writes, authentication, registration, item management,
or subscription changes because those calls can partially succeed in MXAccess.
Each attempt carries its own `DefaultCallTimeout` deadline. The overall budget
for a retried call is `MaxAttempts × DefaultCallTimeout` plus `MaxDelay` for each
backoff, so an attempt that ends in `DeadlineExceeded` leaves the retry its own
full deadline instead of consuming the whole budget.
API key may be loaded from `MXGATEWAY_API_KEY` by the CLI, not implicitly by the
library constructor unless a helper explicitly says it does that.
+43 -5
View File
@@ -137,14 +137,42 @@ gateway's own sentinel observable. Normal events pass through with
`IsReplayGap == false` and `ReplayGap == null`. The gap is only ever produced by
`StreamEvents`; the diagnostic drain path never emits it. If you already consume
the raw `StreamEventsAsync` (or the client-level stream), the
### Caller-owned channels, resuming sessions, diagnostics
`MxGatewayClient.Create(GrpcChannel, MxGatewayClientOptions)` and
`GalaxyRepositoryClient.Create(GrpcChannel, MxGatewayClientOptions)` build a
client over a channel the caller created (shared channel, custom handler or
TLS). The client never disposes a caller-owned channel; `Create(options)` still
creates and owns its own channel.
`MxGatewayClient.AttachSession(sessionId)` returns an `MxGatewaySession` bound
to an existing session id without calling OpenSession, so a restarted process
can resume `StreamEventsAsync(afterWorkerSequence)`. The gateway only allows the
API key that opened the session to attach; any other key gets `PermissionDenied`.
The attached session's `OpenSessionReply` carries only the session id.
`MxGatewaySession` has typed diagnostics: `PingAsync`, `GetSessionStateAsync`,
`GetWorkerInfoAsync`, and `DrainEventsAsync` (removes queued events from the
worker; call until empty). `DisposeAsync` closes the session best-effort and does
not throw when the gateway is unreachable or the client is already disposed.
Retry applies only to idempotent calls (`CloseSessionRawAsync`, ping,
session-state and worker-info commands, Galaxy reads). Each attempt gets its own
`DefaultCallTimeout`; a retried call can take up to `Retry.MaxAttempts ×
DefaultCallTimeout` plus the backoff between attempts. `AcknowledgeAlarmAsync`
is never retried because it is a non-idempotent operator write that re-sends the
operator password and can double-acknowledge in the Historian.
`AsStreamItemsAsync()` extension projects any `IAsyncEnumerable<MxEvent>` into
the same `MxEventStreamItem` surface.
For alarms, the client exposes `QueryActiveAlarmsAsync` (one-shot snapshot of
the active alarms the gateway's central monitor currently holds),
`StreamAlarmsAsync` (server-streaming feed of alarm-state-change messages
keyed by the same monitor), and `AcknowledgeAlarmAsync` (ack by alarm
reference, optional comment, ack target). All three accept a cancellation
keyed by the same monitor), and `AcknowledgeAlarmAsync` (acknowledges as
the operator: the gateway authenticates `OperatorUser`/`OperatorPassword` and
writes the comment to the alarm's `AckMsg`, so the Historian records the
operator). The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. All three accept a cancellation
token and pass through the `MxGateway:Alarms` configuration on the
server — when alarms are disabled, the gateway returns an empty list / empty
stream rather than failing.
@@ -166,6 +194,11 @@ when the gateway's alarm monitor restarts and drops a truncated verdict. Track
it if you need set completeness on a live feed without polling
`QueryActiveAlarmsAsync`.
The CLI's text rows always spell out `degraded=false` and `truncated=false` so
healthy and clearing frames remain visible. Its protobuf-JSON modes retain the
standard proto3 omission rule, so those same false-valued payloads appear as
`{"providerStatus": {}}` and `{"snapshotStatus": {}}`; absent means false.
`MxGatewaySession.CloseAsync` is explicit and idempotent. Repeated calls return
the first `CloseSessionReply` instead of sending another close request.
@@ -285,11 +318,16 @@ dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- advise --s
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write2 --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --timestamp 2026-01-01T00:00:00Z --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-events --session-id <id> --max-events 1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-alarms --filter-prefix Area001 --max-events 1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --comment "ack from cli" --operator operator1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-alarms --filter-prefix Area001 --max-events 4 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --comment "ack from cli" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- smoke --endpoint http://localhost:5000 --api-key-env MXGATEWAY_API_KEY --item Area001.Pump001.Speed --json
```
`acknowledge-alarm` reads the operator password from `--operator-password` or the
environment variable named by `--operator-password-env` (default
`MXGATEWAY_OPERATOR_PASSWORD`); the password is never echoed and is redacted from
any error output. Prefer the environment variable to keep it off the command line.
`smoke` opens a session, registers a client, adds one item, advises it,
optionally writes a value when `--type` and `--value` are supplied, reads a
bounded event stream, and closes the session in a `finally` block. CLI error
@@ -518,7 +556,7 @@ dotnet nuget add source https://gitea.dohertylan.com/api/packages/dohertj2/nuget
Then add the package to your project:
````bash
dotnet add package ZB.MOM.WW.MxGateway.Client --version 0.2.0
dotnet add package ZB.MOM.WW.MxGateway.Client --version 0.3.0
````
The `ZB.MOM.WW.MxGateway.Contracts` package is pulled in transitively.
@@ -185,6 +185,7 @@ public static class MxGatewayClientCli
exception.Message,
apiKey,
TryResolveVerifyUserPassword(arguments),
ResolveOperatorPassword(arguments),
arguments.GetOptional("value"));
if (forceJsonErrors || arguments.HasFlag("json"))
@@ -347,13 +348,19 @@ public static class MxGatewayClientCli
/// <summary>
/// Canonical CLI credential environment variable, shared by every official
/// client CLI (CLI-45) so one exported variable drives the same operator
/// client CLI so one exported variable drives the same operator
/// workflow in all five languages.
/// </summary>
private const string DefaultVerifyPasswordEnvironmentName = "MXGATEWAY_VERIFY_PASSWORD";
/// <summary>
/// Pre-CLI-45 environment variable, still honoured as a deprecated fallback
/// Default environment variable holding the operator password for
/// <c>acknowledge-alarm</c> when <c>--operator-password-env</c> is not given.
/// </summary>
private const string OperatorPasswordEnvironmentName = "MXGATEWAY_OPERATOR_PASSWORD";
/// <summary>
/// Legacy environment variable, still honoured as a deprecated fallback
/// for one release so existing scripts keep working.
/// </summary>
private const string LegacyVerifyPasswordEnvironmentName = "MXGATEWAY_VERIFY_USER_PASSWORD";
@@ -379,14 +386,38 @@ public static class MxGatewayClientCli
}
/// <summary>
/// Resolves the effective MXAccess verify-user credential in the CLI-45
/// Resolves the <c>acknowledge-alarm</c> operator password: <c>--operator-password</c>,
/// then the environment variable named by <c>--operator-password-env</c>
/// (default <c>MXGATEWAY_OPERATOR_PASSWORD</c>). An empty value is treated as
/// absent. The value is never echoed; it is also fed to the error redactor.
/// </summary>
private static string? ResolveOperatorPassword(CliArguments arguments)
{
string? password = arguments.GetOptional("operator-password");
if (!string.IsNullOrEmpty(password))
{
return password;
}
string environmentName = arguments.GetOptional("operator-password-env") ?? string.Empty;
if (string.IsNullOrEmpty(environmentName))
{
environmentName = OperatorPasswordEnvironmentName;
}
password = Environment.GetEnvironmentVariable(environmentName);
return string.IsNullOrEmpty(password) ? null : password;
}
/// <summary>
/// Resolves the effective MXAccess verify-user credential in this
/// order: <c>--password</c>, the deprecated <c>--verify-user-password</c>
/// alias, the environment variable named by <c>--password-env</c> (default
/// <c>MXGATEWAY_VERIFY_PASSWORD</c>), then the deprecated
/// <c>MXGATEWAY_VERIFY_USER_PASSWORD</c>. An empty value from any source is
/// treated as absent. The credential is never echoed; this resolver exists so
/// the error-redaction catch block can strip it from any surfaced error
/// (CLI-04), mirroring <see cref="TryResolveApiKey" />.
/// the error-redaction catch block can strip it from any surfaced error,
/// mirroring <see cref="TryResolveApiKey" />.
/// </summary>
private static string? TryResolveVerifyUserPassword(CliArguments arguments)
{
@@ -416,7 +447,7 @@ public static class MxGatewayClientCli
/// Resolves the verify-user credential for <c>authenticate-user</c>, throwing
/// a redaction-safe error when no source yields a non-empty value. Failing
/// fast keeps a misconfigured environment from becoming a real MXAccess
/// authentication attempt with an empty credential (CLI-45); the thrown
/// authentication attempt with an empty credential; the thrown
/// message names only the option/env var, never the value.
/// </summary>
private static string ResolveVerifyUserPassword(CliArguments arguments)
@@ -758,9 +789,9 @@ public static class MxGatewayClientCli
{
// The credential is resolved from --password or its env var (default
// MXGATEWAY_VERIFY_PASSWORD) and is never echoed; a missing or empty value
// fails fast before the invoke rather than reaching the wire (CLI-45).
// fails fast before the invoke rather than reaching the wire.
// On any surfaced error the RunCoreAsync catch block routes
// it through MxGatewayCliSecretRedactor so it cannot reach stderr (CLI-04).
// it through MxGatewayCliSecretRedactor so it cannot reach stderr.
return InvokeAndWriteAsync(
arguments,
client,
@@ -1425,7 +1456,7 @@ public static class MxGatewayClientCli
else if (gatewayEvent.ReplayGap is { } replayGap)
{
// Render the ReplayGap sentinel as the typed cross-CLI row instead of the raw
// sentinel MxEvent (NEXT-02, mirroring the Go/Python/Rust CLIs).
// sentinel MxEvent, mirroring the Go/Python/Rust CLIs.
output.WriteLine(FormatReplayGapRow(replayGap));
}
else
@@ -1535,6 +1566,8 @@ public static class MxGatewayClientCli
AlarmFullReference = arguments.GetRequired("reference"),
Comment = arguments.GetOptional("comment") ?? string.Empty,
OperatorUser = arguments.GetOptional("operator") ?? string.Empty,
OperatorDomain = arguments.GetOptional("operator-domain") ?? string.Empty,
OperatorPassword = ResolveOperatorPassword(arguments) ?? string.Empty,
};
return WriteReplyAsync(
@@ -1561,9 +1594,9 @@ public static class MxGatewayClientCli
AlarmFeedMessage.PayloadOneofCase.Transition =>
$"transition {ProtobufJsonFormatter.Format(feedMessage.Transition)}",
AlarmFeedMessage.PayloadOneofCase.ProviderStatus =>
$"provider-status {ProtobufJsonFormatter.Format(feedMessage.ProviderStatus)}",
$"provider-status mode={feedMessage.ProviderStatus.Mode} degraded={feedMessage.ProviderStatus.Degraded.ToString().ToLowerInvariant()} reason=\"{feedMessage.ProviderStatus.Reason}\"",
AlarmFeedMessage.PayloadOneofCase.SnapshotStatus =>
$"snapshot-status {ProtobufJsonFormatter.Format(feedMessage.SnapshotStatus)}",
$"snapshot-status truncated={feedMessage.SnapshotStatus.Truncated.ToString().ToLowerInvariant()}",
_ => $"unknown-payload {feedMessage.PayloadCase}",
};
}
@@ -1849,7 +1882,7 @@ public static class MxGatewayClientCli
}
/// <summary>
/// Formats the typed ReplayGap row shared by the CLIs (NEXT-02). Hand-built so the
/// Formats the typed ReplayGap row shared by the CLIs. Hand-built so the
/// cursors are JSON numbers like the Go/Python/Rust rows, not the protobuf JSON
/// formatter's quoted uint64 strings.
/// </summary>
@@ -2465,7 +2498,7 @@ public static class MxGatewayClientCli
writer.WriteLine("mxgw-dotnet bench-read-bulk [--duration-seconds <n>] [--warmup-seconds <n>] [--bulk-size <n>] [--tag-start <n>] [--tag-prefix <s>] [--tag-attribute <s>] [--timeout-ms <n>] [--client-name <name>]");
writer.WriteLine("mxgw-dotnet stream-events --session-id <id> [--max-events <n>] [--json]");
writer.WriteLine("mxgw-dotnet stream-alarms [--filter-prefix <ref>] [--max-events <n>] [--json] [--jsonl]");
writer.WriteLine("mxgw-dotnet acknowledge-alarm --reference <ref> [--comment <text>] [--operator <user>] [--json]");
writer.WriteLine("mxgw-dotnet acknowledge-alarm --reference <ref> --operator <user> [--operator-domain <dom>] [--operator-password <pw> | --operator-password-env <VAR>] [--comment <text>] [--json]");
writer.WriteLine("mxgw-dotnet write --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> [--json]");
writer.WriteLine("mxgw-dotnet write2 --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> [--timestamp <iso>] [--json]");
writer.WriteLine("mxgw-dotnet smoke --item <ref> [--value <value> --type <type>] [--json]");
@@ -36,6 +36,8 @@ public sealed class MxGatewayClientAlarmsTests
AlarmFullReference = "Tank01.Level.HiHi",
Comment = "investigating",
OperatorUser = "alice",
OperatorPassword = "pw-secret",
OperatorDomain = "CORP",
});
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
@@ -45,32 +47,56 @@ public sealed class MxGatewayClientAlarmsTests
Assert.Equal("Tank01.Level.HiHi", call.Request.AlarmFullReference);
Assert.Equal("investigating", call.Request.Comment);
Assert.Equal("alice", call.Request.OperatorUser);
Assert.Equal("pw-secret", call.Request.OperatorPassword);
Assert.Equal("CORP", call.Request.OperatorDomain);
Assert.Equal("Bearer test-api-key", call.CallOptions.Headers?.GetValue("authorization"));
}
/// <summary>AcknowledgeAlarmAsync redacts the operator password from a failing call's message.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AcknowledgeAlarmAsync_FailureMessage_DoesNotContainOperatorPassword()
{
FakeGatewayTransport transport = CreateTransport();
// The real transport maps RpcException to MxGatewayException using the status
// detail; the fake passes exceptions verbatim, so enqueue the mapped shape.
transport.AcknowledgeAlarmExceptions.Enqueue(new MxGatewayException("boom pw-secret"));
await using MxGatewayClient client = CreateClient(transport);
MxGatewayException ex = await Assert.ThrowsAsync<MxGatewayException>(
() => client.AcknowledgeAlarmAsync(new AcknowledgeAlarmRequest
{
AlarmFullReference = "Tank01.Level.HiHi",
OperatorUser = "alice",
OperatorPassword = "pw-secret",
}));
Assert.DoesNotContain("pw-secret", ex.Message);
Assert.Contains("boom", ex.Message);
}
/// <summary>AcknowledgeAlarmAsync honors cancellation.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AcknowledgeAlarmAsync_HonorsCancellation()
{
// Acks are routed through the safe-unary retry pipeline (idempotent at the
// MxAccess level), so the transport-side cancellation token is a linked one
// rather than the caller's original. Verify cancellation by tripping the source
// and asserting the call observes it.
// Acks bypass the retry pipeline (non-idempotent), so the caller's token
// reaches the transport unchanged.
using CancellationTokenSource cancellation = new();
cancellation.Cancel();
FakeGatewayTransport transport = CreateTransport();
await using MxGatewayClient client = CreateClient(transport);
await Assert.ThrowsAnyAsync<OperationCanceledException>(() =>
client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest
{
AlarmFullReference = "Tank01.Level.HiHi",
Comment = string.Empty,
OperatorUser = "alice",
},
cancellation.Token));
await client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest
{
AlarmFullReference = "Tank01.Level.HiHi",
Comment = string.Empty,
OperatorUser = "alice",
},
cancellation.Token);
Assert.Equal(cancellation.Token, Assert.Single(transport.AcknowledgeAlarmCalls).CallOptions.CancellationToken);
}
/// <summary>AcknowledgeAlarmAsync maps unauthenticated RPC exception to typed exception.</summary>
@@ -134,6 +134,7 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>Verifies that write-secured builds a WriteSecured command with the value and user ids.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_WriteSecured_BuildsWriteSecuredCommand()
{
@@ -178,6 +179,7 @@ public sealed class MxGatewayClientCliTests
/// Verifies that authenticate-user builds an AuthenticateUser command sourcing the
/// credential from the flag, and that the credential never appears in stdout/stderr.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_AuthenticateUser_BuildsCommandAndDoesNotEchoCredential()
{
@@ -218,9 +220,10 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>
/// CLI-04: a surfaced error for authenticate-user must have the credential
/// Verifies that a surfaced error for authenticate-user must have the credential
/// redacted (never echoed to stderr), mirroring the API-key redaction seam.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_AuthenticateUser_ErrorOutput_RedactsCredential()
{
@@ -248,7 +251,7 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>
/// CLI-45: <c>--password</c> is the primary credential flag, matching the other
/// <c>--password</c> is the primary credential flag, matching the other
/// four CLIs. The credential reaches the wire but never stdout/stderr.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
@@ -290,10 +293,12 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>
/// CLI-45: the credential is read from the environment variable named by
/// the credential is read from the environment variable named by
/// <c>--password-env</c>, whose default is the canonical
/// <c>MXGATEWAY_VERIFY_PASSWORD</c> shared by all five CLIs.
/// </summary>
/// <param name="passwordEnvArgument">The value passed to <c>--password-env</c>, or null to use the default.</param>
/// <param name="environmentName">The environment variable that holds the credential.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(null, "MXGATEWAY_VERIFY_PASSWORD")]
@@ -340,7 +345,7 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>
/// CLI-45: the pre-rename names stay usable for one release — the deprecated
/// the pre-rename names stay usable for one release — the deprecated
/// <c>--verify-user-password</c> flag and the deprecated
/// <c>MXGATEWAY_VERIFY_USER_PASSWORD</c> environment variable both still resolve.
/// </summary>
@@ -413,7 +418,7 @@ public sealed class MxGatewayClientCliTests
}
/// <summary>
/// CLI-45: a missing or empty credential fails fast before the invoke — the CLI
/// a missing or empty credential fails fast before the invoke — the CLI
/// never sends a fabricated empty password to the wire. The error names the flag
/// and the environment variable, never a value.
/// </summary>
@@ -463,6 +468,9 @@ public sealed class MxGatewayClientCliTests
private readonly string _name;
private readonly string? _original;
/// <summary>Sets the environment variable and remembers its original value.</summary>
/// <param name="name">The environment variable name.</param>
/// <param name="value">The temporary value, or null to clear it.</param>
public EnvironmentVariableScope(string name, string? value)
{
_name = name;
@@ -470,6 +478,7 @@ public sealed class MxGatewayClientCliTests
Environment.SetEnvironmentVariable(name, value);
}
/// <summary>Restores the original environment variable value.</summary>
public void Dispose()
{
Environment.SetEnvironmentVariable(_name, _original);
@@ -588,7 +597,7 @@ public sealed class MxGatewayClientCliTests
/// <summary>
/// Verifies stream-events renders the ReplayGap sentinel as the typed cross-CLI row —
/// numeric cursors under a replayGap key — instead of the raw sentinel MxEvent (NEXT-02).
/// numeric cursors under a replayGap key — instead of the raw sentinel MxEvent.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -705,11 +714,56 @@ public sealed class MxGatewayClientCliTests
Assert.Equal(string.Empty, error.ToString());
}
/// <summary>Verifies that false alarm-feed status flags remain explicit in text while protobuf JSON omits defaults.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_StreamAlarms_StatusFramesDistinguishTextFromProtobufJson()
{
FakeCliClient fakeClient = new();
fakeClient.AlarmFeedMessages.Add(new AlarmFeedMessage
{
ProviderStatus = new AlarmProviderStatus { Degraded = false },
});
fakeClient.AlarmFeedMessages.Add(new AlarmFeedMessage
{
SnapshotStatus = new AlarmSnapshotStatus { Truncated = false },
});
using var textOutput = new StringWriter();
using var textError = new StringWriter();
int textExit = await MxGatewayClientCli.RunAsync(
["stream-alarms", "--endpoint", "http://localhost:5000", "--api-key", "test-api-key", "--max-events", "2"],
textOutput,
textError,
_ => fakeClient);
Assert.Equal(0, textExit);
Assert.Equal(
["provider-status mode=Unspecified degraded=false reason=\"\"", "snapshot-status truncated=false"],
textOutput.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries));
Assert.Equal(string.Empty, textError.ToString());
using var jsonOutput = new StringWriter();
using var jsonError = new StringWriter();
int jsonExit = await MxGatewayClientCli.RunAsync(
["stream-alarms", "--endpoint", "http://localhost:5000", "--api-key", "test-api-key", "--max-events", "2", "--jsonl"],
jsonOutput,
jsonError,
_ => fakeClient);
Assert.Equal(0, jsonExit);
Assert.Equal(
["{ \"providerStatus\": { } }", "{ \"snapshotStatus\": { } }"],
jsonOutput.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries));
Assert.Equal(string.Empty, jsonError.ToString());
}
/// <summary>Verifies that acknowledge-alarm builds a request and prints the JSON reply.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_AcknowledgeAlarm_BuildsRequestAndPrintsJsonReply()
{
Environment.SetEnvironmentVariable("MXGW_TEST_OPPW", "op-pw-secret");
using var output = new StringWriter();
using var error = new StringWriter();
FakeCliClient fakeClient = new();
@@ -733,6 +787,10 @@ public sealed class MxGatewayClientCliTests
"ack from cli",
"--operator",
"operator1",
"--operator-domain",
"CORP",
"--operator-password-env",
"MXGW_TEST_OPPW",
"--json",
],
output,
@@ -744,6 +802,9 @@ public sealed class MxGatewayClientCliTests
Assert.Equal("Tank01.Level.HiHi", request.AlarmFullReference);
Assert.Equal("ack from cli", request.Comment);
Assert.Equal("operator1", request.OperatorUser);
Assert.Equal("CORP", request.OperatorDomain);
Assert.Equal("op-pw-secret", request.OperatorPassword);
Assert.DoesNotContain("op-pw-secret", output.ToString());
Assert.Contains("ack-fixture", output.ToString());
Assert.Equal(string.Empty, error.ToString());
}
@@ -2,6 +2,39 @@ namespace ZB.MOM.WW.MxGateway.Client.Tests;
public sealed class MxGatewayClientOptionsTests
{
/// <summary>
/// The safe-unary budget covers every attempt's deadline plus the backoff between them, so a
/// first attempt that hits its deadline does not consume the retry's time (#197).
/// </summary>
[Fact]
public void SafeUnaryBudget_CoversEveryAttemptAndBackoff()
{
var options = new MxGatewayClientOptions
{
Endpoint = new Uri("http://localhost:5000"),
ApiKey = "test-api-key",
DefaultCallTimeout = TimeSpan.FromSeconds(30),
Retry = new MxGatewayClientRetryOptions { MaxAttempts = 3, MaxDelay = TimeSpan.FromSeconds(2) },
};
Assert.Equal(TimeSpan.FromSeconds(94), MxGatewayClientRetryPolicy.SafeUnaryBudget(options));
}
/// <summary>A budget beyond what a cancellation timer accepts means no outer cap rather than a throw.</summary>
[Fact]
public void SafeUnaryBudget_WhenTooLargeForATimer_ReturnsNull()
{
var options = new MxGatewayClientOptions
{
Endpoint = new Uri("http://localhost:5000"),
ApiKey = "test-api-key",
DefaultCallTimeout = TimeSpan.FromDays(40),
Retry = new MxGatewayClientRetryOptions { MaxAttempts = 2 },
};
Assert.Null(MxGatewayClientRetryPolicy.SafeUnaryBudget(options));
}
/// <summary>Verifies that options with valid endpoint and API key pass validation.</summary>
[Fact]
public void Validate_WithAbsoluteEndpointAndApiKey_Succeeds()
@@ -0,0 +1,241 @@
using Grpc.Core;
using Grpc.Net.Client;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>Tests for ack no-retry, caller-owned channels, safe dispose, typed diagnostics, and AttachSession.</summary>
public sealed class MxGatewayClientRemediationTests
{
/// <summary>An acknowledge is a non-idempotent operator write and is never retried.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AcknowledgeAlarmAsync_DoesNotRetryTransientFailure()
{
FakeGatewayTransport transport = CreateTransport();
transport.AcknowledgeAlarmExceptions.Enqueue(new RpcException(new Status(StatusCode.Unavailable, "down")));
await using MxGatewayClient client = new(transport.Options, transport);
await Assert.ThrowsAsync<RpcException>(() => client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest { OperatorUser = "op", OperatorPassword = "pw" }));
Assert.Single(transport.AcknowledgeAlarmCalls);
}
/// <summary>A client created over a caller-owned channel does not dispose it.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Create_WithExistingChannel_DoesNotDisposeChannel()
{
using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost:1");
MxGatewayClient gateway = MxGatewayClient.Create(channel, Options());
GalaxyRepositoryClient galaxy = GalaxyRepositoryClient.Create(channel, Options());
await gateway.DisposeAsync();
await galaxy.DisposeAsync();
// A disposed channel throws ObjectDisposedException when a call is created.
var invoker = channel.CreateCallInvoker();
Exception? ex = Record.Exception(() => invoker.AsyncUnaryCall(
new Method<byte[], byte[]>(
MethodType.Unary,
"s",
"m",
Marshallers.Create(b => b, b => b),
Marshallers.Create(b => b, b => b)),
null,
new CallOptions(deadline: DateTime.UtcNow.AddMilliseconds(1)),
[]).Dispose());
Assert.IsNotType<ObjectDisposedException>(ex);
}
/// <summary>Create over an existing channel rejects a null channel.</summary>
[Fact]
public void Create_WithNullChannel_Throws()
{
Assert.Throws<ArgumentNullException>(() => MxGatewayClient.Create(null!, Options()));
Assert.Throws<ArgumentNullException>(() => GalaxyRepositoryClient.Create(null!, Options()));
}
/// <summary>DisposeAsync swallows a gateway failure during close and still disposes.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Session_DisposeAsync_SwallowsCloseFailure()
{
FakeGatewayTransport transport = CreateTransport();
transport.CloseSessionExceptions.Enqueue(new MxGatewayException("unreachable"));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await session.DisposeAsync();
Assert.Single(transport.CloseSessionCalls);
await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
}
/// <summary>DisposeAsync after the client is disposed does not throw.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Session_DisposeAsync_AfterClientDisposed_DoesNotThrow()
{
FakeGatewayTransport transport = CreateTransport();
MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await client.DisposeAsync();
await session.DisposeAsync();
await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
}
/// <summary>PingAsync sends a Ping command carrying the message.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task PingAsync_SendsPingCommand()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.Ping));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await session.PingAsync("hello");
MxCommandRequest request = Assert.Single(transport.InvokeCalls).Request;
Assert.Equal(MxCommandKind.Ping, request.Command.Kind);
Assert.Equal("hello", request.Command.Ping.Message);
Assert.Equal("session-fixture", request.SessionId);
}
/// <summary>GetSessionStateAsync returns the typed state.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetSessionStateAsync_ReturnsState()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.GetSessionState);
reply.SessionState = new SessionStateReply { State = SessionState.Ready };
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
SessionState state = await session.GetSessionStateAsync();
Assert.Equal(SessionState.Ready, state);
Assert.Equal(MxCommandKind.GetSessionState, Assert.Single(transport.InvokeCalls).Request.Command.Kind);
}
/// <summary>GetSessionStateAsync throws on a reply without payload.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetSessionStateAsync_MissingPayload_ThrowsMalformed()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.GetSessionState));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await Assert.ThrowsAsync<MxGatewayMalformedReplyException>(() => session.GetSessionStateAsync());
}
/// <summary>GetWorkerInfoAsync returns the worker info payload.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetWorkerInfoAsync_ReturnsWorkerInfo()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.GetWorkerInfo);
reply.WorkerInfo = new WorkerInfoReply { WorkerProcessId = 42, WorkerVersion = "1.2.3" };
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
WorkerInfoReply info = await session.GetWorkerInfoAsync();
Assert.Equal(42, info.WorkerProcessId);
Assert.Equal("1.2.3", info.WorkerVersion);
}
/// <summary>DrainEventsAsync sends max_events and returns the events.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DrainEventsAsync_ReturnsEventsAndSendsMax()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.DrainEvents);
reply.DrainEvents = new DrainEventsReply();
reply.DrainEvents.Events.Add(new MxEvent { SessionId = "session-fixture", WorkerSequence = 5 });
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
IReadOnlyList<MxEvent> events = await session.DrainEventsAsync(10);
Assert.Equal(5UL, Assert.Single(events).WorkerSequence);
Assert.Equal(10U, Assert.Single(transport.InvokeCalls).Request.Command.DrainEvents.MaxEvents);
}
/// <summary>DrainEventsAsync treats a missing payload as an empty drain.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DrainEventsAsync_MissingPayload_ReturnsEmpty()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.DrainEvents));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
Assert.Empty(await session.DrainEventsAsync());
}
/// <summary>AttachSession binds to an existing id without opening a session and resumes the stream.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AttachSession_BindsExistingIdWithoutOpening_AndResumesStream()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddEvent(new MxEvent { SessionId = "existing", WorkerSequence = 8 });
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = client.AttachSession("existing");
List<MxEvent> received = [];
await foreach (MxEvent e in session.StreamEventsAsync(afterWorkerSequence: 7))
{
received.Add(e);
}
Assert.Empty(transport.OpenSessionCalls);
Assert.Equal("existing", session.SessionId);
StreamEventsRequest request = Assert.Single(transport.StreamEventsCalls).Request;
Assert.Equal("existing", request.SessionId);
Assert.Equal(7UL, request.AfterWorkerSequence);
Assert.Single(received);
}
/// <summary>AttachSession rejects blank ids and a disposed client.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AttachSession_InvalidArguments_Throw()
{
FakeGatewayTransport transport = CreateTransport();
MxGatewayClient client = new(transport.Options, transport);
Assert.Throws<ArgumentException>(() => client.AttachSession(" "));
await client.DisposeAsync();
Assert.Throws<ObjectDisposedException>(() => client.AttachSession("x"));
}
private static MxCommandReply OkReply(MxCommandKind kind) => new()
{
SessionId = "session-fixture",
Kind = kind,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
};
private static MxGatewayClientOptions Options() => new()
{
Endpoint = new Uri("http://localhost:5000"),
ApiKey = "test-api-key",
};
private static FakeGatewayTransport CreateTransport() => new(Options());
}
@@ -228,6 +228,7 @@ public sealed class MxGatewayClientSessionTests
/// <see cref="MxEventStreamItem"/> with the gap populated, while normal
/// events pass through unchanged with IsReplayGap false.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task StreamEventItemsAsync_SurfacesReplayGapSentinel()
{
@@ -430,6 +431,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that unregister builds an unregister command with the server handle.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task UnregisterAsync_BuildsUnregisterCommand()
{
@@ -451,6 +453,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that advise-supervisory builds the supervisory advise command.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AdviseSupervisoryAsync_BuildsAdviseSupervisoryCommand()
{
@@ -473,6 +476,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that add-buffered-item returns the item handle from the typed reply.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AddBufferedItemAsync_BuildsCommandAndReturnsItemHandle()
{
@@ -498,6 +502,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that set-buffered-update-interval builds the command with the interval.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SetBufferedUpdateIntervalAsync_BuildsCommandWithInterval()
{
@@ -520,6 +525,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that suspend builds the command and returns the reply status.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SuspendAsync_BuildsCommandAndReturnsStatus()
{
@@ -545,6 +551,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that activate builds the command and returns the reply status.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ActivateAsync_BuildsCommandAndReturnsStatus()
{
@@ -568,6 +575,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that write-secured builds a WriteSecured command with value and user ids.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteSecuredAsync_BuildsWriteSecuredCommand()
{
@@ -598,6 +606,7 @@ public sealed class MxGatewayClientSessionTests
/// natively. The client must surface that scripted failure (as an
/// <see cref="MxAccessException"/>) rather than pre-validating it away.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteSecuredAsync_SurfacesNativeFailureWhenNotAuthenticated()
{
@@ -621,6 +630,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that write-secured2 builds a WriteSecured2 command with value and timestamp.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteSecured2Async_BuildsWriteSecured2Command()
{
@@ -647,6 +657,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that authenticate-user builds the command and returns the resolved user id.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AuthenticateUserAsync_BuildsCommandAndReturnsUserId()
{
@@ -675,6 +686,7 @@ public sealed class MxGatewayClientSessionTests
/// must never contain the credential — the error path is built only from
/// reply-derived diagnostics, not the request payload.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AuthenticateUserAsync_FailureDoesNotLeakCredentialInErrorMessage()
{
@@ -698,6 +710,7 @@ public sealed class MxGatewayClientSessionTests
}
/// <summary>Verifies that archestra-user-to-id builds the command and returns the resolved user id.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ArchestraUserToIdAsync_BuildsCommandAndReturnsUserId()
{
@@ -90,6 +90,41 @@ public sealed class MxGatewayClientWireTests
server.Service.AuthorizationByMethod);
}
/// <summary>
/// A safe unary call whose first attempt runs into its deadline must still get its retry:
/// the outer budget covers every attempt, not just one DefaultCallTimeout (#197).
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SafeUnaryRetry_AfterDeadlineExceededFirstAttempt_RunsTheRetry()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(service =>
service.StallFirstInvokes = 1);
await using MxGatewayClient client = server.CreateClient(
ApiKey,
TimeSpan.FromMilliseconds(500),
new MxGatewayClientRetryOptions
{
MaxAttempts = 2,
Delay = TimeSpan.FromMilliseconds(10),
MaxDelay = TimeSpan.FromMilliseconds(10),
UseJitter = false,
});
MxCommandReply reply = await client.InvokeAsync(new MxCommandRequest
{
SessionId = WireFakeGatewayServer.FakeGatewayService.SessionId,
Command = new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand(),
},
});
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.Equal(2, server.Service.InvokeCount);
}
/// <summary>
/// Verifies the gateway's replay-gap sentinel survives serialization and is
/// surfaced as a typed, non-terminal stream item.
@@ -8,6 +8,7 @@ namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// </summary>
public sealed class MxGatewaySecretRedactionTests
{
/// <summary>Test: Redact replaces every occurrence of secret.</summary>
[Fact]
public void Redact_ReplacesEveryOccurrenceOfSecret()
{
@@ -19,6 +20,7 @@ public sealed class MxGatewaySecretRedactionTests
Assert.Equal("pw=<redacted> retry pw=<redacted> again <redacted>", result);
}
/// <summary>Test: Redact scrubs both secrets when one is substring of the other.</summary>
[Fact]
public void Redact_ScrubsBothSecretsWhenOneIsSubstringOfTheOther()
{
@@ -33,6 +35,7 @@ public sealed class MxGatewaySecretRedactionTests
Assert.DoesNotContain("secret", result, StringComparison.Ordinal);
}
/// <summary>Test: Redact with null secrets array returns message unchanged.</summary>
[Fact]
public void Redact_WithNullSecretsArray_ReturnsMessageUnchanged()
{
@@ -43,6 +46,7 @@ public sealed class MxGatewaySecretRedactionTests
Assert.Equal(message, result);
}
/// <summary>Test: Redact with empty secrets array returns message unchanged.</summary>
[Fact]
public void Redact_WithEmptySecretsArray_ReturnsMessageUnchanged()
{
@@ -53,6 +57,7 @@ public sealed class MxGatewaySecretRedactionTests
Assert.Equal(message, result);
}
/// <summary>Test: Redact ignores whitespace only secret.</summary>
[Fact]
public void Redact_IgnoresWhitespaceOnlySecret()
{
@@ -64,6 +69,7 @@ public sealed class MxGatewaySecretRedactionTests
Assert.Equal(message, result);
}
/// <summary>Test: Redacted preserves concrete subtype and does not chain secret bearing original.</summary>
[Fact]
public void Redacted_PreservesConcreteSubtypeAndDoesNotChainSecretBearingOriginal()
{
@@ -4,15 +4,16 @@ using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Tests for the credential-scrub (CLI-40) and malformed-reply (CLI-41) contracts on the
/// Tests for the credential-scrub and malformed-reply contracts on the
/// credential and id-returning session helpers, driven from shared behavior fixtures.
/// </summary>
public sealed class MxGatewaySessionReplyContractTests
{
/// <summary>
/// CLI-40: when MXAccess echoes the submitted credential back in its failure diagnostic,
/// Verifies that when MXAccess echoes the submitted credential back in its failure diagnostic,
/// the surfaced exception message must scrub it to the library redaction marker.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AuthenticateUserAsync_RedactsEchoedCredentialInFailureMessage()
{
@@ -33,13 +34,14 @@ public sealed class MxGatewaySessionReplyContractTests
}
/// <summary>
/// CLI-40: the redacted exception must not leak the echoed credential through any structured
/// Verifies that the redacted exception must not leak the echoed credential through any structured
/// accessor either — <see cref="MxAccessException.Reply"/> (protocol message, diagnostic
/// message, and each MXSTATUS_PROXY diagnostic text) and <see cref="MxGatewayException.Statuses"/>
/// all carry the server-echoed credential verbatim before the fix. Both the OK+negative-HRESULT
/// and the MXACCESS_FAILURE reply route to <see cref="MxAccessException"/>, so both must scrub.
/// </summary>
/// <param name="fixture">The echoed-credential reply fixture to drive.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData("authenticate-user.echoed-credential.reply.json")]
[InlineData("authenticate-user.echoed-credential-mxaccess-failure.reply.json")]
@@ -71,9 +73,10 @@ public sealed class MxGatewaySessionReplyContractTests
}
/// <summary>
/// CLI-41: an OK reply that carries neither the typed AuthenticateUser payload nor an
/// Verifies that an OK reply that carries neither the typed AuthenticateUser payload nor an
/// int32 return_value is a malformed reply, surfaced as a typed exception rather than an NRE.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AuthenticateUserAsync_MissingPayloadAndReturnValue_ThrowsMalformedReply()
{
@@ -87,9 +90,10 @@ public sealed class MxGatewaySessionReplyContractTests
}
/// <summary>
/// CLI-41: an OK reply that omits the typed payload but carries an int32 return_value
/// Verifies that an OK reply that omits the typed payload but carries an int32 return_value
/// resolves to that return value.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AuthenticateUserAsync_ReturnValueOnly_ResolvesReturnValue()
{
@@ -104,9 +108,10 @@ public sealed class MxGatewaySessionReplyContractTests
}
/// <summary>
/// CLI-41: the AddBufferedItem fallback shares the malformed-reply contract — an OK reply
/// Verifies that the AddBufferedItem fallback shares the malformed-reply contract — an OK reply
/// with neither a typed item handle nor an int32 return_value throws the typed exception.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AddBufferedItemAsync_MissingPayloadAndReturnValue_ThrowsMalformedReply()
{
@@ -32,6 +32,9 @@ public sealed class MxStatusProxyExtensionsTests
}
/// <summary>Verifies that the raw success member never overrides the authoritative category.</summary>
/// <param name="category">The status category under test.</param>
/// <param name="success">The raw success member, which must not influence the result.</param>
/// <param name="expected">The expected <c>IsSuccess</c> result.</param>
[Theory]
[InlineData(MxStatusCategory.Ok, 0, true)]
[InlineData(MxStatusCategory.Ok, 1, true)]
@@ -88,15 +88,30 @@ internal sealed class WireFakeGatewayServer : IAsyncDisposable
/// <param name="apiKey">API key the client should present.</param>
/// <returns>A client that talks to this server over h2c.</returns>
public MxGatewayClient CreateClient(string apiKey) =>
CreateClient(apiKey, TimeSpan.FromSeconds(30), new MxGatewayClientRetryOptions());
/// <summary>
/// Creates a client bound to this server's endpoint with an explicit call timeout and retry policy.
/// </summary>
/// <param name="apiKey">API key the client should present.</param>
/// <param name="defaultCallTimeout">Per-attempt unary deadline.</param>
/// <param name="retry">Retry options for safe unary calls.</param>
/// <returns>A client that talks to this server over h2c.</returns>
public MxGatewayClient CreateClient(
string apiKey,
TimeSpan defaultCallTimeout,
MxGatewayClientRetryOptions retry) =>
MxGatewayClient.Create(new MxGatewayClientOptions
{
Endpoint = Endpoint,
ApiKey = apiKey,
UseTls = false,
DefaultCallTimeout = TimeSpan.FromSeconds(30),
DefaultCallTimeout = defaultCallTimeout,
Retry = retry,
});
/// <inheritdoc />
/// <summary>Stops and disposes the underlying web application.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
await _app.StopAsync().ConfigureAwait(false);
@@ -144,6 +159,17 @@ internal sealed class WireFakeGatewayServer : IAsyncDisposable
/// </summary>
public ReplayGap? ReplayGap { get; set; }
/// <summary>
/// Gets or sets how many leading <c>Invoke</c> calls stall until the client's deadline
/// expires, so the client sees <see cref="StatusCode.DeadlineExceeded"/> for them.
/// </summary>
public int StallFirstInvokes { get; set; }
/// <summary>Gets the number of <c>Invoke</c> calls the server received.</summary>
public int InvokeCount => Volatile.Read(ref _invokeCount);
private int _invokeCount;
/// <summary>
/// Gets the last <c>Invoke</c> request the client sent, as decoded from the wire.
/// </summary>
@@ -178,17 +204,32 @@ internal sealed class WireFakeGatewayServer : IAsyncDisposable
}
/// <inheritdoc />
public override Task<MxCommandReply> Invoke(MxCommandRequest request, ServerCallContext context)
public override async Task<MxCommandReply> Invoke(MxCommandRequest request, ServerCallContext context)
{
Record(context);
InvokeRequest = request;
if (Interlocked.Increment(ref _invokeCount) <= StallFirstInvokes)
{
// Hold the call past the client's deadline; the cancellation is the deadline
// firing (or the client giving up), either way the caller has stopped waiting.
try
{
await Task.Delay(Timeout.InfiniteTimeSpan, context.CancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
}
throw new RpcException(new Status(StatusCode.DeadlineExceeded, "stalled attempt"));
}
if (DenyInvoke)
{
throw new RpcException(new Status(StatusCode.PermissionDenied, "invoke scope required"));
}
return Task.FromResult(new MxCommandReply
return new MxCommandReply
{
SessionId = request.SessionId,
CorrelationId = request.ClientCorrelationId,
@@ -196,7 +237,7 @@ internal sealed class WireFakeGatewayServer : IAsyncDisposable
ProtocolStatus = Ok(),
Hresult = 0,
Register = new RegisterReply { ServerHandle = ServerHandle },
});
};
}
/// <inheritdoc />
@@ -22,6 +22,7 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
private const int BrowseChildrenPageSize = 500;
private readonly GrpcChannel? _channel;
private readonly bool _ownsChannel;
private readonly IGalaxyRepositoryClientTransport _transport;
private readonly ResiliencePipeline _safeUnaryRetryPipeline;
private bool _disposed;
@@ -48,9 +49,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
private GalaxyRepositoryClient(
GrpcChannel channel,
IGalaxyRepositoryClientTransport transport)
IGalaxyRepositoryClientTransport transport,
bool ownsChannel)
{
_channel = channel;
_ownsChannel = ownsChannel;
_transport = transport;
Options = transport.Options;
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -95,7 +98,32 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
channel,
new GrpcGalaxyRepositoryClientTransport(
options,
new GalaxyRepository.GalaxyRepositoryClient(channel)));
new GalaxyRepository.GalaxyRepositoryClient(channel)),
ownsChannel: true);
}
/// <summary>
/// Creates a Galaxy Repository client over a caller-owned <see cref="GrpcChannel"/>.
/// The client does not dispose the channel; the caller controls its lifetime and any
/// TLS/handler configuration.
/// </summary>
/// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
/// <param name="options">Client options (API key, timeouts, retry).</param>
/// <returns>A new client instance that does not own the channel.</returns>
public static GalaxyRepositoryClient Create(
GrpcChannel channel,
MxGatewayClientOptions options)
{
ArgumentNullException.ThrowIfNull(channel);
ArgumentNullException.ThrowIfNull(options);
options.Validate();
return new GalaxyRepositoryClient(
channel,
new GrpcGalaxyRepositoryClientTransport(
options,
new GalaxyRepository.GalaxyRepositoryClient(channel)),
ownsChannel: false);
}
/// <summary>
@@ -436,7 +464,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
}
_disposed = true;
_channel?.Dispose();
if (_ownsChannel)
{
_channel?.Dispose();
}
return ValueTask.CompletedTask;
}
@@ -485,8 +517,15 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
Func<CancellationToken, Task<T>> call,
CancellationToken cancellationToken)
{
// Each attempt carries its own DefaultCallTimeout deadline, so the outer cap must cover
// every attempt plus the backoff between them; capping it at one DefaultCallTimeout let a
// DeadlineExceeded first attempt spend the whole budget and the retry never ran (#197).
using CancellationTokenSource timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeout.CancelAfter(Options.DefaultCallTimeout);
TimeSpan? budget = MxGatewayClientRetryPolicy.SafeUnaryBudget(Options);
if (budget is not null)
{
timeout.CancelAfter(budget.Value);
}
return await _safeUnaryRetryPipeline.ExecuteAsync(
async token => await call(token).ConfigureAwait(false),
@@ -14,7 +14,8 @@ namespace ZB.MOM.WW.MxGateway.Client;
/// </summary>
public sealed class MxGatewayClient : IAsyncDisposable
{
private readonly GrpcChannel _channel;
private readonly GrpcChannel? _channel;
private readonly bool _ownsChannel;
private readonly IMxGatewayClientTransport _transport;
private readonly ResiliencePipeline _safeUnaryRetryPipeline;
private bool _disposed;
@@ -36,14 +37,16 @@ public sealed class MxGatewayClient : IAsyncDisposable
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
options.Retry,
options.LoggerFactory?.CreateLogger<MxGatewayClient>());
_channel = null!;
_channel = null;
}
private MxGatewayClient(
GrpcChannel channel,
IMxGatewayClientTransport transport)
IMxGatewayClientTransport transport,
bool ownsChannel)
{
_channel = channel;
_ownsChannel = ownsChannel;
_transport = transport;
Options = transport.Options;
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -88,7 +91,58 @@ public sealed class MxGatewayClient : IAsyncDisposable
channel,
new GrpcMxGatewayClientTransport(
options,
new MxAccessGateway.MxAccessGatewayClient(channel)));
new MxAccessGateway.MxAccessGatewayClient(channel)),
ownsChannel: true);
}
/// <summary>
/// Creates a gateway client over a caller-owned <see cref="GrpcChannel"/>. The client
/// does not dispose the channel; the caller controls its lifetime and any TLS/handler
/// configuration. <see cref="MxGatewayClientOptions.Endpoint"/> is still validated but
/// the channel's own address is used for transport.
/// </summary>
/// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
/// <param name="options">Client configuration options (API key, timeouts, retry).</param>
/// <returns>A new gateway client instance that does not own the channel.</returns>
public static MxGatewayClient Create(
GrpcChannel channel,
MxGatewayClientOptions options)
{
ArgumentNullException.ThrowIfNull(channel);
ArgumentNullException.ThrowIfNull(options);
options.Validate();
return new MxGatewayClient(
channel,
new GrpcMxGatewayClientTransport(
options,
new MxAccessGateway.MxAccessGatewayClient(channel)),
ownsChannel: false);
}
/// <summary>
/// Binds a session object to an existing gateway session id without calling
/// OpenSession, so a reconnecting process can resume
/// <see cref="MxGatewaySession.StreamEventsAsync"/> with <c>afterWorkerSequence</c>.
/// The gateway only allows the API key that opened the session to stream its events
/// or invoke commands on it; any other key is rejected with PermissionDenied. The
/// returned session's <see cref="MxGatewaySession.OpenSessionReply"/> carries only the
/// session id.
/// </summary>
/// <param name="sessionId">The id of a session previously opened with this client's API key.</param>
/// <returns>A session bound to <paramref name="sessionId"/>.</returns>
public MxGatewaySession AttachSession(string sessionId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(sessionId);
ThrowIfDisposed();
return new MxGatewaySession(
this,
new OpenSessionReply
{
SessionId = sessionId,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
});
}
/// <summary>
@@ -183,24 +237,34 @@ public sealed class MxGatewayClient : IAsyncDisposable
}
/// <summary>
/// Acknowledges an active MXAccess alarm condition through the gateway. The
/// gateway authenticates the request against the API key's <c>invoke:alarm-ack</c>
/// scope and forwards the acknowledge to the worker's MXAccess session;
/// the resulting <see cref="MxStatusProxy"/> is returned in the reply.
/// Acknowledges an alarm as the operator. The gateway authenticates
/// <see cref="AcknowledgeAlarmRequest.OperatorUser"/> with
/// <see cref="AcknowledgeAlarmRequest.OperatorPassword"/> and writes the
/// comment to the alarm's AckMsg attribute, so the alarm history records
/// the operator. The password is redacted from any surfaced error.
/// </summary>
/// <param name="request">The acknowledge request — alarm reference, comment, operator user.</param>
/// <param name="request">The acknowledge request: alarm reference, comment, operator credentials.</param>
/// <param name="cancellationToken">Cancellation token for the operation.</param>
/// <returns>The acknowledge reply with protocol + native MxStatus.</returns>
public Task<AcknowledgeAlarmReply> AcknowledgeAlarmAsync(
public async Task<AcknowledgeAlarmReply> AcknowledgeAlarmAsync(
AcknowledgeAlarmRequest request,
CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(request);
ThrowIfDisposed();
return ExecuteSafeUnaryAsync(
token => _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(token)),
cancellationToken);
// Deliberately not wrapped in the retry pipeline: an acknowledge is a
// non-idempotent operator write that re-sends the operator password and
// can double-ack in the Historian.
try
{
return await _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(cancellationToken))
.ConfigureAwait(false);
}
catch (MxGatewayException ex) when (request.OperatorPassword.Length > 0)
{
throw MxGatewaySecretRedaction.Redacted(ex, request.OperatorPassword);
}
}
/// <summary>
@@ -258,7 +322,11 @@ public sealed class MxGatewayClient : IAsyncDisposable
}
_disposed = true;
_channel?.Dispose();
if (_ownsChannel)
{
_channel?.Dispose();
}
return ValueTask.CompletedTask;
}
@@ -307,8 +375,15 @@ public sealed class MxGatewayClient : IAsyncDisposable
Func<CancellationToken, Task<T>> call,
CancellationToken cancellationToken)
{
// Each attempt carries its own DefaultCallTimeout deadline, so the outer cap must cover
// every attempt plus the backoff between them; capping it at one DefaultCallTimeout let a
// DeadlineExceeded first attempt spend the whole budget and the retry never ran (#197).
using CancellationTokenSource timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeout.CancelAfter(Options.DefaultCallTimeout);
TimeSpan? budget = MxGatewayClientRetryPolicy.SafeUnaryBudget(Options);
if (budget is not null)
{
timeout.CancelAfter(budget.Value);
}
return await _safeUnaryRetryPipeline.ExecuteAsync(
async token => await call(token).ConfigureAwait(false),
@@ -41,6 +41,29 @@ internal static class MxGatewayClientRetryPolicy
.Build();
}
/// <summary>
/// Returns the overall budget for one retried safe unary call: every attempt's
/// <see cref="MxGatewayClientOptions.DefaultCallTimeout"/> deadline plus the longest backoff
/// between attempts. Returns <see langword="null"/> when the budget is too large for a
/// cancellation timer, in which case the per-attempt deadlines and the bounded attempt count
/// already bound the call.
/// </summary>
/// <param name="options">The client options supplying the call timeout and retry settings.</param>
/// <returns>The overall budget, or <see langword="null"/> for no outer cap.</returns>
public static TimeSpan? SafeUnaryBudget(MxGatewayClientOptions options)
{
ArgumentNullException.ThrowIfNull(options);
int attempts = Math.Max(1, options.Retry.MaxAttempts);
double milliseconds = (options.DefaultCallTimeout.TotalMilliseconds * attempts)
+ (options.Retry.MaxDelay.TotalMilliseconds * (attempts - 1));
// CancellationTokenSource.CancelAfter rejects anything above uint.MaxValue - 1 ms.
return milliseconds >= uint.MaxValue - 1
? null
: TimeSpan.FromMilliseconds(milliseconds);
}
/// <summary>Returns whether a command kind is eligible for automatic retry on transient failures.</summary>
/// <param name="kind">The command kind to check.</param>
/// <returns><see langword="true"/> if the command kind may be automatically retried; otherwise, <see langword="false"/>.</returns>
@@ -1,3 +1,4 @@
using Grpc.Core;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client;
@@ -640,7 +641,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// </summary>
/// <param name="serverHandle">The ServerHandle from register.</param>
/// <param name="tagAddresses">Tag addresses to read (one per result).</param>
/// <param name="timeout">Per-call timeout for the snapshot lifecycle path; <see cref="TimeSpan.Zero"/> uses the gateway default.</param>
/// <param name="timeout">One shared snapshot deadline for the whole command, not a per-tag budget: the worker advises every uncached tag, waits once for all of them, then tears them down. <see cref="TimeSpan.Zero"/> uses the worker default (1000 ms).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>One <see cref="BulkReadResult"/> per requested tag, in request order.</returns>
public async Task<IReadOnlyList<BulkReadResult>> ReadBulkAsync(
@@ -854,6 +855,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// </summary>
/// <param name="serverHandle">The ServerHandle from register.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task UnregisterAsync(
int serverHandle,
CancellationToken cancellationToken = default)
@@ -888,6 +890,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <param name="serverHandle">The ServerHandle from register.</param>
/// <param name="itemHandle">The ItemHandle from add-item.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task AdviseSupervisoryAsync(
int serverHandle,
int itemHandle,
@@ -985,6 +988,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <param name="serverHandle">The ServerHandle from register.</param>
/// <param name="updateIntervalMilliseconds">The buffered update interval, in milliseconds.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task SetBufferedUpdateIntervalAsync(
int serverHandle,
int updateIntervalMilliseconds,
@@ -1125,6 +1129,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <param name="currentUserId">The current operator user id.</param>
/// <param name="verifierUserId">The verifier (secondary approver) user id.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task WriteSecuredAsync(
int serverHandle,
int itemHandle,
@@ -1204,6 +1209,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <param name="currentUserId">The current operator user id.</param>
/// <param name="verifierUserId">The verifier (secondary approver) user id.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task WriteSecured2Async(
int serverHandle,
int itemHandle,
@@ -1447,6 +1453,106 @@ public sealed class MxGatewaySession : IAsyncDisposable
return _client.InvokeAsync(request, cancellationToken);
}
/// <summary>
/// Sends a PING diagnostic command to the worker and throws if the reply is a failure.
/// </summary>
/// <param name="message">Optional message echoed to the worker; defaults to empty.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that completes when the worker acknowledges the ping.</returns>
public async Task PingAsync(
string? message = null,
CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand { Message = message ?? string.Empty },
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
}
/// <summary>
/// Gets the gateway's lifecycle state for this session.
/// </summary>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The current session state.</returns>
public async Task<SessionState> GetSessionStateAsync(CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.GetSessionState,
GetSessionState = new GetSessionStateCommand(),
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.SessionState?.State
?? throw CreateMalformed(reply, "GetSessionState");
}
/// <summary>
/// Gets diagnostic information about the worker process serving this session.
/// </summary>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The worker process id, version, and MXAccess COM identifiers.</returns>
public async Task<WorkerInfoReply> GetWorkerInfoAsync(CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.GetWorkerInfo,
GetWorkerInfo = new GetWorkerInfoCommand(),
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.WorkerInfo
?? throw CreateMalformed(reply, "GetWorkerInfo");
}
/// <summary>
/// Drains events queued in the worker through the diagnostic command path. This removes
/// events from the worker queue; it is not a substitute for <see cref="StreamEventsAsync"/>
/// and is not retried automatically. The reply is bounded by server-side caps, so call
/// repeatedly until it returns an empty list.
/// </summary>
/// <param name="maxEvents">Maximum events to return; 0 lets the server choose.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The drained events, possibly empty.</returns>
public async Task<IReadOnlyList<MxEvent>> DrainEventsAsync(
uint maxEvents = 0,
CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.DrainEvents,
DrainEvents = new DrainEventsCommand { MaxEvents = maxEvents },
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.DrainEvents?.Events.ToArray() ?? [];
}
private static MxGatewayMalformedReplyException CreateMalformed(MxCommandReply reply, string operation)
{
return new MxGatewayMalformedReplyException(
$"{operation} returned a malformed reply: OK reply carried no typed payload",
reply.SessionId,
reply.CorrelationId,
reply.ProtocolStatus,
reply.HasHresult ? reply.Hresult : null,
reply.Statuses.ToArray());
}
/// <summary>
/// Streams events from the worker for this session, optionally starting after a given sequence number.
/// </summary>
@@ -1500,8 +1606,20 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
await CloseAsync().ConfigureAwait(false);
_closeLock.Dispose();
try
{
await CloseAsync().ConfigureAwait(false);
}
catch (Exception ex) when (ex is MxGatewayException or ObjectDisposedException or RpcException)
{
// Best-effort close: the gateway may be unreachable or the owning client
// already disposed. Disposal must not throw or leak the lock; the gateway
// reaps the session on its own idle/detach timers.
}
finally
{
_closeLock.Dispose();
}
}
private Task<MxCommandReply> InvokeCommandAsync(
@@ -19,7 +19,7 @@
<PropertyGroup>
<IsPackable>true</IsPackable>
<PackageId>ZB.MOM.WW.MxGateway.Client</PackageId>
<Version>0.2.0</Version>
<Version>0.3.0</Version>
<Description>.NET 10 gRPC client for the MxAccessGateway service. Provides typed wrappers, retry, and a lazy-browse walker over the Galaxy Repository hierarchy.</Description>
<PackageReadmeFile>README.md</PackageReadmeFile>
<!-- Only the shipped library generates XML docs (matching src/Contracts). The Cli and
+4 -1
View File
@@ -146,7 +146,10 @@ type MxAccessError struct { ... }
```
Use `errors.Is` / `errors.As` support. Preserve raw protobuf replies on command
errors.
errors. `GatewayError` matches the sentinels `ErrUnauthenticated` and
`ErrPermissionDenied` through `errors.Is` by gRPC status code, so callers treat
the two authorization failures distinctly (`docs/ClientLibrariesDesign.md`)
without parsing messages.
## Test CLI
+47 -5
View File
@@ -92,7 +92,13 @@ returned subscription owns cancellation and exposes `Close` for deterministic
goroutine cleanup. Raw protobuf messages remain available through the
`mxgateway` package aliases and the `Raw` helper methods. Typed errors support
`errors.As` for `GatewayError`, `CommandError`, and `MxAccessError`; command
errors preserve the raw reply.
errors preserve the raw reply. Authorization failures are distinct:
`errors.Is(err, mxgateway.ErrUnauthenticated)` matches a rejected API key
(gRPC `Unauthenticated` — fix the key) and `errors.Is(err,
mxgateway.ErrPermissionDenied)` matches a valid key missing a scope or another
key's session (gRPC `PermissionDenied` — grant the scope). `GatewayError.Code()`
returns the raw gRPC status code. The match reads only the status code, so it
works through the credential-redaction wrapper.
`EnsureMxAccessSuccess` follows COM semantics: only a **negative** HRESULT is a
failure, so positive success codes such as `S_FALSE` (1) pass. `StatusSucceeded`
@@ -100,6 +106,29 @@ judges each `MXSTATUS_PROXY` entry by its category — an entry fails when
`Category` is not `MX_STATUS_CATEGORY_OK`, and the raw `Success` member is a
diagnostic that never decides the verdict. A nil entry is success.
`Dial` is connect-or-fail: it creates the connection with `grpc.NewClient`
(the endpoint resolves with the `passthrough` scheme unless it names one) and
then waits for the channel to become READY, bounded by `DialTimeout` (default
10 s) or the context deadline. A failure is returned as a `GatewayError` with
`Op == "dial"`. Passing a connection to `NewClient`/`NewGalaxyClient` stays lazy.
### Event buffering
The channel returned by `Session.Events`/`EventsAfter` buffers
`Options.EventBufferSize` data results (default `DefaultEventBufferSize`, 1024).
If the consumer falls behind and the buffer fills, the stream is cancelled; the
results already buffered are still delivered in order, followed by a final
result whose error matches `errors.Is(err, mxgateway.ErrSlowConsumer)`. Raise
`EventBufferSize` for bursty feeds, or use `SubscribeEvents`/`SubscribeEventsAfter`,
which block instead of dropping and are the non-dropping alternative.
### Diagnostics
`Session.Ping`, `GetSessionState`, `GetWorkerInfo`, and `DrainEvents` are typed
wrappers over the diagnostic commands (`PingRaw` remains for the raw reply).
`DrainEvents(ctx, maxEvents)` returns fewer events than requested when the
server caps the reply; call it until it returns an empty slice.
### Reconnect-replay gap
Each `EventResult` carries exactly one of `Event`, `ReplayGap`, or `Err`. When
@@ -140,7 +169,11 @@ sentinel typed and observable; it never synthesizes or swallows it.
For alarms, the package exposes `Client.QueryActiveAlarms` for one-shot
snapshots, `Client.StreamAlarms` for the server-streaming feed, and
`Client.AcknowledgeAlarm` to ack an alarm by full reference. The streaming
`Client.AcknowledgeAlarm` to ack an alarm by full reference. Acks carry the
operator's Galaxy credentials (`OperatorUser` required, `OperatorPassword`,
optional `OperatorDomain`) so the gateway authenticates as the operator and the
Historian records that operator; an empty password is allowed, and the password
is scrubbed from any returned error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The streaming
call returns a `StreamAlarmsClient`; cancel its context to terminate the
stream. All three pass straight through to the gateway's central alarm
monitor.
@@ -162,6 +195,11 @@ on every verdict change — including the clearing frame sent when the gateway's
alarm monitor restarts and drops a truncated verdict. Track it if you need set
completeness on a live feed without polling `QueryActiveAlarms`.
The CLI's text rows always spell out `degraded=false` and `truncated=false`.
Its protobuf-JSON mode retains the standard proto3 omission rule, so those
false-valued payloads appear as `{"providerStatus":{}}` and
`{"snapshotStatus":{}}`; absent means false.
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
@@ -394,7 +432,11 @@ the gateway:
Every subcommand wired into the CLI. All accept the common flags
(`-endpoint`, `-plaintext`, `-api-key` / `-api-key-env`, `-ca-cert`,
`-server-name-override`, `-call-timeout`) and most accept `-json`.
`-server-name-override`, `-call-timeout`, `-require-certificate-validation`) and
most accept `-json`. Over TLS without `-ca-cert` the CLI accepts any gateway
certificate (the self-signed default); `-require-certificate-validation` sets
`Options.RequireCertificateValidation` so the certificate must chain to the
system trust roots instead.
| Command | Purpose |
|---|---|
@@ -419,7 +461,7 @@ Every subcommand wired into the CLI. All accept the common flags
| `bench-read-bulk` | Throughput benchmark (`-duration-seconds`, `-warmup-seconds`, `-bulk-size`). |
| `stream-events` | Stream item-value events for a session (`-session-id`, `-limit`). |
| `stream-alarms` | Stream the alarm feed (`-filter-prefix`, `-limit`). |
| `acknowledge-alarm` | Acknowledge an alarm reference. |
| `acknowledge-alarm` | Acknowledge an alarm as an operator (`-operator`, `-operator-domain`, `-operator-password-env`, default `MXGATEWAY_OPERATOR_PASSWORD`). |
| `smoke` | End-to-end smoke workflow against one item. |
| `galaxy-test-connection` | Probe the Galaxy Repository RPC connection. |
| `galaxy-last-deploy` | Print the most recent deploy event. |
@@ -439,7 +481,7 @@ go run ./cmd/mxgw-go write -session-id <id> -server-handle 1 -item-handle 1 -typ
go run ./cmd/mxgw-go write-bulk -session-id <id> -server-handle 1 -item-handles 1,2 -values 10,20 -type int32 -plaintext -json
go run ./cmd/mxgw-go read-bulk -session-id <id> -item-handles 1,2 -plaintext -json
go run ./cmd/mxgw-go stream-events -session-id <id> -plaintext -json
go run ./cmd/mxgw-go stream-alarms -plaintext -json
go run ./cmd/mxgw-go stream-alarms -plaintext -limit 4 -json
go run ./cmd/mxgw-go smoke -item Area001.Tag.Value -plaintext -json
go run ./cmd/mxgw-go galaxy-test-connection -plaintext -json
go run ./cmd/mxgw-go galaxy-last-deploy -plaintext -json
+45 -15
View File
@@ -40,6 +40,10 @@ type commonOptions struct {
CACertFile string `json:"caCertFile,omitempty"`
ServerName string `json:"serverNameOverride,omitempty"`
CallTimeout string `json:"callTimeout,omitempty"`
// RequireCertificateValidation verifies the gateway certificate against the
// system trust roots when no -ca-cert is pinned, instead of the lenient
// self-signed default.
RequireCertificateValidation bool `json:"requireCertificateValidation,omitempty"`
apiKeyValue string
timeout time.Duration
@@ -571,7 +575,7 @@ func runReadBulk(ctx context.Context, args []string, stdout, stderr io.Writer) e
sessionID := flags.String("session-id", "", "gateway session id")
serverHandle := flags.Int("server-handle", 0, "MXAccess server handle")
items := flags.String("items", "", "comma-separated tag addresses")
timeoutMs := flags.Int("timeout-ms", 0, "per-tag snapshot timeout in milliseconds (0 = worker default)")
timeoutMs := flags.Int("timeout-ms", 0, "snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms)")
if err := flags.Parse(args); err != nil {
return err
@@ -754,7 +758,7 @@ func runBenchReadBulk(ctx context.Context, args []string, stdout, stderr io.Writ
tagStart := flags.Int("tag-start", 1, "first machine number")
tagPrefix := flags.String("tag-prefix", "TestMachine_", "tag prefix (machine number appended as %03d)")
tagAttribute := flags.String("tag-attribute", "TestChangingInt", "attribute appended to each tag prefix")
timeoutMs := flags.Int("timeout-ms", 1500, "per-tag snapshot timeout in milliseconds")
timeoutMs := flags.Int("timeout-ms", 1500, "snapshot deadline in milliseconds, shared by all tags in each ReadBulk command")
if err := flags.Parse(args); err != nil {
return err
@@ -1119,6 +1123,10 @@ func formatAlarmFeedMessage(message *mxgateway.AlarmFeedMessage) string {
}
}
// defaultOperatorPasswordEnv is the CLI environment variable holding the
// operator password for acknowledge-alarm.
const defaultOperatorPasswordEnv = "MXGATEWAY_OPERATOR_PASSWORD"
func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.Writer) error {
flags := flag.NewFlagSet("acknowledge-alarm", flag.ContinueOnError)
flags.SetOutput(stderr)
@@ -1126,7 +1134,12 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
jsonOutput := flags.Bool("json", false, "write JSON output")
reference := flags.String("reference", "", "full alarm reference to acknowledge")
comment := flags.String("comment", "", "operator acknowledge comment")
operator := flags.String("operator", "", "operator user performing the acknowledge")
operator := flags.String("operator", "", "operator user performing the acknowledge (required)")
operatorDomain := flags.String("operator-domain", "", "operator Galaxy domain (optional)")
// Prefer the environment variable so the credential stays out of shell
// history and the process table; an empty password is allowed.
operatorPassword := flags.String("operator-password", "", "operator password (prefer -operator-password-env)")
operatorPasswordEnv := flags.String("operator-password-env", defaultOperatorPasswordEnv, "environment variable containing the operator password")
if err := flags.Parse(args); err != nil {
return err
@@ -1134,6 +1147,17 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
if *reference == "" {
return errors.New("reference is required")
}
if *operator == "" {
return errors.New("operator is required")
}
resolvedOperatorPassword := *operatorPassword
if resolvedOperatorPassword == "" {
envName := *operatorPasswordEnv
if envName == "" {
envName = defaultOperatorPasswordEnv
}
resolvedOperatorPassword = os.Getenv(envName)
}
client, options, err := dialForCommand(ctx, common)
if err != nil {
@@ -1145,6 +1169,8 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
AlarmFullReference: *reference,
Comment: *comment,
OperatorUser: *operator,
OperatorDomain: *operatorDomain,
OperatorPassword: resolvedOperatorPassword,
})
if err != nil {
return err
@@ -1275,6 +1301,8 @@ func bindCommonFlags(flags *flag.FlagSet) *commonOptions {
flags.StringVar(&common.CACertFile, "ca-cert", "", "CA certificate file")
flags.StringVar(&common.ServerName, "server-name-override", "", "TLS server name override")
flags.StringVar(&common.CallTimeout, "call-timeout", "30s", "per-call timeout")
flags.BoolVar(&common.RequireCertificateValidation, "require-certificate-validation", false,
"verify the gateway TLS certificate against the system trust roots when no -ca-cert is set (default: accept any certificate)")
return common
}
@@ -1285,12 +1313,13 @@ func dialForCommand(ctx context.Context, common *commonOptions) (*mxgateway.Clie
}
client, err := mxgateway.Dial(ctx, mxgateway.Options{
Endpoint: options.Endpoint,
APIKey: options.apiKeyValue,
Plaintext: options.Plaintext,
CACertFile: options.CACertFile,
ServerNameOverride: options.ServerName,
CallTimeout: options.timeout,
Endpoint: options.Endpoint,
APIKey: options.apiKeyValue,
Plaintext: options.Plaintext,
CACertFile: options.CACertFile,
ServerNameOverride: options.ServerName,
CallTimeout: options.timeout,
RequireCertificateValidation: options.RequireCertificateValidation,
})
return client, options, err
}
@@ -1501,12 +1530,13 @@ func dialGalaxyForCommand(ctx context.Context, common *commonOptions) (*mxgatewa
}
client, err := mxgateway.DialGalaxy(ctx, mxgateway.Options{
Endpoint: options.Endpoint,
APIKey: options.apiKeyValue,
Plaintext: options.Plaintext,
CACertFile: options.CACertFile,
ServerNameOverride: options.ServerName,
CallTimeout: options.timeout,
Endpoint: options.Endpoint,
APIKey: options.apiKeyValue,
Plaintext: options.Plaintext,
CACertFile: options.CACertFile,
ServerNameOverride: options.ServerName,
CallTimeout: options.timeout,
RequireCertificateValidation: options.RequireCertificateValidation,
})
return client, options, err
}
+62
View File
@@ -283,6 +283,68 @@ type pingFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
// alarmStatusFakeGateway emits both status arms with their proto3 boolean
// defaults so the CLI contract covers values that protobuf JSON omits.
type alarmStatusFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
func (g *alarmStatusFakeGateway) StreamAlarms(
_ *pb.StreamAlarmsRequest,
stream grpc.ServerStreamingServer[pb.AlarmFeedMessage],
) error {
if err := stream.Send(&pb.AlarmFeedMessage{
Payload: &pb.AlarmFeedMessage_ProviderStatus{
ProviderStatus: &pb.AlarmProviderStatus{Degraded: false},
},
}); err != nil {
return err
}
return stream.Send(&pb.AlarmFeedMessage{
Payload: &pb.AlarmFeedMessage_SnapshotStatus{
SnapshotStatus: &pb.AlarmSnapshotStatus{Truncated: false},
},
})
}
func startAlarmStatusGateway(t *testing.T) string {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
server := grpc.NewServer()
pb.RegisterMxAccessGatewayServer(server, &alarmStatusFakeGateway{})
go func() { _ = server.Serve(listener) }()
t.Cleanup(func() {
server.Stop()
_ = listener.Close()
})
return listener.Addr().String()
}
func TestRunStreamAlarmsPrintsStatusArmsInTextAndProtobufJSON(t *testing.T) {
endpoint := startAlarmStatusGateway(t)
baseArgs := []string{"stream-alarms", "-endpoint", endpoint, "-plaintext", "-api-key", "test", "-limit", "2"}
var textOut, textErr bytes.Buffer
if err := runWithIO(t.Context(), baseArgs, &textOut, &textErr); err != nil {
t.Fatalf("text stream-alarms error = %v; stderr = %s", err, textErr.String())
}
if got, want := strings.TrimSpace(textOut.String()), "provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=\"\"\nsnapshot-status truncated=false"; got != want {
t.Fatalf("text stream-alarms output = %q, want %q", got, want)
}
var jsonOut, jsonErr bytes.Buffer
jsonArgs := append(append([]string{}, baseArgs...), "-json")
if err := runWithIO(t.Context(), jsonArgs, &jsonOut, &jsonErr); err != nil {
t.Fatalf("JSON stream-alarms error = %v; stderr = %s", err, jsonErr.String())
}
if got, want := strings.TrimSpace(jsonOut.String()), "{\"providerStatus\":{}}\n{\"snapshotStatus\":{}}"; got != want {
t.Fatalf("JSON stream-alarms output = %q, want protobuf JSON %q", got, want)
}
}
func (g *pingFakeGateway) Invoke(_ context.Context, req *pb.MxCommandRequest) (*pb.MxCommandReply, error) {
echo := "pong:" + req.GetCommand().GetPing().GetMessage()
return &pb.MxCommandReply{
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"bytes"
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"math/big"
"net"
"strings"
"testing"
"time"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
)
// #199 — `--tls` without a CA used to be skip-verify with no way out from the
// CLI. -require-certificate-validation must make the CLI reject a certificate
// the system roots do not trust (the gateway's own self-signed one).
type openSessionFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
func (g *openSessionFakeGateway) OpenSession(context.Context, *pb.OpenSessionRequest) (*pb.OpenSessionReply, error) {
return &pb.OpenSessionReply{
SessionId: "tls-session",
ProtocolStatus: &pb.ProtocolStatus{Code: pb.ProtocolStatusCode_PROTOCOL_STATUS_CODE_OK},
}, nil
}
func startSelfSignedTLSGateway(t *testing.T) string {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatalf("generate key: %v", err)
}
template := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "localhost"},
DNSNames: []string{"localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}
der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key)
if err != nil {
t.Fatalf("create certificate: %v", err)
}
cert := tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
server := grpc.NewServer(grpc.Creds(credentials.NewTLS(&tls.Config{
Certificates: []tls.Certificate{cert},
MinVersion: tls.VersionTLS12,
})))
pb.RegisterMxAccessGatewayServer(server, &openSessionFakeGateway{})
go func() { _ = server.Serve(listener) }()
t.Cleanup(func() {
server.Stop()
_ = listener.Close()
})
return listener.Addr().String()
}
func TestRequireCertificateValidationFlagRejectsSelfSignedGateway(t *testing.T) {
endpoint := startSelfSignedTLSGateway(t)
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
base := []string{"open-session", "-endpoint", endpoint, "-api-key", "test", "-call-timeout", "5s"}
// Positive control: the lenient default accepts the self-signed certificate.
var okOut, okErr bytes.Buffer
if err := runWithIO(ctx, base, &okOut, &okErr); err != nil {
t.Fatalf("default TLS open-session error = %v; stderr = %s", err, okErr.String())
}
if got := strings.TrimSpace(okOut.String()); got != "tls-session" {
t.Fatalf("default TLS open-session output = %q", got)
}
// Strict: the handshake never verifies, so the connected-dial wait runs out.
// The dial error (not a flag-parse error) is what proves the flag reached the
// TLS configuration; the positive control above proves the endpoint is live.
strictCtx, strictCancel := context.WithTimeout(t.Context(), 2*time.Second)
defer strictCancel()
var strictOut, strictErr bytes.Buffer
err := runWithIO(strictCtx, append(append([]string{}, base...), "-require-certificate-validation"), &strictOut, &strictErr)
if err == nil {
t.Fatalf("strict TLS open-session succeeded against a self-signed gateway; stdout = %s", strictOut.String())
}
if !strings.Contains(err.Error(), "dial failed") {
t.Fatalf("strict TLS open-session error = %v; want a dial failure (stderr = %s)", err, strictErr.String())
}
}
File diff suppressed because it is too large Load Diff
+7 -2
View File
@@ -10,6 +10,11 @@ import (
// invoke:alarm-ack scope and forwards the acknowledge to the worker's MXAccess
// session; the resulting native MxStatus is returned in the reply.
//
// OperatorUser is required: the gateway authenticates the operator (with
// OperatorPassword and optional OperatorDomain) so the Historian records the
// real operator. An empty password is allowed. The password is scrubbed from
// any error this method returns.
//
// Acks are idempotent — re-acking an already-acked condition is a no-op at
// the MxAccess layer.
func (c *Client) AcknowledgeAlarm(ctx context.Context, req *AcknowledgeAlarmRequest) (*AcknowledgeAlarmReply, error) {
@@ -22,10 +27,10 @@ func (c *Client) AcknowledgeAlarm(ctx context.Context, req *AcknowledgeAlarmRequ
reply, err := c.raw.AcknowledgeAlarm(callCtx, req)
if err != nil {
return nil, &GatewayError{Op: "acknowledge alarm", Err: err}
return nil, redactSecrets(&GatewayError{Op: "acknowledge alarm", Err: err}, req.GetOperatorPassword())
}
if err := EnsureProtocolSuccess("acknowledge alarm", reply.GetProtocolStatus(), nil); err != nil {
return reply, err
return reply, redactSecrets(err, req.GetOperatorPassword())
}
return reply, nil
+29
View File
@@ -5,6 +5,7 @@ import (
"errors"
"io"
"net"
"strings"
"testing"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
@@ -38,6 +39,8 @@ func TestAcknowledgeAlarmSendsRequestAndReturnsReply(t *testing.T) {
AlarmFullReference: "Tank01.Level.HiHi",
Comment: "investigating",
OperatorUser: "alice",
OperatorPassword: "pw-secret",
OperatorDomain: "CORP",
})
if err != nil {
t.Fatalf("AcknowledgeAlarm() error = %v", err)
@@ -51,11 +54,37 @@ func TestAcknowledgeAlarmSendsRequestAndReturnsReply(t *testing.T) {
if got := fake.acknowledgeRequest.GetComment(); got != "investigating" {
t.Fatalf("captured comment = %q", got)
}
if got := fake.acknowledgeRequest.GetOperatorPassword(); got != "pw-secret" {
t.Fatalf("captured operator password mismatch")
}
if got := fake.acknowledgeRequest.GetOperatorDomain(); got != "CORP" {
t.Fatalf("captured operator domain = %q", got)
}
if got := fake.acknowledgeAuth; got != "Bearer test-api-key" {
t.Fatalf("authorization metadata = %q", got)
}
}
func TestAcknowledgeAlarmRedactsPasswordFromError(t *testing.T) {
fake := &fakeGatewayWithAlarms{
acknowledgeError: status.Error(codes.Internal, "boom pw-secret"),
}
client, cleanup := newBufconnClientWithAlarms(t, fake)
defer cleanup()
_, err := client.AcknowledgeAlarm(context.Background(), &pb.AcknowledgeAlarmRequest{
AlarmFullReference: "Tank01.Level.HiHi",
OperatorUser: "alice",
OperatorPassword: "pw-secret",
})
if err == nil {
t.Fatal("AcknowledgeAlarm() returned no error")
}
if strings.Contains(err.Error(), "pw-secret") {
t.Fatalf("error leaks operator password: %v", err)
}
}
func TestAcknowledgeAlarmRejectsNilRequest(t *testing.T) {
fake := &fakeGatewayWithAlarms{}
client, cleanup := newBufconnClientWithAlarms(t, fake)
+34 -2
View File
@@ -15,10 +15,12 @@ import (
"context"
"crypto/tls"
"errors"
"strings"
"time"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc"
"google.golang.org/grpc/connectivity"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/types/known/durationpb"
@@ -61,11 +63,10 @@ func Dial(ctx context.Context, opts Options) (*Client, error) {
grpc.WithTransportCredentials(transportCredentials),
grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
grpc.WithBlock(),
}
dialOptions = append(dialOptions, opts.DialOptions...)
conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
if err != nil {
return nil, &GatewayError{Op: "dial", Err: err}
}
@@ -265,3 +266,34 @@ func (o OpenSessionOptions) Request() *OpenSessionRequest {
}
return req
}
// dialConnected creates a connection with grpc.NewClient and then waits for it
// to become READY, bounded by ctx. This preserves the connect-or-fail contract
// the removed grpc.DialContext + grpc.WithBlock combination gave Dial callers.
// The endpoint is resolved with the passthrough scheme unless it already names a
// scheme, matching DialContext's historical default (NewClient defaults to dns).
func dialConnected(ctx context.Context, endpoint string, dialOptions []grpc.DialOption) (*grpc.ClientConn, error) {
target := endpoint
if !strings.Contains(target, "://") {
target = "passthrough:///" + target
}
conn, err := grpc.NewClient(target, dialOptions...)
if err != nil {
return nil, err
}
conn.Connect()
for {
state := conn.GetState()
if state == connectivity.Ready {
return conn, nil
}
if state == connectivity.Shutdown {
_ = conn.Close()
return nil, errors.New("connection shut down while connecting")
}
if !conn.WaitForStateChange(ctx, state) {
_ = conn.Close()
return nil, ctx.Err()
}
}
}
+241 -4
View File
@@ -121,7 +121,7 @@ func TestEventsAfterCancelsStreamWhenCompatibilityChannelIsAbandoned(t *testing.
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -155,7 +155,7 @@ func TestEventsSlowConsumerYieldsErrSlowConsumerBeforeClose(t *testing.T) {
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -209,7 +209,7 @@ func TestEventsFullBufferTerminalErrorKeepsRootCause(t *testing.T) {
streamEventCount: eventBufferSize,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -280,7 +280,7 @@ func TestSubscribeEventsFullBufferDeliversTerminalError(t *testing.T) {
streamEventCount: eventBufferSize + eventBufferReservedSlots,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -1047,3 +1047,240 @@ func TestWriteArrayElementsSendsWriteCommandWithSparseOneof(t *testing.T) {
t.Errorf("element float value = %v, want 1.5", elem.GetValue().GetFloatValue())
}
}
// newBufconnClientWithOptions mirrors newBufconnClient but lets a test tune Options.
func newBufconnClientWithOptions(t *testing.T, fake *fakeGatewayServer, opts Options) (*Client, func()) {
t.Helper()
listener := bufconn.Listen(bufSize)
server := grpc.NewServer()
pb.RegisterMxAccessGatewayServer(server, fake)
go func() { _ = server.Serve(listener) }()
opts.Endpoint = "bufnet"
opts.APIKey = "test-api-key"
opts.Plaintext = true
opts.DialOptions = append(opts.DialOptions, grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
return listener.DialContext(ctx)
}))
client, err := Dial(context.Background(), opts)
if err != nil {
t.Fatalf("Dial() error = %v", err)
}
return client, func() {
client.Close()
server.Stop()
listener.Close()
}
}
func TestEventsHonorsConfiguredEventBufferSize(t *testing.T) {
// 100 events exceed the old fixed 16-slot buffer; a 256-slot buffer must
// hold them all without a slow-consumer termination.
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: 100,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 256})
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.Events(context.Background())
if err != nil {
t.Fatalf("Events() error = %v", err)
}
<-fake.streamDone
time.Sleep(250 * time.Millisecond)
count := 0
var last EventResult
for res := range events {
if res.Event != nil {
count++
}
last = res
}
if count != 100 {
t.Fatalf("delivered events = %d, want 100", count)
}
if errors.Is(last.Err, ErrSlowConsumer) || status.Code(last.Err) != codes.Internal {
t.Fatalf("terminal err = %v, want the genuine Internal error", last.Err)
}
}
func TestEventsOverflowDeliversBufferedEventsBeforeSlowConsumerError(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 4})
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.Events(context.Background())
if err != nil {
t.Fatalf("Events() error = %v", err)
}
<-fake.streamStarted
time.Sleep(250 * time.Millisecond)
buffered := 0
var last EventResult
for res := range events {
if res.Event != nil {
buffered++
if got := res.Event.GetWorkerSequence(); got != uint64(buffered) {
t.Fatalf("event %d has sequence %d, want in-order", buffered, got)
}
}
last = res
}
if buffered != 4 {
t.Fatalf("buffered events delivered = %d, want 4 (all data slots)", buffered)
}
if !errors.Is(last.Err, ErrSlowConsumer) {
t.Fatalf("final err = %v, want ErrSlowConsumer", last.Err)
}
}
func TestEventBufferSizeDefault(t *testing.T) {
if got := (Options{}).eventBufferSize(); got != DefaultEventBufferSize {
t.Fatalf("default = %d, want %d", got, DefaultEventBufferSize)
}
if got := (Options{EventBufferSize: 7}).eventBufferSize(); got != 7 {
t.Fatalf("explicit = %d, want 7", got)
}
}
func TestDialFailsWhenEndpointUnreachable(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_, err := Dial(ctx, Options{
Endpoint: "bufnet",
Plaintext: true,
DialTimeout: 300 * time.Millisecond,
DialOptions: []grpc.DialOption{grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
return nil, errors.New("refused")
})},
})
if err == nil {
t.Fatal("Dial() error = nil, want connect failure")
}
var gwErr *GatewayError
if !errors.As(err, &gwErr) || gwErr.Op != "dial" {
t.Fatalf("Dial() error = %v, want *GatewayError Op=dial", err)
}
}
func okStatus() *pb.ProtocolStatus {
return &pb.ProtocolStatus{Code: pb.ProtocolStatusCode_PROTOCOL_STATUS_CODE_OK}
}
func TestSessionPingReturnsEchoedMessage(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_PING,
ProtocolStatus: okStatus(),
DiagnosticMessage: "pong:hi",
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
got, err := session.Ping(context.Background(), "hi")
if err != nil {
t.Fatalf("Ping() error = %v", err)
}
if got != "pong:hi" {
t.Fatalf("Ping() = %q, want pong:hi", got)
}
if fake.invokeRequest.GetCommand().GetPing().GetMessage() != "hi" {
t.Fatalf("ping payload not sent: %v", fake.invokeRequest)
}
}
func TestSessionGetSessionState(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_SessionState{SessionState: &pb.SessionStateReply{
State: pb.SessionState_SESSION_STATE_CREATING,
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
got, err := session.GetSessionState(context.Background())
if err != nil {
t.Fatalf("GetSessionState() error = %v", err)
}
if got != pb.SessionState_SESSION_STATE_CREATING {
t.Fatalf("state = %v", got)
}
if fake.invokeRequest.GetCommand().GetKind() != pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE {
t.Fatalf("kind = %v", fake.invokeRequest.GetCommand().GetKind())
}
}
func TestSessionGetWorkerInfo(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_WorkerInfo{WorkerInfo: &pb.WorkerInfoReply{
WorkerProcessId: 42, WorkerVersion: "1.2.3", MxaccessProgid: "prog", MxaccessClsid: "clsid",
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
info, err := session.GetWorkerInfo(context.Background())
if err != nil {
t.Fatalf("GetWorkerInfo() error = %v", err)
}
if info.GetWorkerProcessId() != 42 || info.GetWorkerVersion() != "1.2.3" {
t.Fatalf("info = %v", info)
}
}
func TestSessionDrainEvents(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_DrainEvents{DrainEvents: &pb.DrainEventsReply{
Events: []*pb.MxEvent{{WorkerSequence: 1}, {WorkerSequence: 2}},
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.DrainEvents(context.Background(), 10)
if err != nil {
t.Fatalf("DrainEvents() error = %v", err)
}
if len(events) != 2 || events[1].GetWorkerSequence() != 2 {
t.Fatalf("events = %v", events)
}
if fake.invokeRequest.GetCommand().GetDrainEvents().GetMaxEvents() != 10 {
t.Fatalf("max_events not sent: %v", fake.invokeRequest)
}
}
func TestSessionDiagnosticsErrorOnMissingPayload(t *testing.T) {
fake := &fakeGatewayServer{}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
if _, err := session.GetWorkerInfo(context.Background()); err == nil {
t.Fatal("GetWorkerInfo() with empty reply: want error")
}
if _, err := session.GetSessionState(context.Background()); err == nil {
t.Fatal("GetSessionState() with empty reply: want error")
}
if _, err := session.DrainEvents(context.Background(), 1); err == nil {
t.Fatal("DrainEvents() with empty reply: want error")
}
}
+42 -1
View File
@@ -6,6 +6,8 @@ import (
"strings"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
)
@@ -180,7 +182,20 @@ func (e *MalformedReplyError) Error() string {
return fmt.Sprintf("mxgateway: %s returned a malformed reply: %s", e.Op, e.Detail)
}
// GatewayError wraps transport-level gRPC failures.
// ErrUnauthenticated matches, via errors.Is, a GatewayError whose gRPC status is
// Unauthenticated: the gateway did not accept the API key (missing, malformed,
// unknown, or revoked). Retrying with the same key will not help.
var ErrUnauthenticated = errors.New("mxgateway: unauthenticated")
// ErrPermissionDenied matches, via errors.Is, a GatewayError whose gRPC status
// is PermissionDenied: the API key is valid but lacks the scope the RPC needs, or
// the session belongs to another key. It is distinct from ErrUnauthenticated so a
// caller can tell "fix the key" from "grant the key a scope".
var ErrPermissionDenied = errors.New("mxgateway: permission denied")
// GatewayError wraps transport-level gRPC failures. errors.Is(err,
// ErrUnauthenticated) and errors.Is(err, ErrPermissionDenied) classify the two
// authorization failures; Code returns the raw gRPC status code.
type GatewayError struct {
// Op names the operation that failed (for example "dial" or "invoke").
Op string
@@ -207,6 +222,32 @@ func (e *GatewayError) Unwrap() error {
return e.Err
}
// Code returns the gRPC status code of the wrapped error, codes.Unknown for a
// non-gRPC error, and codes.OK for a nil receiver or nil wrapped error.
func (e *GatewayError) Code() codes.Code {
if e == nil || e.Err == nil {
return codes.OK
}
return status.Code(e.Err)
}
// Is reports whether target is the sentinel matching this error's gRPC status
// code (ErrUnauthenticated or ErrPermissionDenied). The check reads only the
// status code, never the message, so credential redaction is unaffected.
func (e *GatewayError) Is(target error) bool {
if e == nil {
return false
}
switch target {
case ErrUnauthenticated:
return e.Code() == codes.Unauthenticated
case ErrPermissionDenied:
return e.Code() == codes.PermissionDenied
default:
return false
}
}
// CommandError reports a non-OK gateway protocol status and keeps the raw
// command reply when one exists.
type CommandError struct {
+85
View File
@@ -0,0 +1,85 @@
package mxgateway
import (
"context"
"errors"
"fmt"
"strings"
"testing"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// #198 — the gateway answers a bad key with Unauthenticated and a missing scope
// with PermissionDenied; callers must be able to tell them apart with errors.Is.
func TestGatewayErrorMatchesAuthSentinelsByStatusCode(t *testing.T) {
cases := []struct {
name string
err error
unauthenticated bool
permissionDenied bool
}{
{"unauthenticated", status.Error(codes.Unauthenticated, "bad key"), true, false},
{"permission denied", status.Error(codes.PermissionDenied, "scope required"), false, true},
{"unavailable", status.Error(codes.Unavailable, "down"), false, false},
{"non-grpc", errors.New("plain"), false, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := fmt.Errorf("outer: %w", &GatewayError{Op: "invoke", Err: tc.err})
if got := errors.Is(err, ErrUnauthenticated); got != tc.unauthenticated {
t.Fatalf("errors.Is(ErrUnauthenticated) = %v, want %v", got, tc.unauthenticated)
}
if got := errors.Is(err, ErrPermissionDenied); got != tc.permissionDenied {
t.Fatalf("errors.Is(ErrPermissionDenied) = %v, want %v", got, tc.permissionDenied)
}
})
}
}
func TestGatewayErrorCode(t *testing.T) {
if got := (&GatewayError{Err: status.Error(codes.PermissionDenied, "x")}).Code(); got != codes.PermissionDenied {
t.Fatalf("Code() = %v", got)
}
if got := (*GatewayError)(nil).Code(); got != codes.OK {
t.Fatalf("nil Code() = %v", got)
}
}
func TestAuthSentinelsSurviveTheWireAndRedaction(t *testing.T) {
for _, tc := range []struct {
code codes.Code
sentinel error
other error
}{
{codes.Unauthenticated, ErrUnauthenticated, ErrPermissionDenied},
{codes.PermissionDenied, ErrPermissionDenied, ErrUnauthenticated},
} {
t.Run(tc.code.String(), func(t *testing.T) {
fake := &fakeGatewayWithAlarms{acknowledgeError: status.Error(tc.code, "denied pw-secret")}
client, cleanup := newBufconnClientWithAlarms(t, fake)
defer cleanup()
_, err := client.AcknowledgeAlarm(context.Background(), &pb.AcknowledgeAlarmRequest{
AlarmFullReference: "Tank01.Level.HiHi",
OperatorUser: "alice",
OperatorPassword: "pw-secret",
})
if err == nil {
t.Fatal("AcknowledgeAlarm() returned no error")
}
if !errors.Is(err, tc.sentinel) {
t.Fatalf("errors.Is(%v) = false for %v", tc.sentinel, err)
}
if errors.Is(err, tc.other) {
t.Fatalf("errors.Is(%v) = true for %v", tc.other, err)
}
if strings.Contains(err.Error(), "pw-secret") {
t.Fatalf("error leaked the operator password: %v", err)
}
})
}
}
+1 -2
View File
@@ -99,11 +99,10 @@ func DialGalaxy(ctx context.Context, opts Options) (*GalaxyClient, error) {
grpc.WithTransportCredentials(transportCredentials),
grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
grpc.WithBlock(),
}
dialOptions = append(dialOptions, opts.DialOptions...)
conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
if err != nil {
return nil, &GatewayError{Op: "dial", Err: err}
}
+15
View File
@@ -38,6 +38,21 @@ type Options struct {
// no CACertFile is pinned. Default false: the gateway's self-signed cert is
// accepted without verification (internal-tool posture).
RequireCertificateValidation bool
// EventBufferSize is the number of data results the channel returned by
// Session.Events/EventsAfter buffers before a slow consumer is cancelled with
// ErrSlowConsumer. Zero or negative applies DefaultEventBufferSize. It does not
// affect SubscribeEvents, which blocks instead of dropping.
EventBufferSize int
}
// DefaultEventBufferSize is the default Options.EventBufferSize.
const DefaultEventBufferSize = 1024
func (o Options) eventBufferSize() int {
if o.EventBufferSize <= 0 {
return DefaultEventBufferSize
}
return o.EventBufferSize
}
// BrowseChildrenOptions configures lazy Galaxy hierarchy walks performed by
+76 -8
View File
@@ -18,8 +18,9 @@ import (
const maxBulkItems = 1000
// eventBufferSize is the number of buffered event slots on the
// Events/EventsAfter (cancel-when-full) results channel.
// eventBufferSize is the data-slot count the unit tests use for the legacy
// fixed-size assertions; the runtime value comes from Options.EventBufferSize
// (default DefaultEventBufferSize).
const eventBufferSize = 16
// eventBufferReservedSlots is the extra capacity reserved beyond
@@ -548,8 +549,9 @@ func (s *Session) WriteSecured2Bulk(ctx context.Context, serverHandle int32, ent
// MXAccess COM has no synchronous Read; the worker satisfies this by returning the
// most recent cached OnDataChange value when the tag is already advised (WasCached=true),
// or by taking a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
// otherwise. timeout bounds the wait per tag in the snapshot case; pass zero to use the
// worker default. Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
// otherwise. timeout is one shared deadline for the whole command, not a per-tag budget:
// the worker advises every uncached tag, waits once for all of them, then tears them down.
// Pass zero to use the worker default (1000 ms). Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
// with WasSuccessful=false; the call never returns an error for per-tag MXAccess failures.
//
// A non-nil but empty tagAddresses slice is treated as a no-op and returns an empty
@@ -666,6 +668,68 @@ func buildSparseArrayValue(elementDataType MxDataType, totalLength uint32, eleme
}
}
// Ping sends a diagnostic PING command and returns the message the gateway
// echoed back in the reply's DiagnosticMessage field.
func (s *Session) Ping(ctx context.Context, message string) (string, error) {
reply, err := s.PingRaw(ctx, message)
if err != nil {
return "", err
}
return reply.GetDiagnosticMessage(), nil
}
// GetSessionState returns the gateway's current lifecycle state for the session.
func (s *Session) GetSessionState(ctx context.Context) (SessionState, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
Payload: &pb.MxCommand_GetSessionState{GetSessionState: &pb.GetSessionStateCommand{}},
})
if err != nil {
return 0, err
}
payload := reply.GetSessionState()
if payload == nil {
return 0, errors.New("mxgateway: get session state reply carried no session_state payload")
}
return payload.GetState(), nil
}
// GetWorkerInfo returns diagnostic information about the session's worker
// process (process id, version, MXAccess ProgID and CLSID).
func (s *Session) GetWorkerInfo(ctx context.Context) (*WorkerInfoReply, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
Payload: &pb.MxCommand_GetWorkerInfo{GetWorkerInfo: &pb.GetWorkerInfoCommand{}},
})
if err != nil {
return nil, err
}
payload := reply.GetWorkerInfo()
if payload == nil {
return nil, errors.New("mxgateway: get worker info reply carried no worker_info payload")
}
return payload, nil
}
// DrainEvents removes and returns up to maxEvents queued events through the
// diagnostic DRAIN_EVENTS command. The reply is bounded server-side, so it can
// carry fewer events than requested or queued; call repeatedly until it returns
// an empty slice. It is a diagnostic tool, not a replacement for Events.
func (s *Session) DrainEvents(ctx context.Context, maxEvents uint32) ([]*MxEvent, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
Payload: &pb.MxCommand_DrainEvents{DrainEvents: &pb.DrainEventsCommand{MaxEvents: maxEvents}},
})
if err != nil {
return nil, err
}
payload := reply.GetDrainEvents()
if payload == nil {
return nil, errors.New("mxgateway: drain events reply carried no drain_events payload")
}
return payload.GetEvents(), nil
}
// PingRaw sends a diagnostic PING command and returns the raw reply.
// The message is echoed back by the gateway in the reply's DiagnosticMessage field.
func (s *Session) PingRaw(ctx context.Context, message string) (*MxCommandReply, error) {
@@ -1001,7 +1065,9 @@ func stringSecrets(values ...*MxValue) []string {
// The returned channel is buffered. If the consumer falls behind and the buffer
// overflows with data, the stream is terminated and a final EventResult carrying
// a GatewayError that wraps ErrSlowConsumer is delivered before the channel
// closes; match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
// closes. Results already buffered are delivered first, in order, ahead of that
// terminal result. The buffer holds Options.EventBufferSize data results
// (default DefaultEventBufferSize); match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
// slow-consumer drop from a graceful server end. A genuine stream error is
// reported as itself even under overflow — it is never relabeled as
// ErrSlowConsumer, so the underlying gRPC status stays inspectable. Use
@@ -1046,7 +1112,8 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
return nil, err
}
results := make(chan EventResult, eventBufferSize+eventBufferReservedSlots)
dataSlots := s.client.opts.eventBufferSize()
results := make(chan EventResult, dataSlots+eventBufferReservedSlots)
done := make(chan struct{})
go func() {
defer close(results)
@@ -1062,7 +1129,7 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
if gap := event.GetReplayGap(); gap != nil {
result = EventResult{ReplayGap: gap}
}
if !sendEventResult(streamCtx, results, result, cancelWhenResultBufferFull, cancel) {
if !sendEventResult(streamCtx, results, result, dataSlots, cancelWhenResultBufferFull, cancel) {
return
}
continue
@@ -1126,6 +1193,7 @@ func sendEventResult(
ctx context.Context,
results chan<- EventResult,
result EventResult,
dataSlots int,
cancelWhenBufferFull bool,
cancel context.CancelFunc,
) bool {
@@ -1136,7 +1204,7 @@ func sendEventResult(
// grows by our own sends and shrinks as the consumer reads; the reserve
// therefore always survives to carry ErrSlowConsumer. A plain buffered
// send would instead consume the reserved slot as ordinary data.
if len(results) >= eventBufferSize {
if len(results) >= dataSlots {
// The consumer fell behind and the data slots are full. Cancel the
// stream, then use the reserved terminal slot to enqueue a loud
// ErrSlowConsumer result so overflow is always observable rather
+9 -2
View File
@@ -28,6 +28,8 @@ type (
MxCommandReply = pb.MxCommandReply
// MxCommand is the discriminated union of MXAccess command payloads.
MxCommand = pb.MxCommand
// WorkerInfoReply is the GetWorkerInfo diagnostic reply payload.
WorkerInfoReply = pb.WorkerInfoReply
// MxEvent is one ordered event delivered on a session event stream.
MxEvent = pb.MxEvent
// ReplayGap is the gateway sentinel payload signalling that a resumed event
@@ -127,9 +129,14 @@ type (
QueryActiveAlarmsRequest = pb.QueryActiveAlarmsRequest
// StreamAlarmsRequest is the gateway StreamAlarms request message.
StreamAlarmsRequest = pb.StreamAlarmsRequest
// AlarmFeedMessage is one message on the StreamAlarms feed — an
// active-alarm snapshot row, a snapshot-complete sentinel, or a transition.
// AlarmFeedMessage is one message on the StreamAlarms feed — an active-alarm
// snapshot row, snapshot-complete sentinel, transition, provider status, or
// snapshot-completeness status.
AlarmFeedMessage = pb.AlarmFeedMessage
// AlarmProviderStatus reports the alarm provider mode and degradation state.
AlarmProviderStatus = pb.AlarmProviderStatus
// AlarmSnapshotStatus reports whether the cached alarm snapshot is truncated.
AlarmSnapshotStatus = pb.AlarmSnapshotStatus
// ActiveAlarmSnapshot is one row in a ConditionRefresh stream.
ActiveAlarmSnapshot = pb.ActiveAlarmSnapshot
// OnAlarmTransitionEvent is the body carried by alarm-transition MxEvents.
+18
View File
@@ -0,0 +1,18 @@
package mxgateway_test
import (
"testing"
"gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/mxgateway"
)
// TestAlarmFeedStatusAliasesArePublic is a compile-time consumer contract for
// the two status payloads carried by AlarmFeedMessage.
func TestAlarmFeedStatusAliasesArePublic(t *testing.T) {
t.Helper()
// Pointers, not values: protobuf messages embed a mutex and must not be copied.
var provider *mxgateway.AlarmProviderStatus
var snapshot *mxgateway.AlarmSnapshotStatus
_ = provider
_ = snapshot
}
+1 -1
View File
@@ -3,7 +3,7 @@ package mxgateway
const (
// ClientVersion is the released semantic version of this Go client module.
// Keep it in sync with the module tag applied by scripts/tag-go-module.ps1.
ClientVersion = "0.2.0"
ClientVersion = "0.2.1"
// GatewayProtocolVersion matches GatewayContractInfo.GatewayProtocolVersion
// in the shared .NET contracts.
+37 -5
View File
@@ -79,6 +79,30 @@ only stops the client from waiting; it does not abort an in-flight MXAccess COM
call on the worker STA. It is a **single-consumer** surface: drive
`hasNext()`/`next()` (or `nextItem()`) from one thread only.
### Event buffer, overflow, and interrupts
The iterator buffers events in a bounded queue whose size is
`MxGatewayClientOptions.builder().eventQueueCapacity(n)` (default 1024; it was a
hard-coded 16, which a short burst could overflow). If the consumer falls more
than `n` events behind, the client cancels the call, keeps the events already
buffered, and throws `MxGatewayException` only after you have drained them. Resume
with `streamEventsAfter(lastSeenWorkerSequence)`. If the consuming thread is
interrupted, the call is cancelled, the interrupt flag is restored, and the
iterator throws a `CANCELLED` `StatusRuntimeException`.
`streamEventsAsync(request, StreamObserver<MxEvent>)` is the non-dropping
alternative: it delivers each event to your observer on the gRPC thread with no
client-side queue, so nothing is discarded by the client (a slow observer
instead applies normal gRPC flow control).
### Session diagnostics
`MxGatewaySession` exposes typed helpers for the worker diagnostic commands:
`ping(message)` (returns the raw `MxCommandReply`), `getSessionState()`
(`SessionState`), `getWorkerInfo()` (`WorkerInfoReply`: process id, version,
ProgID, CLSID) and `drainEvents(maxEvents)` (`List<MxEvent>`; replies are
bounded server-side, so loop until an empty list).
### Reconnect-replay gap signal
When you resume a stream with `streamEventsAfter(afterWorkerSequence)` and the
@@ -114,8 +138,11 @@ try (MxEventStream events = session.streamEventsAfter(lastSeenSequence)) {
For alarms, `MxGatewayClient` exposes `queryActiveAlarms` (one-shot snapshot),
`streamAlarms` (returns an `MxGatewayAlarmFeedSubscription` whose iterator
yields alarm-feed messages from the gateway's central monitor), and
`acknowledgeAlarm` (ack by full alarm reference with an optional comment and
ack target). Close the subscription to cancel the underlying gRPC stream.
`acknowledgeAlarm` (ack by full alarm reference with an optional comment).
The ack carries operator credentials: `operator_user` is required, and
`operator_password` / `operator_domain` let the gateway authenticate as that
operator so the Historian records the real operator. An empty password is
allowed. The password is scrubbed from exceptions the client throws. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. Close the subscription to cancel the underlying gRPC stream.
`ActiveAlarmSnapshot.getFromTruncatedSnapshot()` reports that the record came
from a provider fetch which hit the per-fetch cap: the snapshot set may omit
@@ -135,6 +162,11 @@ the clearing frame sent when the gateway's alarm monitor restarts and drops a
truncated verdict. Track it if you need set completeness on a live feed without
polling `queryActiveAlarms`.
The CLI's text rows always spell out `degraded=false` and `truncated=false`.
Its protobuf-JSON mode retains the standard proto3 omission rule, so those
false-valued payloads appear as `{"providerStatus":{}}` and
`{"snapshotStatus":{}}`; absent means false.
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
@@ -411,8 +443,8 @@ gradle :zb-mom-ww-mxgateway-cli:run --args="advise-supervisory --endpoint localh
gradle :zb-mom-ww-mxgateway-cli:run --args="authenticate-user --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --server-handle 1 --verify-user operator --password-env MXGATEWAY_VERIFY_PASSWORD --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="write-secured --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --current-user-id 100 --verifier-user-id 100 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-events --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --limit 1 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-alarms --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --filter-prefix Galaxy --limit 1 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="acknowledge-alarm --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --reference \"\\Galaxy\Area001.Pump001.PumpFault\" --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-alarms --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --filter-prefix Galaxy --limit 4 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="acknowledge-alarm --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --reference \"\\Galaxy\Area001.Pump001.PumpFault\" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="smoke --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --item TestObject.TestInt --json"
```
@@ -483,7 +515,7 @@ repositories {
}
dependencies {
implementation 'com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client:0.2.1'
implementation 'com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client:0.2.2'
}
````
+7 -4
View File
@@ -16,11 +16,14 @@ subprojects {
// 0.2.0 was already published to the Gitea Maven feed on 2026-06-26,
// before the CLI-37/38/40/41 conformance fixes changed the client's
// observable behavior (status.category-based validation, hresult < 0,
// exact-secret redaction, typed malformed-reply errors). Bump to 0.2.1
// so the published coordinate matches the conformant behavior the other
// four clients ship at 0.2.0 for the first time. See CLI-39 and the
// exact-secret redaction, typed malformed-reply errors). Bumped to 0.2.1
// so the published coordinate matched the conformant behavior the other
// four clients shipped at 0.2.0 for the first time. See CLI-39 and the
// "Versioning" section of docs/ClientPackaging.md.
version = '0.2.1'
// 0.2.1 was itself published (Maven, 2026-08-07); this is the next
// release-train bump (CLI-46), one ahead of the other four clients'
// 0.2.1 because the Java coordinate has always trailed by one number.
version = '0.2.2'
pluginManager.withPlugin('java') {
java {
File diff suppressed because it is too large Load Diff
@@ -216,6 +216,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
*/
WORKER_FAULT_CATEGORY_UNSPECIFIED(0),
/**
* <pre>
* Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
* arguments, gateway authentication) exit via WorkerExitCode before the pipe
* exists, so there is no channel to carry a fault. Names kept so the numbers
* are not reused.
* </pre>
*
* <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
*/
WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS(1),
@@ -276,6 +283,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
*/
public static final int WORKER_FAULT_CATEGORY_UNSPECIFIED_VALUE = 0;
/**
* <pre>
* Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
* arguments, gateway authentication) exit via WorkerExitCode before the pipe
* exists, so there is no channel to carry a fault. Names kept so the numbers
* are not reused.
* </pre>
*
* <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
*/
public static final int WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS_VALUE = 1;
@@ -1262,7 +1262,7 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(
names = "--timeout-ms",
defaultValue = "0",
description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
int timeoutMs;
ReadBulkCommand(MxGatewayCliClientFactory clientFactory) {
@@ -1510,7 +1510,7 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(
names = "--timeout-ms",
defaultValue = "0",
description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
int timeoutMs;
@Option(names = "--iterations", defaultValue = "10", description = "Number of ReadBulk calls to perform.")
@@ -1822,6 +1822,31 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(names = "--operator", description = "Operator user performing the acknowledge.")
String operator = "";
@Option(names = "--operator-domain", description = "Operator Galaxy domain (optional).")
String operatorDomain = "";
@Option(
names = "--operator-password",
description = "Operator Galaxy password (credential; prefer --operator-password-env). Never echoed.")
String operatorPassword;
@Option(
names = "--operator-password-env",
defaultValue = "MXGATEWAY_OPERATOR_PASSWORD",
description = "Environment variable holding the operator password when --operator-password is omitted.")
String operatorPasswordEnv;
private String resolveOperatorPassword() {
if (operatorPassword != null && !operatorPassword.isEmpty()) {
return operatorPassword;
}
String name = operatorPasswordEnv == null || operatorPasswordEnv.isBlank()
? "MXGATEWAY_OPERATOR_PASSWORD"
: operatorPasswordEnv;
String value = System.getenv(name);
return value == null ? "" : value;
}
AcknowledgeAlarmCommand(MxGatewayCliClientFactory clientFactory) {
super(clientFactory);
}
@@ -1833,6 +1858,8 @@ public final class MxGatewayCli implements Callable<Integer> {
.setAlarmFullReference(reference)
.setComment(comment)
.setOperatorUser(operator)
.setOperatorDomain(operatorDomain == null ? "" : operatorDomain)
.setOperatorPassword(resolveOperatorPassword())
.build());
writeOutput(
"acknowledge-alarm",
@@ -27,6 +27,8 @@ import mxaccess_gateway.v1.MxaccessGateway.ActiveAlarmSnapshot;
import mxaccess_gateway.v1.MxaccessGateway.AddItemReply;
import mxaccess_gateway.v1.MxaccessGateway.AlarmConditionState;
import mxaccess_gateway.v1.MxaccessGateway.AlarmFeedMessage;
import mxaccess_gateway.v1.MxaccessGateway.AlarmProviderStatus;
import mxaccess_gateway.v1.MxaccessGateway.AlarmSnapshotStatus;
import mxaccess_gateway.v1.MxaccessGateway.AlarmTransitionKind;
import mxaccess_gateway.v1.MxaccessGateway.BulkReadResult;
import mxaccess_gateway.v1.MxaccessGateway.BulkWriteResult;
@@ -60,7 +62,7 @@ final class MxGatewayCliTests {
assertEquals(0, run.exitCode());
assertEquals("", run.errors());
assertTrue(run.output().contains("mxgateway-java 0.2.1"));
assertTrue(run.output().contains("mxgateway-java 0.2.2"));
assertTrue(run.output().contains("gatewayProtocolVersion=3"));
assertTrue(run.output().contains("workerProtocolVersion=1"));
}
@@ -90,7 +92,7 @@ final class MxGatewayCliTests {
CliRun run = execute(new FakeClientFactory(), "version", "--json");
assertEquals(0, run.exitCode());
assertTrue(run.output().contains("\"clientVersion\":\"0.2.1\""));
assertTrue(run.output().contains("\"clientVersion\":\"0.2.2\""));
assertTrue(run.output().contains("\"gatewayProtocolVersion\":3"));
}
@@ -685,6 +687,21 @@ final class MxGatewayCliTests {
assertTrue(run.output().contains("\"snapshotComplete\""), run.output());
}
@Test
void streamAlarmsCommandPrintsStatusArmsInTextAndProtobufJson() {
FakeClientFactory factory = new FakeClientFactory();
CliRun text = execute(factory, "stream-alarms");
List<String> textRows = text.output().lines().filter(line -> !line.isBlank()).toList();
assertEquals("provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=", textRows.get(3));
assertEquals("snapshot-status truncated=false", textRows.get(4));
CliRun json = execute(factory, "stream-alarms", "--json");
List<String> jsonRows = json.output().lines().filter(line -> !line.isBlank()).toList();
assertEquals("{\"providerStatus\":{}}", jsonRows.get(3));
assertEquals("{\"snapshotStatus\":{}}", jsonRows.get(4));
}
@Test
void acknowledgeAlarmCommandForwardsOptionsAndPrintsReply() {
FakeClientFactory factory = new FakeClientFactory();
@@ -697,9 +714,16 @@ final class MxGatewayCliTests {
"checked",
"--operator",
"operator1",
"--operator-domain",
"CORP",
"--operator-password",
"pw-secret",
"--json");
assertEquals(0, run.exitCode());
assertEquals("CORP", factory.client.lastAcknowledgeAlarmRequest.getOperatorDomain());
assertEquals("pw-secret", factory.client.lastAcknowledgeAlarmRequest.getOperatorPassword());
assertFalse(run.output().contains("pw-secret"), run.output());
assertEquals("Tank01.Level.HiHi", factory.client.lastAcknowledgeAlarmRequest.getAlarmFullReference());
assertEquals("checked", factory.client.lastAcknowledgeAlarmRequest.getComment());
assertEquals("operator1", factory.client.lastAcknowledgeAlarmRequest.getOperatorUser());
@@ -753,6 +777,10 @@ final class MxGatewayCliTests {
"--plaintext",
"--reference",
"\\Galaxy\\Area001.Pump001.PumpFault",
"--operator",
"operator1",
"--operator-password-env",
"MXGATEWAY_OPERATOR_PASSWORD",
"--json"
};
assertReadmeExampleParses(args);
@@ -1409,6 +1437,12 @@ final class MxGatewayCliTests {
.setTransitionKind(AlarmTransitionKind.ALARM_TRANSITION_KIND_ACKNOWLEDGE)
.setSeverity(700))
.build());
observer.onNext(AlarmFeedMessage.newBuilder()
.setProviderStatus(AlarmProviderStatus.newBuilder().setDegraded(false))
.build());
observer.onNext(AlarmFeedMessage.newBuilder()
.setSnapshotStatus(AlarmSnapshotStatus.newBuilder().setTruncated(false))
.build());
observer.onCompleted();
return new MxGatewayAlarmFeedSubscription();
}
@@ -19,8 +19,12 @@ import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
* <p>Events arrive on a background gRPC thread and are buffered in a bounded
* blocking queue; the iterator drains them on the calling thread. Closing the
* stream cancels the underlying gRPC call. If the queue overflows the call is
* cancelled and a follow-up call to {@link #next()} throws
* {@link MxGatewayException}.
* cancelled, the events already buffered remain readable, and the
* {@link MxGatewayException} is thrown only after they are drained (the consumer
* can then resume with {@code after_worker_sequence}). The capacity comes from
* {@link MxGatewayClientOptions#eventQueueCapacity()}. Callers that must never be
* dropped should use {@link MxGatewayClient#streamEventsAsync}, which delivers on
* the gRPC thread with the caller's own flow control.
*
* <p><strong>Single consumer.</strong> This stream is not safe to drain from
* more than one thread. Interleave {@link #hasNext()}/{@link #next()} (or
@@ -46,10 +50,15 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
private final BlockingQueue<Object> queue;
private volatile ClientCallStreamObserver<StreamEventsRequest> requestStream;
private volatile boolean closed;
private final int capacity;
private boolean overflowed;
private Object next;
MxEventStream(int capacity) {
queue = new ArrayBlockingQueue<>(capacity);
this.capacity = capacity;
// Two extra slots are reserved so the overflow error and END sentinel
// can always be enqueued behind a full buffer of received events.
queue = new ArrayBlockingQueue<>(capacity + 2);
}
ClientResponseObserver<StreamEventsRequest, MxEvent> observer() {
@@ -143,12 +152,16 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
return queue.take();
} catch (InterruptedException error) {
Thread.currentThread().interrupt();
ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
if (stream != null) {
stream.cancel("interrupted while reading events", error);
}
return new StatusRuntimeException(Status.CANCELLED.withDescription("interrupted while reading events"));
}
}
}
private void offer(Object value) {
private synchronized void offer(Object value) {
Objects.requireNonNull(value, "value");
if (value == END) {
if (!queue.offer(value)) {
@@ -157,14 +170,29 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
}
return;
}
if (!queue.offer(value)) {
if (overflowed) {
return;
}
if (value instanceof MxEvent) {
if (queue.size() < capacity) {
queue.offer(value);
return;
}
// Keep the events already received so the consumer can drain them
// and resume from the last one via after_worker_sequence; the
// overflow error is delivered behind them.
overflowed = true;
ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
if (stream != null) {
stream.cancel("client event stream queue overflowed", null);
}
queue.clear();
queue.offer(new MxGatewayException("gateway stream events queue overflowed"));
queue.offer(END);
return;
}
if (!queue.offer(value)) {
queue.clear();
queue.offer(value);
}
}
}
@@ -245,7 +245,7 @@ public final class MxGatewayClient implements AutoCloseable {
* @return an iterator-style stream of events
*/
public MxEventStream streamEvents(StreamEventsRequest request) {
MxEventStream stream = new MxEventStream(16);
MxEventStream stream = new MxEventStream(options.eventQueueCapacity());
withStreamDeadline(rawAsyncStub()).streamEvents(request, stream.observer());
return stream;
}
@@ -273,6 +273,11 @@ public final class MxGatewayClient implements AutoCloseable {
* worker's MXAccess session; the resulting native MxStatus is returned
* in the reply. Acks are idempotent at the MxAccess layer.
*
* <p>{@code operator_user} is required; the gateway authenticates it with
* {@code operator_password} (and optional {@code operator_domain}) and the
* Historian records that operator. An empty password is allowed. The
* password is scrubbed from any exception this method throws.
*
* @param request the {@code AcknowledgeAlarmRequest}
* @return the raw acknowledge reply
* @throws MxGatewayException on transport or protocol failure
@@ -283,10 +288,14 @@ public final class MxGatewayClient implements AutoCloseable {
MxGatewayErrors.ensureProtocolSuccess("acknowledge alarm", reply.getProtocolStatus(), null);
return reply;
} catch (RuntimeException error) {
if (error instanceof MxGatewayException) {
throw error;
MxGatewayException mapped = error instanceof MxGatewayException gatewayError
? gatewayError
: (MxGatewayException) MxGatewayErrors.fromGrpc("acknowledge alarm", error);
String secret = request.getOperatorPassword();
if (secret.isEmpty()) {
throw mapped;
}
throw MxGatewayErrors.fromGrpc("acknowledge alarm", error);
throw MxGatewayErrors.redacted(mapped, secret);
}
}
@@ -299,10 +308,24 @@ public final class MxGatewayClient implements AutoCloseable {
*/
public CompletableFuture<AcknowledgeAlarmReply> acknowledgeAlarmAsync(AcknowledgeAlarmRequest request) {
CompletableFuture<AcknowledgeAlarmReply> future = toCompletable(rawFutureStub().acknowledgeAlarm(request));
return future.thenApply(reply -> {
String secret = request.getOperatorPassword();
CompletableFuture<AcknowledgeAlarmReply> checked = future.thenApply(reply -> {
MxGatewayErrors.ensureProtocolSuccess("acknowledge alarm", reply.getProtocolStatus(), null);
return reply;
});
if (secret.isEmpty()) {
return checked;
}
// Scrub the operator password from any failure surfaced through the future.
return checked.exceptionally(error -> {
Throwable cause = error instanceof java.util.concurrent.CompletionException && error.getCause() != null
? error.getCause()
: error;
if (cause instanceof MxGatewayException gatewayError) {
throw new java.util.concurrent.CompletionException(MxGatewayErrors.redacted(gatewayError, secret));
}
throw new java.util.concurrent.CompletionException(cause);
});
}
/**
@@ -15,6 +15,8 @@ public final class MxGatewayClientOptions {
private static final Duration DEFAULT_CONNECT_TIMEOUT = Duration.ofSeconds(10);
private static final Duration DEFAULT_CALL_TIMEOUT = Duration.ofSeconds(30);
private static final int DEFAULT_MAX_GRPC_MESSAGE_BYTES = 16 * 1024 * 1024;
/** Default {@code StreamEvents} iterator buffer; sized to absorb bursts without dropping a slow consumer. */
static final int DEFAULT_EVENT_QUEUE_CAPACITY = 1024;
private final String endpoint;
private final String apiKey;
@@ -26,6 +28,7 @@ public final class MxGatewayClientOptions {
private final Duration callTimeout;
private final Duration streamTimeout;
private final int maxGrpcMessageBytes;
private final int eventQueueCapacity;
private MxGatewayClientOptions(Builder builder) {
endpoint = requireText(builder.endpoint, "endpoint");
@@ -40,6 +43,10 @@ public final class MxGatewayClientOptions {
maxGrpcMessageBytes = builder.maxGrpcMessageBytes <= 0
? DEFAULT_MAX_GRPC_MESSAGE_BYTES
: builder.maxGrpcMessageBytes;
if (builder.eventQueueCapacity <= 0) {
throw new IllegalArgumentException("eventQueueCapacity must be positive");
}
eventQueueCapacity = builder.eventQueueCapacity;
}
/**
@@ -149,6 +156,17 @@ public final class MxGatewayClientOptions {
return maxGrpcMessageBytes;
}
/**
* Returns the capacity of the buffer behind {@link MxGatewayClient#streamEvents}.
* A consumer that falls more than this many events behind has its stream
* cancelled with an overflow error after the buffered events are drained.
*
* @return the event buffer capacity (default 1024)
*/
public int eventQueueCapacity() {
return eventQueueCapacity;
}
@Override
public String toString() {
return "MxGatewayClientOptions{"
@@ -199,6 +217,7 @@ public final class MxGatewayClientOptions {
private Duration callTimeout;
private Duration streamTimeout;
private int maxGrpcMessageBytes;
private int eventQueueCapacity = DEFAULT_EVENT_QUEUE_CAPACITY;
private Builder() {
}
@@ -314,6 +333,17 @@ public final class MxGatewayClientOptions {
return this;
}
/**
* Sets the capacity of the iterator-style event stream buffer. Defaults to 1024.
*
* @param value the buffer capacity, must be positive
* @return this builder
*/
public Builder eventQueueCapacity(int value) {
eventQueueCapacity = value;
return this;
}
/**
* Builds an immutable {@link MxGatewayClientOptions} from the current state.
*
@@ -9,7 +9,7 @@ package com.zb.mom.ww.mxgateway.client;
public final class MxGatewayClientVersion {
private static final int GATEWAY_PROTOCOL_VERSION = 3;
private static final int WORKER_PROTOCOL_VERSION = 1;
private static final String CLIENT_VERSION = "0.2.1";
private static final String CLIENT_VERSION = "0.2.2";
private MxGatewayClientVersion() {
}
@@ -28,6 +28,43 @@ final class MxGatewayErrors {
return new MxGatewayException("gateway " + operation + " failed: " + error.getMessage(), error);
}
/**
* Rebuilds a gateway exception with an exact secret scrubbed from its
* message and protocol status. The original is not chained as a cause
* because its message and cause chain may carry the secret.
*/
static MxGatewayException redacted(MxGatewayException error, String secret) {
String message = MxGatewaySecrets.redactExact(error.getMessage(), secret);
if (error instanceof MxAccessException ex) {
return new MxAccessException(message, scrub(ex.protocolStatus(), secret), ex.reply(), null);
}
if (error instanceof MxGatewayCommandException ex) {
return new MxGatewayCommandException(message, scrub(ex.protocolStatus(), secret), ex.reply(), null);
}
if (error instanceof MxGatewaySessionException ex) {
return new MxGatewaySessionException(message, scrub(ex.protocolStatus(), secret), null);
}
if (error instanceof MxGatewayWorkerException ex) {
return new MxGatewayWorkerException(message, scrub(ex.protocolStatus(), secret), null);
}
if (error instanceof MxGatewayAuthenticationException) {
return new MxGatewayAuthenticationException(message, null);
}
if (error instanceof MxGatewayAuthorizationException) {
return new MxGatewayAuthorizationException(message, null);
}
return new MxGatewayException(message);
}
private static ProtocolStatus scrub(ProtocolStatus status, String secret) {
if (status == null) {
return null;
}
return status.toBuilder()
.setMessage(MxGatewaySecrets.redactExact(status.getMessage(), secret))
.build();
}
static void ensureProtocolSuccess(String operation, ProtocolStatus status, MxCommandReply reply) {
if (status == null || status.getCode() == ProtocolStatusCode.PROTOCOL_STATUS_CODE_OK) {
return;
@@ -37,6 +37,13 @@ import mxaccess_gateway.v1.MxaccessGateway.RegisterCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemCommand;
import mxaccess_gateway.v1.MxaccessGateway.SetBufferedUpdateIntervalCommand;
import mxaccess_gateway.v1.MxaccessGateway.DrainEventsCommand;
import mxaccess_gateway.v1.MxaccessGateway.GetSessionStateCommand;
import mxaccess_gateway.v1.MxaccessGateway.GetWorkerInfoCommand;
import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
import mxaccess_gateway.v1.MxaccessGateway.PingCommand;
import mxaccess_gateway.v1.MxaccessGateway.SessionState;
import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -548,15 +555,17 @@ public final class MxGatewaySession implements AutoCloseable {
* {@code OnDataChange} value for any tag that is already advised
* ({@code wasCached == true}) without modifying the existing subscription,
* and falls back to a full AddItem + Advise + wait + UnAdvise + RemoveItem
* snapshot lifecycle otherwise. The supplied {@code timeout} bounds the
* per-tag wait in the snapshot case; pass {@link Duration#ZERO} (or
* {@code null}) to use the worker default (1000 ms). Per-tag failures
* snapshot lifecycle otherwise. The supplied {@code timeout} is one shared
* deadline for the whole command, not a per-tag budget: the worker advises
* every uncached tag, waits once for all of them, then tears them down.
* Pass {@link Duration#ZERO} (or {@code null}) to use the worker default
* (1000 ms). Per-tag failures
* appear as {@link BulkReadResult} entries with {@code wasSuccessful == false};
* this method does not throw for per-tag MXAccess failures.
*
* @param serverHandle the {@code ServerHandle} owning the items
* @param tagAddresses the tag addresses to read
* @param timeout per-tag snapshot timeout (zero or null = worker default)
* @param timeout snapshot deadline shared by all tags in the command (zero or null = worker default)
* @return a per-tag {@link BulkReadResult} list
* @throws MxGatewayException on transport or protocol failure
* @throws NullPointerException if {@code tagAddresses} is {@code null}
@@ -1033,6 +1042,67 @@ public final class MxGatewaySession implements AutoCloseable {
.build());
}
/**
* Sends a {@code Ping} diagnostic command to the session's worker.
*
* @param message free-form text carried in the command
* @return the raw command reply
* @throws MxGatewayException on transport or protocol failure
*/
public MxCommandReply ping(String message) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_PING)
.setPing(PingCommand.newBuilder().setMessage(message == null ? "" : message))
.build());
}
/**
* Queries the worker-side state of this session.
*
* @return the reported {@link SessionState}
* @throws MxGatewayException on transport or protocol failure
*/
public SessionState getSessionState() {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE)
.setGetSessionState(GetSessionStateCommand.getDefaultInstance())
.build())
.getSessionState()
.getState();
}
/**
* Queries the worker process and MXAccess COM identity.
*
* @return the {@link WorkerInfoReply} (process id, version, ProgID, CLSID)
* @throws MxGatewayException on transport or protocol failure
*/
public WorkerInfoReply getWorkerInfo() {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_GET_WORKER_INFO)
.setGetWorkerInfo(GetWorkerInfoCommand.getDefaultInstance())
.build())
.getWorkerInfo();
}
/**
* Pulls queued events from the worker without opening a stream. The reply
* is bounded server-side and may carry fewer than {@code maxEvents}; drain
* iteratively until an empty list is returned.
*
* @param maxEvents the maximum number of events to return
* @return the drained events, possibly empty
* @throws MxGatewayException on transport or protocol failure
*/
public List<MxEvent> drainEvents(int maxEvents) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_DRAIN_EVENTS)
.setDrainEvents(DrainEventsCommand.newBuilder().setMaxEvents(maxEvents))
.build())
.getDrainEvents()
.getEventsList();
}
/**
* Sends a pre-built {@link MxCommand} for this session and returns the raw
* reply, attaching a freshly generated client correlation id.
@@ -0,0 +1,93 @@
package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import io.grpc.stub.ClientCallStreamObserver;
import io.grpc.stub.ClientResponseObserver;
import java.util.concurrent.atomic.AtomicInteger;
import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import org.junit.jupiter.api.Test;
final class MxEventStreamTests {
@Test
void overflowKeepsRetainedEventsThenSurfacesError() {
MxEventStream stream = new MxEventStream(3);
RecordingCall call = new RecordingCall();
ClientResponseObserver<StreamEventsRequest, MxEvent> observer = stream.observer();
observer.beforeStart(call);
for (long i = 1; i <= 5; i++) {
observer.onNext(MxEvent.newBuilder().setWorkerSequence(i).build());
}
assertTrue(call.cancelCount.get() >= 1, "overflow cancels the call");
assertEquals(1L, stream.next().getWorkerSequence());
assertEquals(2L, stream.next().getWorkerSequence());
assertEquals(3L, stream.next().getWorkerSequence());
MxGatewayException error = assertThrows(MxGatewayException.class, stream::hasNext);
assertTrue(error.getMessage().contains("overflowed"));
assertFalse(stream.hasNext());
}
@Test
void interruptCancelsCallAndRestoresInterruptFlag() {
MxEventStream stream = new MxEventStream(4);
RecordingCall call = new RecordingCall();
stream.observer().beforeStart(call);
Thread.currentThread().interrupt();
try {
assertThrows(io.grpc.StatusRuntimeException.class, stream::hasNext);
assertTrue(Thread.currentThread().isInterrupted(), "interrupt flag restored");
assertEquals(1, call.cancelCount.get(), "call cancelled on interrupt");
} finally {
Thread.interrupted();
}
}
private static final class RecordingCall extends ClientCallStreamObserver<StreamEventsRequest> {
final AtomicInteger cancelCount = new AtomicInteger();
@Override
public void cancel(String message, Throwable cause) {
cancelCount.incrementAndGet();
}
@Override
public boolean isReady() {
return true;
}
@Override
public void setOnReadyHandler(Runnable onReadyHandler) {
}
@Override
public void request(int count) {
}
@Override
public void setMessageCompression(boolean enable) {
}
@Override
public void disableAutoInboundFlowControl() {
}
@Override
public void onNext(StreamEventsRequest value) {
}
@Override
public void onError(Throwable t) {
}
@Override
public void onCompleted() {
}
}
}
@@ -51,7 +51,10 @@ import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
import mxaccess_gateway.v1.MxaccessGateway.QueryActiveAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.RegisterReply;
import mxaccess_gateway.v1.MxaccessGateway.ReplayGap;
import mxaccess_gateway.v1.MxaccessGateway.DrainEventsReply;
import mxaccess_gateway.v1.MxaccessGateway.SessionState;
import mxaccess_gateway.v1.MxaccessGateway.SessionStateReply;
import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -727,6 +730,63 @@ final class MxGatewayClientSessionTests {
}
}
@Test
void diagnosticsHelpersBuildTypedCommandsAndDecodeReplies() throws Exception {
AtomicReference<MxCommandRequest> last = new AtomicReference<>();
TestGatewayService service = new TestGatewayService() {
@Override
public void invoke(MxCommandRequest request, StreamObserver<MxCommandReply> responseObserver) {
last.set(request);
MxCommandReply.Builder reply = MxCommandReply.newBuilder()
.setSessionId(request.getSessionId())
.setKind(request.getCommand().getKind())
.setProtocolStatus(ok());
switch (request.getCommand().getKind()) {
case MX_COMMAND_KIND_GET_SESSION_STATE -> reply.setSessionState(
SessionStateReply.newBuilder().setState(SessionState.SESSION_STATE_READY));
case MX_COMMAND_KIND_GET_WORKER_INFO -> reply.setWorkerInfo(
WorkerInfoReply.newBuilder().setWorkerProcessId(99).setWorkerVersion("1.2.3"));
case MX_COMMAND_KIND_DRAIN_EVENTS -> reply.setDrainEvents(DrainEventsReply.newBuilder()
.addEvents(MxEvent.newBuilder().setWorkerSequence(5)));
default -> { }
}
responseObserver.onNext(reply.build());
responseObserver.onCompleted();
}
};
try (InProcessGateway gateway = InProcessGateway.start(service, new AtomicReference<>());
MxGatewayClient client = gateway.client("", Duration.ofSeconds(5))) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "existing-session");
MxCommandReply ping = session.ping("hello");
assertEquals(MxCommandKind.MX_COMMAND_KIND_PING, ping.getKind());
assertEquals("hello", last.get().getCommand().getPing().getMessage());
assertEquals(SessionState.SESSION_STATE_READY, session.getSessionState());
assertEquals(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE, last.get().getCommand().getKind());
WorkerInfoReply info = session.getWorkerInfo();
assertEquals(99, info.getWorkerProcessId());
assertEquals("1.2.3", info.getWorkerVersion());
List<MxEvent> events = session.drainEvents(10);
assertEquals(1, events.size());
assertEquals(5L, events.get(0).getWorkerSequence());
assertEquals(10, last.get().getCommand().getDrainEvents().getMaxEvents());
}
}
@Test
void eventQueueCapacityOptionDefaultsAndValidates() {
assertEquals(1024, MxGatewayClientOptions.builder().endpoint("x").build().eventQueueCapacity());
assertEquals(
7, MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(7).build().eventQueueCapacity());
assertThrows(
IllegalArgumentException.class,
() -> MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(0).build());
}
private static TestGatewayService okInvokeService(AtomicReference<MxCommandRequest> commandRequest) {
return new TestGatewayService() {
@Override
@@ -74,6 +74,52 @@ final class MxGatewayCredentialReplyTests {
}
}
@Test
void acknowledgeAlarmAsyncRedactsOperatorPasswordFromFailure() throws Exception {
String serverName = "mxgw-java-ack-" + UUID.randomUUID();
MxAccessGatewayGrpc.MxAccessGatewayImplBase service = new MxAccessGatewayGrpc.MxAccessGatewayImplBase() {
@Override
public void acknowledgeAlarm(
mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmRequest request,
StreamObserver<mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmReply> observer) {
observer.onNext(mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmReply.newBuilder()
.setProtocolStatus(ProtocolStatus.newBuilder()
.setCode(ProtocolStatusCode.PROTOCOL_STATUS_CODE_MXACCESS_FAILURE)
.setMessage("auth failed for operator1 password " + CREDENTIAL))
.build());
observer.onCompleted();
}
};
Server server = InProcessServerBuilder.forName(serverName)
.directExecutor().addService(service).build().start();
ManagedChannel channel = InProcessChannelBuilder.forName(serverName).directExecutor().build();
try (MxGatewayClient client = new MxGatewayClient(
channel,
MxGatewayClientOptions.builder()
.endpoint("in-process").apiKey("").plaintext(true)
.callTimeout(Duration.ofSeconds(5)).build())) {
var request = mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmRequest.newBuilder()
.setAlarmFullReference("Tank01.Level.HiHi")
.setOperatorUser("operator1")
.setOperatorPassword(CREDENTIAL)
.build();
java.util.concurrent.ExecutionException async = assertThrows(
java.util.concurrent.ExecutionException.class,
() -> client.acknowledgeAlarmAsync(request).get());
assertTrue(async.getCause() instanceof MxGatewayException, String.valueOf(async.getCause()));
assertFalse(async.getCause().getMessage().contains(CREDENTIAL));
assertFalse(((MxGatewayCommandException) async.getCause()).protocolStatus().getMessage()
.contains(CREDENTIAL));
MxGatewayException sync = assertThrows(MxGatewayException.class, () -> client.acknowledgeAlarm(request));
assertFalse(sync.getMessage().contains(CREDENTIAL));
} finally {
channel.shutdownNow();
server.shutdownNow();
}
}
@Test
void authenticateUserMissingPayloadThrowsMalformedReply() throws Exception {
MxCommandReply reply = loadReply("authenticate-user.missing-payload.reply.json");
+35 -5
View File
@@ -111,9 +111,15 @@ MXAccess commands and preserve raw replies on typed command exceptions.
For alarms, the client exposes `GatewayClient.query_active_alarms` (one-shot
snapshot), `GatewayClient.stream_alarms` (async generator yielding alarm-feed
messages from the gateway's central monitor), and
`GatewayClient.acknowledge_alarm` (ack by alarm reference, optional comment
and ack target). Cancel the surrounding task or `aclose()` the iterator to
terminate the stream.
`GatewayClient.acknowledge_alarm` (ack by alarm reference with an optional
comment). The ack is made as the operator: set `operator_user` (required) and
`operator_password` (may be empty) and optionally `operator_domain` on
`AcknowledgeAlarmRequest`. The gateway authenticates the operator and writes the
comment to the alarm's `AckMsg`, so the alarm history records the operator. The
password is redacted from any raised error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account.
Cancel the surrounding task or `aclose()` an alarm iterator to terminate the
stream.
`ActiveAlarmSnapshot.from_truncated_snapshot` reports that the record came from
a provider fetch which hit the per-fetch cap: the snapshot set may omit active
@@ -132,6 +138,14 @@ sent when the gateway's alarm monitor restarts and drops a truncated verdict.
Track it if you need set completeness on a live feed without polling
`query_active_alarms`.
`mxgw-py stream-alarms` writes one JSON row as each feed message arrives. The
default `--max-messages 0` is unbounded; use a positive value for a finite
sample, remembering that the provider-status and snapshot-status preamble both
count toward it. `--timeout` ends the command after an idle wait and the stream
is closed in every exit path. Protobuf JSON omits false-valued scalar defaults,
so healthy/clearing frames appear as `{"providerStatus": {}}` and
`{"snapshotStatus": {}}`; absent means false.
Canceling a Python task cancels the client-side gRPC call or stream wait. It
does not abort an in-flight MXAccess COM call inside the worker process.
@@ -448,8 +462,8 @@ mxgw-py register --session-id <id> --client-name python-client --json
mxgw-py add-item --session-id <id> --server-handle 1 --item Object.Attribute --json
mxgw-py advise --session-id <id> --server-handle 1 --item-handle 2 --json
mxgw-py stream-events --session-id <id> --max-events 1 --json
mxgw-py stream-alarms --max-messages 1 --json
mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --json
mxgw-py stream-alarms --max-messages 4 --json
mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json
mxgw-py write --session-id <id> --server-handle 1 --item-handle 2 --type int32 --value 123 --json
```
@@ -484,6 +498,15 @@ mxgw-py smoke --endpoint $env:MXGATEWAY_ENDPOINT --plaintext --api-key-env MXGAT
## Installing from the Gitea PyPI Feed
**The published `0.2.0` artifact cannot connect.** Its `connect()` built the
`grpc.aio` channel inside `asyncio.to_thread`, and a `grpc.aio` channel binds
to the event loop current on the *constructing* thread — so every
non-stub connection raised `RuntimeError: There is no current event loop in
thread`. Fixed in-tree by commit `a8f86b5` (`resolve_channel_security` now
runs off-loop, `create_channel` runs on-loop). The fix is superseded by
`0.2.1`; do not install `0.2.0` from the feed below — use `0.2.1` or later,
or install from source at a commit at or after `a8f86b5`.
The client publishes to the internal Gitea PyPI feed:
````bash
@@ -500,3 +523,10 @@ a `~/.netrc` entry or `PIP_INDEX_URL=https://<user>:<token>@gitea.dohertylan.com
- [Client Packaging](../../docs/ClientPackaging.md)
- [Client Proto Generation](../../docs/ClientProtoGeneration.md)
- [Python Client Detailed Design](./PythonClientDesign.md)
## Typed diagnostics
`Session.ping(message)`, `get_session_state()`, `get_worker_info()`, and
`drain_events(max_events)` wrap the worker diagnostic commands. `drain_events`
returns a list that may be shorter than `max_events`; call it until it returns
an empty list.
+1 -1
View File
@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "zb-mom-ww-mxaccess-gateway-client"
version = "0.2.0"
version = "0.2.1"
description = "Async Python client for MXAccess Gateway."
readme = "README.md"
requires-python = ">=3.12"
@@ -9,7 +9,7 @@ from typing import Any
import grpc
from .auth import merge_metadata
from .errors import ensure_protocol_success, map_rpc_error
from .errors import MxGatewayError, ensure_protocol_success, map_rpc_error
from .generated import mxaccess_gateway_pb2 as pb
from .generated import mxaccess_gateway_pb2_grpc as pb_grpc
from .options import ClientOptions, create_channel, resolve_channel_security
@@ -148,17 +148,23 @@ class GatewayClient:
self,
request: pb.AcknowledgeAlarmRequest,
) -> pb.AcknowledgeAlarmReply:
"""Acknowledge an active MXAccess alarm condition through the gateway.
"""Acknowledge an alarm as the operator.
The gateway authenticates the request against the API key's
``invoke:alarm-ack`` scope and forwards the acknowledge to the worker's
MXAccess session; the resulting native ``MxStatus`` is returned in the
reply. Acks are idempotent — re-acking an already-acked condition is a
no-op at the MxAccess layer.
The gateway authenticates ``operator_user`` with ``operator_password`` and
writes the comment to the alarm's ``AckMsg``, so the alarm history records
the operator. The password is redacted from any raised error.
"""
reply = await self._unary("acknowledge alarm", self.raw_stub.AcknowledgeAlarm, request)
ensure_protocol_success("acknowledge alarm", reply.protocol_status, reply)
return reply
try:
reply = await self._unary("acknowledge alarm", self.raw_stub.AcknowledgeAlarm, request)
ensure_protocol_success("acknowledge alarm", reply.protocol_status, reply)
return reply
except MxGatewayError as error:
if request.operator_password:
# Imported lazily: session.py imports this module at its tail.
from .session import _redact_error
_redact_error(error, [request.operator_password])
raise
def query_active_alarms(
self,
File diff suppressed because one or more lines are too long
@@ -3,6 +3,7 @@
from __future__ import annotations
from collections.abc import AsyncIterator, Sequence
from typing import Any
from .auth import redact_secret
from .errors import MalformedReplyError, MxGatewayError, ensure_mxaccess_success
@@ -447,8 +448,10 @@ class Session:
``OnDataChange`` value for any tag that is already advised (``was_cached =
True``) without modifying the existing subscription, and falls back to
a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
otherwise. ``timeout_ms`` bounds the per-tag wait in the snapshot case;
pass ``0`` to use the worker default (1000 ms).
otherwise. ``timeout_ms`` is one shared deadline for the whole command,
not a per-tag budget: the worker advises every uncached tag, waits once
for all of them, then tears them down. Pass ``0`` to use the worker
default (1000 ms).
"""
if tag_addresses is None:
raise TypeError("tag_addresses is required")
@@ -837,6 +840,55 @@ class Session:
)
return reply.activate.status
async def ping(self, message: str = "", *, correlation_id: str = "") -> pb.MxCommandReply:
"""Invoke the worker `Ping` diagnostic and return its reply."""
return await self.invoke(
pb.MxCommand(kind=pb.MX_COMMAND_KIND_PING, ping=pb.PingCommand(message=message)),
correlation_id=correlation_id,
)
async def get_session_state(self, *, correlation_id: str = "") -> int:
"""Return the worker-reported `SessionState` enum value."""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_GET_SESSION_STATE,
get_session_state=pb.GetSessionStateCommand(),
),
correlation_id=correlation_id,
)
return reply.session_state.state
async def get_worker_info(self, *, correlation_id: str = "") -> pb.WorkerInfoReply:
"""Return worker process id, version, and MXAccess ProgID/CLSID."""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_GET_WORKER_INFO,
get_worker_info=pb.GetWorkerInfoCommand(),
),
correlation_id=correlation_id,
)
return reply.worker_info
async def drain_events(
self,
max_events: int = 0,
*,
correlation_id: str = "",
) -> list[pb.MxEvent]:
"""Drain queued worker events (diagnostic).
A reply may carry fewer than *max_events* (server count and frame-size
caps); call repeatedly until an empty list is returned.
"""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_DRAIN_EVENTS,
drain_events=pb.DrainEventsCommand(max_events=max_events),
),
correlation_id=correlation_id,
)
return list(reply.drain_events.events)
def stream_events(
self,
*,
@@ -943,22 +995,26 @@ def _redact_error(error: MxGatewayError, secrets: Sequence[str | None]) -> None:
error.raw_reply = _redact_reply(error.raw_reply, scrubbed)
def _redact_reply(reply: pb.MxCommandReply, secrets: Sequence[str]) -> pb.MxCommandReply:
def _redact_reply(reply: Any, secrets: Sequence[str]) -> Any:
"""Return a deep copy of *reply* with credential text scrubbed from diagnostics.
Operates on a clone so the caller's original reply object is never mutated.
Only the free-text diagnostic fields that can echo a client-supplied secret
are scrubbed; the structured/enum fields the gateway itself sets are left as-is.
Works for any reply message (``MxCommandReply``, ``AcknowledgeAlarmReply``, ...):
each field is scrubbed only when the message type declares it.
"""
clone = type(reply)()
clone.CopyFrom(reply)
if clone.protocol_status.message:
fields = clone.DESCRIPTOR.fields_by_name
if "protocol_status" in fields and clone.protocol_status.message:
clone.protocol_status.message = redact_secret(clone.protocol_status.message, secrets)
if clone.diagnostic_message:
if "diagnostic_message" in fields and clone.diagnostic_message:
clone.diagnostic_message = redact_secret(clone.diagnostic_message, secrets)
for status in clone.statuses:
if status.diagnostic_text:
status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
if "statuses" in fields:
for status in clone.statuses:
if status.diagnostic_text:
status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
return clone
@@ -1,3 +1,3 @@
"""Package version information."""
__version__ = "0.2.0"
__version__ = "0.2.1"
@@ -36,6 +36,7 @@ MAX_AGGREGATE_EVENTS = 10_000
#: CLI (CLI-45) so one exported variable drives the same operator workflow in all
#: five languages.
DEFAULT_VERIFY_PASSWORD_ENV = "MXGATEWAY_VERIFY_PASSWORD"
DEFAULT_OPERATOR_PASSWORD_ENV = "MXGATEWAY_OPERATOR_PASSWORD"
_BATCH_EOR = "__MXGW_BATCH_EOR__"
@@ -391,7 +392,7 @@ def unsubscribe_bulk(**kwargs: Any) -> None:
@click.option("--server-handle", required=True, type=int, help="MXAccess server handle.")
@click.option("--items", required=True, help="Comma-separated MXAccess tag addresses.")
@click.option("--timeout-ms", default=0, type=int, show_default=True,
help="Per-tag snapshot timeout in milliseconds. 0 = worker default.")
help="Snapshot deadline in milliseconds, shared by all tags in the command. 0 = worker default (1000 ms).")
@click.option("--correlation-id", default="", help="Client correlation id.")
@click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
def read_bulk(**kwargs: Any) -> None:
@@ -511,16 +512,15 @@ def stream_events(**kwargs: Any) -> None:
@main.command("stream-alarms")
@gateway_options
@click.option("--filter-prefix", default="", help="Alarm-reference prefix filter.")
@click.option("--max-messages", default=1, type=int, show_default=True)
@click.option("--max-messages", default=0, type=int, show_default=True, help="Maximum rows to emit; 0 means unbounded.")
@click.option("--timeout", default=5.0, type=float, show_default=True)
@click.option("--correlation-id", default="", help="Client correlation id.")
@click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
def stream_alarms(**kwargs: Any) -> None:
"""Stream a bounded number of messages from the gateway's central alarm feed."""
"""Stream messages from the gateway's central alarm feed as they arrive."""
_run(
_run_streaming(
_stream_alarms(**kwargs),
output_json=kwargs["output_json"],
secrets=_secrets(kwargs),
)
@@ -529,16 +529,30 @@ def stream_alarms(**kwargs: Any) -> None:
@gateway_options
@click.option("--reference", required=True, help="Alarm full reference to acknowledge.")
@click.option("--comment", default="", help="Acknowledgement comment.")
@click.option("--operator", default="", help="Operator user name.")
@click.option("--operator", required=True, help="Operator user name (recorded in the alarm history).")
@click.option("--operator-domain", default="", help="Operator Windows/Galaxy domain (optional).")
@click.option(
"--operator-password",
default=None,
help="Operator password. Prefer --operator-password-env so the secret is not visible on the command line.",
)
@click.option(
"--operator-password-env",
default=DEFAULT_OPERATOR_PASSWORD_ENV,
show_default=True,
help="Environment variable holding the operator password (an unset variable means an empty password).",
)
@click.option("--correlation-id", default="", help="Client correlation id.")
@click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
def acknowledge_alarm(**kwargs: Any) -> None:
"""Acknowledge an active MXAccess alarm condition (session-less)."""
"""Acknowledge an active MXAccess alarm condition as the operator (session-less)."""
password = _resolve_operator_password(kwargs)
kwargs["operator_password"] = password
_run(
_acknowledge_alarm(**kwargs),
output_json=kwargs["output_json"],
secrets=_secrets(kwargs),
secrets=_secrets(kwargs) + [password],
)
@@ -837,6 +851,21 @@ async def _authenticate_user(**kwargs: Any) -> dict[str, Any]:
return {"userId": user_id}
def _resolve_operator_password(kwargs: dict[str, Any]) -> str:
"""Resolve the alarm-ack operator password from --operator-password or its env var.
Unlike :func:`_resolve_password`, an unset or empty value is allowed: Galaxy
accounts may have empty passwords. The value is never echoed; callers add it
to the ``secrets`` redaction list.
"""
env_name = kwargs.get("operator_password_env") or DEFAULT_OPERATOR_PASSWORD_ENV
password = kwargs.get("operator_password")
if password is None:
password = os.environ.get(env_name, "")
return password
def _resolve_password(kwargs: dict[str, Any]) -> str:
"""Resolve the authenticate-user password from --password or --password-env.
@@ -1119,19 +1148,39 @@ async def _stream_events(**kwargs: Any) -> dict[str, Any]:
return {"events": [_event_row(event) for event in events]}
async def _stream_alarms(**kwargs: Any) -> dict[str, Any]:
async with await _connect(kwargs) as client:
messages = await _collect_alarm_messages(
client.stream_alarms(
pb.StreamAlarmsRequest(
client_correlation_id=kwargs["correlation_id"],
alarm_filter_prefix=kwargs["filter_prefix"],
),
),
max_messages=kwargs["max_messages"],
timeout=kwargs["timeout"],
async def _stream_alarms(**kwargs: Any) -> None:
max_messages = int(kwargs["max_messages"])
if max_messages > MAX_AGGREGATE_EVENTS:
raise click.BadParameter(
f"must be less than or equal to {MAX_AGGREGATE_EVENTS}",
param_hint="--max-messages",
)
return {"messages": [_message_dict(message) for message in messages]}
async with await _connect(kwargs) as client:
messages = client.stream_alarms(
pb.StreamAlarmsRequest(
client_correlation_id=kwargs["correlation_id"],
alarm_filter_prefix=kwargs["filter_prefix"],
),
)
iterator = messages.__aiter__()
emitted = 0
try:
while max_messages == 0 or emitted < max_messages:
try:
message = await asyncio.wait_for(
iterator.__anext__(),
timeout=kwargs["timeout"],
)
except (StopAsyncIteration, asyncio.TimeoutError):
break
_emit(_message_dict(message), output_json=kwargs["output_json"])
sys.stdout.flush()
emitted += 1
finally:
close = getattr(iterator, "aclose", None)
if close is not None:
await close()
async def _acknowledge_alarm(**kwargs: Any) -> dict[str, Any]:
@@ -1142,6 +1191,8 @@ async def _acknowledge_alarm(**kwargs: Any) -> dict[str, Any]:
alarm_full_reference=kwargs["reference"],
comment=kwargs["comment"],
operator_user=kwargs["operator"],
operator_domain=kwargs["operator_domain"],
operator_password=kwargs["operator_password"],
),
)
return _message_dict(reply)
@@ -1489,6 +1540,17 @@ def _run(
_emit(payload, output_json=output_json)
def _run_streaming(
awaitable: Awaitable[None],
*,
secrets: list[str | None],
) -> None:
try:
asyncio.run(awaitable)
except MxGatewayError as error:
raise click.ClickException(redact_secret(str(error), secrets)) from error
def _emit(
payload: dict[str, Any],
*,
@@ -1536,34 +1598,6 @@ async def _collect_events(
return collected
async def _collect_alarm_messages(
messages: Any,
*,
max_messages: int,
timeout: float,
) -> list[pb.AlarmFeedMessage]:
if max_messages > MAX_AGGREGATE_EVENTS:
raise click.BadParameter(
f"must be less than or equal to {MAX_AGGREGATE_EVENTS}",
param_hint="--max-messages",
)
collected: list[pb.AlarmFeedMessage] = []
iterator = messages.__aiter__()
try:
while len(collected) < max_messages:
collected.append(await asyncio.wait_for(iterator.__anext__(), timeout=timeout))
except StopAsyncIteration:
pass
except asyncio.TimeoutError:
pass
finally:
close = getattr(iterator, "aclose", None)
if close is not None:
await close()
return collected
async def _collect_deploy_events(
events: Any,
*,
+61
View File
@@ -34,6 +34,8 @@ async def test_acknowledge_alarm_sends_request_and_returns_reply() -> None:
alarm_full_reference="Tank01.Level.HiHi",
comment="investigating",
operator_user="alice",
operator_password="pw-secret",
operator_domain="CORP",
),
)
@@ -44,9 +46,68 @@ async def test_acknowledge_alarm_sends_request_and_returns_reply() -> None:
assert captured_request.alarm_full_reference == "Tank01.Level.HiHi"
assert captured_request.comment == "investigating"
assert captured_request.operator_user == "alice"
assert captured_request.operator_password == "pw-secret"
assert captured_request.operator_domain == "CORP"
assert stub.acknowledge_alarm.metadata == (("authorization", "Bearer mxgw_test_secret"),)
@pytest.mark.asyncio
async def test_acknowledge_alarm_error_redacts_operator_password() -> None:
stub = FakeGatewayStub()
stub.acknowledge_alarm.exception = FakeRpcError(
grpc.StatusCode.UNAUTHENTICATED, "bad credentials pw-secret",
)
client = await GatewayClient.connect(
ClientOptions(endpoint="fake", api_key="mxgw_test_secret", plaintext=True),
stub=stub,
)
with pytest.raises(MxGatewayAuthenticationError) as excinfo:
await client.acknowledge_alarm(
pb.AcknowledgeAlarmRequest(
alarm_full_reference="Tank01.Level.HiHi",
operator_user="alice",
operator_password="pw-secret",
),
)
assert "pw-secret" not in str(excinfo.value)
@pytest.mark.asyncio
async def test_acknowledge_alarm_protocol_failure_redacts_operator_password() -> None:
from zb_mom_ww_mxgateway.errors import MxGatewayError
stub = FakeGatewayStub()
stub.acknowledge_alarm.replies = [
pb.AcknowledgeAlarmReply(
protocol_status=pb.ProtocolStatus(
code=pb.PROTOCOL_STATUS_CODE_INVALID_REQUEST,
message="rejected credential pw-secret",
),
),
]
client = await GatewayClient.connect(
ClientOptions(endpoint="fake", api_key="mxgw_test_secret", plaintext=True),
stub=stub,
)
with pytest.raises(MxGatewayError) as excinfo:
await client.acknowledge_alarm(
pb.AcknowledgeAlarmRequest(
alarm_full_reference="Tank01.Level.HiHi",
operator_user="alice",
operator_password="pw-secret",
),
)
assert "pw-secret" not in str(excinfo.value)
raw = excinfo.value.raw_reply
assert isinstance(raw, pb.AcknowledgeAlarmReply)
assert "pw-secret" not in raw.protocol_status.message
assert "pw-secret" not in str(raw)
@pytest.mark.asyncio
async def test_acknowledge_alarm_unauthenticated_raises_typed_error() -> None:
stub = FakeGatewayStub()
@@ -7,6 +7,7 @@ fixed source.
from __future__ import annotations
import asyncio
import json
import re
import time as _time_module_ref
@@ -296,6 +297,22 @@ class _AlarmFakeStream:
self.cancelled = True
class _IncrementalAlarmFakeStream(_AlarmFakeStream):
def __init__(self, messages: list[pb.AlarmFeedMessage], emitted: list[str]) -> None:
super().__init__(messages)
self._emitted = emitted
self._yielded = 0
async def __anext__(self) -> pb.AlarmFeedMessage:
if self._yielded > 0:
assert len(self._emitted) >= self._yielded, "previous alarm row was buffered"
if not self._messages:
await asyncio.Event().wait()
raise AssertionError("unreachable")
self._yielded += 1
return self._messages.pop(0)
class _BulkFakeUnary:
def __init__(self, replies: list[Any]) -> None:
self.replies = replies
@@ -667,8 +684,15 @@ def test_cli_write_bulk_happy_path(monkeypatch) -> None:
assert cmd.kind == pb.MX_COMMAND_KIND_WRITE_BULK
def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
transitions = [
def test_cli_stream_alarms_default_streams_status_cached_and_live_rows_incrementally(
monkeypatch,
) -> None:
messages = [
pb.AlarmFeedMessage(provider_status=pb.AlarmProviderStatus(degraded=False)),
pb.AlarmFeedMessage(snapshot_status=pb.AlarmSnapshotStatus(truncated=False)),
pb.AlarmFeedMessage(
active_alarm=pb.ActiveAlarmSnapshot(alarm_full_reference="Tank01.Level.HiHi"),
),
pb.AlarmFeedMessage(
transition=pb.OnAlarmTransitionEvent(
alarm_full_reference="Tank01.Level.HiHi",
@@ -676,7 +700,15 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
),
),
]
stream = _AlarmFakeStream(transitions)
emitted: list[str] = []
real_echo = cli_commands.click.echo
def recording_echo(message=None, **kwargs) -> None:
emitted.append(str(message))
real_echo(message, **kwargs)
monkeypatch.setattr(cli_commands.click, "echo", recording_echo)
stream = _IncrementalAlarmFakeStream(messages, emitted)
stub = _BulkFakeStub()
stub.set_alarm_stream(stream)
_install_fake_connect(monkeypatch, stub)
@@ -689,10 +721,8 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
"--endpoint",
"localhost:5000",
"--plaintext",
"--max-messages",
"1",
"--timeout",
"5.0",
"0.01",
"--filter-prefix",
"Tank01.",
"--json",
@@ -700,11 +730,48 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
)
assert result.exit_code == 0, result.output
payload = json.loads(result.output)
assert payload["messages"][0]["transition"]["alarmFullReference"] == "Tank01.Level.HiHi"
rows = [json.loads(line) for line in result.output.splitlines()]
assert [next(iter(row)) for row in rows] == [
"providerStatus",
"snapshotStatus",
"activeAlarm",
"transition",
]
assert stream.cancelled
def test_cli_stream_alarms_positive_max_messages_remains_finite(monkeypatch) -> None:
messages = [
pb.AlarmFeedMessage(provider_status=pb.AlarmProviderStatus()),
pb.AlarmFeedMessage(snapshot_status=pb.AlarmSnapshotStatus()),
pb.AlarmFeedMessage(active_alarm=pb.ActiveAlarmSnapshot()),
]
stream = _AlarmFakeStream(messages)
stub = _BulkFakeStub()
stub.set_alarm_stream(stream)
_install_fake_connect(monkeypatch, stub)
result = CliRunner().invoke(
main,
[
"stream-alarms",
"--endpoint",
"localhost:5000",
"--plaintext",
"--max-messages",
"2",
"--json",
],
)
assert result.exit_code == 0, result.output
rows = [json.loads(line) for line in result.output.splitlines()]
assert [next(iter(row)) for row in rows] == ["providerStatus", "snapshotStatus"]
assert stream.cancelled
def test_cli_acknowledge_alarm_happy_path(monkeypatch) -> None:
monkeypatch.setenv("MXGW_TEST_OPPW", "pw-secret")
stub = _BulkFakeStub()
stub.acknowledge_alarm = _BulkFakeUnary(
[
@@ -732,12 +799,19 @@ def test_cli_acknowledge_alarm_happy_path(monkeypatch) -> None:
"investigating",
"--operator",
"alice",
"--operator-domain",
"CORP",
"--operator-password-env",
"MXGW_TEST_OPPW",
"--json",
],
)
assert result.exit_code == 0, result.output
assert "pw-secret" not in result.output
captured_request = stub.acknowledge_alarm.requests[0]
assert captured_request.operator_domain == "CORP"
assert captured_request.operator_password == "pw-secret"
assert captured_request.alarm_full_reference == "Tank01.Level.HiHi"
assert captured_request.comment == "investigating"
assert captured_request.operator_user == "alice"
@@ -234,3 +234,61 @@ async def test_suspend_and_activate_return_status() -> None:
)
status = await session.activate(12, 34)
assert status.category == pb.MX_STATUS_CATEGORY_OK
@pytest.mark.asyncio
async def test_ping_sends_message_and_returns_reply() -> None:
session, stub = await _session_with([_ok(pb.MX_COMMAND_KIND_PING)])
reply = await session.ping("hello")
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_PING
assert command.ping.message == "hello"
assert reply.kind == pb.MX_COMMAND_KIND_PING
@pytest.mark.asyncio
async def test_get_session_state_returns_state() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_GET_SESSION_STATE,
session_state=pb.SessionStateReply(state=pb.SESSION_STATE_CLOSED),
),
],
)
state = await session.get_session_state()
assert state == pb.SESSION_STATE_CLOSED
assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_SESSION_STATE
@pytest.mark.asyncio
async def test_get_worker_info_returns_info() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_GET_WORKER_INFO,
worker_info=pb.WorkerInfoReply(worker_process_id=42, worker_version="1.2"),
),
],
)
info = await session.get_worker_info()
assert info.worker_process_id == 42
assert info.worker_version == "1.2"
assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_WORKER_INFO
@pytest.mark.asyncio
async def test_drain_events_sends_max_and_returns_events() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_DRAIN_EVENTS,
drain_events=pb.DrainEventsReply(events=[pb.MxEvent(), pb.MxEvent()]),
),
],
)
events = await session.drain_events(max_events=5)
assert len(events) == 2
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_DRAIN_EVENTS
assert command.drain_events.max_events == 5
+2 -2
View File
@@ -590,7 +590,7 @@ checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"
[[package]]
name = "mxgw-cli"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"clap",
"futures-util",
@@ -1490,7 +1490,7 @@ dependencies = [
[[package]]
name = "zb-mom-ww-mxgateway-client"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"futures-core",
"futures-util",
+3 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "zb-mom-ww-mxgateway-client"
version = "0.2.0"
version = "0.2.1"
edition = "2021"
authors = ["Joseph Doherty"]
description = "Async Rust client for the MxAccessGateway gRPC service, including a lazy-browse walker over the Galaxy Repository hierarchy."
@@ -25,7 +25,7 @@ resolver = "2"
[workspace.package]
edition = "2021"
version = "0.2.0"
version = "0.2.1"
authors = ["Joseph Doherty"]
license = "Proprietary"
repository = "https://gitea.dohertylan.com/dohertj2/mxaccessgw"
@@ -40,7 +40,7 @@ prost-types = "0.13.5"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.145"
thiserror = "2.0.17"
tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time"] }
tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time", "fs"] }
tokio-stream = { version = "0.1.17", features = ["net"] }
tonic = { version = "0.13.1", features = ["transport", "tls-ring", "tls-native-roots"] }
tonic-build = "0.13.1"
+23 -2
View File
@@ -77,8 +77,9 @@ cargo run -p mxgw-cli -- register --session-id <session-id> --client-name mxgw-r
cargo run -p mxgw-cli -- add-item --session-id <session-id> --server-handle 1 --item TestChildObject.TestInt --json
cargo run -p mxgw-cli -- advise --session-id <session-id> --server-handle 1 --item-handle 1 --json
cargo run -p mxgw-cli -- stream-events --session-id <session-id> --max-events 1 --json
cargo run -p mxgw-cli -- stream-alarms --session-id <session-id> --max-messages 1 --json
cargo run -p mxgw-cli -- acknowledge-alarm --session-id <session-id> --alarm-reference "\\Galaxy\Area001.Pump001.PumpFault" --json
cargo run -p mxgw-cli -- stream-alarms --session-id <session-id> --max-messages 4 --json
cargo run -p mxgw-cli -- acknowledge-alarm --reference "Area001.Pump001.PumpFault" --operator <user> [--operator-domain <domain>] --json
# operator password: MXGATEWAY_OPERATOR_PASSWORD, or --operator-password-env / --operator-password
cargo run -p mxgw-cli -- write --session-id <session-id> --server-handle 1 --item-handle 1 --value-type int32 --value 123 --json
```
@@ -124,6 +125,9 @@ creates an authenticated `tonic` client and attaches `authorization: Bearer
`stream_alarms`, `acknowledge_alarm`, and `raw_client`. `stream_alarms`
returns an `AlarmFeedStream` async stream of alarm-feed messages and
shares the gateway's central alarm monitor with every other client.
`acknowledge_alarm` requires `operator_user` and carries `operator_password` (may be
empty) and `operator_domain`; the gateway authenticates as that operator so the Historian
records them. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The password is scrubbed from any returned error's `Display`/`Debug`.
The `from_truncated_snapshot` field on `ActiveAlarmSnapshot` reports that the record came from
a provider fetch which hit the per-fetch cap: the snapshot set may omit active
@@ -141,6 +145,11 @@ every verdict change — including the clearing frame sent when the gateway's
alarm monitor restarts and drops a truncated verdict. Track it if you need set
completeness on a live feed without polling `query_active_alarms`.
Unlike the protobuf-JSON formatters used by the other CLIs, the Rust CLI's
hand-built JSON status rows always include `"degraded": false` and
`"truncated": false`. This deliberate representational difference is
documented in the cross-language smoke matrix; both shapes mean the same thing.
The session helpers keep MXAccess handles visible:
```rust
@@ -169,6 +178,18 @@ redaction. Per-item bulk failures are reported inside each result entry
`invoke_raw` / `client.invoke_raw` escape hatch performs neither check and
returns the unvalidated reply.
### Diagnostic commands
`Session` wraps the worker diagnostic commands as typed calls: `ping(message)`
(returns `()` on success), `get_session_state()` (returns `SessionState`),
`get_worker_info()` (returns `WorkerInfoReply`: pid, version, MXAccess
ProgID/CLSID) and `drain_events(max_events)` (returns `Vec<MxEvent>`; the reply
is bounded server-side, so drain until an empty result). A reply missing its
expected payload surfaces as `Error::MalformedReply`.
TLS note: `ClientOptions::with_ca_file` is read asynchronously (`tokio::fs`)
during `connect`, so connecting no longer blocks the executor thread.
## Event Streaming And Reconnect-Replay Gaps
`session.events()` / `session.events_after(after_worker_sequence)` (and the
+3 -1
View File
@@ -174,6 +174,8 @@ alarm monitor — no worker session is opened — so any number of clients may
attach. Dropping the stream cancels the gRPC call cooperatively.
`acknowledge_alarm` is idempotent at the MxAccess layer; the returned
`AcknowledgeAlarmReply` carries the native MxStatus from the worker.
`operator_user` is required; `operator_password` may be empty and is scrubbed from any
returned error via `Error::redact_secrets`.
## Galaxy Repository
@@ -346,7 +348,7 @@ mxgw write-secured-bulk ...
mxgw write-secured2-bulk ...
mxgw stream-events --session-id <id> --json
mxgw stream-alarms [--filter-prefix <prefix>] [--max-events <n>]
mxgw acknowledge-alarm --reference <full-ref> [--comment <txt>] [--operator <user>]
mxgw acknowledge-alarm --reference <full-ref> [--comment <txt>] --operator <user> [--operator-domain <domain>] [--operator-password-env <VAR>=MXGATEWAY_OPERATOR_PASSWORD] [--operator-password <pw>]
mxgw bench-read-bulk [--duration-seconds <n>] [--warmup-seconds <n>] [--bulk-size <n>]
mxgw smoke --endpoint http://localhost:5000 --api-key-env MXGATEWAY_API_KEY --item TestChildObject.TestInt
mxgw batch
+34 -2
View File
@@ -215,7 +215,7 @@ enum Command {
server_handle: i32,
#[arg(long, value_delimiter = ',')]
items: Vec<String>,
/// Per-tag snapshot timeout in milliseconds. `0` uses the worker default (1000 ms).
/// Snapshot deadline in milliseconds, shared by all tags in the command. `0` uses the worker default (1000 ms).
#[arg(long, default_value_t = 0)]
timeout_ms: u32,
#[arg(long)]
@@ -374,8 +374,19 @@ enum Command {
reference: String,
#[arg(long, default_value = "")]
comment: String,
#[arg(long, default_value = "")]
/// Galaxy account of the acknowledging operator (required; the gateway
/// rejects an empty operator so the real operator is recorded).
#[arg(long)]
operator: String,
/// Galaxy domain of the operator; empty authenticates the bare user name.
#[arg(long, default_value = "")]
operator_domain: String,
/// Operator password (prefer `--operator-password-env`; empty is allowed).
#[arg(long)]
operator_password: Option<String>,
/// Environment variable holding the operator password.
#[arg(long, default_value = "MXGATEWAY_OPERATOR_PASSWORD")]
operator_password_env: String,
#[arg(long)]
json: bool,
},
@@ -1081,8 +1092,16 @@ async fn dispatch(command: Command) -> Result<(), Error> {
reference,
comment,
operator,
operator_domain,
operator_password,
operator_password_env,
json,
} => {
// An empty password is allowed (Galaxy accounts may have one), so
// an unset flag and env resolve to an empty string.
let operator_password = operator_password
.or_else(|| env::var(&operator_password_env).ok())
.unwrap_or_default();
let client = connect(connection).await?;
let reply = client
.acknowledge_alarm(AcknowledgeAlarmRequest {
@@ -1090,6 +1109,8 @@ async fn dispatch(command: Command) -> Result<(), Error> {
alarm_full_reference: reference,
comment,
operator_user: operator,
operator_password,
operator_domain,
})
.await?;
print_acknowledge_alarm_reply(&reply, json);
@@ -2771,6 +2792,17 @@ mod tests {
assert!(parsed.is_ok(), "parse failed: {parsed:?}");
}
#[test]
fn acknowledge_alarm_requires_operator() {
let parsed = Cli::try_parse_from([
"mxgw",
"acknowledge-alarm",
"--reference",
"Tank01.Level.HiHi",
]);
assert!(parsed.is_err());
}
#[test]
fn acknowledge_alarm_requires_reference() {
let parsed = Cli::try_parse_from(["mxgw", "acknowledge-alarm"]);
+50 -17
View File
@@ -130,6 +130,7 @@ message MxCommand {
WriteSecuredBulkCommand write_secured_bulk = 41;
WriteSecured2BulkCommand write_secured2_bulk = 42;
ReadBulkCommand read_bulk = 43;
UpdateAlarmWatchListCommand update_alarm_watch_list = 44;
PingCommand ping = 100;
GetSessionStateCommand get_session_state = 101;
GetWorkerInfoCommand get_worker_info = 102;
@@ -174,6 +175,7 @@ enum MxCommandKind {
MX_COMMAND_KIND_WRITE_SECURED_BULK = 32;
MX_COMMAND_KIND_WRITE_SECURED2_BULK = 33;
MX_COMMAND_KIND_READ_BULK = 34;
MX_COMMAND_KIND_UPDATE_ALARM_WATCH_LIST = 35;
MX_COMMAND_KIND_PING = 100;
MX_COMMAND_KIND_GET_SESSION_STATE = 101;
MX_COMMAND_KIND_GET_WORKER_INFO = 102;
@@ -344,8 +346,8 @@ enum AlarmProviderMode {
// Subscription expression follows the canonical
// `\\<machine>\Galaxy!<area>` format (literal "Galaxy" provider). The
// worker spins up a wnwrapConsumer-backed subscription on its STA on
// first call; subsequent calls are an error (use UnsubscribeAlarms then
// SubscribeAlarms to reconfigure).
// first call; subsequent calls are an error. Use UpdateAlarmWatchList to
// replace the standby targets without restarting the primary subscription.
message SubscribeAlarmsCommand {
string subscription_expression = 1;
// UNSPECIFIED = auto-failover/failback. ALARMMGR/SUBTAG force one provider.
@@ -354,6 +356,17 @@ message SubscribeAlarmsCommand {
// alarmmgr mode; in subtag mode it bounds what the consumer can observe.
repeated AlarmSubtagTarget watch_list = 3;
AlarmFailoverConfig failover = 4;
// True when auto mode should retain subtag-fallback capability even if the
// gateway's initial discovery produced an empty watch list. False leaves an
// empty list as an intentional alarm-manager-only subscription.
bool fallback_enabled = 5;
}
// Replace the live alarm subscription's subtag standby targets. The worker
// constructs and attaches the replacement before swapping it into service, so
// a setup failure leaves the existing standby and active provider unchanged.
message UpdateAlarmWatchListCommand {
repeated AlarmSubtagTarget watch_list = 1;
}
// Tear down the worker's alarm consumer. No-op if no subscription is
@@ -378,10 +391,13 @@ message AlarmFailoverConfig {
int32 failback_stable_probes = 3; // clean probes before switching back (>=1)
}
// Acknowledge a single alarm by its GUID. Operator identity fields are
// recorded atomically with the ack transition in the alarm-history log.
// The reply's hresult / native_status surfaces AVEVA's
// AlarmAckByGUID return code.
// Acknowledge a single alarm by its GUID. The worker resolves the GUID to the
// alarm's Galaxy reference from its active-alarm snapshot and acknowledges it
// by writing `comment` to `<TAGNAME>.AckMsg` over MXAccess after
// AuthenticateUser as the operator, so the alarm history records the real
// operator. The reply's hresult / native_status is 0 when the write was
// applied (or sent but unconfirmed), otherwise the AuthenticateUser HRESULT or
// the OnWriteComplete status detail.
message AcknowledgeAlarmCommand {
// Canonical 8-4-4-4-12 GUID string (e.g. "BCC47053-9542-4D65-BDAA-BCDEA6A32A73").
string alarm_guid = 1;
@@ -390,6 +406,10 @@ message AcknowledgeAlarmCommand {
string operator_node = 4;
string operator_domain = 5;
string operator_full_name = 6;
// Operator credential used for AuthenticateUser before the AckMsg write.
// Implementations must keep this field out of logs, metrics labels, command
// lines, and diagnostics.
string operator_password = 7;
}
// Snapshot the currently-active alarm set. Optional filter prefix scopes
@@ -401,9 +421,7 @@ message QueryActiveAlarmsCommand {
// Acknowledge a single alarm by its (name, provider, group) tuple. Used
// when the public RPC's AlarmFullReference (Provider!Group.Tag) cannot
// be resolved to a GUID directly. The worker invokes
// wwAlarmConsumerClass.AlarmAckByName which reaches the same alarm
// history path as AlarmAckByGUID.
// be resolved to a GUID directly. The worker acknowledges by writing `comment` to `<alarm_name>.AckMsg` (Galaxy provider only) after AuthenticateUser as the operator.
message AcknowledgeAlarmByNameCommand {
// Tag/alarm name (e.g. "TestMachine_001.TestAlarm001"). Tag itself
// may contain dots; the gateway-side parser splits on the first dot
@@ -418,6 +436,10 @@ message AcknowledgeAlarmByNameCommand {
string operator_node = 6;
string operator_domain = 7;
string operator_full_name = 8;
// Operator credential used for AuthenticateUser before the AckMsg write.
// Implementations must keep this field out of logs, metrics labels, command
// lines, and diagnostics.
string operator_password = 9;
}
message UnsubscribeBulkCommand {
@@ -537,11 +559,14 @@ message MxCommandReply {
MxValue return_value = 6;
// Correlated per-item outcome rows. For WRITE / WRITE2 / WRITE_SECURED /
// WRITE_SECURED2 replies the worker holds the reply for a bounded window
// (default 1.5 s, MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS) waiting for
// (default 1.5 s, MxGateway:Worker:WriteCompletionWaitMilliseconds) waiting for
// the matching MXAccess OnWriteComplete callback and copies its status rows
// here, so statuses[0] carries the real MXAccess commit outcome (success OR
// failure) while protocol_status/hresult still describe command acceptance
// only. Empty statuses on a write reply means the completion did not arrive
// only. Callback correlation is recency-based per item handle: after the
// bounded wait, a sequential write's callback supersedes a timed-out
// predecessor, and a later stale callback for that predecessor is ignored.
// Empty statuses on a write reply means the completion did not arrive
// within the window — the write is unconfirmed, not failed. Correlation is
// best-effort per (server_handle, item_handle): MXAccess's callback carries
// no transaction id, so concurrent writes to the same item within the
@@ -709,11 +734,11 @@ message DrainEventsReply {
// Reply payload for AcknowledgeAlarmCommand AND
// AcknowledgeAlarmByNameCommand — both ack command kinds reuse this
// payload case (`MxCommandReply.acknowledge_alarm`); there is no
// dedicated by-name reply case. Surfaces AVEVA's native ack return
// code (AlarmAckByGUID for the GUID arm, AlarmAckByName for the
// by-name arm); 0 means success. The MxCommandReply's hresult field
// dedicated by-name reply case. Surfaces the ack outcome: 0 when the
// AckMsg write was applied or sent but unconfirmed; otherwise the
// AuthenticateUser HRESULT or the OnWriteComplete status detail. The MxCommandReply's hresult field
// carries the same value and is preferred for protocol consumers —
// this payload exists so the gateway-side WorkerAlarmRpcDispatcher
// this payload exists so GatewayAlarmMonitor
// can echo native_status into AcknowledgeAlarmReply.hresult without
// unpacking the outer envelope.
message AcknowledgeAlarmReplyPayload {
@@ -971,6 +996,15 @@ message AcknowledgeAlarmRequest {
// Operator principal performing the acknowledgement. The lmxopcua side
// resolves this from the OPC UA session prior to invoking the RPC.
string operator_user = 5;
// Operator's Galaxy (or OS) password. The gateway authenticates as
// operator_user with it before writing the acknowledgement so the alarm
// history records the real operator. May be empty for accounts without a
// password. Credential-sensitive: implementations must keep this field out
// of logs, metrics labels, command lines, and diagnostics.
string operator_password = 6;
// Optional domain for OS-authenticated galaxies. When set the gateway
// authenticates as "<operator_domain>\<operator_user>".
string operator_domain = 7;
}
message AcknowledgeAlarmReply {
@@ -981,8 +1015,7 @@ message AcknowledgeAlarmReply {
ProtocolStatus protocol_status = 3;
// Native ack return code echoed from the worker. The worker carries the
// ack outcome as a single int32 (AcknowledgeAlarmReplyPayload.native_status,
// = AlarmAckByName / AlarmAckByGUID return code; 0 = success); the gateway's
// WorkerAlarmRpcDispatcher copies that value here. This is the authoritative
// 0 = applied or sent; otherwise the AuthenticateUser HRESULT or OnWriteComplete detail); GatewayAlarmMonitor copies that value here. This is the authoritative
// ack-outcome field for the public RPC. Absent only when the worker reply
// omitted the value entirely (a protocol violation).
optional int32 hresult = 4;
+16 -7
View File
@@ -187,7 +187,7 @@ impl GatewayClient {
})?;
endpoint = endpoint.connect_timeout(options.connect_timeout());
if let Some(tls) = build_tls_config(&options)? {
if let Some(tls) = build_tls_config(&options).await? {
endpoint = endpoint.tls_config(tls)?;
}
@@ -327,6 +327,9 @@ impl GatewayClient {
/// MXAccess session; the resulting native MxStatus is returned in the
/// reply. Acks are idempotent at the MxAccess layer.
///
/// `operator_user` is required; `operator_password` may be empty. The
/// password is scrubbed from any returned error's `Display`/`Debug`.
///
/// # Errors
///
/// Returns [`Error::ProtocolStatus`] when the gateway accepts the call but
@@ -336,13 +339,19 @@ impl GatewayClient {
&self,
request: AcknowledgeAlarmRequest,
) -> Result<AcknowledgeAlarmReply, Error> {
let secret = request.operator_password.clone();
let mut client = self.inner.clone();
let response = client
.acknowledge_alarm(self.unary_request(request))
.await?;
let reply = response.into_inner();
ensure_protocol_success("acknowledge alarm", reply.protocol_status.as_ref())?;
Ok(reply)
let result = async {
let response = client
.acknowledge_alarm(self.unary_request(request))
.await?;
let reply = response.into_inner();
ensure_protocol_success("acknowledge alarm", reply.protocol_status.as_ref())?;
Ok(reply)
}
.await;
// The operator password must never surface in a returned error.
result.map_err(|error: Error| error.redact_secrets(&[secret.as_str()]))
}
/// Open the server-streaming `QueryActiveAlarms` RPC — the gateway's
+70
View File
@@ -314,6 +314,68 @@ impl std::fmt::Display for MxAccessError {
impl std::error::Error for MxAccessError {}
impl Error {
/// Scrub exact caller-supplied secrets (for example an operator password)
/// from the error's rendered text and any wrapped `tonic::Status`, so that
/// neither `Display` nor `Debug` can reveal them. Empty secrets are ignored.
#[must_use]
pub fn redact_secrets(self, secrets: &[&str]) -> Self {
let scrub = |text: &str| {
let mut out = text.to_owned();
for secret in secrets.iter().filter(|secret| !secret.is_empty()) {
out = out.replace(*secret, "<redacted>");
}
out
};
let scrub_status = |status: Box<tonic::Status>| {
Box::new(tonic::Status::new(status.code(), scrub(status.message())))
};
match self {
Self::Authentication { message, status } => Self::Authentication {
message: scrub(&message),
status: scrub_status(status),
},
Self::Authorization { message, status } => Self::Authorization {
message: scrub(&message),
status: scrub_status(status),
},
Self::Timeout { message, status } => Self::Timeout {
message: scrub(&message),
status: scrub_status(status),
},
Self::Cancelled { message, status } => Self::Cancelled {
message: scrub(&message),
status: scrub_status(status),
},
Self::Unavailable { message, status } => Self::Unavailable {
message: scrub(&message),
status: scrub_status(status),
},
Self::Status(status) => Self::Status(scrub_status(status)),
Self::ProtocolStatus {
operation,
code,
message,
} => Self::ProtocolStatus {
operation,
code,
message: scrub(&message),
},
Self::MalformedReply { detail } => Self::MalformedReply {
detail: scrub(&detail),
},
Self::InvalidArgument { name, detail } => Self::InvalidArgument {
name,
detail: scrub(&detail),
},
Self::MxAccess(error) => Self::MxAccess(Box::new(
error.with_secrets(secrets.iter().map(|secret| (*secret).to_owned()).collect()),
)),
other => other,
}
}
}
impl From<tonic::Status> for Error {
fn from(status: tonic::Status) -> Self {
let message = redact_credentials(status.message());
@@ -461,6 +523,14 @@ fn redact_credentials(message: &str) -> String {
#[cfg(test)]
mod tests {
#[test]
fn redact_secrets_scrubs_status_display_and_debug() {
let error: super::Error = tonic::Status::internal("boom pw-secret").into();
let error = error.redact_secrets(&["pw-secret", ""]);
assert!(!format!("{error:?}").contains("pw-secret"));
assert!(!error.to_string().contains("pw-secret"));
}
use tonic::{Code, Status};
use super::{ensure_mxaccess_success, Error};
+1 -1
View File
@@ -182,7 +182,7 @@ impl GalaxyClient {
})?;
endpoint = endpoint.connect_timeout(options.connect_timeout());
if let Some(tls) = build_tls_config(&options)? {
if let Some(tls) = build_tls_config(&options).await? {
endpoint = endpoint.tls_config(tls)?;
}
+9 -3
View File
@@ -13,7 +13,10 @@ pub mod mxaccess_gateway {
/// `mxaccess_gateway.v1` package — the v1 wire contract surfaced by the
/// gateway to language clients.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
// tonic's generated server traits use async_trait, whose expansion adds
// redundant must_use attributes to Future returns under Rust 1.99.
#![allow(clippy::double_must_use)]
tonic::include_proto!("mxaccess_gateway.v1");
}
@@ -24,7 +27,7 @@ pub mod mxaccess_worker {
/// `mxaccess_worker.v1` package — frame and envelope types used inside
/// the named-pipe transport between gateway and worker.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
tonic::include_proto!("mxaccess_worker.v1");
}
@@ -35,7 +38,10 @@ pub mod galaxy_repository {
/// `galaxy_repository.v1` package — types for the Galaxy hierarchy
/// discovery and deploy-event watch RPCs.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
// tonic's generated server traits use async_trait, whose expansion adds
// redundant must_use attributes to Future returns under Rust 1.99.
#![allow(clippy::double_must_use)]
tonic::include_proto!("galaxy_repository.v1");
}
+28 -15
View File
@@ -3,7 +3,6 @@
//! chain of `with_*` setters; the `Debug` impl redacts the API key.
use std::fmt;
use std::fs;
use std::path::PathBuf;
use std::time::Duration;
@@ -235,7 +234,9 @@ pub(crate) fn tls_trust_decision(options: &ClientOptions) -> TlsTrustDecision {
/// rejects the configuration early with an actionable error. The strict opt-in
/// instead loads the system trust roots so a certificate chaining to an
/// already-trusted root validates.
pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientTlsConfig>, Error> {
pub(crate) async fn build_tls_config(
options: &ClientOptions,
) -> Result<Option<ClientTlsConfig>, Error> {
let decision = tls_trust_decision(options);
if decision == TlsTrustDecision::None {
return Ok(None);
@@ -248,10 +249,13 @@ pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientT
match decision {
TlsTrustDecision::PinnedCa => {
let ca_file = options.ca_file().expect("PinnedCa implies a CA file");
let certificate = fs::read(ca_file).map_err(|source| Error::InvalidEndpoint {
endpoint: options.endpoint().to_owned(),
detail: format!("failed to read CA file {}: {source}", ca_file.display()),
})?;
let certificate =
tokio::fs::read(ca_file)
.await
.map_err(|source| Error::InvalidEndpoint {
endpoint: options.endpoint().to_owned(),
detail: format!("failed to read CA file {}: {source}", ca_file.display()),
})?;
tls = tls.ca_certificate(Certificate::from_pem(certificate));
}
TlsTrustDecision::SystemRoots => {
@@ -334,11 +338,11 @@ mod tests {
assert!(!debug.contains("mxgw_secret"));
}
#[test]
fn plaintext_needs_no_tls() {
#[tokio::test]
async fn plaintext_needs_no_tls() {
let options = ClientOptions::new("http://127.0.0.1:5000").with_plaintext(true);
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::None);
assert!(build_tls_config(&options).unwrap().is_none());
assert!(build_tls_config(&options).await.unwrap().is_none());
}
#[test]
@@ -349,8 +353,8 @@ mod tests {
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::PinnedCa);
}
#[test]
fn strict_without_ca_uses_system_roots() {
#[tokio::test]
async fn strict_without_ca_uses_system_roots() {
// Regression for Client.Rust-031: strict verification with no pinned CA
// must verify against the system trust roots, not produce a config with
// zero trust anchors. The trust decision proves roots are consulted; the
@@ -365,15 +369,24 @@ mod tests {
"strict-no-CA must request the system trust roots"
);
assert!(
build_tls_config(&options).unwrap().is_some(),
build_tls_config(&options).await.unwrap().is_some(),
"strict-no-CA must build a usable TLS config"
);
}
#[test]
fn lenient_without_ca_is_rejected() {
#[tokio::test]
async fn lenient_without_ca_is_rejected() {
let options = ClientOptions::new("https://127.0.0.1:5000").with_plaintext(false);
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::RejectNoCa);
assert!(build_tls_config(&options).is_err());
assert!(build_tls_config(&options).await.is_err());
}
#[tokio::test]
async fn missing_pinned_ca_file_errors_without_blocking_read() {
let options = ClientOptions::new("https://127.0.0.1:5000")
.with_plaintext(false)
.with_ca_file("/nonexistent/definitely-missing-ca.pem");
let error = build_tls_config(&options).await.unwrap_err();
assert!(error.to_string().contains("failed to read CA file"));
}
}
+96 -8
View File
@@ -17,14 +17,15 @@ use crate::generated::mxaccess_gateway::v1::mx_command_reply;
use crate::generated::mxaccess_gateway::v1::{
ActivateCommand, AddBufferedItemCommand, AddItem2Command, AddItemBulkCommand, AddItemCommand,
AdviseCommand, AdviseItemBulkCommand, AdviseSupervisoryCommand, ArchestrAUserToIdCommand,
AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest, MxCommand,
MxCommandKind, MxCommandReply, MxCommandRequest, MxDataType, MxSparseArray, MxSparseElement,
MxValue as ProtoMxValue, OpenSessionRequest, ReadBulkCommand, RegisterCommand,
RemoveItemBulkCommand, RemoveItemCommand, SetBufferedUpdateIntervalCommand,
AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest,
DrainEventsCommand, GetSessionStateCommand, GetWorkerInfoCommand, MxCommand, MxCommandKind,
MxCommandReply, MxCommandRequest, MxDataType, MxEvent, MxSparseArray, MxSparseElement,
MxValue as ProtoMxValue, OpenSessionRequest, PingCommand, ReadBulkCommand, RegisterCommand,
RemoveItemBulkCommand, RemoveItemCommand, SessionState, SetBufferedUpdateIntervalCommand,
StreamEventsRequest, SubscribeBulkCommand, SubscribeResult, SuspendCommand, UnAdviseCommand,
UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, Write2BulkCommand,
Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry, WriteCommand,
WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, WorkerInfoReply,
Write2BulkCommand, Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry,
WriteCommand, WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
WriteSecuredBulkCommand, WriteSecuredBulkEntry, WriteSecuredCommand,
};
use crate::value::{MxStatus, MxValue, MxValueProjection};
@@ -532,7 +533,10 @@ impl Session {
/// returning the most recent cached `OnDataChange` value when the tag is
/// already advised (`was_cached = true`), or by taking a full AddItem +
/// Advise + wait + UnAdvise + RemoveItem snapshot lifecycle otherwise.
/// `timeout_ms == 0` lets the worker pick its default (1000 ms).
/// `timeout_ms` is one shared deadline for the whole command, not a
/// per-tag budget: the worker advises every uncached tag, waits once for
/// all of them, then tears them down. `timeout_ms == 0` lets the worker
/// pick its default (1000 ms).
/// Per-tag failures appear as `BulkReadResult` entries with
/// `was_successful = false`; the call never errors on per-tag failure.
///
@@ -960,6 +964,90 @@ impl Session {
.await
}
/// Send a `Ping` diagnostic command to the worker. A successful return
/// means the worker answered; `message` is forwarded for log correlation.
///
/// # Errors
///
/// Returns [`Error::Command`] for non-OK statuses plus transport errors.
pub async fn ping(&self, message: &str) -> Result<(), Error> {
self.invoke(
MxCommandKind::Ping,
Payload::Ping(PingCommand {
message: message.to_owned(),
}),
)
.await?;
Ok(())
}
/// Fetch the gateway-side [`SessionState`] of this session.
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a session-state
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn get_session_state(&self) -> Result<SessionState, Error> {
let reply = self
.invoke(
MxCommandKind::GetSessionState,
Payload::GetSessionState(GetSessionStateCommand {}),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::SessionState(state)) => {
Ok(SessionState::try_from(state.state).unwrap_or(SessionState::Unspecified))
}
_ => Err(Error::MalformedReply {
detail: "get_session_state reply did not carry a SessionState payload".to_owned(),
}),
}
}
/// Fetch worker process details (pid, version, MXAccess ProgID/CLSID).
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a worker-info
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn get_worker_info(&self) -> Result<WorkerInfoReply, Error> {
let reply = self
.invoke(
MxCommandKind::GetWorkerInfo,
Payload::GetWorkerInfo(GetWorkerInfoCommand {}),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::WorkerInfo(info)) => Ok(info),
_ => Err(Error::MalformedReply {
detail: "get_worker_info reply did not carry a WorkerInfo payload".to_owned(),
}),
}
}
/// Drain up to `max_events` queued events through the diagnostic command
/// (distinct from the streaming [`events`](Self::events) surface). The
/// reply is bounded server-side, so call repeatedly until it is empty.
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a drain-events
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn drain_events(&self, max_events: u32) -> Result<Vec<MxEvent>, Error> {
let reply = self
.invoke(
MxCommandKind::DrainEvents,
Payload::DrainEvents(DrainEventsCommand { max_events }),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::DrainEvents(drain)) => Ok(drain.events),
_ => Err(Error::MalformedReply {
detail: "drain_events reply did not carry a DrainEvents payload".to_owned(),
}),
}
}
/// Issue a raw `Invoke` for an arbitrary command, without filtering on
/// the protocol status. Useful when callers need the full reply for
/// commands not yet wrapped by `Session`.
+163 -6
View File
@@ -26,12 +26,12 @@ use zb_mom_ww_mxgateway_client::generated::mxaccess_gateway::v1::{
AddItem2Reply, AddItemReply, AlarmConditionState, AlarmFeedMessage, AlarmTransitionKind,
AuthenticateUserCommand, AuthenticateUserReply, BulkReadReply, BulkReadResult,
BulkSubscribeReply, BulkWriteReply, BulkWriteResult, CloseSessionReply, CloseSessionRequest,
MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily, MxSparseArray,
MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
DrainEventsReply, MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily,
MxSparseArray, MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
OnAlarmTransitionEvent, OpenSessionReply, OpenSessionRequest, ProtocolStatus,
ProtocolStatusCode, QueryActiveAlarmsRequest, RegisterReply, ReplayGap, SessionState,
StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, Write2BulkEntry, WriteBulkEntry,
WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
SessionStateReply, StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, WorkerInfoReply,
Write2BulkEntry, WriteBulkEntry, WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
};
use zb_mom_ww_mxgateway_client::{
next_correlation_id, ApiKey, ClientOptions, CommandError, Error, EventItem, GatewayClient,
@@ -229,6 +229,8 @@ async fn acknowledge_alarm_returns_reply_with_native_status() {
alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
comment: "investigating".to_owned(),
operator_user: "alice".to_owned(),
operator_password: String::new(),
operator_domain: String::new(),
})
.await
.unwrap();
@@ -548,6 +550,81 @@ async fn unary_invoke_maps_status_unavailable_to_error_unavailable() {
);
}
#[tokio::test]
async fn ping_round_trips_through_the_fake_gateway() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
session.ping("hello").await.unwrap();
assert_eq!(
*state.last_command_kind.lock().await,
Some(MxCommandKind::Ping as i32)
);
}
#[tokio::test]
async fn get_session_state_returns_typed_state() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let session_state = session.get_session_state().await.unwrap();
assert_eq!(session_state, SessionState::Ready);
}
#[tokio::test]
async fn get_worker_info_returns_typed_reply() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let info = session.get_worker_info().await.unwrap();
assert_eq!(info.worker_process_id, 4242);
assert_eq!(info.worker_version, "1.2.3");
}
#[tokio::test]
async fn drain_events_returns_events() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let events = session.drain_events(10).await.unwrap();
assert_eq!(events.len(), 1);
}
#[tokio::test]
async fn get_session_state_malformed_without_payload() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
*state.invoke_override.lock().await = Some(InvokeOverride::OkWithoutPayload);
let error = session.get_session_state().await.unwrap_err();
assert!(matches!(error, Error::MalformedReply { .. }));
}
#[tokio::test]
async fn read_bulk_round_trips_through_the_fake_gateway() {
let state = Arc::new(FakeState::default());
@@ -1036,6 +1113,30 @@ async fn stream_alarms_emits_snapshot_then_complete_then_transition_in_order() {
));
}
#[tokio::test]
async fn acknowledge_alarm_error_does_not_leak_operator_password() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let err = client
.acknowledge_alarm(AcknowledgeAlarmRequest {
client_correlation_id: "corr-pw".to_owned(),
alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
comment: String::new(),
operator_user: "alice".to_owned(),
operator_password: "pw-secret".to_owned(),
operator_domain: String::new(),
})
.await
.unwrap_err();
assert!(!format!("{err:?}").contains("pw-secret"));
assert!(!err.to_string().contains("pw-secret"));
}
#[tokio::test]
async fn cli_subcommands_propagate_unique_correlation_ids_from_next_correlation_id() {
// The CLI's `stream-alarms` and `acknowledge-alarm` paths used to
@@ -1073,6 +1174,8 @@ async fn cli_subcommands_propagate_unique_correlation_ids_from_next_correlation_
alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
comment: String::new(),
operator_user: String::new(),
operator_password: String::new(),
operator_domain: String::new(),
})
.await
.unwrap();
@@ -1309,6 +1412,57 @@ impl MxAccessGateway for FakeGateway {
}));
}
if kind == MxCommandKind::Ping as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::GetSessionState as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::SessionState(SessionStateReply {
state: SessionState::Ready as i32,
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::GetWorkerInfo as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::WorkerInfo(WorkerInfoReply {
worker_process_id: 4242,
worker_version: "1.2.3".to_owned(),
..WorkerInfoReply::default()
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::DrainEvents as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::DrainEvents(DrainEventsReply {
events: vec![MxEvent::default()],
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::ReadBulk as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
@@ -1424,8 +1578,11 @@ impl MxAccessGateway for FakeGateway {
&self,
request: Request<AcknowledgeAlarmRequest>,
) -> Result<Response<AcknowledgeAlarmReply>, Status> {
*self.state.last_correlation_id.lock().await =
Some(request.into_inner().client_correlation_id);
let request = request.into_inner();
if request.operator_password == "pw-secret" {
return Err(Status::internal("boom pw-secret"));
}
*self.state.last_correlation_id.lock().await = Some(request.client_correlation_id);
Ok(Response::new(AcknowledgeAlarmReply {
correlation_id: "corr-1".to_owned(),
protocol_status: Some(ok_status("ack ok")),
+4 -1
View File
@@ -1,5 +1,8 @@
# Code Review — IntegrationTests
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/mxaccessgw
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
| Field | Value |
|---|---|
| Module | `src/ZB.MOM.WW.MxGateway.IntegrationTests` |
@@ -710,4 +713,4 @@ The prior `DashboardAuthenticator` ctor took `IOptions<GatewayOptions>`, so the
**Recommendation:** Add AddItem2/Write2 to the parity test (or a dedicated test) asserting each produces a real reply (not InvalidRequest) against a valid handle and item-definition split.
**Resolution:** 2026-06-16: requires a live MXAccess rig + provider state not available on this dev box; add the AddItem2/Write2 parity assertions when running on the MXAccess host.
**Resolution:** 2026-06-16: requires a live MXAccess rig + provider state not available on this dev box; add the AddItem2/Write2 parity assertions when running on the MXAccess host. → #212
+139 -38
View File
@@ -621,8 +621,9 @@ returned to normal but is unacknowledged — i.e., visible in the
"current alarms" list because operator hasn't acked it yet) and
`UNACK_ALM` (the alarm is currently active and unacknowledged).
The other states from `eAlmState` (`ACK_RTN`, `ACK_ALM`) would
appear when an ack is performed — `wwAlarmConsumerClass.AlarmAckByGUID`
is the method to call.
appear when an ack is performed. The alarm-client ack methods
(`AlarmAckByGUID` and friends) do not produce them headless; an `AckMsg`
write does (`AlarmProbeFindings.md`).
### `GetStatistics` AV — unrelated quirk
@@ -660,7 +661,9 @@ alarm-consumer surface unblocks A.2 fully. Outline:
events for added/changed/removed records.
- `AlarmAckByGUID(VBGUID, comment, oprName, node, domain,
fullName)` for client-driven acknowledgements (matches
PR A.5's `AlarmAckCommand` payload).
PR A.5's `AlarmAckCommand` payload). *Ack step superseded
2026-10-19: acks write `AckMsg` as the authenticated operator
(see "Acknowledge in subtag mode" and `AlarmProbeFindings.md`).*
- Lifecycle teardown: `DeregisterConsumer` +
`UninitializeConsumer` + `Marshal.FinalReleaseComObject`.
3. **Conversion layer:** map XML record fields to
@@ -701,8 +704,9 @@ alarm-consumer surface unblocks A.2 fully. Outline:
`bRetainHiddenAlarms` parameter at all.
- Subscription expression format: `\\<machine>\Galaxy!<area>`
(literal `Galaxy` provider) for both libraries.
- Native ack: `AlarmAckByGUID(VBGUID guid, comment, oprName,
node, domain, fullName)` on the v2 surface; ID 5-arg
- Native ack surface (not used by the gateway; superseded 2026-10-19 by
the authenticated `AckMsg` write): `AlarmAckByGUID(VBGUID guid,
comment, oprName, node, domain, fullName)` on the v2 surface; ID 5-arg
variant on the legacy `IwwAlarmConsumer` interface.
These findings retire the open follow-up probes from the
@@ -735,25 +739,32 @@ for `SetXmlAlarmQuery`.
the canonical `\\<machine>\Galaxy!<area>` form into the XML's
NODE/PROVIDER/GROUP fields. Mangled or not, the call enables reads.
### 2. Two consumers required: read-side vs. ack-side
### 2. Two consumers required: read-side vs. ack-side (historical)
> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
> see `AlarmProbeFindings.md`. The worker no longer provisions an ack-only
> wnwrap consumer. The text below records why the split existed.
`SetXmlAlarmQuery` enables reads but **breaks `AlarmAckByName` on
the same consumer instance**. With SetXml applied, AlarmAckByName
returns -55 even with valid name+provider+group+operator. Without
SetXml, AlarmAckByName succeeds with rc=0.
The production consumer therefore provisions **two** wnwrap COM
The consumer at the time provisioned **two** wnwrap COM
instances:
- Primary consumer (`client`): runs full lifecycle including
`SetXmlAlarmQuery` for `GetXmlCurrentAlarms2` polls.
- Ack-only consumer (`ackClient`): runs Initialize → Register →
Subscribe via the v1-prefixed methods, **no SetXmlAlarmQuery**.
All `AcknowledgeByName` calls dispatch through this instance.
All by-name ack calls dispatched through this instance.
Both consumers subscribe to the same expression. Disposal cleans up
both via a shared `ReleaseConsumerCom` helper.
### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg
### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg (historical)
> Superseded 2026-10-19: the gateway no longer calls `AlarmAckByName`; see
> `AlarmProbeFindings.md`.
`wwAlarmConsumerClass` exposes two `AlarmAckByName` overloads:
- `IwwAlarmConsumer2` v2: 8 args (`name, provider, group, comment,
@@ -762,8 +773,8 @@ both via a shared `ReleaseConsumerCom` helper.
The v2 8-arg method returns -55 on this AVEVA build regardless of
operator-identity inputs — looks like a stub. The v1 6-arg method
works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the
6-arg overload and discards the proto's `domain` + `full_name` fields.
works. `WnWrapAlarmConsumer.AcknowledgeByName` called the
6-arg overload and discarded the proto's `domain` + `full_name` fields.
**Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves
of the paragraph above are narrower than they read. The -55 tracks the
@@ -775,23 +786,58 @@ snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm
extension's own `.Acked` attribute stay `False` for 16 s after each
call. On this rig the wnwrap ack is accepted and then inert, so a zero
return code must not be read as an applied acknowledgement.
The proto contract keeps the 8 fields for forward compatibility if
AVEVA fixes the v2 method later.
### 4. `AlarmAckByGUID` is not implemented
### 4. `AlarmAckByGUID` is not implemented (historical)
> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
> see `AlarmProbeFindings.md`.
The v2 `AlarmAckByGUID(VBGUID, …)` throws `NotImplementedException`
(COM `E_NOTIMPL`) on `wwAlarmConsumerClass` against this AVEVA
build. The reference→GUID lookup that we initially planned to wire
through `AlarmAckByGUID` is therefore not viable on wnwrap; all acks
must go through `AlarmAckByName`.
through `AlarmAckByGUID` is therefore not viable on wnwrap.
The proto `AcknowledgeAlarmCommand` (GUID-based) and the worker's
`MxAccessCommandExecutor.ExecuteAcknowledgeAlarm` switch arm remain
in the codebase for the forward-compat shape, but the gateway-side
`WorkerAlarmRpcDispatcher.AcknowledgeAsync` now always routes through
`AcknowledgeAlarmByName` when the public RPC supplies a recognizable
`Provider!Group.Tag` reference.
#### Managed GUID ack experiment — did not work (live-tested 2026-10-18, removed 2026-10-19)
An opt-in experiment drove `aaAlarmManagedClient.AlarmClient` by reflection
(its members are non-public) to take GUID-keyed acks in place of the wnwrap
method. The live test showed it could not work, and the experiment and its
enabling flag were removed once the `AckMsg` write replaced it. The findings
are kept because they explain why the alarm-client route was abandoned.
The test raised `TestMachine_001.TestAlarm001` with a secured write,
read its GUID from the wnwrap XML snapshot (the public API exposes no
alarm GUIDs), and acknowledged it through the `AcknowledgeAlarm` RPC
with the experiment enabled:
| Step | Result |
|---|---|
| RPC ack by GUID, experiment off (control) | `NotImplementedException`; `.Acked` stays `False` |
| RPC ack by GUID, experiment on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty |
| `AlarmClient.RegisterConsumer` in a process with a registered wnwrap consumer | `-65` — and wnwrap gets `-65` if the managed client registered first; a real `hWnd` does not help |
| Managed client alone in its own process, records loaded | `AlarmAckByGUID` → `-1`, `SelectByGUID` → `-1` for the wnwrap GUID in both byte orders, `AlarmAckSelected` → `-57` |
Three separate problems follow:
1. **The two libraries cannot share a process.** The worker always runs
a wnwrap reader, so the managed `RegisterConsumer` fails with `-65`.
The dispatcher is built to drop back to the wnwrap ack path on a
lifecycle failure (so alarm reads keep working), and it does so
silently — which is why the experiment-on RPC was indistinguishable from the
control.
2. **The managed client does not recognise the wnwrap GUID.** `-1` from
`SelectByGUID` is "not found", not a security refusal, even with the
client holding records. Getting the client's own identifier for the
alarm is not reliable either: `GetHighPriAlarm` mostly throws the
FILETIME `ArgumentOutOfRangeException` described above. The
`MxSecurityOperate` question was therefore never reached.
3. **The gateway does not send the identity fields.** Its GUID arm fills
`OperatorNode`, `OperatorDomain` and `OperatorFullName` with empty
strings, so the domain and full name the managed method carries
would not arrive through the public RPC even if 1 and 2 were solved.
The route was dropped in favour of the authenticated `AckMsg` write
(`AlarmProbeFindings.md`, "Fourth attempt").
### 5. STA / threading — resolved
@@ -835,7 +881,7 @@ carries `degraded = true`.
### Watch-list discovery
`GatewayAlarmMonitor` resolves the subtag watch-list at subscribe time by
calling `IAlarmWatchListResolver.GetAlarmAttributesAsync`. The resolver merges:
calling `IAlarmWatchListResolver.ResolveAsync`. The resolver merges:
1. Galaxy Repository SQL (`GetAlarmAttributesAsync`) — objects that have alarm
extensions in the configured area.
@@ -847,8 +893,47 @@ The resolved list is a set of `AlarmSubtagTarget` messages sent to the worker
inside `SubscribeAlarmsCommand.watch_list`. Each target carries the composed
MXAccess item addresses for the `InAlarm`, `Acked`, `AckMsg`, and `Priority`
subtags (confirmed AVEVA `AlarmExtension` field names, verified against the live
ZB Galaxy `attribute_definition` rows). The gateway re-runs discovery on its
reconcile cadence and pushes an updated watch-list when the model changes.
ZB Galaxy `attribute_definition` rows).
The gateway refreshes this list on the existing alarm reconciliation cadence
and immediately before publishing a failover-to-subtag status. It compares a
canonical fingerprint of all six target fields and sends
`UpdateAlarmWatchListCommand` only for a material change. The worker constructs,
subscribes, and attaches the replacement standby on its STA before swapping it
in; a setup failure leaves the previous standby and the current primary/subtag
mode unchanged. The gateway advances its acknowledged fingerprint only after an
OK reply, so a rejected update remains degraded and is retried.
`SubscribeAlarmsCommand.fallback_enabled` distinguishes an initially empty but
discovery-enabled fallback from an intentionally alarm-manager-only
subscription. In the former case the worker creates an empty standby that can be
hot-swapped after discovery recovers. `ForceAlarmManager` leaves the field false
and does not create a standby.
Because the list is fixed, the resolution is **graded** rather than merely
logged (GWC-37). `ResolveAsync` returns an `AlarmWatchListResolution` carrying
whether Galaxy Repository discovery was attempted and whether it failed, and the
monitor folds the verdict into the alarm feed's `AlarmProviderStatus`:
- Zero targets while the effective mode can need them (`Auto` or `ForceSubtag`)
sets `degraded = true` with reason `AlarmProviderReasons.SubtagWatchListEmpty`
— the subtag provider covers no alarms.
- Discovery enabled but failing adds
`AlarmProviderReasons.SubtagDiscoveryFailed` and the failure message, so a
Galaxy Repository outage at startup is a standing status rather than one log
line emitted possibly weeks before the failover that needs the list.
- If the worker rejects a material update, the monitor retains the previous
acknowledged fingerprint and reports
`AlarmProviderReasons.SubtagWatchListStale` with the rejection diagnostic.
The next reconciliation or failover edge retries the same resolved targets.
`degraded` on the status is therefore the OR of "the subtag provider is the one
serving" and "the subtag watch-list cannot cover anything". In `Auto` the second
can be true while the mode is still `ALARMMGR`: the live feed is healthy and the
standby behind it is empty. The dashboard renders that combination as the amber
`Alarm Manager (no subtag standby)` badge rather than the failover label, so the
badge never names the wrong live provider. `ForceAlarmManager` never builds a
subtag consumer, so its watch-list is not graded at all.
Each target's canonical `AlarmFullReference` is composed as
`Galaxy!{area}.{reference}` (literal `Galaxy` provider). The `{area}` is the
@@ -908,20 +993,24 @@ AVEVA-internal GUID.
### Acknowledge in subtag mode
`AlarmDispatcher` routes ack calls by active provider mode:
Both provider modes acknowledge through the worker's `AlarmAckWriter`, which
calls `AuthenticateUser` as the requesting operator and writes the comment to
the ack item over MXAccess on its own `LMXProxyServer`:
- **Alarm-manager mode:** `AlarmAckByName` on `wwAlarmConsumerClass` (unchanged).
- **Subtag mode:** `SubtagAlarmConsumer.AcknowledgeByName` resolves the
watch-list entry's `ack_comment_subtag` and issues a `Write(comment)` on
the STA via `LmxSubtagAlarmSource`. Writing the `AckMsg` subtag performs
the acknowledge in AVEVA (`AckMsg` is the confirmed `AlarmExtension` ack-comment
write target).
- **Alarm-manager mode:** the item is `<TagName>.AckMsg` for Galaxy-provider
alarms. An alarm from another provider is not ackable.
- **Subtag mode:** `SubtagAlarmConsumer` resolves the watch-list entry's
`ack_comment_subtag` and the writer writes that item verbatim. Writing the
`AckMsg` subtag performs the acknowledge in AVEVA (`AckMsg` is the confirmed
`AlarmExtension` ack-comment write target).
If the alarm has no writable ack-comment subtag (`AckComment` config key is
empty, or the entry's `ack_comment_subtag` field is empty), the ack call
returns a failure code that the gateway surfaces as `FailedPrecondition`.
`AcknowledgeByGuid` maps the synthetic GUID back to its reference via an
internal dictionary, then calls the same write path.
Because the write is authenticated, the Historian records the operator's name,
domain and full name, and the writer's MXAccess client name
(`MxGateway:Alarms:AckClientName`) as OperatorNode. If the alarm has no ack
target (`AckComment` config key is empty, the entry's `ack_comment_subtag` is
empty, or the reference or synthetic GUID is unknown), the worker reports it
as not found or not ackable and the gateway surfaces `InvalidRequest` with
`hresult` -1.
`SubtagAlarmConsumer.Subscribe` advises the ack-comment subtag alongside the
observed ones (active/acked/priority). This is required: MXAccess rejects a
@@ -939,7 +1028,7 @@ The subtag path was validated against live MXAccess on the dev rig
`…​.Priority` → `500` (Int32), `…​.AckMsg` → string — confirming the field
names **and** the runtime reference shape `<Object>.<AlarmAttr>.<field>`
with **no** intermediate alarm-condition segment.
- `AcknowledgeByName` (AckMsg write) returned `0` once the ack-comment subtag
- The `AckMsg` write returned `0` once the ack-comment subtag
was advised — confirming the ack-by-comment-write mechanism end to end.
### Fidelity limitations
@@ -962,6 +1051,18 @@ Every transition and snapshot record carries `degraded = true` and
`source_provider = ALARM_PROVIDER_MODE_SUBTAG`. Clients that require full
fidelity must wait for failback to the alarm manager.
### Retrigger transitions
`ALARM_TRANSITION_KIND_RETRIGGER` is reserved in the contract and the worker
never emits it, in either provider mode. The alarmmgr XML reports only the
four states `UNACK_ALM`, `ACK_ALM`, `UNACK_RTN`, and `ACK_RTN`, so a condition
that returns to alarm after clearing (`*_RTN` → `*_ALM`) looks the same as a
new raise, and `AlarmRecordTransitionMapper.MapTransition` reports it as
`ALARM_TRANSITION_KIND_RAISE`. The subtag source has no re-alarm counter to
observe either. The worker does not infer a retrigger from timing or
from a raise that follows a clear, because the gateway forwards only
transitions its source reports.
### Provider mode reflection
When `FailoverAlarmConsumer` switches between providers, it raises
+127 -2
View File
@@ -24,13 +24,15 @@ the blocker rather than rediscovering it.
|---|---|
| GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) |
| GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID |
| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state; 2026-08-18 (third attempt) identifies the reason as the test attribute's `MxSecurityOperate` classification, which no non-interactive ack surface on the rig can satisfy |
| GUID stable across `UNACK → ACK` | Blocker resolved 2026-10-19 — the rig is driven to `ACK_ALM` by an `AckMsg` write, and the `MxSecurityOperate` classification was not the cause (see "Fourth attempt"); GUID survival across that transition has not been recorded yet |
| Acknowledge an alarm headless | Answered 2026-10-19 — **an `AckMsg` write as the authenticated operator**; every alarm-client ack is inert (see "Fourth attempt") |
| `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** |
The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The
2026-08-18 run cleared that blocker with `AuthenticateUser` + `WriteSecured` and answered
two of the three questions; the acknowledge leg is now blocked on something narrower and
different, described in "Second attempt" and diagnosed in "Third attempt".
different, described in "Second attempt" and diagnosed in "Third attempt"
(resolved in the Fourth attempt, 2026-10-19).
## First attempt (2026-08-17): plain `Write`
@@ -420,3 +422,126 @@ The second attempt's inventory, unchanged since: the three
snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only
`UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on
the next `alarmmgr` restart.
## Fourth attempt (2026-10-19): authenticated AckMsg write
The third attempt left the acknowledge leg blocked. This run found a route that
acknowledges and identifies the operator, and ruled out the alarm-client routes.
It drove `TestMachine_001.TestAlarm001`, raised with
`mxa write TestMachine_001.TestAlarm001 true --type bool --secured -u Administrator`.
### Writing `AckMsg` acknowledges
Writing the comment to `TestMachine_001.TestAlarm001.AckMsg` over MXAccess
acknowledges the alarm: `.Acked` becomes `True` and the wnwrap record moves to
`STATE=ACK_ALM` with an `ACK_COMMENT`. `AckMsg` is `MxSecurityFreeAccess` even
though the alarm UDA itself is `MxSecurityOperate`, so no Galaxy change is
needed.
What the Historian records, read from `Runtime.dbo.v_AlarmEventHistory2`:
| Write | Operator | DomainName | UserFullName | OperatorNode | Description |
|---|---|---|---|---|---|
| Unauthenticated `Write` | `NULL` | `NULL` | `NULL` | `NULL` | the comment |
| `AuthenticateUser` + `Write(value, userId)` | `Administrator` | `ArchestrA` | `Administrator` | `mxa` (the MXAccess `Register()` client name) | the comment |
The wnwrap XML also gains `ACK_OPERATOR_NAME`, `ACK_DOMAIN_NAME`,
`ACK_OPERATOR_NODE` and `ACK_COMMENT` on the acknowledged record; the worker's
snapshot parser reads them for Acknowledge transitions. An unauthenticated write
therefore acknowledges but leaves no operator in the Historian, which is why the
gateway requires `operator_user` and authenticates it.
Another actor's `AckMsg` writes on the same day (authenticated as another
operator account) logged that account as `Operator`, `Domain="."` and that machine's name as the node,
which matches the client-name rule. By this date unauthenticated writes to
`MxSecurityOperate` attributes were also accepted Galaxy-wide on the rig, where
August refused them with 1008.
### Alarm-client acks are inert in every variant
All of these returned `rc=0` or `-1` and changed neither the snapshot, `.Acked`
nor the Historian:
- wnwrap `AlarmAckByName` and `AlarmAckByGUID` (`E_NOTIMPL`) on the ack-only and
reader consumers.
- wnwrap `IwwAlarmConsumer2` on a sole consumer with `SetXmlAlarmQuery` applied
(list populated, `GetAlarm(0)` returns the row): `AlarmAck` with ids
0/1/2/3/-1/65536, and `Select(0)` + `AlarmAckSelected`, with operators
`TestOperator`, `Administrator` and a third account. `SelectByGUID` is `E_NOTIMPL`.
- The same calls with `TestAlarm001` temporarily reclassified to
`MxSecurityFreeAccess` (template and instance, restored afterwards), so the
security classification is not the cause.
- Managed `aaAlarmManagedClient`: `AlarmAckByGUID` and `SelectByGUID` return
`-1` (the client's list never fills headless, even with a real `hWnd` and a
message pump); `AckSelectedTagAlarms_Ex` returns `0` even for a
`NOSUCHPROVIDER` provider and writes no Historian row. The managed and wnwrap
consumers cannot share a process (`RegisterConsumer` returns `-65`).
- A 20 s continuous message pump after `AlarmAck`, and registering as `InTouch`,
`WindowViewer` or `Wonderware InTouch`.
Static reverse engineering matched: the five-string COM acks forward domain and
full name with an empty password, and InTouch's `AlarmViewer.ocx` also sends an
empty password. The user-request call is synchronous request-to-return-code with
no completion channel, so the drop happens provider-side. Hooking the user
request in a live WindowViewer that acknowledges successfully and diffing it
against the headless probe is the one definitive step not taken.
### Consequence
The gateway acknowledges by authenticating the operator and writing `AckMsg`
(`docs/DesignDecisions.md`, "Alarm acknowledgement writes AckMsg as the
authenticated operator"). The managed ack consumer and the second wnwrap ack
consumer were removed.
### Gateway-level verification
The same result holds end to end through the gateway. With the feature build deployed on
the windev rig and `TestAlarm001` raised, an `AcknowledgeAlarm` RPC for
`Galaxy!TestArea.TestMachine_001.TestAlarm001` with `operator_user=Administrator` and an
empty password returned `PROTOCOL_STATUS_CODE_OK`, `hresult` 0. `.Acked` turned True,
`.AckMsg` held the comment, and the Historian logged the `ACK_ALM` row with
`Operator=Administrator`, `DomainName=ArchestrA`, `UserFullName=Administrator`,
`OperatorNode=MxGateway` (the default `AckClientName`), and `Description` = the comment.
A wrong password, or a user Galaxy security does not know, makes `AuthenticateUser`
throw `ArgumentException` (`E_INVALIDARG`, 0x80070057) rather than a `COMException`.
That is why the writer treats `ArgumentException` from the MXAccess call itself as an
interop failure alongside `COMException`, and reports the HRESULT the exception carries.
Such an ack returns `PROTOCOL_STATUS_CODE_MXACCESS_FAILURE`, the alarm stays unacked,
the Historian gains no ack row, and the password string does not appear in the gateway
log, `stdout.log`, or `stderr.log`. A non-Galaxy provider reference returns
`PROTOCOL_STATUS_CODE_INVALID_REQUEST` with `hresult` -1, and an empty `operator_user`
returns gRPC `InvalidArgument`.
The GUID arm of the ack command was not driven live. The public alarm RPCs expose only
the full reference, so clients reach the by-name arm; worker unit tests cover the GUID
arm.
## Item handle reuse
`AlarmAckWriter` correlates an ack with its `OnWriteComplete` by the
(server handle, item handle) pair, because the callback carries no per-write id.
After an unconfirmed ack it drops the `AckMsg` item and re-adds it so the next
ack waits on a new handle that a late completion for the old one cannot match.
That only works if MXAccess does not hand the same number out again.
`ItemHandleReuseProbeTests` (Skip-gated, worker tests) checked this on windev
(`10.100.0.48`, Galaxy `DEV`) on 2026-10-20 against
`TestMachine_001.TestAlarm001.AckMsg`, without writing:
| Sequence on one registration (server handle 1) | Item handles returned |
|---|---|
| `AddItem` + `Advise`, then `UnAdvise` + `RemoveItem`, five times | 1, 2, 3, 4, 5 |
| `AddItem`, then `RemoveItem`, five times | 6, 7, 8, 9, 10 |
| Add and remove `AckMsg`, add `Acked`, add `AckMsg` again | 11, then 12 (`Acked`), 13 |
| Add `AckMsg` twice while both are held | 14, 15 |
Within a registration the numbers increase and a freed number is never reused,
so the drop-and-re-add mitigation holds. Numbering is per registration:
`Unregister` + `Register` on the same COM object returned server handle 2 and
item handle 1, and a new `LMXProxyServerClass` returned server handle 1 and
item handle 1 again. The writer is not exposed to that restart because its
teardown detaches its `OnWriteComplete` handler from the old COM object and
clears that server handle's completion entries before it creates a new one.
No per-handle generation counter is needed.
+13 -1
View File
@@ -113,6 +113,16 @@ library:
library returns carries no expiry timestamp, so the cache cannot cap an entry at the
key's expiry (capping it needs the donor library to surface expiry on the
verification identity). The default 15 s TTL bounds both windows.
Misses are **single-flight**: concurrent misses for the same token share one inner
verification, so a TTL expiry under load costs one store read instead of one per
in-flight RPC. The flight is keyed on the token hash *and* the key's generation, so
a miss that starts after an `Invalidate` never joins a verification that started
before it and cannot receive the pre-revoke identity. The shared inner call is not
bound to any one caller's cancellation token; a caller that gives up stops waiting
without cancelling the others. The per-key index `Invalidate` uses to find every
cached secret variant is pruned by a post-eviction callback when the cache drops an
entry on its own (TTL expiry, compaction), so it tracks the live cache instead of
growing with every token ever verified.
- **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
most one `MarkUsed` write per key per
`MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a
@@ -138,7 +148,9 @@ would otherwise let any peer throttle a key it does not hold — and it joins th
partition only when the presented token is validly shaped
(`mxgw_<keyId>_<secret>`, key id at most 64 characters), with at most 32 key-id
partitions per address before the overflow collapses onto that address's fallback
partition. An over-limit state admits one probe per
partition. That peer-layer collapse does not exempt a validly shaped key id from
the cross-peer aggregate: its failures still feed and consult the key-id aggregate,
whose own cardinality is bounded by the configured LRU cap. An over-limit state admits one probe per
`ApiKeyFailureProbeIntervalSeconds` through to the real verifier and refuses
everything else with `ResourceExhausted` before the store read, so a legitimate
holder presenting the correct secret always reaches the constant-time compare and
+34 -8
View File
@@ -10,7 +10,7 @@ Authorization runs as a single gRPC server interceptor registered for every call
The participating types live under `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/`:
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for unary and server-streaming calls.
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for every call shape: unary, server-streaming, client-streaming, and duplex-streaming.
- `GatewayGrpcScopeResolver` maps a request message (and, for `MxCommandRequest`, the inner `MxCommandKind`) to the scope string that must be present on the caller.
- `GatewayScopes` exposes the canonical scope constants used by the resolver and any downstream consumer.
- `GatewayRequestIdentityAccessor` and `IGatewayRequestIdentityAccessor` expose the verified identity to handlers and any service code that runs inside the call.
@@ -32,7 +32,7 @@ Centralizing the policy in `GatewayGrpcAuthorizationInterceptor` produces three
## Interceptor Flow
`GatewayGrpcAuthorizationInterceptor` overrides both `UnaryServerHandler` and `ServerStreamingServerHandler`. Both call the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then push the resolved identity onto the accessor for the duration of the call.
`GatewayGrpcAuthorizationInterceptor` overrides all four server handlers: `UnaryServerHandler`, `ServerStreamingServerHandler`, `ClientStreamingServerHandler`, and `DuplexStreamingServerHandler`. Each calls the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then pushes the resolved identity onto the accessor for the duration of the call. The base `Interceptor` passes any handler it does not override straight through, so a client- or duplex-streaming RPC added to a mapped service would otherwise run with no API key check at all. The unary and server-streaming handlers resolve the required scope from the request message; the client- and duplex-streaming handlers have no single request to inspect before the handler reads the stream, so they fail closed to `admin` — the same scope the resolver assigns an unrecognized request type. No public RPC uses either shape today.
```csharp
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
@@ -40,7 +40,10 @@ public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
scopeResolver.ResolveRequiredScope(request),
context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
@@ -96,7 +99,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is anonymous by default (`MxGateway:Metrics:RequireAuthentication` is opt-in), so neither key ids nor peer addresses may appear there.
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
@@ -125,7 +128,7 @@ public string ResolveRequiredScope(object request)
}
```
The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*`. It stays `invoke:write` rather than `invoke:secure` because the request carries the operator's own Galaxy credential, which the worker authenticates before the `AckMsg` write; the credential, not the API-key scope, authorizes the identity the Historian records. The trade-off is that an `invoke:write` key can tell `AuthenticationFailed` (an `MXACCESS_FAILURE` whose diagnostic reports operator authentication failure) apart from every other outcome, which makes `AcknowledgeAlarm` a Galaxy-password oracle for that key — whereas the `AuthenticateUser` command itself requires `invoke:secure`. What bounds it: the caller must already hold a valid, admin-issued, revocable `invoke:write` key, and every acknowledgement is serialized through the single monitor worker session's STA, so guesses run one at a time. The gateway does **not** throttle failed operator authentications: the `ApiKeyFailureLimiter` counts only failed API-key verifications, and the ASP.NET Core rate limiter covers only the dashboard `/login` endpoint. Any lockout after repeated bad passwords comes from the Galaxy security configuration, not from the gateway. Grant `invoke:write` only to clients trusted with that exposure. `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
`MxCommandRequest` is special because it multiplexes many MxAccess operations through a single RPC. The resolver inspects the embedded `MxCommandKind` so each operation gets its own scope:
@@ -180,6 +183,21 @@ Supported constraints are:
| `read_historized_only` | Read/subscription commands must target objects with historized attributes. |
| `dashboard_tags` | Dashboard event-visibility tags. **Not a data-access constraint** — see below. |
`browse_subtrees` is applied by `GatewayBrowseScopeProvider`, the host seam the
shared `ZB.MOM.WW.GalaxyRepository` package calls through `IGalaxyBrowseScopeProvider`;
it reads the constraint off the ambient per-call identity that
`GatewayGrpcAuthorizationInterceptor` pushed earlier in the same call. An empty
`browse_subtrees` list is the library's "no scoping" signal, so the provider has to
distinguish two states that look alike (GWC-38): an authenticated key with no browse
constraint is legitimately unscoped and returns the empty list, whereas **no ambient
identity at all** means the interceptor did not run before the Galaxy service and the
request's scope cannot be established. In that second case, with
`MxGateway:Authentication:Mode = ApiKey`, the provider logs a warning naming the broken
ordering invariant and fails the call with `PermissionDenied` rather than serving the full
hierarchy — a narrowing control's silent absence is invisible in normal operation, so it is
made loud instead. Under `Mode = Disabled` an absent identity is the expected steady state
(nothing authenticates, so no key and no constraints exist) and browse stays unscoped.
`dashboard_tags` is the one member of the blob that constrains nothing on the
gRPC data path. No read, write, browse, or subscribe check consults it, and
`HasReadConstraints` / `HasWriteConstraints` deliberately ignore it: adding a tag
@@ -247,9 +265,17 @@ the gateway fails closed.
Non-bulk constraint failures return gRPC `PermissionDenied`. Bulk read
commands preserve input order and return a failed `SubscribeResult` for each
denied item while still forwarding allowed items to the worker. Every denial
adds an `api_key_audit` entry with the key id, command kind, target, and
blocking constraint; secured values and raw credentials are never logged.
denied item while still forwarding allowed items to the worker. A non-bulk
denial adds one `constraint-denied` audit event (canonical `audit_event` table)
with the key id, command kind, target, and blocking constraint. A bulk command
adds **one aggregated** `constraint-denied` event per command, not one per
denied item: its target is `<commandKind>:bulk` and its details carry the
denied and total item counts, the distinct blocking constraint names, a sample
of at most 20 denied targets (`sampleTruncated` says whether more were
denied), and the client correlation id. The audit writer is a bounded channel
that drops on full, so per-item events let a single low-privilege key flush
every other pending audit record with one large bulk command. Secured values
and raw credentials are never logged.
## Scope Catalog
+49
View File
@@ -72,6 +72,23 @@ carries the equivalent guard for the Go module: it refuses to create a
`ClientVersion` already equals `X.Y.Z` (CLI-21/CLI-39), so a forgotten
version bump fails the tag instead of shipping a mismatched module.
Both of those guards only fire when someone actually runs a publish.
CLI-46 found the same hazard recurring silently *between* publishes: a
real bug fix (the pypi 0.2.0 `connect()` fix above, commit `a8f86b5`) plus
later wire-surface changes shipped into the tree while every version
constant still read the already-published number, and nothing recorded
that the registry artifact and the in-repo source had diverged.
`scripts/check-client-versions.ps1` is the PR-time half of that guard —
wired into CI (`.gitea/workflows/ci.yml`, the `portable` job, next to the
codegen freshness checks): for each client it finds the latest publish tag
(`clients/go/vX.Y.Z` for Go — the only client with a git tag at all; .NET,
Python, Rust, and Java publish straight to a package registry via
`pack-clients.ps1 -Publish` with no corresponding git tag, so the check
reports "no tags found" and passes for those four today) and fails if
`clients/<lang>` differs from that tag while the version constant still
equals the tag's version. A client with no publish tag is not a
violation — there is no "since the last publish" window to check yet.
As of 2026-08-07 (CLI-39) all five clients — plus `ZB.MOM.WW.MxGateway.Contracts`,
which releases in lockstep with the .NET Client — moved to **0.2.0**,
converging on one number after four of the five had drifted onto the
@@ -111,6 +128,38 @@ confirmed present through the Gitea package API after the push, and
`go list -m` resolves the Go tag. These are the numbers a future release
bumps off.
On 2026-11-06 (#214) the .NET pair alone moved to **0.3.0**: `nuget`
`ZB.MOM.WW.MxGateway.Contracts` **0.3.0** and `ZB.MOM.WW.MxGateway.Client`
**0.3.0**. The release carries the additive `AcknowledgeAlarmRequest`
`operator_password` / `operator_domain` fields from `488649c`. The feed's
0.2.1 predates them, so OtOpcUa could not consume operator-authenticated
alarm ack. The minor bump marks a new wire field. The Python, Rust, Java and
Go clients were not republished in that release, and their manifests were
left alone.
**The published pypi `0.2.0` artifact has a broken `connect()` (CLI-46).**
Its `connect()` built the `grpc.aio` channel inside `asyncio.to_thread`, and
a `grpc.aio` channel binds to the event loop current on the *constructing*
thread, so every non-stub connection raised `RuntimeError: There is no
current event loop in thread` — TST-24's first real-wire run caught it.
Fixed in-tree by commit `a8f86b5` (2026-08-10): `resolve_channel_security`
now runs off-loop, `create_channel` runs on-loop. That fix, plus later
wire-surface additions (`d9ea8a8`, `c748361`), shipped after the 2026-08-07
publish while the version constants still read the already-published
0.2.0/0.2.1 numbers above — see `clients/python/README.md` for the
user-facing warning. **0.2.0 is superseded by 0.2.1; do not install it.**
On 2026-08-25 the superseding release shipped from commit
`5776021970cbbe59813ba57579abe00d9fc61a46`. Published coordinates on
`gitea.dohertylan.com` are `nuget` `ZB.MOM.WW.MxGateway.Client` **0.2.1** and
`ZB.MOM.WW.MxGateway.Contracts` **0.2.1**, `pypi`
`zb-mom-ww-mxaccess-gateway-client` **0.2.1**, `cargo`
`zb-mom-ww-mxgateway-client` **0.2.1**, and `maven`
`com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client` **0.2.2**. The Go module
tag `clients/go/v0.2.1` points to that same commit. Every exact package
coordinate returned HTTP 200 through the Gitea package API after publication,
and `go list -m` resolved `v0.2.1` to the tagged commit.
## .NET
The .NET client uses .NET 10 and references
+36
View File
@@ -40,6 +40,26 @@ the corresponding MXAccess `AddItem`, `Advise`, `UnAdvise`, and `RemoveItem`
calls sequentially on the session STA and preserves input order in the result
list.
The shared alarm command model includes `SubscribeAlarms` and
`UpdateAlarmWatchList`. `SubscribeAlarmsCommand.fallback_enabled` makes an empty
initial watch list explicitly fallback-capable, allowing discovery recovery to
populate a standby without restarting the primary alarm-manager subscription.
`UpdateAlarmWatchListCommand.watch_list` is the complete replacement list. The
worker prepares the replacement on its STA before swapping it into the existing
failover consumer and returns a normal `MxCommandReply`; callers must treat only
`protocol_status = OK` as acknowledgement of the new list.
`AcknowledgeAlarmRequest` carries the operator's identity: `operator_user`
(required), `operator_password` (field 6) and `operator_domain` (field 7). The
gateway copies them onto `AcknowledgeAlarmCommand` (`operator_password` = 7) and
`AcknowledgeAlarmByNameCommand` (`operator_password` = 9); the worker
authenticates the operator and writes `AckMsg` as that user. The fields are
additive, so `GatewayProtocolVersion` stays `3`. `operator_password` is a
credential and follows the same never-log rule as
`AuthenticateUserCommand.verify_user_password`: it must not reach logs, metric
labels, `DiagnosticMessage`, `ProtocolStatus.Message`, or exception text. A
not-found or not-ackable alarm is reported with `hresult` -1.
The command model also includes bulk write/read command kinds:
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
@@ -94,6 +114,15 @@ only and does not share types with `mxaccess_gateway.proto`. See
[Galaxy Repository Browse](./GalaxyRepository.md) for the RPC catalog and
behavior.
The retained proto remains the client-generation source of truth while the
gateway serves the wire-identical descriptor from the
`ZB.MOM.WW.GalaxyRepository` package. `GalaxyContractParityTests` converts both
loaded descriptors to `FileDescriptorProto`, removes source locations, and
normalizes the C# namespace and file name before comparing deterministic bytes.
Its mutation test also requires drift failures to identify the first differing
protobuf message. Any parity failure must be resolved before publishing either
Contracts or clients.
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
hand-edit generated files.
@@ -140,6 +169,13 @@ Run the focused contract tests after changing either `.proto` file:
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter ProtobufContractRoundTripTests
```
When changing `galaxy_repository.proto` or the served Galaxy Repository
package, also run the retained-vs-served parity guard:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter GalaxyContractParityTests
```
The full solution build also regenerates the C# contracts before compiling
gateway and test projects:
+17
View File
@@ -66,6 +66,23 @@ declaration-ordered JSON; and the row sits alone on its own line for Go and for
Rust's `--jsonl`, but inside an `events` array for Python and for Rust's
aggregate `--json`.
Alarm-feed status rows deliberately preserve each CLI's established JSON
mapping while making false values explicit in every text formatter:
| CLI | Healthy provider text | Clearing snapshot text | JSON false-value shape |
|-----|-----------------------|------------------------|------------------------|
| `mxgw-dotnet` | `provider-status mode=Unspecified degraded=false reason=""` | `snapshot-status truncated=false` | `{"providerStatus": {}}` / `{"snapshotStatus": {}}` |
| `mxgw-go` | `provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=""` | `snapshot-status truncated=false` | `{"providerStatus":{}}` / `{"snapshotStatus":{}}` |
| `mxgw-py` | same JSON row as `--json` | same JSON row as `--json` | `{"providerStatus": {}}` / `{"snapshotStatus": {}}`, emitted one row at a time |
| `mxgw-java` | `provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=` | `snapshot-status truncated=false` | `{"providerStatus":{}}` / `{"snapshotStatus":{}}` |
| `mxgw-rs` | explicit `degraded=false` summary | explicit `truncated=false` summary | always includes `"degraded":false` / `"truncated":false` |
For the protobuf-JSON rows, an absent proto3 boolean means `false`; Rust's
always-explicit booleans are semantically equivalent. Python `stream-alarms`
defaults to an unbounded feed and emits each row immediately; a positive
`--max-messages` value provides a bounded sample and includes both status
preamble rows in the count.
## Integration Gate
Cross-language smoke execution is opt-in. Runners should skip the matrix unless
+91 -11
View File
@@ -90,9 +90,9 @@ detach-grace and replay-retention windows are on by default, the reconnect
surface is a trust boundary — a retained session outlives the stream that opened
it. The session records the opening key id (`GatewaySession.OwnerKeyId`) and
every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless
the caller's key id matches the owner. Gating on the `event` scope alone was
rejected: it would let any `event`-scoped key that learned a session id attach to
another key's retained session and receive its replayed and live data. The check
the caller's key id matches the owner. Gating on the `events:read` scope alone was
rejected: it would let any `events:read`-scoped key that learned a session id
attach to another key's retained session and receive its replayed and live data. The check
runs before any subscriber is attached, so a foreign key never touches the
replay ring. Admin-scope override is deferred.
@@ -430,6 +430,44 @@ Suggested mapping:
- Galaxy SQL metadata endpoints if added: `metadata:read`
- worker shutdown diagnostics and key management: `admin`
## Metrics Endpoint Authentication (SEC-14)
Decision: `/metrics` stays anonymous by default and can be put behind
authentication with `MxGateway:Metrics:RequireAuthentication=true`; `/health`
stays anonymous unconditionally.
- **Opt-in rather than on by default.** Existing Prometheus scrapers on the
deployed hosts scrape anonymously; flipping the default would silently break
them on upgrade. Hosts whose network exposure makes the counters sensitive
enable it explicitly.
- **Health is never gated.** The family Overview dashboard (another repo) polls
`/health/active` without credentials, and liveness/readiness probes are
conventionally anonymous. Health responses carry status only.
- **Credentials reuse existing plumbing.** One `MxGateway.Metrics` scheme
accepts an API key with the `admin` scope (verified through the same cached
`IApiKeyVerifier` and the same per-peer failure limiter as gRPC, so `/metrics`
is not a way around the gRPC brute-force throttle) or, when no authorization
header is sent, the dashboard login cookie with a real dashboard role. No new
credential type or store is introduced.
- **`admin` scope, any dashboard role.** Metrics are an operator surface, so a
data-plane client key (`invoke:*`, `events:read`) is not enough. A dashboard
`Viewer` qualifies because every dashboard role already sees the same aggregate
counters on the dashboard; the exported metrics carry no session ids or tag
values.
- **No environment bypass.** The requirement reads no options, so anonymous
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
dashboard Viewer policy — do not satisfy it. Nor does dev-only
`Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
name, so the metrics scheme skips the cookie fallback while it is on rather than
letting a dev flag silently open an endpoint the operator asked to protect.
- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
the remote IP, while the gRPC one is the gRPC peer string (per connection), so
only the per-key-id aggregate is shared between the two paths.
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
scheme that falls back to the cookie internally keeps the challenge a `401`
with `WWW-Authenticate: Bearer`, which scrapers understand.
## Worker Process Identity
Decision: run workers as the gateway service identity for v1.
@@ -703,14 +741,15 @@ latch in subtag mode.
### Acknowledge via ack-comment write
In subtag mode, `AcknowledgeAlarm` writes the operator comment to the alarm
attribute's ack-comment subtag (`Fallback:Subtags:AckComment`). The write
performs the native ack in AVEVA. This differs from alarmmgr mode, where
`AlarmAckByName` on `wwAlarmConsumerClass` is called directly. The `AckComment`
subtag name is empty by default; configuring it is required for ack to work in
subtag mode. The exact AVEVA subtag names are not hard-coded — the `Subtags`
config block exists precisely so names are not guessed without validation
against the live MXAccess attribute set.
Both provider modes acknowledge by writing the operator comment to `AckMsg` as
the authenticated operator (see "Alarm acknowledgement writes AckMsg as the
authenticated operator" below). In alarm-manager mode the target is
`<TagName>.AckMsg`; in subtag mode it is the watch-list entry's ack-comment
subtag (`Fallback:Subtags:AckComment`, default `AckMsg`). The write performs
the ack in AVEVA. An entry with an empty ack-comment subtag is not ackable and
the RPC reports `InvalidRequest`. The exact AVEVA subtag names are not
hard-coded — the `Subtags` config block exists so names are not guessed
without validation against the live MXAccess attribute set.
### Related documentation
@@ -784,6 +823,47 @@ The following items were previously listed here and have since shipped:
`MxGateway:Sessions:MaxEventSubscribersPerSession`.
See [Event Subscribers](#event-subscribers) above and [Sessions](./Sessions.md).
## Alarm acknowledgement writes AckMsg as the authenticated operator
**Decision (2026-10-19).** `AcknowledgeAlarm` authenticates the requesting
operator with `AuthenticateUser` (`operator_user`, `operator_password`,
optional `operator_domain`) and writes the comment to `<TagName>.AckMsg` over
MXAccess as that user, on a dedicated `LMXProxyServer` registered as
`MxGateway:Alarms:AckClientName`. Both provider modes use this path.
**Why.** Live tests (2026-10-18/19) showed every alarm-client ack is inert on a
headless consumer: wnwrap `AlarmAckByName`/`AlarmAckByGUID`/`AlarmAck`/
`AlarmAckSelected`, and the managed `aaAlarmManagedClient` GUID ack, return
rc=0 or -1 and change nothing, regardless of security classification,
operator name or consumer name. An MXAccess write to `AckMsg` does acknowledge
the alarm, and when the write is made under an authenticated user the AVEVA
runtime historizes that identity (`Operator`, `DomainName`, `UserFullName`),
with the MXAccess client name as `OperatorNode`. The requirement is that the
real operator appears in the Historian, as with InTouch, so a shared gateway
service account was rejected. Operators have Galaxy accounts, so the
per-request credential is workable. See
[Alarm Probe Findings](./AlarmProbeFindings.md).
**Consequences.**
- `operator_user` is required; `operator_password` may be empty.
- `operator_password` is a credential. It is forwarded to the worker over the
pipe but never logged: `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are
credential-bearing commands and any property named like `password` is
redacted.
- `Applied` and `Unconfirmed` (write sent, no `OnWriteComplete` in the bounded
wait) both report `OK`; clients that need certainty watch for the transition.
- The managed ack consumer, the WNAL native ack consumer, the second wnwrap
ack-only consumer and the opt-in managed-ack environment flag are removed.
**Upgrade note.** This is a breaking change for callers. They must now send
`operator_user` (required; empty is rejected with `InvalidArgument`) and the
operator's Galaxy password; a caller that sends no password for a
password-protected account gets `MXACCESS_FAILURE` from `AuthenticateUser`
instead of an acknowledgement. OtOpcUa's `GalaxyDriver` alarm acknowledger is a
known caller that must be updated to forward the operator's credentials before
the gateway is upgraded.
## Related Documentation
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
+9 -4
View File
@@ -4,7 +4,7 @@ The diagnostics subsystem provides structured logging, credential redaction, and
## Goals
The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, `AcknowledgeAlarm` (its `operator_password`), or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
A second goal is parity-test diagnosability. Because MXAccess sessions, workers, correlation ids, and command methods are the units of comparison, every log entry produced inside a request scope must carry those identifiers without each call site having to format them.
@@ -70,7 +70,11 @@ private static readonly HashSet<string> SensitiveCommandMethods = new(StringComp
{
"AuthenticateUser",
"WriteSecured",
"WriteSecured2"
"WriteSecured2",
"WriteSecuredBulk",
"WriteSecured2Bulk",
"AcknowledgeAlarm",
"AcknowledgeAlarmByName"
};
public static bool IsCredentialBearingCommand(string? commandMethod)
@@ -80,7 +84,7 @@ public static bool IsCredentialBearingCommand(string? commandMethod)
}
```
The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below.
The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below. `WriteSecured`, `WriteSecured2`, `WriteSecuredBulk` and `WriteSecured2Bulk` are in the set because every secured write — single-item or bulk — carries a credential-sensitive value. `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are in the set because their commands carry the operator's `operator_password`. `GatewayLogRedactorTests` walks the `MxCommand` payload descriptors and fails if a secured or password-bearing command kind is missing from the set. Independently of the method set, the name-based seam redacts any property whose name contains `password`, so a credential field added to a payload later is redacted without a code change.
### API key redaction
@@ -214,11 +218,12 @@ The order matters: putting the logging scope first ensures that authentication f
## Health Checks
The shared `ZB.MOM.WW.Health` package maps three endpoints — `/healthz` (live), `/health/ready`, and
`/health/active` — and each registered check opts into a tier by tag. The gateway registers two:
`/health/active` — and each registered check opts into a tier by tag. The gateway registers three:
| Check | Endpoint tier | Fails when |
|---|---|---|
| `auth-store` | `ready` | The SQLite auth store cannot be opened. Every gRPC call authenticates against it, so its reachability genuinely gates whether the process should receive traffic. |
| `worker-executable` | `ready` | The configured `MxGateway:Worker:ExecutablePath` is not launchable — invalid path, not a `.exe`, missing, or not a PE image of `RequiredArchitecture`. It applies the launcher's own rule (`WorkerExecutableValidator.ResolveAndValidate`), so the probe and `OpenSession` cannot disagree. Every session launches the worker, so a gateway without one cannot serve MXAccess. It reports only the launch error code as entry `data` (`error_code`), never the path, because `/health/ready` is anonymous. It does not depend on session count. |
| `mxaccess-sessions` | `active` | Sessions exist and their workers have faulted. Reports `total` / `ready` / `faulted` / `starting` / `closing` as entry `data`. |
**Zero sessions is Healthy, and the tier choice follows from that.** The gateway opens an MXAccess

Some files were not shown because too many files have changed in this diff Show More