fix(go-client): go vet copylocks in types_alias_test (declare pointer types)

Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.

CLAUDE.md

@@ -126,7 +126,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
 - **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `docs/DesignDecisions.md` (Session-Resilience Epic Scope, session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`.
 - **Gateway restart does not reattach orphan workers.** The first version terminates orphaned workers on startup; do not design code paths that assume reattachment.
 - **No Blazor UI component libraries.** Dashboard uses local Bootstrap CSS/JS only — do not introduce MudBlazor, Radzen, FluentUI, etc.
-- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, and `AuthenticateUser` credentials must never reach logs. Value logging is opt-in and redacted.
+- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, `AuthenticateUser` credentials, and `AcknowledgeAlarm` `operator_password` must never reach logs. Value logging is opt-in and redacted.
 - **Generated code** under `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`, `clients/*/generated*/`, `clients/python/src/zb_mom_ww_mxgateway/generated/`, etc., is build output. Don't hand-edit. To regenerate, build the contracts project (`dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj`) or run the per-client generation step in that client's README.
 - **Documentation style** (`StyleGuide.md`): PascalCase filenames, no marketing language, present tense, explain *why* not *what*.
 - **Update docs in the same change as the source.** When public APIs, contracts, configuration, build steps, security behavior, event shapes, value conversion, status mapping, or lifecycle rules change, the affected docs (`gateway.md`, `docs/`, client READMEs, design docs) must change in the same commit. Don't leave stale prose describing old behavior.

archreview/2026-08-19/remediation/00-tracking.md

@@ -27,7 +27,7 @@ This plan is executed as **seven parallel batches**, each by one subagent workin
 |---|---|---|
 | TST-32 | Blocked (external) | Gitea runner registration-token reset is a web-UI action on the Gitea admin page; no repo change. Operator to-do. |
 | SEC-37 | Blocked (external) | Family-wide: TLS on the shared GLAuth lives in `scadaproj/infra/glauth/` (separate repo + docker host). Plan separately; after it lands, move the deployed service to `Production`. |
-| CLI-47/48/49/50, IPC-36/37, TST-35 | Not started | Activated by the 2026-08-25 P2 continuation below. |
+| CLI-47/48/49/50, IPC-36/37, TST-35 | Done | Activated by the 2026-08-25 P2 continuation below and closed there: CLI-47/48/49/50 in batch K (`5cdb3da`), IPC-36/37 in batch J (`d6c5a11`), TST-35 in batch M (`cc143b5`). |
 | GWC-33/35, SEC-38/39 | Done | Closed by continuation batch I; regression counts and commits are recorded below. |
 | GWC-36, IPC-38, TST-33/34/36 | Done | Closed by continuation batch H; verification and commit are recorded below. |
 | WRK-34 | Done | Completed opportunistically in batch F (`3679e59`): watchdog faults use `CommandMethod=Watchdog` and carry the stuck correlation ID in the diagnostic message. Tracker correction recorded by the continuation audit. |

archreview/2026-10-21/00-overall.md

@@ -0,0 +1,27 @@
+# MxAccessGateway — System Review (2026-10-21)
+
+Summary of the fourth review cycle. Per-finding status is in [`remediation/00-tracking.md`](remediation/00-tracking.md); the per-domain finding text is not kept in this directory, so the tracker rows carry the description.
+
+## Dimensions
+
+Six reviewers each took one dimension of the system.
+
+| Dimension | Reviewed | Outcome |
+|---|---|---|
+| Sessions | Session lifecycle, sweep, shutdown, lease, metrics accounting (`GatewaySession`, `SessionManager`) | One High: a kill or shutdown during worker startup orphaned the worker. Nine findings, all fixed in stream A. Replay-ring byte cap remains open (low) |
+| Worker | STA runtime, alarm polling, `ReadBulk`, caches, process exit paths (`ZB.MOM.WW.MxGateway.Worker`) | Eight findings fixed on `fix/review-2026-10-21-e`, not yet merged; four deferred (one needs a parity probe, one is a security tradeoff) |
+| gRPC | Interceptor coverage, audit, alarm feed, correlation ids, `/metrics` | Eleven findings fixed. `/metrics` authentication is opt-in |
+| Dashboard | Hub fan-out, browse, snapshot feed, health status, design doc | Ten findings fixed; three low races left open |
+| Clients | Five language clients: ack retry, buffers, dispose, typed diagnostics | All fixed. The Go `go vet` copylocks error in a test predates the review and remains |
+| Docs vs code | Proto comments, `stillpending.md`, tracker headers, untested helpers | In progress in stream L |
+
+## Cross-cutting observations
+
+- The two-process lifecycle is the most failure-prone area: most of the nine session findings are races between open, close, kill and shutdown. Each fix added a regression test; the pattern to keep is that every exit path accounts for the session exactly once.
+- Silent failure was the recurring theme in the worker (STA death, alarm-poll failure, missing unhandled-exception hook). Each now ends in a typed fault or a distinct exit code.
+- Bounded fan-out was the recurring theme in the dashboard: every hub send now has a bound, a timeout or a drop policy.
+- Three external items are outside this repository: shared GLAuth TLS, the Gitea runner token, and an OtOpcUa session leak.
+
+## Cycle-1 backlog
+
+The review also re-triaged 68 first-cycle findings that no later tracker had re-listed: 21 are done, 13 are closed as won't fix with a reason, and 34 remain open and are promoted as `R21-O1` to `R21-O34`. The highest-weight open items are the missing reply to a command that races worker shutdown, the worker pipe without `CurrentUserOnly`, the no-op startup probe with retry options that cannot retry, the redactor gap for `WriteSecuredBulk`/`WriteSecured2Bulk`, the .NET retry budget that a `DeadlineExceeded` consumes, and the Go client's missing typed auth errors and strict-TLS flag.

archreview/2026-10-21/remediation/00-tracking.md

@@ -0,0 +1,224 @@
+# MxAccessGateway — Remediation Tracking (2026-10-21 system review)
+
+Source review: [`../00-overall.md`](../00-overall.md) (summary of the six review dimensions). Generated 2026-10-22 from the review outcomes and the integration branch `fix/review-2026-10-21`.
+Tracks the findings of the 2026-10-21 system review and the cycle-1 findings that no later tracker had re-listed. The earlier trackers remain the record for their own IDs: [cycle 1](../../remediation/00-tracking.md), [cycle 2](../../2026-07-12/remediation/00-tracking.md), [cycle 3](../../2026-08-19/remediation/00-tracking.md).
+
+## How to use this document
+
+- IDs in this cycle use the prefix `R21-` so they cannot collide with the `GWC`/`WRK`/`IPC`/`SEC`/`CLI`/`TST` registers of earlier cycles. `R21-S` is sessions, `R21-W` is the worker client, `R21-G` is the gRPC surface, `R21-D` is the dashboard, `R21-K` is the Windows worker, `R21-C*` are the client libraries, `R21-L` is docs and test gaps, `R21-X` is external, and `R21-O` is a cycle-1 finding re-triaged as still open.
+- Where a finding is the same as an earlier-cycle ID, the description names it (for example `R21-G11` is `SEC-14`).
+- The review's own severity table is not in the repository. Only `R21-S1` carries a severity from the review (High); every other severity here is the triage estimate from the finding's impact and is open to correction.
+- The **Status** column is the single source of truth. Update it in the same change that lands the fix, and cite the finding ID in commit messages.
+
+**Status legend:** `Done` · `Done (merge 2013573)` · `Done (merge 2fad124)` · `Deferred` · `Won't fix` · `Open` · `Blocked (external)`.
+
+## Execution model
+
+The review was remediated by ten parallel streams on branches `fix/review-2026-10-21-{a..j}`, merged into the integration branch `fix/review-2026-10-21`, plus stream K (this tracker triage) and stream L (docs and test gaps).
+
+| Stream | Branch | Scope | Status |
+|---|---|---|---|
+| A | `fix/review-2026-10-21-a` | Sessions (`R21-S*`) | Done, merged (`7f0720f`) |
+| B | `fix/review-2026-10-21-b` | WorkerClient (`R21-W1`) | Done, merged (`81d4cd8`) |
+| C | `fix/review-2026-10-21-c` | gRPC surface (`R21-G*`) | Done, merged (`3f9448f`) |
+| D | `fix/review-2026-10-21-d` | Dashboard (`R21-D*`) | Done, merged (`8212b0e`) |
+| E | `fix/review-2026-10-21-e` | Worker (`R21-K*`) | Done (merge 2013573) — `7618caf`, follow-up `38fbbde` |
+| F | `fix/review-2026-10-21-f` | .NET client | Done, merged (`3a1ee28`) |
+| G | `fix/review-2026-10-21-g` | Java client | Done, merged (`c18566b`) |
+| H | `fix/review-2026-10-21-h` | Go client | Done, merged (`7636573`) |
+| I | `fix/review-2026-10-21-i` | Rust client | Done, merged (`b2a7fc5`) |
+| J | `fix/review-2026-10-21-j` | Python client | Done, merged (`29d4e51`) |
+| K | `fix/review-2026-10-21-k` | Tracker triage (this document) | Done |
+| L | (stream L branch) | Proto comments, docs, test gaps | Done (merge 2fad124) |
+
+Worker changes (stream E) need the windev x86 build and `Worker.Tests` run after merge; stream E's own live facts passed on windev.
+
+## Sessions (stream A)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-S1 | High | Sessions | A kill or shutdown during worker startup orphaned the worker | Done | A: `87f23d9`, `ec9f264` | Gated attach, discard on any exception, `OpenRecorded`; a late worker is never orphaned |
+| R21-S2 | Medium | Sessions | Sweep-driven closes skipped `SessionClosed()`, so the open-sessions gauge drifted | Done | A: `09666cc` | |
+| R21-S3 | Medium | Sessions | `OpenSession` was still accepted while the host was stopping | Done | A: `37ab629` | `GatewayStopping` returns `Unavailable`; `ShutdownAsync` re-snapshots until the registry is empty |
+| R21-S4 | Low | Sessions | A shutdown timeout whose kill succeeded was reported as `CloseFailed` | Done | A: `2b165cd` | Now `Closed`, with `WorkerShutdownFailure` recorded |
+| R21-S5 | Low | Sessions | A failed open left a metrics entry behind | Done | A: `87f23d9` | Open rollback goes through the removal path |
+| R21-S6 | Medium | Sessions | Sessions that never stream pinned a worker for the whole lease | Done | A: `285ef64`, `ec9f264` | Config-gated `UnattachedSessionGraceSeconds` (default 0 = off); `ExtendLease` deleted; internal subscribers count as attached; idle clock starts at Ready |
+| R21-S7 | Low | Sessions | A kill racing a close double-decremented the gauge | Done | A: `ec9f264` | Found in review of stream A |
+| R21-S8 | Low | Sessions | A sweep that lost the race to a close was reported as a failure | Done | A: `ec9f264` | Found in review of stream A |
+| R21-S9 | Medium | Sessions | After `ShutdownWorker` the session stayed `Ready` | Done | A: `cc1e895` | The session faults and the sweep reaps it. Found live |
+| R21-S10 | Low | Sessions | The replay ring has a count cap but no byte cap | Open | — | Documented as bounded by count. Next step: add a byte cap if large-array events make the count cap insufficient |
+
+## WorkerClient (stream B)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-W1 | Medium | WorkerClient | `ShutdownAsync` could block past `ShutdownTimeout` | Done | B: `b0100e7` | The timeout CTS now covers the envelope enqueue |
+
+## gRPC surface (stream C)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-G1 | Medium | gRPC | Denied bulk commands flooded the audit channel | Done | C: `ca0cf19` | One aggregated event per bulk call |
+| R21-G2 | Medium | gRPC | The authorization interceptor did not cover client-streaming or duplex calls (cycle-1 `SEC-17`) | Done | C: `19cc74f` | Both fail closed to the admin scope |
+| R21-G3 | Low | gRPC | `StreamAlarms` overflow returned `Unavailable` | Done | C: `0b4a168` | Now `ResourceExhausted` |
+| R21-G4 | Low | gRPC | `StreamAlarms` cloned the snapshot under the lock on every attach | Done | C: `0b4a168` | Memoized projection |
+| R21-G5 | Low | gRPC | `WriteAsync` was not given the call's cancellation token | Done | C: `0b4a168` | |
+| R21-G6 | Low | gRPC | `correlation_id` handling was inconsistent across replies | Done | C: `0b4a168`, `82a856b` | `client_correlation_id` is echoed; a Debug log records the client/envelope pair |
+| R21-G7 | Low | gRPC | The `StreamDisconnected` reason was a constant (cycle-1 `GWC-22`) | Done | C: `6ae5ed9` | |
+| R21-G8 | Low | Security | `CachingApiKeyVerifier` index pruning and single-flight misses | Done | C: `ed3a2c9` | |
+| R21-G9 | Low | gRPC | `ConstraintEnforcer` allocated a `Task` per item | Done | C: `ca0cf19` | `ValueTask` |
+| R21-G10 | Low | Metrics | `Enum.ToString` was called per event | Done | C: `68d6b5f` | `MxEventFamilyNames` cache |
+| R21-G11 | Medium | Security | `/metrics` was unauthenticated (cycle-1 `SEC-14`) | Done | C: `65bca80`, `82a856b` | Opt-in `Metrics:RequireAuthentication` (admin key, or dashboard Admin/Viewer cookie); `DisableLogin` does not bypass it; `/health` stays open by design |
+| R21-G12 | Low | Metrics | The `/metrics` per-peer limiter is keyed by IP while the gRPC limiter is keyed by connection | Won't fix | — | Documented difference; the two surfaces have different peer identities |
+| R21-G13 | Low | Security | The verifier generation guard depends on key-id case | Open | — | Latent only. Next step: normalize the key id (ordinal-ignore-case) at the guard and add a test |
+
+## Dashboard (stream D)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-D1 | Medium | Dashboard | Unbounded events-hub sends | Done | D: `f1db002`, `bd7abf4` | One drop-oldest pump per session; a lock re-check race was fixed in review |
+| R21-D2 | Medium | Dashboard | One slow client stalled the alarms hub | Done | D: `3688fa1` | 2 s send timeout plus an `AlarmFeedReset` marker. Tradeoff documented: an abandoned send is not followed by a reset |
+| R21-D3 | Low | Dashboard | Browse search ran on every keystroke | Done | D: `db78336` | Debounce, off-dispatcher, indexed |
+| R21-D4 | Medium | Dashboard | The full snapshot plus Configuration went out every second | Done | D: `6733519` | Configuration is sent once per connection; sends have a timeout |
+| R21-D5 | Low | Dashboard | `AlarmsPage` re-sorted twice per render | Done | D: `2a7d6fa` | `DashboardAlarmView` memo |
+| R21-D6 | Low | Dashboard | `BrowsePage` disposal was unbounded | Done | D: `db78336` | |
+| R21-D7 | Low | Dashboard | Two viewers thrashed the advise LRU | Done | D: `5d4aefc` | Per-viewer share |
+| R21-D8 | Low | Dashboard | Disconnected circuits kept polling | Done | D: `db78336` | 30 s retention plus a connection tracker |
+| R21-D9 | Low | Dashboard | `GatewayStatus` was hardcoded (cycle-1 `SEC-27`) | Done | D: `6733519` | Derived from health checks |
+| R21-D10 | Low | Docs | `docs/GatewayDashboardDesign.md` described counters that do not exist | Done | D: `2c963b7` | Doc rewritten to match the shipped pages |
+| R21-D11 | Low | Dashboard | `BrowsePage` search CTS cancel/dispose race | Open | — | Only a caught `ObjectDisposedException`. Next step: swap-and-dispose under a lock when the page is next touched |
+| R21-D12 | Low | Dashboard | Cached health status is stale while the publisher is idle | Open | — | Next step: refresh on first subscribe |
+| R21-D13 | Low | Dashboard | Hub pumps are not awaited on broadcaster dispose | Open | — | Next step: await the pumps with a bounded timeout in `DisposeAsync` |
+
+## Worker (stream E)
+
+All rows are on `fix/review-2026-10-21-e` (`7618caf`, follow-up `38fbbde`), not yet merged. After merge, run `windev-worker-ci.ps1 -Mode test` and update these rows to plain `Done`.
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-K1 | Medium | Worker | The write-completion cache and value cache were never evicted | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Ghost-entry eviction on `AddItem` added in the follow-up |
+| R21-K2 | Medium | Worker | `ReadBulk` was serial (N x timeout) | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Shared deadline; dedupe and periodic message pumping in the follow-up |
+| R21-K3 | Medium | Worker | One alarm-poll failure faulted the session (cycle-1 `WRK-05`) | Done (merge 2013573) | E: `7618caf` | Consecutive-failure threshold of 3 |
+| R21-K4 | Low | Worker | The primary alarm consumer leaked when the standby build failed | Done (merge 2013573) | E: `7618caf` | |
+| R21-K5 | Low | Worker | `StaRuntime` drained the whole queue before pumping messages | Done (merge 2013573) | E: `7618caf` | |
+| R21-K6 | Medium | Worker | STA thread death after start was silent (cycle-1 `WRK-02`) | Done (merge 2013573) | E: `7618caf` | `LoopFault` becomes a `StaHung` WorkerFault |
+| R21-K7 | Low | Worker | No unhandled-exception hook (cycle-1 `WRK-09`) | Done (merge 2013573) | E: `7618caf` | Exit code 7 |
+| R21-K8 | Low | Worker | `RETRIGGER` handling was undocumented | Done (merge 2fad124) | stream L | Documented as reserved, never synthesized; the proto comment is stream L's |
+| R21-K9 | Medium | Tests | Live control-command facts for the worker (cycle-1 `TST-05` residual) | Done (merge 2013573) | E: `069972a`, `4ea3672` | Facts added and passing on windev |
+| R21-K10 | Low | Worker | Unregister-first shutdown cleanup | Deferred | — | Needs a parity probe against the installed MXAccess before it can change cleanup order |
+| R21-K11 | Medium | Security | A per-operator `AuthenticateUser` cache | Deferred | — | Security tradeoff (caching credentials' outcome); decide with the security owner before building |
+| R21-K12 | Low | Worker | Batch task-completion-source allocation rework | Deferred | — | Low value |
+| R21-K13 | Low | Worker | The alarm poll builds a full-BSTR `XmlDocument` every 500 ms | Deferred | — | Steady GC cost, not a correctness issue; revisit if profiling flags it |
+
+## Clients (streams F to J)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-CN1 | Medium | .NET client | Acknowledge retried on transient failure | Done | F: `6f0b408` | No ack retry |
+| R21-CN2 | Medium | .NET client | Client always owned its channel | Done | F: `6f0b408` | Caller-owned channel constructors |
+| R21-CN3 | Medium | .NET client | `DisposeAsync` could throw or block (cycle-1 `CLI-06`) | Done | F: `6f0b408` | Safe, best-effort dispose |
+| R21-CN4 | Low | .NET client | No typed Ping/GetSessionState/GetWorkerInfo/DrainEvents | Done | F: `6f0b408` | |
+| R21-CN5 | Medium | .NET client | A session could not be re-attached by id (cycle-1 `CLI-05`) | Done | F: `6f0b408` | `AttachSession` |
+| R21-CJ1 | Medium | Java client | Event queue was fixed at 16 | Done | G: `48c9323` | Configurable, default 1024 |
+| R21-CJ2 | Medium | Java client | Overflow dropped buffered events | Done | G: `48c9323` | Overflow keeps buffered events |
+| R21-CJ3 | Low | Java client | Interrupt did not cancel the call | Done | G: `48c9323` | |
+| R21-CJ4 | Low | Java client | No typed diagnostics | Done | G: `48c9323` | |
+| R21-CG1 | Medium | Go client | Event buffer fixed at 16 | Done | H: `a3a7bd2` | `EventBufferSize` option, default 1024 |
+| R21-CG2 | Medium | Go client | Deprecated `DialContext` + `WithBlock` (cycle-1 `CLI-10`) | Done | H: `a3a7bd2` | `grpc.NewClient` |
+| R21-CG3 | Low | Go client | No typed diagnostics | Done | H: `a3a7bd2` | |
+| R21-CG4 | Low | Go client | `go vet` copylocks error in `types_alias_test.go` | Done | see commit "fix(go-client): go vet copylocks" | The test declares pointer types; `go vet ./...` is clean |
+| R21-CR1 | Medium | Rust client | Blocking file read on an async path (CA file) | Done | I: `0345726` | Async read |
+| R21-CR2 | Low | Rust client | No typed diagnostics | Done | I: `0345726` | |
+| R21-CP1 | Low | Python client | No typed diagnostics | Done | J: `99347d3`, `d716dec` | The client already had no ack retry, no event buffer, and cancels on exit |
+
+## Docs and test gaps (stream L)
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-L1 | Low | Docs | `operator_full_name` is silently dropped; proto comment and docs do not say so | Done (merge 2fad124) | stream L | |
+| R21-L2 | Low | Contracts | `RETRIGGER` and the `WorkerFaultCategory` values `INVALID_ARGUMENTS` and `GATEWAY_AUTHENTICATION_FAILED` are reserved but not labeled so | Done (merge 2fad124) | stream L | Proto comments |
+| R21-L3 | Low | Docs | Stale notes in `stillpending.md` (sections 1.4, 3.4, 7.1) and the deferred-followups header | Done (merge 2fad124) | stream L | |
+| R21-L4 | Low | Docs | The 2026-08-19 tracker header says CLI-47..50, IPC-36/37, TST-35 are `Not started` while the rows say `Done` | Done | K | Header row corrected in this stream |
+| R21-L5 | Low | Tests | No tests for `GatewayConfigPathRules` | Done (merge 2fad124) | stream L | |
+| R21-L6 | Low | Tests | No tests for `DashboardRedactor` | Done (merge 2fad124) | stream L | |
+| R21-L7 | Low | Tests | No tests for `EventSubscriberReplayAttachment` | Done (merge 2fad124) | stream L | |
+| R21-L8 | Low | Tests | No tests for the `MxStatusDetailText` table | Deferred | — | Worker-side test; needs a windev run, not done in stream L |
+
+## External / blocked
+
+| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
+|---|---|---|---|---|---|---|
+| R21-X1 | Medium | Security | TLS on the shared GLAuth (cycle-3 `SEC-37`) | Blocked (external) | — | Lives in `scadaproj/infra/glauth/` (separate repo and docker host); after it lands, move the deployed service to `Production` |
+| R21-X2 | Low | CI | Gitea runner registration-token reset (cycle-3 `TST-32`) | Blocked (external) | — | Web-UI action on the Gitea admin page |
+| R21-X3 | Medium | OtOpcUa | `GalaxyDriver.ShutdownAsync` leaks sessions | Blocked (external) | — | Defect is in the OtOpcUa repo; the gateway's lease expiry and detach grace eventually reap a leaked session |
+
+## Cycle-1 rows re-triaged as still open (promoted)
+
+These are cycle-1 findings that no later tracker re-listed. Each was checked against the integration branch, and the cycle-1 row now points here. The `Medium` rows carry correctness, security or operability weight (`R21-O1`, `R21-O8`, `R21-O9`, `R21-O16`, `R21-O21` to `R21-O23`, `R21-O30`, `R21-O31`); the `Low` rows are hygiene to batch with the next change that touches the file.
+
+| ID | Cycle-1 ID | Sev | Area | Description | Status | Next step |
+|---|---|---|---|---|---|---|
+| R21-O1 | WRK-03 | Medium | Worker | `TryStartCommandTask` returns silently when `_acceptingCommands` is false, so a command racing shutdown gets no reply and no log (`Ipc/WorkerPipeSession.cs:1240`) | Open | Reply `WorkerUnavailable` (at minimum log `WorkerCommandResultDropped`) and add `CommandAfterShutdown_RepliesWorkerUnavailable`; verify on windev |
+| R21-O2 | WRK-08 | Low | Worker | Residual events queued after `WorkerShutdown` are discarded, and the shutdown section of `docs/MxAccessWorkerInstanceDesign.md` does not say so | Open | Document the discard as v1 behavior (docs-only) |
+| R21-O3 | WRK-10 | Low | Worker | `WorkerApplication.Run` catch blocks log the exception type but never the message | Open | Add `exception_message` through `WorkerLogRedactor` in the three catch blocks; add a `WorkerApplicationTests` assertion |
+| R21-O4 | WRK-16 | Low | Worker | Seven near-identical `CreateEnvelope` overload pairs and eight `WorkerPipeClient` constructors | Open | Refactor when `WorkerPipeSession` is next touched; existing IPC tests are the guard |
+| R21-O5 | WRK-17 | Low | Worker | Gateway death surfaces as `EndOfStream` and exits 6 (`ProtocolViolation`) instead of 5 | Open | Map `EndOfStream` to `PipeConnectionFailed`; add a `WorkerApplicationTests` case |
+| R21-O6 | WRK-18 | Low | Worker | Event-queue overflow exits as generic `UnexpectedFailure` (1) | Open | Add a dedicated exit code (stream E already uses 7 for unhandled exceptions, so use 8) and update the overflow paragraph of `docs/MxAccessWorkerInstanceDesign.md` |
+| R21-O7 | WRK-19 | Low | Worker | The documented per-command start/end log with correlation id and latency does not exist | Open | Add level-gated `WorkerCommandStarted`/`WorkerCommandCompleted` in `StaCommandDispatcher.ExecuteQueuedCommandAsync` |
+| R21-O8 | GWC-05 | Medium | Gateway | Worker pipe is created with no ACL and no `CurrentUserOnly` (`SessionWorkerClientFactory.CreatePipe`); only the unguessable name and `maxNumberOfServerInstances: 1` limit access | Open | Add `PipeOptions.CurrentUserOnly` (or a `PipeSecurity` for the worker account); confirm the worker runs as the same user on windev; add a launch test |
+| R21-O9 | GWC-09 | Medium | Gateway | `WorkerProcessStartedProbe` only checks `HasExited`, so `StartupProbeRetryAttempts` and `StartupProbeRetryDelayMilliseconds` can never retry anything | Open | Either make the probe wait for a real readiness signal, or remove the two options and their docs so config does not promise a retry that cannot happen |
+| R21-O10 | GWC-13 | Low | Gateway | Worker-ready wait is a 25 ms poll loop (`GatewaySession`, `pollIntervalMs = 25`) | Open | Replace with a `TaskCompletionSource` signalled on state change |
+| R21-O11 | GWC-17 | Low | Gateway | `SparseArrayExpander` throws `Grpc.Core.RpcException` from the Sessions layer | Open | Throw a domain exception and map it to `InvalidArgument` in the gRPC layer |
+| R21-O12 | GWC-18 | Low | Gateway | `GatewaySession` exposes `DisposeAsync` without implementing `IAsyncDisposable` | Open | Declare the interface; check `await using` call sites |
+| R21-O13 | GWC-19 | Low | Gateway | `GatewaySession.KillWorker(string)` has no production callers (only test comments mention it) | Open | Delete it and fix the comments, or document it as the synchronous kill primitive |
+| R21-O14 | GWC-20 | Low | Gateway | `HeartbeatCheckInterval` is fed from `HeartbeatIntervalSeconds`, conflating the send and check intervals | Open | Split the option or rename it, and update `docs/GatewayConfiguration.md` |
+| R21-O15 | GWC-21 | Low | Gateway | `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` are compile-time constants (`WorkerClientOptions`) | Open | Expose both under `MxGateway:Worker:*` with validation, or record them as intentionally fixed |
+| R21-O16 | SEC-13 | Medium | Security | `GatewayLogRedactor.SensitiveCommandMethods` lacks `WriteSecuredBulk` and `WriteSecured2Bulk`. Latent in cycle 1, but `GatewayLogRedactorSeam` now calls `RedactCommandValue`, so a secured-bulk payload would be logged unredacted if value logging is enabled | Open | Add both bulk kinds plus a redactor test; cross-check every command kind that carries a password |
+| R21-O17 | SEC-16 | Low | Security | `apikey --pepper` is accepted on the command line (`ApiKeyAdminCommandLineParser`) and lands in shell history and the process list | Open | Prefer an env var or stdin prompt; keep the flag only with a warning |
+| R21-O18 | SEC-18 | Low | Security | `DashboardApiKeyManagementService` detects a missing pepper by matching `"pepper unavailable"` in an exception message | Open | Catch a typed exception from `ZB.MOM.WW.Auth.ApiKeys`, or check the pepper before the call |
+| R21-O19 | SEC-23 | Low | Security | The options validator does not reject a `__Host-` `CookieName` combined with `RequireHttpsCookie=false`, a cookie browsers silently drop | Open | Add the cross-field check to `ValidateDashboard` |
+| R21-O20 | SEC-24 | Low | Security | `EffectiveDashboardConfiguration` omits `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie` and `CookieName`, so the Settings page cannot show that login is disabled | Open | Project the four fields in `GatewayConfigurationProvider` |
+| R21-O21 | CLI-07 | Medium | .NET client | `ExecuteSafeUnaryAsync` wraps the retry pipeline in `CancelAfter(DefaultCallTimeout)` while each attempt has the same deadline, so a `DeadlineExceeded` first attempt consumes the whole budget and retries never run | Open | Make the outer budget `MaxAttempts x DefaultCallTimeout` (or drop the outer cap); add a wire test with a slow first attempt |
+| R21-O22 | CLI-09 | Medium | Go client | Go wraps every RPC failure in `GatewayError`; there is no typed `Unauthenticated` vs `PermissionDenied` mapping, though `ClientLibrariesDesign.md` requires one | Open | Add sentinel or typed errors usable with `errors.Is`; keep credential redaction |
+| R21-O23 | CLI-11 | Medium | Go client | `mxgw-go` has no flag for strict TLS validation, so `--tls` without a CA is always skip-verify | Open | Add `--require-certificate-validation` wired to `Options.RequireCertificateValidation` |
+| R21-O24 | CLI-19 | Low | .NET client | `InternalsVisibleTo` is declared in both `Properties/AssemblyInfo.cs` and the csproj | Open | Delete one (cosmetic) |
+| R21-O25 | CLI-22 | Low | Go client | `newCorrelationID` returns an empty string if `crypto/rand` fails | Open | Fall back to a counter-based id |
+| R21-O26 | CLI-27 | Low | Python client | `Session.close()` has no lock around `_closed`, and a repeat close returns a synthesized reply | Open | Guard with an `asyncio.Lock` and cache the real reply, as .NET and Go do |
+| R21-O27 | CLI-31 | Low | Rust client | The Rust CLI `main.rs` is now 3,023 lines | Open | Split by command group when next touched |
+| R21-O28 | CLI-33 | Low | Clients | Slow-consumer behavior is documented in the Go README but not in the Java README or the cross-language `docs/ClientBehaviorFixtures.md` | Open | Add one comparison table now that this cycle changed the Go and Java buffers |
+| R21-O29 | TST-07 | Low | Tests | Real-clock sleeps with negative assertions remain (for example `WorkerClientTests.cs:975`, 150 ms) | Open | Convert to a fake time provider where one exists; otherwise leave |
+| R21-O30 | TST-09 | Medium | Ops | Health checks cover the auth store and sessions only; Galaxy SQL, LDAP and worker-executable presence are unchecked (`GatewayApplication.cs:118`) | Open | Add a Ready-tag worker-exe presence check first, then a bounded Galaxy SQL ping; zero sessions must stay Healthy because the Overview dashboard polls the Active tier |
+| R21-O31 | TST-10 | Medium | Ops | No deploy or upgrade script or runbook for the NSSM-wrapped service | Open | Write a deployment doc from the windev/wonder deploy notes; respect the current ban on remote execution on the wonder boxes |
+| R21-O32 | TST-18 | Low | Tests | `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` timer/startup wiring has no direct tests | Open | Add timer and startup tests with a fake time provider |
+| R21-O33 | TST-21 | Low | Ops | Serilog file sink rolls daily with no `retainedFileCountLimit` or size cap | Open | Set `retainedFileCountLimit` and `fileSizeLimitBytes` in `appsettings.json`; document |
+| R21-O34 | TST-22 | Low | Docs | The Configuration Shape JSON block in `docs/GatewayConfiguration.md` omits `DisableLogin`, `AutoLoginUser`, `CookieName`, `Tls` and `Alarms:Fallback` | Open | Add the keys to the block (docs-only) |
+
+## Cycle-1 triage summary
+
+The 68 cycle-1 rows that were `Not started` with no later re-listing resolved as follows.
+
+| Verdict | Count | Rows |
+|---|:-:|---|
+| Done | 21 | WRK-02, WRK-05, WRK-09, WRK-13, WRK-20, GWC-10, GWC-12, GWC-22, IPC-10, IPC-12, SEC-14, SEC-17, SEC-19, SEC-21, SEC-26, SEC-27, CLI-05, CLI-06, CLI-10, CLI-13, TST-14 |
+| Won't fix | 13 | GWC-11, IPC-11, SEC-15, CLI-14, CLI-17, CLI-20, CLI-23, CLI-25, CLI-28, CLI-32, TST-19, TST-20, WRK-14 |
+| Open (promoted) | 34 | `R21-O1` through `R21-O34` above |
+
+`WRK-02`, `WRK-05` and `WRK-09` are closed by stream E and become plain `Done` when `fix/review-2026-10-21-e` merges.
+
+The seven rows singled out for assessment: `WRK-03` Open (`R21-O1`: the worker still returns silently from `TryStartCommandTask` after shutdown starts); `WRK-05` Done (stream E); `CLI-06` Done (stream F); `CLI-07` Open (`R21-O21`: the outer `CancelAfter` still equals the per-attempt deadline); `CLI-09` Open (`R21-O22`: no typed auth errors in Go); `CLI-11` Open (`R21-O23`: no strict-TLS flag on `mxgw-go`); `CLI-13` Done (stream G).
+
+## Change log
+
+| Date | Change |
+|---|---|
+| 2026-10-22 | Initial tracker from the 2026-10-21 review outcomes. Stream E is recorded as done but unmerged; stream L items are in progress. 68 cycle-1 rows re-triaged and the 34 still-open ones promoted. |
+
+## Final integration review (stream M)
+
+| ID | Severity | Area | Finding | Status | Commit | Notes |
+|---|---|---|---|---|---|---|
+| R21-M1 | Medium | Sessions | A worker that dies right after its handshake left the session `Ready` with a dead worker until the lease expired | Done | M: `a178d68` | Attach refuses a terminal worker; the pump observes terminal state on its first read |
+| R21-M2 | Low | Metrics | One worker fault was counted 1+K times (K = number of `StreamEvents` subscribers) | Done | M: `a0b5e91` | |
+| R21-M3 | Low | Sessions | A cancelled open, or one drained at shutdown, was reported as `OpenFailed` | Done | M: `a178d68` | Now `Cancelled` and `GatewayStopping` respectively |
+| R21-M4 | Low | Docs | Unattached-grace exemptions, subscriber-limit status code, stale event-stream description in Grpc.md | Done | M: `3ffba83` | |
+| R21-M5 | Low | Docs | Client docs described the ReadBulk timeout as per-tag | Done | M: `dd62335` | Worker-side comments (`MxAccessCommandExecutor.cs:15`, `MxAccessValueCache.cs:219`, `MxAccessValueCacheTests.cs:246`) still say per-tag: Open |

archreview/remediation/00-tracking.md

@@ -11,7 +11,7 @@ Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation
 - Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` (depends-on) column within a tier.
 - When a fix lands: flip Status to `Done`, and per the repo rule, update the affected docs in the same commit.
 
-**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action).
+**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action) · `Open` (re-triaged 2026-10-21: still present, promoted to the [2026-10-21 tracker](../2026-10-21/remediation/00-tracking.md)).
 
 ## Severity roll-up
 
@@ -63,24 +63,24 @@ Full design + implementation for each row lives in the linked domain doc under i
 | GWC-02 | High | P0 | M | — | Done | Faulted sessions are never swept |
 | GWC-03 | High | P0 | S | — | Done | Documented sparse-array max-length bound is unimplemented |
 | GWC-04 | Medium | P1 | M | — | Done | Full event channel stalls the worker read loop behind command replies |
-| GWC-05 | Medium | — | S | — | Not started | Worker pipe created with no ACL / no CurrentUserOnly |
+| GWC-05 | Medium | — | S | — | Open | Worker pipe created with no ACL / no CurrentUserOnly — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O8 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | GWC-06 | Medium | P2 | S | GWC-07,08 | Done | Stopwatch allocated per streamed event |
 | GWC-07 | Medium | P2 | S | GWC-06,08 | Done | Every mapped event is deep-cloned |
 | GWC-08 | Medium | P2 | M | GWC-06,07 | Done | Pipe framing allocates per frame and writes twice |
-| GWC-09 | Medium | — | M | — | Not started | Startup probe is a no-op; its retry pipeline can never retry |
-| GWC-10 | Medium | — | S | — | Not started | Envelope `sequence` monotonicity specified but not enforced |
-| GWC-11 | Low | — | S | — | Not started | `_workerClient` written under lock but read lock-free |
-| GWC-12 | Low | — | S | — | Not started | `WorkerClient.DisposeAsync` not safe against double-dispose |
-| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
+| GWC-09 | Medium | — | M | — | Open | Startup probe is a no-op; its retry pipeline can never retry — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O9 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-10 | Medium | — | S | — | Done | Envelope `sequence` monotonicity specified but not enforced — **2026-10-21 triage:** resolved by spec change, not enforcement: `gateway.md:440` states `sequence` is a diagnostic aid, stamped in each side's single write path and not validated on receive |
+| GWC-11 | Low | — | S | — | Won't fix | `_workerClient` written under lock but read lock-free — **2026-10-21 triage:** reference reads are atomic and every ready/state decision reads `_workerClient` under `_syncRoot` (`EvaluateReadyUnderLock`); only diagnostic properties read it lock-free |
+| GWC-12 | Low | — | S | — | Done | `WorkerClient.DisposeAsync` not safe against double-dispose — **2026-10-21 triage:** closed by cycle-3 GWC-32 (`DisposeAsync` claims `_disposed` under `_syncRoot`; see the 2026-08-19 tracker) |
+| GWC-13 | Low | — | M | — | Open | Worker-ready wait is a 25 ms poll loop — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O10 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | GWC-14 | Low | P2 | M | — | Done | Replay ring is a `LinkedList` with a node alloc per event |
 | GWC-15 | Low | P2 | S | — | Done | Per-event gauge reads `ChannelReader.Count` every event |
 | GWC-16 | Low | — | S | — | Won't fix | `Invoke` resolves the session twice per command — measured negligible, deliberately not collapsed (see domain doc) |
-| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
-| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
-| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
-| GWC-20 | Low | — | S | — | Not started | Heartbeat config semantics conflated (send vs check interval) |
-| GWC-21 | Low | — | S | GWC-04 | Not started | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable |
-| GWC-22 | Low | — | S | — | Not started | `StreamDisconnected` always labeled `"Detached"` |
+| GWC-17 | Low | — | M | — | Open | Sessions layer throws `Grpc.Core.RpcException` (layering leak) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O11 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-18 | Low | — | S | — | Open | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O12 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-19 | Low | — | S | — | Open | Dead `GatewaySession.KillWorker(string)` + stale doc — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O13 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-20 | Low | — | S | — | Open | Heartbeat config semantics conflated (send vs check interval) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O14 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-21 | Low | — | S | GWC-04 | Open | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O15 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| GWC-22 | Low | — | S | — | Done | `StreamDisconnected` always labeled `"Detached"` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G7), `6ae5ed9`: `StreamDisconnected` records the real reason |
 | GWC-23 | Info | — | — | — | N/A | `MaxEventSubscribersPerSession` dead in default single-subscriber mode |
 
 ### Worker — [20-worker.md](20-worker.md)
@@ -88,25 +88,25 @@ Full design + implementation for each row lives in the linked domain doc under i
 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |---|---|:-:|:-:|---|---|---|
 | WRK-01 | High | P0 | M | — | Done | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
-| WRK-02 | Medium | — | M | — | Not started | STA thread death after startup is silent; future work hangs forever |
-| WRK-03 | Medium | — | S | WRK-02 | Not started | Commands after shutdown starts are dropped with no reply |
+| WRK-02 | Medium | — | M | — | Done | STA thread death after startup is silent; future work hangs forever — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K6): an STA loop fault raises a `StaHung` WorkerFault; stream E `7618caf`, merged  |
+| WRK-03 | Medium | — | S | WRK-02 | Open | Commands after shutdown starts are dropped with no reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O1 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | WRK-04 | Medium | — | M | — | Done | Envelope `sequence` can appear out of order on the wire |
-| WRK-05 | Medium | — | M | — | Not started | One transient alarm-poll failure kills the whole session |
+| WRK-05 | Medium | — | M | — | Done | One transient alarm-poll failure kills the whole session — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K3): consecutive alarm-poll failure threshold of 3 in `MxAccessStaSession`; stream E `7618caf`, merged  |
 | WRK-06 | Medium | P2 | S | — | Done | `MXSTATUS_PROXY` conversion reflects per field, per event |
 | WRK-07 | Medium | P1 | M | WRK-04 | Done | Documented outbound write priority not implemented |
-| WRK-08 | Low | — | S | — | Not started | Residual event queue discarded at graceful shutdown, undocumented |
-| WRK-09 | Low | — | S | — | Not started | No `AppDomain.UnhandledException` hook |
-| WRK-10 | Low | — | S | — | Not started | Top-level catch logs exception type but never message |
+| WRK-08 | Low | — | S | — | Open | Residual event queue discarded at graceful shutdown, undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O2 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| WRK-09 | Low | — | S | — | Done | No `AppDomain.UnhandledException` hook — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K7): `AppDomain.UnhandledException` hook, exit code 7; stream E `7618caf`, merged  |
+| WRK-10 | Low | — | S | — | Open | Top-level catch logs exception type but never message — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O3 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | WRK-11 | Low | P2 | S | — | Done | Every accepted event is defensively cloned on enqueue |
 | WRK-12 | Low | P2 | M | WRK-04 | Done | No event batching per envelope; one flush per event; 25 ms poll |
-| WRK-13 | Low | — | S | — | Not started | Frame writer allocates a fresh buffer per frame; reader pools |
-| WRK-14 | Low | — | M | — | Not started | Private-field naming split `_camelCase` vs `camelCase` |
+| WRK-13 | Low | — | S | — | Done | Frame writer allocates a fresh buffer per frame; reader pools — **2026-10-21 triage:** worker `WorkerFrameWriter` rents the frame from `ArrayPool` (`Ipc/WorkerFrameWriter.cs:675`); the reader was already pooled |
+| WRK-14 | Low | — | M | — | Won't fix | Private-field naming split `_camelCase` vs `camelCase` — **2026-10-21 triage:** cosmetic; the style guide permits both `_camelCase` and bare `camelCase` where established, and a mass rename is churn without behavior change |
 | WRK-15 | Low | P2 | S | — | Done | Doc drift: STA thread name and heartbeat-counter note |
-| WRK-16 | Low | — | S | — | Not started | Boilerplate duplication in IPC envelope/ctor overloads |
-| WRK-17 | Low | — | S | — | Not started | Gateway death exits with wrong exit code (6 not 5) |
-| WRK-18 | Low | — | S | — | Not started | Event-queue overflow exits as generic `UnexpectedFailure` |
-| WRK-19 | Low | — | S | — | Not started | Command start/end logging with correlation id absent |
-| WRK-20 | Low | — | M | WRK-01..04 | Not started | Test-coverage gaps for the failure modes above |
+| WRK-16 | Low | — | S | — | Open | Boilerplate duplication in IPC envelope/ctor overloads — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O4 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| WRK-17 | Low | — | S | — | Open | Gateway death exits with wrong exit code (6 not 5) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O5 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| WRK-18 | Low | — | S | — | Open | Event-queue overflow exits as generic `UnexpectedFailure` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O6 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| WRK-19 | Low | — | S | — | Open | Command start/end logging with correlation id absent — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O7 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| WRK-20 | Low | — | M | WRK-01..04 | Done | Test-coverage gaps for the failure modes above — **2026-10-21 triage:** STA-death, sequence-monotonic, long-ReadBulk and late-reply tests exist; the one residual (command-after-shutdown reply) is carried by WRK-03 as R21-O1 |
 
 ### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
 
@@ -121,9 +121,9 @@ Full design + implementation for each row lives in the linked domain doc under i
 | IPC-07 | Medium | P2 | S | — | Done | `docs/Grpc.md` says six RPCs; there are seven |
 | IPC-08 | Medium | — | M | — | Done | `WorkerCancel` defined and handled but never sent — resolved by `7c1ea12`/`a1a38b5`; residual (caller-cancellation leg) tracked as cycle-3 IPC-34 |
 | IPC-09 | Medium | P1 | M | IPC-01 | Done | Known codegen fragilities have no in-repo guards |
-| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
-| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
-| IPC-12 | Low | — | S | — | Not started | Gateway frame writer has no write lock; integrity rests on undocumented invariant |
+| IPC-10 | Low | — | S | — | Done | Envelope `sequence` is write-only; monotonicity never validated — **2026-10-21 triage:** same resolution as GWC-10: `gateway.md:440` documents `sequence` as a diagnostic aid |
+| IPC-11 | Low | — | S | — | Won't fix | No protocol version negotiation despite `supported_protocol_version` name — **2026-10-21 triage:** the review itself recommended no change; exact-match protocol version is correct for a lockstep-deployed pair, and the next bump should introduce a min/max range |
+| IPC-12 | Low | — | S | — | Done | Gateway frame writer has no write lock; integrity rests on undocumented invariant — **2026-10-21 triage:** IPC-13 collapsed the gateway write to one pooled-buffer write, and `WorkerClient.WriteLoopAsync` is its only caller |
 | IPC-13 | Low | P2 | S | IPC-05 | Done | Gateway writer serializes twice and issues two stream writes |
 | IPC-14 | Low | P2 | S | IPC-05 | Done | Gateway reader allocates fresh array per frame; worker rents from `ArrayPool` |
 | IPC-15 | Low | P2 | M | — | Done | Worker event delivery 25 ms poll with per-event write+flush, no batching |
@@ -151,21 +151,21 @@ Full design + implementation for each row lives in the linked domain doc under i
 | SEC-10 | Medium | P1 | L | — | Done | No API-key expiry |
 | SEC-11 | Medium | P1 | M | SEC-08 | Done | No rate limiting or lockout on either auth surface |
 | SEC-12 | Medium | P1 | M | — | Done | Dashboard Close/Kill bypass the canonical audit store |
-| SEC-13 | Low | — | S | SEC-30 | Not started | Redactor's credential-command list omits secured-bulk variants |
-| SEC-14 | Low | — | S | SEC-02,20 | Not started | `/metrics` + `/health` unauthenticated; a metric leaks session ids |
-| SEC-15 | Low | — | S | — | Not started | GET `/logout` skips antiforgery |
-| SEC-16 | Low | — | S | — | Not started | CLI accepts the pepper as a command-line argument |
-| SEC-17 | Low | — | S | — | Not started | Interceptor does not override client-streaming/duplex handlers |
-| SEC-18 | Low | — | S | — | Not started | Pepper-unavailable detection matches library message text |
-| SEC-19 | Low | — | S | — | Not started | Canonical audit store re-issues `CREATE TABLE` per write/read |
+| SEC-13 | Low | — | S | SEC-30 | Open | Redactor's credential-command list omits secured-bulk variants — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O16 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| SEC-14 | Low | — | S | SEC-02,20 | Done | `/metrics` + `/health` unauthenticated; a metric leaks session ids — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G11), `65bca80` + `82a856b`: opt-in `Metrics:RequireAuthentication`; `/health` stays open by design |
+| SEC-15 | Low | — | S | — | Won't fix | GET `/logout` skips antiforgery — **2026-10-21 triage:** accepted nuisance: GET `/logout` skips antiforgery deliberately (comment in `DashboardEndpointRouteBuilderExtensions.cs`); the POST path validates antiforgery |
+| SEC-16 | Low | — | S | — | Open | CLI accepts the pepper as a command-line argument — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O17 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| SEC-17 | Low | — | S | — | Done | Interceptor does not override client-streaming/duplex handlers — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G2), `19cc74f`: client- and duplex-streaming calls authenticate and fail closed to admin |
+| SEC-18 | Low | — | S | — | Open | Pepper-unavailable detection matches library message text — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O18 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| SEC-19 | Low | — | S | — | Done | Canonical audit store re-issues `CREATE TABLE` per write/read — **2026-10-21 triage:** `SqliteCanonicalAuditStore` latches schema creation, so `CREATE TABLE IF NOT EXISTS` runs once (a failure leaves the latch open) |
 | SEC-20 | Low | P1 | S | — | Done | `session_id` metric tag is unbounded cardinality |
-| SEC-21 | Low | — | M | — | Not started | Snapshot publisher works every second regardless of audience |
+| SEC-21 | Low | — | M | — | Done | Snapshot publisher works every second regardless of audience — **2026-10-21 triage:** `DashboardSnapshotFeed` is idle-gated (no viewers, no timer), API-key summaries refresh every 15 s, and configuration is sent once per connection (R21-D4, `6733519`) |
 | SEC-22 | Low | P2 | M | — | Done | `docs/Authentication.md` documents types no longer in this repo |
-| SEC-23 | Low | — | S | SEC-03,04 | Not started | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` |
-| SEC-24 | Low | — | S | SEC-04 | Not started | Effective-config view omits the riskiest dashboard flags |
+| SEC-23 | Low | — | S | SEC-03,04 | Open | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O19 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| SEC-24 | Low | — | S | SEC-04 | Open | Effective-config view omits the riskiest dashboard flags — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O20 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | SEC-25 | Low | P2 | M | SEC-02 | Done | Per-session EventsHub ACL is an acknowledged TODO |
-| SEC-26 | Low | — | M | — | Not started | Audit trail has no retention or pruning |
-| SEC-27 | Low | — | S | — | Not started | Dashboard `GatewayStatus` is hardcoded Healthy |
+| SEC-26 | Low | — | M | — | Done | Audit trail has no retention or pruning — **2026-10-21 triage:** `Security/Audit/AuditDrainService.cs` prunes by `AuditRetentionDays` (validated in `GatewayOptionsValidator`) on an hourly sweep |
+| SEC-27 | Low | — | S | — | Done | Dashboard `GatewayStatus` is hardcoded Healthy — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-D9), `6733519`: `GatewayStatus` derives from health checks |
 | SEC-28 | Info | — | S | — | N/A | Positive security observations (preserve under change) |
 | SEC-29 | Info | — | S | — | N/A | UI-stack rule verified compliant |
 | SEC-30 | Info | P2 | S | SEC-13 | Done | Value-logging feature is unwired end-to-end |
@@ -178,35 +178,35 @@ Full design + implementation for each row lives in the linked domain doc under i
 | CLI-02 | High | P1 | M | — | Done | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) |
 | CLI-03 | High | P0 | M | — | Done | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
 | CLI-04 | High | P2 | L | CLI-15 | Done | Typed-command parity gap across all five clients |
-| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
-| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
-| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
+| CLI-05 | Medium | — | S | — | Done | .NET session cannot be re-attached to an existing session id — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN5), `6f0b408`: `MxGatewayClient.AttachSession` |
+| CLI-06 | Medium | — | S | — | Done | .NET `DisposeAsync` blocks/throws on unreachable gateway — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN3), `6f0b408`: `MxGatewaySession.DisposeAsync` is best-effort and never throws |
+| CLI-07 | Medium | — | S | — | Open | .NET retry budget self-defeats on `DeadlineExceeded` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O21 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | CLI-08 | Medium | — | S | CLI-03 | Done | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) — landed via 2026-07-12 [CLI-38](../2026-07-12/remediation/50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1) |
-| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
-| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
-| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
+| CLI-09 | Medium | — | M | — | Open | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O22 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| CLI-10 | Medium | — | M | — | Done | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CG2), `a3a7bd2`: `grpc.NewClient` replaces `DialContext` + `WithBlock` |
+| CLI-11 | Medium | — | S | — | Open | Go CLI cannot opt into strict TLS validation — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O23 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | CLI-12 | Medium | P2 | S | — | Done | Java docs still say "Java 21" after the JDK 17 retarget |
-| CLI-13 | Medium | — | M | — | Not started | Java event-stream buffer hardcoded 16, cancel-on-overflow |
-| CLI-14 | Medium | — | S | — | Not started | Python default TLS is blocking TOFU pin with silent `localhost` SNI |
+| CLI-13 | Medium | — | M | — | Done | Java event-stream buffer hardcoded 16, cancel-on-overflow — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CJ1/CJ2), `48c9323`: event queue is configurable (default 1024) and overflow keeps buffered events |
+| CLI-14 | Medium | — | S | — | Won't fix | Python default TLS is blocking TOFU pin with silent `localhost` SNI — **2026-10-21 triage:** documented and intentional: the Python README describes the trust-on-first-use default and `--require-certificate-validation`; the `localhost` SNI default matches the gateway's self-signed SAN (`options.py:160`) |
 | CLI-15 | Medium | P2 | M | — | Done | No client handles `ReplayGap` or offers a reconnect helper |
 | CLI-16 | Medium | P2 | S | CLI-12 | Done | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` |
-| CLI-17 | Medium | — | M | CLI-14 | Not started | TLS default posture inconsistent across the five clients |
+| CLI-17 | Medium | — | M | CLI-14 | Won't fix | TLS default posture inconsistent across the five clients — **2026-10-21 triage:** documented divergence (`docs/CrossLanguageSmokeMatrix.md`); a strict default would break every deployment on the gateway's self-signed certificate. Strict opt-in exists per client; the Go CLI gap is R21-O23 |
 | CLI-18 | Low | P2 | S | — | Done | .NET csproj records no `<Version>` |
-| CLI-19 | Low | — | S | — | Not started | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) |
-| CLI-20 | Low | — | S | CLI-17 | Not started | .NET `--tls` without CA installs accept-all callback |
+| CLI-19 | Low | — | S | — | Open | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O24 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| CLI-20 | Low | — | S | CLI-17 | Won't fix | .NET `--tls` without CA installs accept-all callback — **2026-10-21 triage:** same decision as CLI-17: documented, intentional default; `RequireCertificateValidation` opts in |
 | CLI-21 | Low | P2 | S | — | Done | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases |
-| CLI-22 | Low | — | S | — | Not started | Go `newCorrelationID` swallows `crypto/rand` error → empty id |
-| CLI-23 | Low | — | S | — | Not started | Go nil-vs-empty bulk short-circuit asymmetry |
+| CLI-22 | Low | — | S | — | Open | Go `newCorrelationID` swallows `crypto/rand` error → empty id — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O25 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| CLI-23 | Low | — | S | — | Won't fix | Go nil-vs-empty bulk short-circuit asymmetry — **2026-10-21 triage:** cosmetic; an empty bulk is a local no-op in some helpers and a wire call in others |
 | CLI-24 | Low | — | S | — | Done | Java `MxEventStream` single-consumer constraint undocumented (closed 2026-08-07 per 2026-07-12 review old-tracker action; documented at MxEventStream.java:25 "Single consumer") |
-| CLI-25 | Low | — | S | — | Not started | Java `close()` does not await channel termination |
+| CLI-25 | Low | — | S | — | Won't fix | Java `close()` does not await channel termination — **2026-10-21 triage:** `closeAndAwaitTermination()` is the documented awaiting path; the leak window is bounded by JVM exit |
 | CLI-26 | Low | P2 | S | — | Done | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) |
-| CLI-27 | Low | — | S | — | Not started | Python `Session.close()` not concurrency-safe; synthesizes reply |
-| CLI-28 | Low | — | S | — | Not started | Python circular-import workaround at end of `session.py` |
+| CLI-27 | Low | — | S | — | Open | Python `Session.close()` not concurrency-safe; synthesizes reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O26 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| CLI-28 | Low | — | S | — | Won't fix | Python circular-import workaround at end of `session.py` — **2026-10-21 triage:** the import cycle is deliberate, marked `# noqa: E402`, and covered by the Python test suite |
 | CLI-29 | Low | P2 | S | — | Done | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) |
 | CLI-30 | Low | — | S | CLI-04 | Done | Rust has no `unregister` typed helper |
-| CLI-31 | Low | — | M | — | Not started | Rust CLI is a single 2,699-line `main.rs` |
-| CLI-32 | Low | — | S | — | Not started | Client-side bulk caps differ (.NET/Java unbounded) |
-| CLI-33 | Low | — | S | CLI-01,13 | Not started | Per-language event backpressure semantics undocumented |
+| CLI-31 | Low | — | M | — | Open | Rust CLI is a single 2,699-line `main.rs` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O27 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| CLI-32 | Low | — | S | — | Won't fix | Client-side bulk caps differ (.NET/Java unbounded) — **2026-10-21 triage:** the gateway is the authority for bulk caps; the divergence affects only the error type for an oversized call |
+| CLI-33 | Low | — | S | CLI-01,13 | Open | Per-language event backpressure semantics undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O28 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | CLI-34 | Low | — | S | — | Done | Python `build/`/`.pytest_cache/` present on disk (untracked) (closed 2026-08-07 per 2026-07-12 review old-tracker action; both gitignored in clients/python/.gitignore) |
 
 ### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)
@@ -219,22 +219,22 @@ Full design + implementation for each row lives in the linked domain doc under i
 | TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
 | TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *` → `scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
 | TST-06 | Medium | — | M | — | Materially narrowed | Dashboard live-data path untested — `DashboardLiveDataServiceTests`, `EventsHubTests`, `EventsHubViewerRegistryTests`, `AlarmsHubPublisherTests`, `SessionDetailsPageEventAclTests` now exist (landed with SEC-25/TST-16); needs a re-audit-and-close pass, not re-implementation |
-| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
+| TST-07 | Medium | — | S | — | Open | Real-clock sleeps with negative assertions are latent flakes — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O29 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
-| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
-| TST-10 | Medium | — | M | — | Not started | Deployment/upgrade undocumented and hand-rolled |
+| TST-09 | Medium | — | M | — | Open | Health checks cover only the auth store — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O30 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| TST-10 | Medium | — | M | — | Open | Deployment/upgrade undocumented and hand-rolled — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O31 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | TST-11 | Medium | P2 | M | — | Done | No version discipline on server; client versions drift |
 | TST-12 | Medium | P0 | S | — | Done | CLAUDE.md misstates default retention behaviour |
 | TST-13 | Medium | P2 | S | — | Done | gateway.md carries stale design-era sketches |
-| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
+| TST-14 | Medium | P2 | S | — | Done | Repo-root working artifacts need triage — **2026-10-21 triage:** root `oldtasks.md` and the `*-docs-*.md` working files are no longer in the tree; `stillpending.md` remains as the backlog and stream L is refreshing it |
 | TST-15 | Medium | P2 | M | TST-04 | Done | Dashboard EventsHub has no per-session ACL |
 | TST-16 | Medium | P1 | S | — | Done | `Dashboard:ShowTagValues` is a dead flag |
 | TST-17 | Medium | — | S | — | Materially changed | Vendor-gated alarm parity residuals silently lossy — `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName`; the vendor 8-arg leg is recorded "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, `d1ae43d`, `540a118`); "silent drop, unchanged" is no longer accurate — needs a worker-domain refresh pass |
-| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
-| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
-| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
-| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
-| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
+| TST-18 | Low | — | S | — | Open | Hosted-service wrappers untested — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O32 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| TST-19 | Low | — | S | — | Won't fix | Keep FakeWorkerHarness canned replies in lockstep — **2026-10-21 triage:** ongoing discipline rather than a work item; the harness shares production framing and the nightly live tier (`nightly-windev`) catches canned-reply drift |
+| TST-20 | Low | — | S | — | Won't fix | E2E script papers over a real advise-without-consumer sharp edge — **2026-10-21 triage:** documented sharp edge in `docs/GatewayTesting.md`; the e2e script's interleaved drains are the intended workaround |
+| TST-21 | Low | — | S | — | Open | Log rotation configured but minimal — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O33 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
+| TST-22 | Low | — | S | — | Open | Config-shape JSON block omits documented keys — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O34 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
 | TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
 | TST-24 | Low | P2 | M | TST-03 | Done | Client wire behaviour has no automated verification — closed 2026-08-10. Go/Rust/Java already had real-server wire tests (the finding's premise was stale); the genuine gaps were .NET (transport-interface fake everywhere, no server package) and Python (stub monkeypatch everywhere but one opt-in TLS test). Added `MxGatewayClientWireTests` + `WireFakeGatewayServer` (Kestrel h2c) and `tests/test_wire_fake_gateway.py` (`grpc.aio` loopback), plus a `dotnet test` step for the .NET client in the `portable` CI job. Caught a real bug: Python `connect()` built the `grpc.aio` channel inside `asyncio.to_thread` and failed for every non-stub connection |
 
@@ -253,6 +253,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
 
 | Date | Change |
 |---|---|
+| 2026-10-22 | **Cycle-1 `Not started` rows re-triaged against the 2026-10-21 integration branch** (stream K). All 68 rows that no later tracker had re-listed now carry a verdict: 21 `Done` (closed by earlier work or by the 2026-10-21 review), 13 `Won't fix` (reason on the row), 34 `Open` (promoted as `R21-O1`..`R21-O34` in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) with a next step). Rows closed by stream E are marked pending merge of that stream. |
 | 2026-08-17 | **TST-16 → `Done`: `Dashboard:ShowTagValues` now gates every dashboard value surface** (branch `feat/followup-closeout`). The flag was live on one seam only (the events-hub mirror, shipped with SEC-25); the two recorded residuals are closed, so it is no longer a partly-dead flag. **Alarms hub:** `AlarmsHubPublisher` took `IOptions<GatewayOptions>` and now clears `current_value`/`limit_value` from a **deep clone** of each `AlarmFeedMessage` on both value-bearing payload arms (`transition`, `active_alarm`) before the `/hubs/alarms` broadcast. Clone-only, matching `DashboardEventBroadcaster.RedactValues`: the source message fans out from one feed to gRPC `StreamAlarms` subscribers and the alarms page, none of which this dashboard-display flag governs. The redaction switches on the value-bearing arms and passes everything else through untouched and uncloned, so a payload arm added later (the `snapshot_status` arm landing alongside this change) is forwarded unbroken rather than silently dropped. **`/browse`:** the gate lives at the service boundary per TST-16's own recommendation — `DashboardLiveDataService` substitutes `DashboardTagValue.RedactedValueText` (`[redacted]`) for the formatted value text, so the value never enters the render tree and `BrowsePage.razor` needed no change; data type, quality, source timestamp, and read errors stay real, keeping the panel diagnostic. A visible marker, not a blank, so a suppressed value is distinguishable from a tag that read back empty. Tests: new `AlarmsHubPublisherTests` (4 — both arms redacted with metadata intact, source not mutated, `true` passes values through as the same instance, valueless arms forwarded as the same instance) and two `DashboardLiveDataServiceTests` cases (`false` → `[redacted]`, `true` → formatted); `DashboardEventBroadcasterTests` re-run as regression. Docs: `docs/GatewayConfiguration.md`'s `ShowTagValues` row rewritten from "gates only the events hub mirror, two still-open residuals" to the three seams it now covers, with the "no gRPC client is affected" scope statement; `docs/GatewayDashboardDesign.md` gained the redaction note on the `AlarmsHub` row and had its Browse-panel paragraph corrected (it claimed the panel "always shows values regardless of `Dashboard:ShowTagValues`"). |
 | 2026-08-17 | **Alarm-snapshot truncation now has a structural degraded-status signal** (branch `feat/deferred-closeout`, commits `693a78d` + `b9fb0dd`). No review ID — this is branch work outside the 153-finding register, recorded here so the tracker is not silent on a shipped change to the alarm surface. Before it, a capped `GetXmlCurrentAlarms2` fetch suppressed absence-implies-Clear inference and said so only in a rate-limited worker stderr warning, so no client and no operator could tell a complete active set from a capped one. Two additive proto3 booleans carry the verdict out — `QueryActiveAlarmsReplyPayload.snapshot_truncated = 2` and `ActiveAlarmSnapshot.from_truncated_snapshot = 16` (per record, because `QueryActiveAlarms` is a bare `stream ActiveAlarmSnapshot` with no envelope; the reply payload states it too, since a prefix filter can leave zero records and a capped fetch with nothing to report still has to say so). Flow: `WnWrapAlarmConsumer` → `AlarmDispatcher` / `IAlarmCommandHandler` → `MxAccessCommandExecutor` reply → `GatewayAlarmMonitor` → `IGatewayAlarmService.SnapshotTruncated` → `AlarmsPage` banner. `b9fb0dd` then made the pairing structural after review: `IMxAccessAlarmConsumer` and `IAlarmCommandHandler` expose one accessor (`SnapshotActiveAlarms(out bool truncated)` / `QueryActive(..., out bool snapshotTruncated)`) satisfied from a single lock acquisition, so the snapshot and its verdict can no longer be read across a poll; the separate `LastSnapshotTruncated` property is gone from every layer. Detection is deliberately unchanged (`fetchedRecordCount >= maxAlarmsPerFetch`); switching to `ALARM_RECORDS/@COUNT` stays blocked on probe evidence (`docs/AlarmProbeFindings.md`). Not latched, and dropped with the cache generation by `ClearCache`. Additive gateway metadata about our fetch mechanics, not MXAccess behaviour — no synthesized event, so not a parity deviation. Docs: `gateway.md` alarm surface, `docs/DesignDecisions.md`. |
 | 2026-08-17 | **TST-15 → `Done` (discharges the ACL half of SEC-25): per-session dashboard event ACL shipped** (branch `feat/deferred-closeout`, commits `693a78d` + `7ec0b35`). Implements `docs/plans/2026-07-10-dashboard-session-acl-tst15.md`, whose header is now `Implemented` with as-built notes in its §12. `IDashboardSessionAcl.CanViewSession` is the single decision **both** subscribe seams consult — `EventsHub.SubscribeSession` (denial is a `HubException`; the caller is neither joined to the group nor registered in `EventsHubViewerRegistry`, so the mirror stays off) and `SessionDetailsPage`'s in-process subscribe (inline denial, no subscription) — so neither path is the weaker one and the `TODO(per-session-acl)` is gone. Decision order, fail-closed on every branch: authenticated Administrator → allow (evaluated **before** the registry lookup, so Admin × unknown-session allows — pinned by a test because reordering the two checks is a plausible refactor); session not found → deny; untagged session → `Dashboard:UntaggedSessionVisibility` (`AdminOnly` default); else allow iff `session.Tags ∩ zb:dashboardtag` claims, ordinal-ignore-case. Session tags are inherited from the owning API key's `dashboard_tags` constraint (`apikey --dashboard-tags`, already in the `ApiKeyConstraints` JSON blob — no SQLite migration) and never from the client's wire request. Viewer grants come from `Dashboard:GroupToTag` over the user's LDAP groups, stamped at cookie login (`DashboardAuthenticator.CreatePrincipal`) and **re-resolved, not copied**, at hub-token mint (`HubTokenService.Issue`), so the 5-minute token lifetime bounds a stale grant. Anonymous localhost is an empty-grant Viewer; `Dashboard:DisableLogin` auto-login carries both roles and so takes the admin bypass unchanged. Tests: `DashboardSessionAclTests` (decision table, every branch asserted in its denying direction too), `EventsHubTests`, `DashboardAuthenticatorTests`, `HubTokenServiceTests`, a `GatewayOptionsTests` case proving `Dashboard:GroupToTag` keeps its ordinal-ignore-case lookup through configuration binding, and two `[LiveLdapFact]`s in `DashboardLdapLiveTests` that drive a real bind against the shared GLAuth (`gw-viewer` → `team-a` grant admits the `team-a` session and refuses the `team-b` one; `multi-role` bypasses on the sessions its own grant does not cover). The live pair needed **no GLAuth change** — the tag layer is config-side, keyed on the existing `GwAdmin`/`GwReader` groups (recorded in `glauth.md`). Docs: `docs/Sessions.md` (session-tag model), `gateway.md` + CLAUDE.md dashboard-auth paragraphs, `docs/GatewayDashboardDesign.md` (three passages that described the ACL as outstanding), `docs/GatewayConfiguration.md` (`ShowTagValues` row: redaction is now the second of two layers, not the only one), `docs/Authentication.md` (`--dashboard-tags` is the only *constraint* flag that splits on commas). |

clients/dotnet/README.md

@@ -137,14 +137,40 @@ gateway's own sentinel observable. Normal events pass through with
 `IsReplayGap == false` and `ReplayGap == null`. The gap is only ever produced by
 `StreamEvents`; the diagnostic drain path never emits it. If you already consume
 the raw `StreamEventsAsync` (or the client-level stream), the
+### Caller-owned channels, resuming sessions, diagnostics
+
+`MxGatewayClient.Create(GrpcChannel, MxGatewayClientOptions)` and
+`GalaxyRepositoryClient.Create(GrpcChannel, MxGatewayClientOptions)` build a
+client over a channel the caller created (shared channel, custom handler or
+TLS). The client never disposes a caller-owned channel; `Create(options)` still
+creates and owns its own channel.
+
+`MxGatewayClient.AttachSession(sessionId)` returns an `MxGatewaySession` bound
+to an existing session id without calling OpenSession, so a restarted process
+can resume `StreamEventsAsync(afterWorkerSequence)`. The gateway only allows the
+API key that opened the session to attach; any other key gets `PermissionDenied`.
+The attached session's `OpenSessionReply` carries only the session id.
+
+`MxGatewaySession` has typed diagnostics: `PingAsync`, `GetSessionStateAsync`,
+`GetWorkerInfoAsync`, and `DrainEventsAsync` (removes queued events from the
+worker; call until empty). `DisposeAsync` closes the session best-effort and does
+not throw when the gateway is unreachable or the client is already disposed.
+
+Retry applies only to idempotent calls (`CloseSessionRawAsync`, ping,
+session-state and worker-info commands, Galaxy reads). `AcknowledgeAlarmAsync`
+is never retried because it is a non-idempotent operator write that re-sends the
+operator password and can double-acknowledge in the Historian.
+
 `AsStreamItemsAsync()` extension projects any `IAsyncEnumerable<MxEvent>` into
 the same `MxEventStreamItem` surface.
 
 For alarms, the client exposes `QueryActiveAlarmsAsync` (one-shot snapshot of
 the active alarms the gateway's central monitor currently holds),
 `StreamAlarmsAsync` (server-streaming feed of alarm-state-change messages
-keyed by the same monitor), and `AcknowledgeAlarmAsync` (ack by alarm
-reference, optional comment, ack target). All three accept a cancellation
+keyed by the same monitor), and `AcknowledgeAlarmAsync` (acknowledges as
+the operator: the gateway authenticates `OperatorUser`/`OperatorPassword` and
+writes the comment to the alarm's `AckMsg`, so the Historian records the
+operator). The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. All three accept a cancellation
 token and pass through the `MxGateway:Alarms` configuration on the
 server — when alarms are disabled, the gateway returns an empty list / empty
 stream rather than failing.
@@ -291,10 +317,15 @@ dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write --se
 dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write2 --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --timestamp 2026-01-01T00:00:00Z --json
 dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-events --session-id <id> --max-events 1 --json
 dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-alarms --filter-prefix Area001 --max-events 4 --json
-dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --comment "ack from cli" --operator operator1 --json
+dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --comment "ack from cli" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json
 dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- smoke --endpoint http://localhost:5000 --api-key-env MXGATEWAY_API_KEY --item Area001.Pump001.Speed --json
 ```
 
+`acknowledge-alarm` reads the operator password from `--operator-password` or the
+environment variable named by `--operator-password-env` (default
+`MXGATEWAY_OPERATOR_PASSWORD`); the password is never echoed and is redacted from
+any error output. Prefer the environment variable to keep it off the command line.
+
 `smoke` opens a session, registers a client, adds one item, advises it,
 optionally writes a value when `--type` and `--value` are supplied, reads a
 bounded event stream, and closes the session in a `finally` block. CLI error

clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs

@@ -185,6 +185,7 @@ public static class MxGatewayClientCli
                 exception.Message,
                 apiKey,
                 TryResolveVerifyUserPassword(arguments),
+                ResolveOperatorPassword(arguments),
                 arguments.GetOptional("value"));
 
             if (forceJsonErrors || arguments.HasFlag("json"))
@@ -352,6 +353,12 @@ public static class MxGatewayClientCli
     /// </summary>
     private const string DefaultVerifyPasswordEnvironmentName = "MXGATEWAY_VERIFY_PASSWORD";
 
+    /// <summary>
+    ///     Default environment variable holding the operator password for
+    ///     <c>acknowledge-alarm</c> when <c>--operator-password-env</c> is not given.
+    /// </summary>
+    private const string OperatorPasswordEnvironmentName = "MXGATEWAY_OPERATOR_PASSWORD";
+
     /// <summary>
     ///     Legacy environment variable, still honoured as a deprecated fallback
     ///     for one release so existing scripts keep working.
@@ -378,6 +385,30 @@ public static class MxGatewayClientCli
             : environmentName;
     }
 
+    /// <summary>
+    ///     Resolves the <c>acknowledge-alarm</c> operator password: <c>--operator-password</c>,
+    ///     then the environment variable named by <c>--operator-password-env</c>
+    ///     (default <c>MXGATEWAY_OPERATOR_PASSWORD</c>). An empty value is treated as
+    ///     absent. The value is never echoed; it is also fed to the error redactor.
+    /// </summary>
+    private static string? ResolveOperatorPassword(CliArguments arguments)
+    {
+        string? password = arguments.GetOptional("operator-password");
+        if (!string.IsNullOrEmpty(password))
+        {
+            return password;
+        }
+
+        string environmentName = arguments.GetOptional("operator-password-env") ?? string.Empty;
+        if (string.IsNullOrEmpty(environmentName))
+        {
+            environmentName = OperatorPasswordEnvironmentName;
+        }
+
+        password = Environment.GetEnvironmentVariable(environmentName);
+        return string.IsNullOrEmpty(password) ? null : password;
+    }
+
     /// <summary>
     ///     Resolves the effective MXAccess verify-user credential in this
     ///     order: <c>--password</c>, the deprecated <c>--verify-user-password</c>
@@ -1535,6 +1566,8 @@ public static class MxGatewayClientCli
             AlarmFullReference = arguments.GetRequired("reference"),
             Comment = arguments.GetOptional("comment") ?? string.Empty,
             OperatorUser = arguments.GetOptional("operator") ?? string.Empty,
+            OperatorDomain = arguments.GetOptional("operator-domain") ?? string.Empty,
+            OperatorPassword = ResolveOperatorPassword(arguments) ?? string.Empty,
         };
 
         return WriteReplyAsync(
@@ -2465,7 +2498,7 @@ public static class MxGatewayClientCli
         writer.WriteLine("mxgw-dotnet bench-read-bulk [--duration-seconds <n>] [--warmup-seconds <n>] [--bulk-size <n>] [--tag-start <n>] [--tag-prefix <s>] [--tag-attribute <s>] [--timeout-ms <n>] [--client-name <name>]");
         writer.WriteLine("mxgw-dotnet stream-events --session-id <id> [--max-events <n>] [--json]");
         writer.WriteLine("mxgw-dotnet stream-alarms [--filter-prefix <ref>] [--max-events <n>] [--json] [--jsonl]");
-        writer.WriteLine("mxgw-dotnet acknowledge-alarm --reference <ref> [--comment <text>] [--operator <user>] [--json]");
+        writer.WriteLine("mxgw-dotnet acknowledge-alarm --reference <ref> --operator <user> [--operator-domain <dom>] [--operator-password <pw> | --operator-password-env <VAR>] [--comment <text>] [--json]");
         writer.WriteLine("mxgw-dotnet write --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> [--json]");
         writer.WriteLine("mxgw-dotnet write2 --session-id <id> --server-handle <n> --item-handle <n> --type <type> --value <value> [--timestamp <iso>] [--json]");
         writer.WriteLine("mxgw-dotnet smoke --item <ref> [--value <value> --type <type>] [--json]");

clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientAlarmsTests.cs

@@ -36,6 +36,8 @@ public sealed class MxGatewayClientAlarmsTests
             AlarmFullReference = "Tank01.Level.HiHi",
             Comment = "investigating",
             OperatorUser = "alice",
+            OperatorPassword = "pw-secret",
+            OperatorDomain = "CORP",
         });
 
         Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
@@ -45,32 +47,56 @@ public sealed class MxGatewayClientAlarmsTests
         Assert.Equal("Tank01.Level.HiHi", call.Request.AlarmFullReference);
         Assert.Equal("investigating", call.Request.Comment);
         Assert.Equal("alice", call.Request.OperatorUser);
+        Assert.Equal("pw-secret", call.Request.OperatorPassword);
+        Assert.Equal("CORP", call.Request.OperatorDomain);
         Assert.Equal("Bearer test-api-key", call.CallOptions.Headers?.GetValue("authorization"));
     }
 
+    /// <summary>AcknowledgeAlarmAsync redacts the operator password from a failing call's message.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task AcknowledgeAlarmAsync_FailureMessage_DoesNotContainOperatorPassword()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+
+        // The real transport maps RpcException to MxGatewayException using the status
+        // detail; the fake passes exceptions verbatim, so enqueue the mapped shape.
+        transport.AcknowledgeAlarmExceptions.Enqueue(new MxGatewayException("boom pw-secret"));
+        await using MxGatewayClient client = CreateClient(transport);
+
+        MxGatewayException ex = await Assert.ThrowsAsync<MxGatewayException>(
+            () => client.AcknowledgeAlarmAsync(new AcknowledgeAlarmRequest
+            {
+                AlarmFullReference = "Tank01.Level.HiHi",
+                OperatorUser = "alice",
+                OperatorPassword = "pw-secret",
+            }));
+
+        Assert.DoesNotContain("pw-secret", ex.Message);
+        Assert.Contains("boom", ex.Message);
+    }
+
     /// <summary>AcknowledgeAlarmAsync honors cancellation.</summary>
     /// <returns>A task that represents the asynchronous operation.</returns>
     [Fact]
     public async Task AcknowledgeAlarmAsync_HonorsCancellation()
     {
-        // Acks are routed through the safe-unary retry pipeline (idempotent at the
-        // MxAccess level), so the transport-side cancellation token is a linked one
-        // rather than the caller's original. Verify cancellation by tripping the source
-        // and asserting the call observes it.
+        // Acks bypass the retry pipeline (non-idempotent), so the caller's token
+        // reaches the transport unchanged.
         using CancellationTokenSource cancellation = new();
-        cancellation.Cancel();
         FakeGatewayTransport transport = CreateTransport();
         await using MxGatewayClient client = CreateClient(transport);
 
-        await Assert.ThrowsAnyAsync<OperationCanceledException>(() =>
-            client.AcknowledgeAlarmAsync(
-                new AcknowledgeAlarmRequest
-                {
-                    AlarmFullReference = "Tank01.Level.HiHi",
-                    Comment = string.Empty,
-                    OperatorUser = "alice",
-                },
-                cancellation.Token));
+        await client.AcknowledgeAlarmAsync(
+            new AcknowledgeAlarmRequest
+            {
+                AlarmFullReference = "Tank01.Level.HiHi",
+                Comment = string.Empty,
+                OperatorUser = "alice",
+            },
+            cancellation.Token);
+
+        Assert.Equal(cancellation.Token, Assert.Single(transport.AcknowledgeAlarmCalls).CallOptions.CancellationToken);
     }
 
     /// <summary>AcknowledgeAlarmAsync maps unauthenticated RPC exception to typed exception.</summary>

clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs

@@ -763,6 +763,7 @@ public sealed class MxGatewayClientCliTests
     [Fact]
     public async Task RunAsync_AcknowledgeAlarm_BuildsRequestAndPrintsJsonReply()
     {
+        Environment.SetEnvironmentVariable("MXGW_TEST_OPPW", "op-pw-secret");
         using var output = new StringWriter();
         using var error = new StringWriter();
         FakeCliClient fakeClient = new();
@@ -786,6 +787,10 @@ public sealed class MxGatewayClientCliTests
                 "ack from cli",
                 "--operator",
                 "operator1",
+                "--operator-domain",
+                "CORP",
+                "--operator-password-env",
+                "MXGW_TEST_OPPW",
                 "--json",
             ],
             output,
@@ -797,6 +802,9 @@ public sealed class MxGatewayClientCliTests
         Assert.Equal("Tank01.Level.HiHi", request.AlarmFullReference);
         Assert.Equal("ack from cli", request.Comment);
         Assert.Equal("operator1", request.OperatorUser);
+        Assert.Equal("CORP", request.OperatorDomain);
+        Assert.Equal("op-pw-secret", request.OperatorPassword);
+        Assert.DoesNotContain("op-pw-secret", output.ToString());
         Assert.Contains("ack-fixture", output.ToString());
         Assert.Equal(string.Empty, error.ToString());
     }

clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientRemediationTests.cs

@@ -0,0 +1,241 @@
+using Grpc.Core;
+using Grpc.Net.Client;
+using ZB.MOM.WW.MxGateway.Contracts.Proto;
+
+namespace ZB.MOM.WW.MxGateway.Client.Tests;
+
+/// <summary>Tests for ack no-retry, caller-owned channels, safe dispose, typed diagnostics, and AttachSession.</summary>
+public sealed class MxGatewayClientRemediationTests
+{
+    /// <summary>An acknowledge is a non-idempotent operator write and is never retried.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task AcknowledgeAlarmAsync_DoesNotRetryTransientFailure()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.AcknowledgeAlarmExceptions.Enqueue(new RpcException(new Status(StatusCode.Unavailable, "down")));
+        await using MxGatewayClient client = new(transport.Options, transport);
+
+        await Assert.ThrowsAsync<RpcException>(() => client.AcknowledgeAlarmAsync(
+            new AcknowledgeAlarmRequest { OperatorUser = "op", OperatorPassword = "pw" }));
+
+        Assert.Single(transport.AcknowledgeAlarmCalls);
+    }
+
+    /// <summary>A client created over a caller-owned channel does not dispose it.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task Create_WithExistingChannel_DoesNotDisposeChannel()
+    {
+        using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost:1");
+        MxGatewayClient gateway = MxGatewayClient.Create(channel, Options());
+        GalaxyRepositoryClient galaxy = GalaxyRepositoryClient.Create(channel, Options());
+
+        await gateway.DisposeAsync();
+        await galaxy.DisposeAsync();
+
+        // A disposed channel throws ObjectDisposedException when a call is created.
+        var invoker = channel.CreateCallInvoker();
+        Exception? ex = Record.Exception(() => invoker.AsyncUnaryCall(
+            new Method<byte[], byte[]>(
+                MethodType.Unary,
+                "s",
+                "m",
+                Marshallers.Create(b => b, b => b),
+                Marshallers.Create(b => b, b => b)),
+            null,
+            new CallOptions(deadline: DateTime.UtcNow.AddMilliseconds(1)),
+            []).Dispose());
+        Assert.IsNotType<ObjectDisposedException>(ex);
+    }
+
+    /// <summary>Create over an existing channel rejects a null channel.</summary>
+    [Fact]
+    public void Create_WithNullChannel_Throws()
+    {
+        Assert.Throws<ArgumentNullException>(() => MxGatewayClient.Create(null!, Options()));
+        Assert.Throws<ArgumentNullException>(() => GalaxyRepositoryClient.Create(null!, Options()));
+    }
+
+    /// <summary>DisposeAsync swallows a gateway failure during close and still disposes.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task Session_DisposeAsync_SwallowsCloseFailure()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.CloseSessionExceptions.Enqueue(new MxGatewayException("unreachable"));
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        await session.DisposeAsync();
+
+        Assert.Single(transport.CloseSessionCalls);
+        await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
+    }
+
+    /// <summary>DisposeAsync after the client is disposed does not throw.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task Session_DisposeAsync_AfterClientDisposed_DoesNotThrow()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+        await client.DisposeAsync();
+
+        await session.DisposeAsync();
+
+        await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
+    }
+
+    /// <summary>PingAsync sends a Ping command carrying the message.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task PingAsync_SendsPingCommand()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.AddInvokeReply(OkReply(MxCommandKind.Ping));
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        await session.PingAsync("hello");
+
+        MxCommandRequest request = Assert.Single(transport.InvokeCalls).Request;
+        Assert.Equal(MxCommandKind.Ping, request.Command.Kind);
+        Assert.Equal("hello", request.Command.Ping.Message);
+        Assert.Equal("session-fixture", request.SessionId);
+    }
+
+    /// <summary>GetSessionStateAsync returns the typed state.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task GetSessionStateAsync_ReturnsState()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        MxCommandReply reply = OkReply(MxCommandKind.GetSessionState);
+        reply.SessionState = new SessionStateReply { State = SessionState.Ready };
+        transport.AddInvokeReply(reply);
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        SessionState state = await session.GetSessionStateAsync();
+
+        Assert.Equal(SessionState.Ready, state);
+        Assert.Equal(MxCommandKind.GetSessionState, Assert.Single(transport.InvokeCalls).Request.Command.Kind);
+    }
+
+    /// <summary>GetSessionStateAsync throws on a reply without payload.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task GetSessionStateAsync_MissingPayload_ThrowsMalformed()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.AddInvokeReply(OkReply(MxCommandKind.GetSessionState));
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        await Assert.ThrowsAsync<MxGatewayMalformedReplyException>(() => session.GetSessionStateAsync());
+    }
+
+    /// <summary>GetWorkerInfoAsync returns the worker info payload.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task GetWorkerInfoAsync_ReturnsWorkerInfo()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        MxCommandReply reply = OkReply(MxCommandKind.GetWorkerInfo);
+        reply.WorkerInfo = new WorkerInfoReply { WorkerProcessId = 42, WorkerVersion = "1.2.3" };
+        transport.AddInvokeReply(reply);
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        WorkerInfoReply info = await session.GetWorkerInfoAsync();
+
+        Assert.Equal(42, info.WorkerProcessId);
+        Assert.Equal("1.2.3", info.WorkerVersion);
+    }
+
+    /// <summary>DrainEventsAsync sends max_events and returns the events.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task DrainEventsAsync_ReturnsEventsAndSendsMax()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        MxCommandReply reply = OkReply(MxCommandKind.DrainEvents);
+        reply.DrainEvents = new DrainEventsReply();
+        reply.DrainEvents.Events.Add(new MxEvent { SessionId = "session-fixture", WorkerSequence = 5 });
+        transport.AddInvokeReply(reply);
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        IReadOnlyList<MxEvent> events = await session.DrainEventsAsync(10);
+
+        Assert.Equal(5UL, Assert.Single(events).WorkerSequence);
+        Assert.Equal(10U, Assert.Single(transport.InvokeCalls).Request.Command.DrainEvents.MaxEvents);
+    }
+
+    /// <summary>DrainEventsAsync treats a missing payload as an empty drain.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task DrainEventsAsync_MissingPayload_ReturnsEmpty()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.AddInvokeReply(OkReply(MxCommandKind.DrainEvents));
+        await using MxGatewayClient client = new(transport.Options, transport);
+        MxGatewaySession session = await client.OpenSessionAsync();
+
+        Assert.Empty(await session.DrainEventsAsync());
+    }
+
+    /// <summary>AttachSession binds to an existing id without opening a session and resumes the stream.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task AttachSession_BindsExistingIdWithoutOpening_AndResumesStream()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        transport.AddEvent(new MxEvent { SessionId = "existing", WorkerSequence = 8 });
+        await using MxGatewayClient client = new(transport.Options, transport);
+
+        MxGatewaySession session = client.AttachSession("existing");
+        List<MxEvent> received = [];
+        await foreach (MxEvent e in session.StreamEventsAsync(afterWorkerSequence: 7))
+        {
+            received.Add(e);
+        }
+
+        Assert.Empty(transport.OpenSessionCalls);
+        Assert.Equal("existing", session.SessionId);
+        StreamEventsRequest request = Assert.Single(transport.StreamEventsCalls).Request;
+        Assert.Equal("existing", request.SessionId);
+        Assert.Equal(7UL, request.AfterWorkerSequence);
+        Assert.Single(received);
+    }
+
+    /// <summary>AttachSession rejects blank ids and a disposed client.</summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task AttachSession_InvalidArguments_Throw()
+    {
+        FakeGatewayTransport transport = CreateTransport();
+        MxGatewayClient client = new(transport.Options, transport);
+
+        Assert.Throws<ArgumentException>(() => client.AttachSession(" "));
+        await client.DisposeAsync();
+        Assert.Throws<ObjectDisposedException>(() => client.AttachSession("x"));
+    }
+
+    private static MxCommandReply OkReply(MxCommandKind kind) => new()
+    {
+        SessionId = "session-fixture",
+        Kind = kind,
+        ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
+    };
+
+    private static MxGatewayClientOptions Options() => new()
+    {
+        Endpoint = new Uri("http://localhost:5000"),
+        ApiKey = "test-api-key",
+    };
+
+    private static FakeGatewayTransport CreateTransport() => new(Options());
+}

clients/dotnet/ZB.MOM.WW.MxGateway.Client/GalaxyRepositoryClient.cs

@@ -22,6 +22,7 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
     private const int BrowseChildrenPageSize = 500;
 
     private readonly GrpcChannel? _channel;
+    private readonly bool _ownsChannel;
     private readonly IGalaxyRepositoryClientTransport _transport;
     private readonly ResiliencePipeline _safeUnaryRetryPipeline;
     private bool _disposed;
@@ -48,9 +49,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
 
     private GalaxyRepositoryClient(
         GrpcChannel channel,
-        IGalaxyRepositoryClientTransport transport)
+        IGalaxyRepositoryClientTransport transport,
+        bool ownsChannel)
     {
         _channel = channel;
+        _ownsChannel = ownsChannel;
         _transport = transport;
         Options = transport.Options;
         _safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -95,7 +98,32 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
             channel,
             new GrpcGalaxyRepositoryClientTransport(
                 options,
-                new GalaxyRepository.GalaxyRepositoryClient(channel)));
+                new GalaxyRepository.GalaxyRepositoryClient(channel)),
+            ownsChannel: true);
+    }
+
+    /// <summary>
+    /// Creates a Galaxy Repository client over a caller-owned <see cref="GrpcChannel"/>.
+    /// The client does not dispose the channel; the caller controls its lifetime and any
+    /// TLS/handler configuration.
+    /// </summary>
+    /// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
+    /// <param name="options">Client options (API key, timeouts, retry).</param>
+    /// <returns>A new client instance that does not own the channel.</returns>
+    public static GalaxyRepositoryClient Create(
+        GrpcChannel channel,
+        MxGatewayClientOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(channel);
+        ArgumentNullException.ThrowIfNull(options);
+        options.Validate();
+
+        return new GalaxyRepositoryClient(
+            channel,
+            new GrpcGalaxyRepositoryClientTransport(
+                options,
+                new GalaxyRepository.GalaxyRepositoryClient(channel)),
+            ownsChannel: false);
     }
 
     /// <summary>
@@ -436,7 +464,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
         }
 
         _disposed = true;
-        _channel?.Dispose();
+        if (_ownsChannel)
+        {
+            _channel?.Dispose();
+        }
+
         return ValueTask.CompletedTask;
     }
 

clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewayClient.cs

@@ -14,7 +14,8 @@ namespace ZB.MOM.WW.MxGateway.Client;
 /// </summary>
 public sealed class MxGatewayClient : IAsyncDisposable
 {
-    private readonly GrpcChannel _channel;
+    private readonly GrpcChannel? _channel;
+    private readonly bool _ownsChannel;
     private readonly IMxGatewayClientTransport _transport;
     private readonly ResiliencePipeline _safeUnaryRetryPipeline;
     private bool _disposed;
@@ -36,14 +37,16 @@ public sealed class MxGatewayClient : IAsyncDisposable
         _safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
             options.Retry,
             options.LoggerFactory?.CreateLogger<MxGatewayClient>());
-        _channel = null!;
+        _channel = null;
     }
 
     private MxGatewayClient(
         GrpcChannel channel,
-        IMxGatewayClientTransport transport)
+        IMxGatewayClientTransport transport,
+        bool ownsChannel)
     {
         _channel = channel;
+        _ownsChannel = ownsChannel;
         _transport = transport;
         Options = transport.Options;
         _safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -88,7 +91,58 @@ public sealed class MxGatewayClient : IAsyncDisposable
             channel,
             new GrpcMxGatewayClientTransport(
                 options,
-                new MxAccessGateway.MxAccessGatewayClient(channel)));
+                new MxAccessGateway.MxAccessGatewayClient(channel)),
+            ownsChannel: true);
+    }
+
+    /// <summary>
+    /// Creates a gateway client over a caller-owned <see cref="GrpcChannel"/>. The client
+    /// does not dispose the channel; the caller controls its lifetime and any TLS/handler
+    /// configuration. <see cref="MxGatewayClientOptions.Endpoint"/> is still validated but
+    /// the channel's own address is used for transport.
+    /// </summary>
+    /// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
+    /// <param name="options">Client configuration options (API key, timeouts, retry).</param>
+    /// <returns>A new gateway client instance that does not own the channel.</returns>
+    public static MxGatewayClient Create(
+        GrpcChannel channel,
+        MxGatewayClientOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(channel);
+        ArgumentNullException.ThrowIfNull(options);
+        options.Validate();
+
+        return new MxGatewayClient(
+            channel,
+            new GrpcMxGatewayClientTransport(
+                options,
+                new MxAccessGateway.MxAccessGatewayClient(channel)),
+            ownsChannel: false);
+    }
+
+    /// <summary>
+    /// Binds a session object to an existing gateway session id without calling
+    /// OpenSession, so a reconnecting process can resume
+    /// <see cref="MxGatewaySession.StreamEventsAsync"/> with <c>afterWorkerSequence</c>.
+    /// The gateway only allows the API key that opened the session to stream its events
+    /// or invoke commands on it; any other key is rejected with PermissionDenied. The
+    /// returned session's <see cref="MxGatewaySession.OpenSessionReply"/> carries only the
+    /// session id.
+    /// </summary>
+    /// <param name="sessionId">The id of a session previously opened with this client's API key.</param>
+    /// <returns>A session bound to <paramref name="sessionId"/>.</returns>
+    public MxGatewaySession AttachSession(string sessionId)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(sessionId);
+        ThrowIfDisposed();
+
+        return new MxGatewaySession(
+            this,
+            new OpenSessionReply
+            {
+                SessionId = sessionId,
+                ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
+            });
     }
 
     /// <summary>
@@ -183,24 +237,34 @@ public sealed class MxGatewayClient : IAsyncDisposable
     }
 
     /// <summary>
-    /// Acknowledges an active MXAccess alarm condition through the gateway. The
-    /// gateway authenticates the request against the API key's <c>invoke:alarm-ack</c>
-    /// scope and forwards the acknowledge to the worker's MXAccess session;
-    /// the resulting <see cref="MxStatusProxy"/> is returned in the reply.
+    /// Acknowledges an alarm as the operator. The gateway authenticates
+    /// <see cref="AcknowledgeAlarmRequest.OperatorUser"/> with
+    /// <see cref="AcknowledgeAlarmRequest.OperatorPassword"/> and writes the
+    /// comment to the alarm's AckMsg attribute, so the alarm history records
+    /// the operator. The password is redacted from any surfaced error.
     /// </summary>
-    /// <param name="request">The acknowledge request — alarm reference, comment, operator user.</param>
+    /// <param name="request">The acknowledge request: alarm reference, comment, operator credentials.</param>
     /// <param name="cancellationToken">Cancellation token for the operation.</param>
     /// <returns>The acknowledge reply with protocol + native MxStatus.</returns>
-    public Task<AcknowledgeAlarmReply> AcknowledgeAlarmAsync(
+    public async Task<AcknowledgeAlarmReply> AcknowledgeAlarmAsync(
         AcknowledgeAlarmRequest request,
         CancellationToken cancellationToken = default)
     {
         ArgumentNullException.ThrowIfNull(request);
         ThrowIfDisposed();
 
-        return ExecuteSafeUnaryAsync(
-            token => _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(token)),
-            cancellationToken);
+        // Deliberately not wrapped in the retry pipeline: an acknowledge is a
+        // non-idempotent operator write that re-sends the operator password and
+        // can double-ack in the Historian.
+        try
+        {
+            return await _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(cancellationToken))
+                .ConfigureAwait(false);
+        }
+        catch (MxGatewayException ex) when (request.OperatorPassword.Length > 0)
+        {
+            throw MxGatewaySecretRedaction.Redacted(ex, request.OperatorPassword);
+        }
     }
 
     /// <summary>
@@ -258,7 +322,11 @@ public sealed class MxGatewayClient : IAsyncDisposable
         }
 
         _disposed = true;
-        _channel?.Dispose();
+        if (_ownsChannel)
+        {
+            _channel?.Dispose();
+        }
+
         return ValueTask.CompletedTask;
     }
 

clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs

@@ -1,3 +1,4 @@
+using Grpc.Core;
 using ZB.MOM.WW.MxGateway.Contracts.Proto;
 
 namespace ZB.MOM.WW.MxGateway.Client;
@@ -640,7 +641,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
     /// </summary>
     /// <param name="serverHandle">The ServerHandle from register.</param>
     /// <param name="tagAddresses">Tag addresses to read (one per result).</param>
-    /// <param name="timeout">Per-call timeout for the snapshot lifecycle path; <see cref="TimeSpan.Zero"/> uses the gateway default.</param>
+    /// <param name="timeout">One shared snapshot deadline for the whole command, not a per-tag budget: the worker advises every uncached tag, waits once for all of them, then tears them down. <see cref="TimeSpan.Zero"/> uses the worker default (1000 ms).</param>
     /// <param name="cancellationToken">Cancellation token.</param>
     /// <returns>One <see cref="BulkReadResult"/> per requested tag, in request order.</returns>
     public async Task<IReadOnlyList<BulkReadResult>> ReadBulkAsync(
@@ -1452,6 +1453,106 @@ public sealed class MxGatewaySession : IAsyncDisposable
         return _client.InvokeAsync(request, cancellationToken);
     }
 
+    /// <summary>
+    /// Sends a PING diagnostic command to the worker and throws if the reply is a failure.
+    /// </summary>
+    /// <param name="message">Optional message echoed to the worker; defaults to empty.</param>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>A task that completes when the worker acknowledges the ping.</returns>
+    public async Task PingAsync(
+        string? message = null,
+        CancellationToken cancellationToken = default)
+    {
+        MxCommandReply reply = await InvokeCommandAsync(
+                new MxCommand
+                {
+                    Kind = MxCommandKind.Ping,
+                    Ping = new PingCommand { Message = message ?? string.Empty },
+                },
+                cancellationToken)
+            .ConfigureAwait(false);
+        reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
+    }
+
+    /// <summary>
+    /// Gets the gateway's lifecycle state for this session.
+    /// </summary>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>The current session state.</returns>
+    public async Task<SessionState> GetSessionStateAsync(CancellationToken cancellationToken = default)
+    {
+        MxCommandReply reply = await InvokeCommandAsync(
+                new MxCommand
+                {
+                    Kind = MxCommandKind.GetSessionState,
+                    GetSessionState = new GetSessionStateCommand(),
+                },
+                cancellationToken)
+            .ConfigureAwait(false);
+        reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
+
+        return reply.SessionState?.State
+            ?? throw CreateMalformed(reply, "GetSessionState");
+    }
+
+    /// <summary>
+    /// Gets diagnostic information about the worker process serving this session.
+    /// </summary>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>The worker process id, version, and MXAccess COM identifiers.</returns>
+    public async Task<WorkerInfoReply> GetWorkerInfoAsync(CancellationToken cancellationToken = default)
+    {
+        MxCommandReply reply = await InvokeCommandAsync(
+                new MxCommand
+                {
+                    Kind = MxCommandKind.GetWorkerInfo,
+                    GetWorkerInfo = new GetWorkerInfoCommand(),
+                },
+                cancellationToken)
+            .ConfigureAwait(false);
+        reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
+
+        return reply.WorkerInfo
+            ?? throw CreateMalformed(reply, "GetWorkerInfo");
+    }
+
+    /// <summary>
+    /// Drains events queued in the worker through the diagnostic command path. This removes
+    /// events from the worker queue; it is not a substitute for <see cref="StreamEventsAsync"/>
+    /// and is not retried automatically. The reply is bounded by server-side caps, so call
+    /// repeatedly until it returns an empty list.
+    /// </summary>
+    /// <param name="maxEvents">Maximum events to return; 0 lets the server choose.</param>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>The drained events, possibly empty.</returns>
+    public async Task<IReadOnlyList<MxEvent>> DrainEventsAsync(
+        uint maxEvents = 0,
+        CancellationToken cancellationToken = default)
+    {
+        MxCommandReply reply = await InvokeCommandAsync(
+                new MxCommand
+                {
+                    Kind = MxCommandKind.DrainEvents,
+                    DrainEvents = new DrainEventsCommand { MaxEvents = maxEvents },
+                },
+                cancellationToken)
+            .ConfigureAwait(false);
+        reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
+
+        return reply.DrainEvents?.Events.ToArray() ?? [];
+    }
+
+    private static MxGatewayMalformedReplyException CreateMalformed(MxCommandReply reply, string operation)
+    {
+        return new MxGatewayMalformedReplyException(
+            $"{operation} returned a malformed reply: OK reply carried no typed payload",
+            reply.SessionId,
+            reply.CorrelationId,
+            reply.ProtocolStatus,
+            reply.HasHresult ? reply.Hresult : null,
+            reply.Statuses.ToArray());
+    }
+
     /// <summary>
     /// Streams events from the worker for this session, optionally starting after a given sequence number.
     /// </summary>
@@ -1505,8 +1606,20 @@ public sealed class MxGatewaySession : IAsyncDisposable
     /// <returns>A task that represents the asynchronous operation.</returns>
     public async ValueTask DisposeAsync()
     {
-        await CloseAsync().ConfigureAwait(false);
-        _closeLock.Dispose();
+        try
+        {
+            await CloseAsync().ConfigureAwait(false);
+        }
+        catch (Exception ex) when (ex is MxGatewayException or ObjectDisposedException or RpcException)
+        {
+            // Best-effort close: the gateway may be unreachable or the owning client
+            // already disposed. Disposal must not throw or leak the lock; the gateway
+            // reaps the session on its own idle/detach timers.
+        }
+        finally
+        {
+            _closeLock.Dispose();
+        }
     }
 
     private Task<MxCommandReply> InvokeCommandAsync(

clients/go/README.md

@@ -100,6 +100,29 @@ judges each `MXSTATUS_PROXY` entry by its category — an entry fails when
 `Category` is not `MX_STATUS_CATEGORY_OK`, and the raw `Success` member is a
 diagnostic that never decides the verdict. A nil entry is success.
 
+`Dial` is connect-or-fail: it creates the connection with `grpc.NewClient`
+(the endpoint resolves with the `passthrough` scheme unless it names one) and
+then waits for the channel to become READY, bounded by `DialTimeout` (default
+10 s) or the context deadline. A failure is returned as a `GatewayError` with
+`Op == "dial"`. Passing a connection to `NewClient`/`NewGalaxyClient` stays lazy.
+
+### Event buffering
+
+The channel returned by `Session.Events`/`EventsAfter` buffers
+`Options.EventBufferSize` data results (default `DefaultEventBufferSize`, 1024).
+If the consumer falls behind and the buffer fills, the stream is cancelled; the
+results already buffered are still delivered in order, followed by a final
+result whose error matches `errors.Is(err, mxgateway.ErrSlowConsumer)`. Raise
+`EventBufferSize` for bursty feeds, or use `SubscribeEvents`/`SubscribeEventsAfter`,
+which block instead of dropping and are the non-dropping alternative.
+
+### Diagnostics
+
+`Session.Ping`, `GetSessionState`, `GetWorkerInfo`, and `DrainEvents` are typed
+wrappers over the diagnostic commands (`PingRaw` remains for the raw reply).
+`DrainEvents(ctx, maxEvents)` returns fewer events than requested when the
+server caps the reply; call it until it returns an empty slice.
+
 ### Reconnect-replay gap
 
 Each `EventResult` carries exactly one of `Event`, `ReplayGap`, or `Err`. When
@@ -140,7 +163,11 @@ sentinel typed and observable; it never synthesizes or swallows it.
 
 For alarms, the package exposes `Client.QueryActiveAlarms` for one-shot
 snapshots, `Client.StreamAlarms` for the server-streaming feed, and
-`Client.AcknowledgeAlarm` to ack an alarm by full reference. The streaming
+`Client.AcknowledgeAlarm` to ack an alarm by full reference. Acks carry the
+operator's Galaxy credentials (`OperatorUser` required, `OperatorPassword`,
+optional `OperatorDomain`) so the gateway authenticates as the operator and the
+Historian records that operator; an empty password is allowed, and the password
+is scrubbed from any returned error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The streaming
 call returns a `StreamAlarmsClient`; cancel its context to terminate the
 stream. All three pass straight through to the gateway's central alarm
 monitor.
@@ -424,7 +451,7 @@ Every subcommand wired into the CLI. All accept the common flags
 | `bench-read-bulk` | Throughput benchmark (`-duration-seconds`, `-warmup-seconds`, `-bulk-size`). |
 | `stream-events` | Stream item-value events for a session (`-session-id`, `-limit`). |
 | `stream-alarms` | Stream the alarm feed (`-filter-prefix`, `-limit`). |
-| `acknowledge-alarm` | Acknowledge an alarm reference. |
+| `acknowledge-alarm` | Acknowledge an alarm as an operator (`-operator`, `-operator-domain`, `-operator-password-env`, default `MXGATEWAY_OPERATOR_PASSWORD`). |
 | `smoke` | End-to-end smoke workflow against one item. |
 | `galaxy-test-connection` | Probe the Galaxy Repository RPC connection. |
 | `galaxy-last-deploy` | Print the most recent deploy event. |

clients/go/cmd/mxgw-go/main.go

@@ -571,7 +571,7 @@ func runReadBulk(ctx context.Context, args []string, stdout, stderr io.Writer) e
 	sessionID := flags.String("session-id", "", "gateway session id")
 	serverHandle := flags.Int("server-handle", 0, "MXAccess server handle")
 	items := flags.String("items", "", "comma-separated tag addresses")
-	timeoutMs := flags.Int("timeout-ms", 0, "per-tag snapshot timeout in milliseconds (0 = worker default)")
+	timeoutMs := flags.Int("timeout-ms", 0, "snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms)")
 
 	if err := flags.Parse(args); err != nil {
 		return err
@@ -754,7 +754,7 @@ func runBenchReadBulk(ctx context.Context, args []string, stdout, stderr io.Writ
 	tagStart := flags.Int("tag-start", 1, "first machine number")
 	tagPrefix := flags.String("tag-prefix", "TestMachine_", "tag prefix (machine number appended as %03d)")
 	tagAttribute := flags.String("tag-attribute", "TestChangingInt", "attribute appended to each tag prefix")
-	timeoutMs := flags.Int("timeout-ms", 1500, "per-tag snapshot timeout in milliseconds")
+	timeoutMs := flags.Int("timeout-ms", 1500, "snapshot deadline in milliseconds, shared by all tags in each ReadBulk command")
 
 	if err := flags.Parse(args); err != nil {
 		return err
@@ -1119,6 +1119,10 @@ func formatAlarmFeedMessage(message *mxgateway.AlarmFeedMessage) string {
 	}
 }
 
+// defaultOperatorPasswordEnv is the CLI environment variable holding the
+// operator password for acknowledge-alarm.
+const defaultOperatorPasswordEnv = "MXGATEWAY_OPERATOR_PASSWORD"
+
 func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.Writer) error {
 	flags := flag.NewFlagSet("acknowledge-alarm", flag.ContinueOnError)
 	flags.SetOutput(stderr)
@@ -1126,7 +1130,12 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
 	jsonOutput := flags.Bool("json", false, "write JSON output")
 	reference := flags.String("reference", "", "full alarm reference to acknowledge")
 	comment := flags.String("comment", "", "operator acknowledge comment")
-	operator := flags.String("operator", "", "operator user performing the acknowledge")
+	operator := flags.String("operator", "", "operator user performing the acknowledge (required)")
+	operatorDomain := flags.String("operator-domain", "", "operator Galaxy domain (optional)")
+	// Prefer the environment variable so the credential stays out of shell
+	// history and the process table; an empty password is allowed.
+	operatorPassword := flags.String("operator-password", "", "operator password (prefer -operator-password-env)")
+	operatorPasswordEnv := flags.String("operator-password-env", defaultOperatorPasswordEnv, "environment variable containing the operator password")
 
 	if err := flags.Parse(args); err != nil {
 		return err
@@ -1134,6 +1143,17 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
 	if *reference == "" {
 		return errors.New("reference is required")
 	}
+	if *operator == "" {
+		return errors.New("operator is required")
+	}
+	resolvedOperatorPassword := *operatorPassword
+	if resolvedOperatorPassword == "" {
+		envName := *operatorPasswordEnv
+		if envName == "" {
+			envName = defaultOperatorPasswordEnv
+		}
+		resolvedOperatorPassword = os.Getenv(envName)
+	}
 
 	client, options, err := dialForCommand(ctx, common)
 	if err != nil {
@@ -1145,6 +1165,8 @@ func runAcknowledgeAlarm(ctx context.Context, args []string, stdout, stderr io.W
 		AlarmFullReference: *reference,
 		Comment:            *comment,
 		OperatorUser:       *operator,
+		OperatorDomain:     *operatorDomain,
+		OperatorPassword:   resolvedOperatorPassword,
 	})
 	if err != nil {
 		return err

clients/go/internal/generated/mxaccess_gateway.pb.go

@@ -3583,10 +3583,13 @@ func (x *AlarmFailoverConfig) GetFailbackStableProbes() int32 {
 	return 0
 }
 
-// Acknowledge a single alarm by its GUID. Operator identity fields are
-// recorded atomically with the ack transition in the alarm-history log.
-// The reply's hresult / native_status surfaces AVEVA's
-// AlarmAckByGUID return code.
+// Acknowledge a single alarm by its GUID. The worker resolves the GUID to the
+// alarm's Galaxy reference from its active-alarm snapshot and acknowledges it
+// by writing `comment` to `<TAGNAME>.AckMsg` over MXAccess after
+// AuthenticateUser as the operator, so the alarm history records the real
+// operator. The reply's hresult / native_status is 0 when the write was
+// applied (or sent but unconfirmed), otherwise the AuthenticateUser HRESULT or
+// the OnWriteComplete status detail.
 type AcknowledgeAlarmCommand struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
 	// Canonical 8-4-4-4-12 GUID string (e.g. "BCC47053-9542-4D65-BDAA-BCDEA6A32A73").
@@ -3596,6 +3599,10 @@ type AcknowledgeAlarmCommand struct {
 	OperatorNode     string `protobuf:"bytes,4,opt,name=operator_node,json=operatorNode,proto3" json:"operator_node,omitempty"`
 	OperatorDomain   string `protobuf:"bytes,5,opt,name=operator_domain,json=operatorDomain,proto3" json:"operator_domain,omitempty"`
 	OperatorFullName string `protobuf:"bytes,6,opt,name=operator_full_name,json=operatorFullName,proto3" json:"operator_full_name,omitempty"`
+	// Operator credential used for AuthenticateUser before the AckMsg write.
+	// Implementations must keep this field out of logs, metrics labels, command
+	// lines, and diagnostics.
+	OperatorPassword string `protobuf:"bytes,7,opt,name=operator_password,json=operatorPassword,proto3" json:"operator_password,omitempty"`
 	unknownFields    protoimpl.UnknownFields
 	sizeCache        protoimpl.SizeCache
 }
@@ -3672,6 +3679,13 @@ func (x *AcknowledgeAlarmCommand) GetOperatorFullName() string {
 	return ""
 }
 
+func (x *AcknowledgeAlarmCommand) GetOperatorPassword() string {
+	if x != nil {
+		return x.OperatorPassword
+	}
+	return ""
+}
+
 // Snapshot the currently-active alarm set. Optional filter prefix scopes
 // the snapshot to alarms whose alarm_full_reference starts with the
 // supplied string (matches QueryActiveAlarmsRequest.alarm_filter_prefix).
@@ -3721,9 +3735,7 @@ func (x *QueryActiveAlarmsCommand) GetAlarmFilterPrefix() string {
 
 // Acknowledge a single alarm by its (name, provider, group) tuple. Used
 // when the public RPC's AlarmFullReference (Provider!Group.Tag) cannot
-// be resolved to a GUID directly. The worker invokes
-// wwAlarmConsumerClass.AlarmAckByName which reaches the same alarm
-// history path as AlarmAckByGUID.
+// be resolved to a GUID directly. The worker acknowledges by writing `comment` to `<alarm_name>.AckMsg` (Galaxy provider only) after AuthenticateUser as the operator.
 type AcknowledgeAlarmByNameCommand struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
 	// Tag/alarm name (e.g. "TestMachine_001.TestAlarm001"). Tag itself
@@ -3739,6 +3751,10 @@ type AcknowledgeAlarmByNameCommand struct {
 	OperatorNode     string `protobuf:"bytes,6,opt,name=operator_node,json=operatorNode,proto3" json:"operator_node,omitempty"`
 	OperatorDomain   string `protobuf:"bytes,7,opt,name=operator_domain,json=operatorDomain,proto3" json:"operator_domain,omitempty"`
 	OperatorFullName string `protobuf:"bytes,8,opt,name=operator_full_name,json=operatorFullName,proto3" json:"operator_full_name,omitempty"`
+	// Operator credential used for AuthenticateUser before the AckMsg write.
+	// Implementations must keep this field out of logs, metrics labels, command
+	// lines, and diagnostics.
+	OperatorPassword string `protobuf:"bytes,9,opt,name=operator_password,json=operatorPassword,proto3" json:"operator_password,omitempty"`
 	unknownFields    protoimpl.UnknownFields
 	sizeCache        protoimpl.SizeCache
 }
@@ -3829,6 +3845,13 @@ func (x *AcknowledgeAlarmByNameCommand) GetOperatorFullName() string {
 	return ""
 }
 
+func (x *AcknowledgeAlarmByNameCommand) GetOperatorPassword() string {
+	if x != nil {
+		return x.OperatorPassword
+	}
+	return ""
+}
+
 type UnsubscribeBulkCommand struct {
 	state         protoimpl.MessageState `protogen:"open.v1"`
 	ServerHandle  int32                  `protobuf:"varint,1,opt,name=server_handle,json=serverHandle,proto3" json:"server_handle,omitempty"`
@@ -6127,11 +6150,11 @@ func (x *DrainEventsReply) GetEvents() []*MxEvent {
 // Reply payload for AcknowledgeAlarmCommand AND
 // AcknowledgeAlarmByNameCommand — both ack command kinds reuse this
 // payload case (`MxCommandReply.acknowledge_alarm`); there is no
-// dedicated by-name reply case. Surfaces AVEVA's native ack return
-// code (AlarmAckByGUID for the GUID arm, AlarmAckByName for the
-// by-name arm); 0 means success. The MxCommandReply's hresult field
+// dedicated by-name reply case. Surfaces the ack outcome: 0 when the
+// AckMsg write was applied or sent but unconfirmed; otherwise the
+// AuthenticateUser HRESULT or the OnWriteComplete status detail. The MxCommandReply's hresult field
 // carries the same value and is preferred for protocol consumers —
-// this payload exists so the gateway-side WorkerAlarmRpcDispatcher
+// this payload exists so GatewayAlarmMonitor
 // can echo native_status into AcknowledgeAlarmReply.hresult without
 // unpacking the outer envelope.
 type AcknowledgeAlarmReplyPayload struct {
@@ -7217,9 +7240,18 @@ type AcknowledgeAlarmRequest struct {
 	Comment string `protobuf:"bytes,4,opt,name=comment,proto3" json:"comment,omitempty"`
 	// Operator principal performing the acknowledgement. The lmxopcua side
 	// resolves this from the OPC UA session prior to invoking the RPC.
-	OperatorUser  string `protobuf:"bytes,5,opt,name=operator_user,json=operatorUser,proto3" json:"operator_user,omitempty"`
-	unknownFields protoimpl.UnknownFields
-	sizeCache     protoimpl.SizeCache
+	OperatorUser string `protobuf:"bytes,5,opt,name=operator_user,json=operatorUser,proto3" json:"operator_user,omitempty"`
+	// Operator's Galaxy (or OS) password. The gateway authenticates as
+	// operator_user with it before writing the acknowledgement so the alarm
+	// history records the real operator. May be empty for accounts without a
+	// password. Credential-sensitive: implementations must keep this field out
+	// of logs, metrics labels, command lines, and diagnostics.
+	OperatorPassword string `protobuf:"bytes,6,opt,name=operator_password,json=operatorPassword,proto3" json:"operator_password,omitempty"`
+	// Optional domain for OS-authenticated galaxies. When set the gateway
+	// authenticates as "<operator_domain>\<operator_user>".
+	OperatorDomain string `protobuf:"bytes,7,opt,name=operator_domain,json=operatorDomain,proto3" json:"operator_domain,omitempty"`
+	unknownFields  protoimpl.UnknownFields
+	sizeCache      protoimpl.SizeCache
 }
 
 func (x *AcknowledgeAlarmRequest) Reset() {
@@ -7280,14 +7312,27 @@ func (x *AcknowledgeAlarmRequest) GetOperatorUser() string {
 	return ""
 }
 
+func (x *AcknowledgeAlarmRequest) GetOperatorPassword() string {
+	if x != nil {
+		return x.OperatorPassword
+	}
+	return ""
+}
+
+func (x *AcknowledgeAlarmRequest) GetOperatorDomain() string {
+	if x != nil {
+		return x.OperatorDomain
+	}
+	return ""
+}
+
 type AcknowledgeAlarmReply struct {
 	state          protoimpl.MessageState `protogen:"open.v1"`
 	CorrelationId  string                 `protobuf:"bytes,2,opt,name=correlation_id,json=correlationId,proto3" json:"correlation_id,omitempty"`
 	ProtocolStatus *ProtocolStatus        `protobuf:"bytes,3,opt,name=protocol_status,json=protocolStatus,proto3" json:"protocol_status,omitempty"`
 	// Native ack return code echoed from the worker. The worker carries the
 	// ack outcome as a single int32 (AcknowledgeAlarmReplyPayload.native_status,
-	// = AlarmAckByName / AlarmAckByGUID return code; 0 = success); the gateway's
-	// WorkerAlarmRpcDispatcher copies that value here. This is the authoritative
+	// 0 = applied or sent; otherwise the AuthenticateUser HRESULT or OnWriteComplete detail); GatewayAlarmMonitor copies that value here. This is the authoritative
 	// ack-outcome field for the public RPC. Absent only when the worker reply
 	// omitted the value entirely (a protocol violation).
 	Hresult *int32 `protobuf:"varint,4,opt,name=hresult,proto3,oneof" json:"hresult,omitempty"`
@@ -8992,7 +9037,7 @@ const file_mxaccess_gateway_proto_rawDesc = "" +
 	"\x13AlarmFailoverConfig\x12B\n" +
 	"\x1dconsecutive_failure_threshold\x18\x01 \x01(\x05R\x1bconsecutiveFailureThreshold\x12E\n" +
 	"\x1ffailback_probe_interval_seconds\x18\x02 \x01(\x05R\x1cfailbackProbeIntervalSeconds\x124\n" +
-	"\x16failback_stable_probes\x18\x03 \x01(\x05R\x14failbackStableProbes\"\xf3\x01\n" +
+	"\x16failback_stable_probes\x18\x03 \x01(\x05R\x14failbackStableProbes\"\xa0\x02\n" +
 	"\x17AcknowledgeAlarmCommand\x12\x1d\n" +
 	"\n" +
 	"alarm_guid\x18\x01 \x01(\tR\talarmGuid\x12\x18\n" +
@@ -9000,9 +9045,10 @@ const file_mxaccess_gateway_proto_rawDesc = "" +
 	"\roperator_user\x18\x03 \x01(\tR\foperatorUser\x12#\n" +
 	"\roperator_node\x18\x04 \x01(\tR\foperatorNode\x12'\n" +
 	"\x0foperator_domain\x18\x05 \x01(\tR\x0eoperatorDomain\x12,\n" +
-	"\x12operator_full_name\x18\x06 \x01(\tR\x10operatorFullName\"J\n" +
+	"\x12operator_full_name\x18\x06 \x01(\tR\x10operatorFullName\x12+\n" +
+	"\x11operator_password\x18\a \x01(\tR\x10operatorPassword\"J\n" +
 	"\x18QueryActiveAlarmsCommand\x12.\n" +
-	"\x13alarm_filter_prefix\x18\x01 \x01(\tR\x11alarmFilterPrefix\"\xbd\x02\n" +
+	"\x13alarm_filter_prefix\x18\x01 \x01(\tR\x11alarmFilterPrefix\"\xea\x02\n" +
 	"\x1dAcknowledgeAlarmByNameCommand\x12\x1d\n" +
 	"\n" +
 	"alarm_name\x18\x01 \x01(\tR\talarmName\x12#\n" +
@@ -9013,7 +9059,8 @@ const file_mxaccess_gateway_proto_rawDesc = "" +
 	"\roperator_user\x18\x05 \x01(\tR\foperatorUser\x12#\n" +
 	"\roperator_node\x18\x06 \x01(\tR\foperatorNode\x12'\n" +
 	"\x0foperator_domain\x18\a \x01(\tR\x0eoperatorDomain\x12,\n" +
-	"\x12operator_full_name\x18\b \x01(\tR\x10operatorFullName\"`\n" +
+	"\x12operator_full_name\x18\b \x01(\tR\x10operatorFullName\x12+\n" +
+	"\x11operator_password\x18\t \x01(\tR\x10operatorPassword\"`\n" +
 	"\x16UnsubscribeBulkCommand\x12#\n" +
 	"\rserver_handle\x18\x01 \x01(\x05R\fserverHandle\x12!\n" +
 	"\fitem_handles\x18\x02 \x03(\x05R\vitemHandles\"v\n" +
@@ -9262,12 +9309,14 @@ const file_mxaccess_gateway_proto_rawDesc = "" +
 	"limitValue\x12\x1a\n" +
 	"\bdegraded\x18\x0e \x01(\bR\bdegraded\x12O\n" +
 	"\x0fsource_provider\x18\x0f \x01(\x0e2&.mxaccess_gateway.v1.AlarmProviderModeR\x0esourceProvider\x126\n" +
-	"\x17from_truncated_snapshot\x18\x10 \x01(\bR\x15fromTruncatedSnapshot\"\xd0\x01\n" +
+	"\x17from_truncated_snapshot\x18\x10 \x01(\bR\x15fromTruncatedSnapshot\"\xa6\x02\n" +
 	"\x17AcknowledgeAlarmRequest\x122\n" +
 	"\x15client_correlation_id\x18\x02 \x01(\tR\x13clientCorrelationId\x120\n" +
 	"\x14alarm_full_reference\x18\x03 \x01(\tR\x12alarmFullReference\x12\x18\n" +
 	"\acomment\x18\x04 \x01(\tR\acomment\x12#\n" +
-	"\roperator_user\x18\x05 \x01(\tR\foperatorUserJ\x04\b\x01\x10\x02R\n" +
+	"\roperator_user\x18\x05 \x01(\tR\foperatorUser\x12+\n" +
+	"\x11operator_password\x18\x06 \x01(\tR\x10operatorPassword\x12'\n" +
+	"\x0foperator_domain\x18\a \x01(\tR\x0eoperatorDomainJ\x04\b\x01\x10\x02R\n" +
 	"session_id\"\xb4\x02\n" +
 	"\x15AcknowledgeAlarmReply\x12%\n" +
 	"\x0ecorrelation_id\x18\x02 \x01(\tR\rcorrelationId\x12L\n" +

clients/go/mxgateway/alarms.go

@@ -10,6 +10,11 @@ import (
 // invoke:alarm-ack scope and forwards the acknowledge to the worker's MXAccess
 // session; the resulting native MxStatus is returned in the reply.
 //
+// OperatorUser is required: the gateway authenticates the operator (with
+// OperatorPassword and optional OperatorDomain) so the Historian records the
+// real operator. An empty password is allowed. The password is scrubbed from
+// any error this method returns.
+//
 // Acks are idempotent — re-acking an already-acked condition is a no-op at
 // the MxAccess layer.
 func (c *Client) AcknowledgeAlarm(ctx context.Context, req *AcknowledgeAlarmRequest) (*AcknowledgeAlarmReply, error) {
@@ -22,10 +27,10 @@ func (c *Client) AcknowledgeAlarm(ctx context.Context, req *AcknowledgeAlarmRequ
 
 	reply, err := c.raw.AcknowledgeAlarm(callCtx, req)
 	if err != nil {
-		return nil, &GatewayError{Op: "acknowledge alarm", Err: err}
+		return nil, redactSecrets(&GatewayError{Op: "acknowledge alarm", Err: err}, req.GetOperatorPassword())
 	}
 	if err := EnsureProtocolSuccess("acknowledge alarm", reply.GetProtocolStatus(), nil); err != nil {
-		return reply, err
+		return reply, redactSecrets(err, req.GetOperatorPassword())
 	}
 
 	return reply, nil

clients/go/mxgateway/alarms_test.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"io"
 	"net"
+	"strings"
 	"testing"
 
 	pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
@@ -38,6 +39,8 @@ func TestAcknowledgeAlarmSendsRequestAndReturnsReply(t *testing.T) {
 		AlarmFullReference:  "Tank01.Level.HiHi",
 		Comment:             "investigating",
 		OperatorUser:        "alice",
+		OperatorPassword:    "pw-secret",
+		OperatorDomain:      "CORP",
 	})
 	if err != nil {
 		t.Fatalf("AcknowledgeAlarm() error = %v", err)
@@ -51,11 +54,37 @@ func TestAcknowledgeAlarmSendsRequestAndReturnsReply(t *testing.T) {
 	if got := fake.acknowledgeRequest.GetComment(); got != "investigating" {
 		t.Fatalf("captured comment = %q", got)
 	}
+	if got := fake.acknowledgeRequest.GetOperatorPassword(); got != "pw-secret" {
+		t.Fatalf("captured operator password mismatch")
+	}
+	if got := fake.acknowledgeRequest.GetOperatorDomain(); got != "CORP" {
+		t.Fatalf("captured operator domain = %q", got)
+	}
 	if got := fake.acknowledgeAuth; got != "Bearer test-api-key" {
 		t.Fatalf("authorization metadata = %q", got)
 	}
 }
 
+func TestAcknowledgeAlarmRedactsPasswordFromError(t *testing.T) {
+	fake := &fakeGatewayWithAlarms{
+		acknowledgeError: status.Error(codes.Internal, "boom pw-secret"),
+	}
+	client, cleanup := newBufconnClientWithAlarms(t, fake)
+	defer cleanup()
+
+	_, err := client.AcknowledgeAlarm(context.Background(), &pb.AcknowledgeAlarmRequest{
+		AlarmFullReference: "Tank01.Level.HiHi",
+		OperatorUser:       "alice",
+		OperatorPassword:   "pw-secret",
+	})
+	if err == nil {
+		t.Fatal("AcknowledgeAlarm() returned no error")
+	}
+	if strings.Contains(err.Error(), "pw-secret") {
+		t.Fatalf("error leaks operator password: %v", err)
+	}
+}
+
 func TestAcknowledgeAlarmRejectsNilRequest(t *testing.T) {
 	fake := &fakeGatewayWithAlarms{}
 	client, cleanup := newBufconnClientWithAlarms(t, fake)

clients/go/mxgateway/client.go

@@ -15,10 +15,12 @@ import (
 	"context"
 	"crypto/tls"
 	"errors"
+	"strings"
 	"time"
 
 	pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
 	"google.golang.org/grpc"
+	"google.golang.org/grpc/connectivity"
 	"google.golang.org/grpc/credentials"
 	"google.golang.org/grpc/credentials/insecure"
 	"google.golang.org/protobuf/types/known/durationpb"
@@ -61,11 +63,10 @@ func Dial(ctx context.Context, opts Options) (*Client, error) {
 		grpc.WithTransportCredentials(transportCredentials),
 		grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
 		grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
-		grpc.WithBlock(),
 	}
 	dialOptions = append(dialOptions, opts.DialOptions...)
 
-	conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
+	conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
 	if err != nil {
 		return nil, &GatewayError{Op: "dial", Err: err}
 	}
@@ -265,3 +266,34 @@ func (o OpenSessionOptions) Request() *OpenSessionRequest {
 	}
 	return req
 }
+
+// dialConnected creates a connection with grpc.NewClient and then waits for it
+// to become READY, bounded by ctx. This preserves the connect-or-fail contract
+// the removed grpc.DialContext + grpc.WithBlock combination gave Dial callers.
+// The endpoint is resolved with the passthrough scheme unless it already names a
+// scheme, matching DialContext's historical default (NewClient defaults to dns).
+func dialConnected(ctx context.Context, endpoint string, dialOptions []grpc.DialOption) (*grpc.ClientConn, error) {
+	target := endpoint
+	if !strings.Contains(target, "://") {
+		target = "passthrough:///" + target
+	}
+	conn, err := grpc.NewClient(target, dialOptions...)
+	if err != nil {
+		return nil, err
+	}
+	conn.Connect()
+	for {
+		state := conn.GetState()
+		if state == connectivity.Ready {
+			return conn, nil
+		}
+		if state == connectivity.Shutdown {
+			_ = conn.Close()
+			return nil, errors.New("connection shut down while connecting")
+		}
+		if !conn.WaitForStateChange(ctx, state) {
+			_ = conn.Close()
+			return nil, ctx.Err()
+		}
+	}
+}

clients/go/mxgateway/client_session_test.go

@@ -121,7 +121,7 @@ func TestEventsAfterCancelsStreamWhenCompatibilityChannelIsAbandoned(t *testing.
 		streamDone:       make(chan struct{}),
 		streamEventCount: 64,
 	}
-	client, cleanup := newBufconnClient(t, fake)
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
 	defer cleanup()
 	session := NewSessionForID(client, "session-1")
 
@@ -155,7 +155,7 @@ func TestEventsSlowConsumerYieldsErrSlowConsumerBeforeClose(t *testing.T) {
 		streamDone:       make(chan struct{}),
 		streamEventCount: 64,
 	}
-	client, cleanup := newBufconnClient(t, fake)
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
 	defer cleanup()
 	session := NewSessionForID(client, "session-1")
 
@@ -209,7 +209,7 @@ func TestEventsFullBufferTerminalErrorKeepsRootCause(t *testing.T) {
 		streamEventCount:  eventBufferSize,
 		streamTerminalErr: status.Error(codes.Internal, "boom"),
 	}
-	client, cleanup := newBufconnClient(t, fake)
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
 	defer cleanup()
 	session := NewSessionForID(client, "session-1")
 
@@ -280,7 +280,7 @@ func TestSubscribeEventsFullBufferDeliversTerminalError(t *testing.T) {
 		streamEventCount:  eventBufferSize + eventBufferReservedSlots,
 		streamTerminalErr: status.Error(codes.Internal, "boom"),
 	}
-	client, cleanup := newBufconnClient(t, fake)
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
 	defer cleanup()
 	session := NewSessionForID(client, "session-1")
 
@@ -1047,3 +1047,240 @@ func TestWriteArrayElementsSendsWriteCommandWithSparseOneof(t *testing.T) {
 		t.Errorf("element float value = %v, want 1.5", elem.GetValue().GetFloatValue())
 	}
 }
+
+// newBufconnClientWithOptions mirrors newBufconnClient but lets a test tune Options.
+func newBufconnClientWithOptions(t *testing.T, fake *fakeGatewayServer, opts Options) (*Client, func()) {
+	t.Helper()
+	listener := bufconn.Listen(bufSize)
+	server := grpc.NewServer()
+	pb.RegisterMxAccessGatewayServer(server, fake)
+	go func() { _ = server.Serve(listener) }()
+	opts.Endpoint = "bufnet"
+	opts.APIKey = "test-api-key"
+	opts.Plaintext = true
+	opts.DialOptions = append(opts.DialOptions, grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
+		return listener.DialContext(ctx)
+	}))
+	client, err := Dial(context.Background(), opts)
+	if err != nil {
+		t.Fatalf("Dial() error = %v", err)
+	}
+	return client, func() {
+		client.Close()
+		server.Stop()
+		listener.Close()
+	}
+}
+
+func TestEventsHonorsConfiguredEventBufferSize(t *testing.T) {
+	// 100 events exceed the old fixed 16-slot buffer; a 256-slot buffer must
+	// hold them all without a slow-consumer termination.
+	fake := &fakeGatewayServer{
+		streamStarted:     make(chan struct{}),
+		streamDone:        make(chan struct{}),
+		streamEventCount:  100,
+		streamTerminalErr: status.Error(codes.Internal, "boom"),
+	}
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 256})
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	events, err := session.Events(context.Background())
+	if err != nil {
+		t.Fatalf("Events() error = %v", err)
+	}
+	<-fake.streamDone
+	time.Sleep(250 * time.Millisecond)
+
+	count := 0
+	var last EventResult
+	for res := range events {
+		if res.Event != nil {
+			count++
+		}
+		last = res
+	}
+	if count != 100 {
+		t.Fatalf("delivered events = %d, want 100", count)
+	}
+	if errors.Is(last.Err, ErrSlowConsumer) || status.Code(last.Err) != codes.Internal {
+		t.Fatalf("terminal err = %v, want the genuine Internal error", last.Err)
+	}
+}
+
+func TestEventsOverflowDeliversBufferedEventsBeforeSlowConsumerError(t *testing.T) {
+	fake := &fakeGatewayServer{
+		streamStarted:    make(chan struct{}),
+		streamDone:       make(chan struct{}),
+		streamEventCount: 64,
+	}
+	client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 4})
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	events, err := session.Events(context.Background())
+	if err != nil {
+		t.Fatalf("Events() error = %v", err)
+	}
+	<-fake.streamStarted
+	time.Sleep(250 * time.Millisecond)
+
+	buffered := 0
+	var last EventResult
+	for res := range events {
+		if res.Event != nil {
+			buffered++
+			if got := res.Event.GetWorkerSequence(); got != uint64(buffered) {
+				t.Fatalf("event %d has sequence %d, want in-order", buffered, got)
+			}
+		}
+		last = res
+	}
+	if buffered != 4 {
+		t.Fatalf("buffered events delivered = %d, want 4 (all data slots)", buffered)
+	}
+	if !errors.Is(last.Err, ErrSlowConsumer) {
+		t.Fatalf("final err = %v, want ErrSlowConsumer", last.Err)
+	}
+}
+
+func TestEventBufferSizeDefault(t *testing.T) {
+	if got := (Options{}).eventBufferSize(); got != DefaultEventBufferSize {
+		t.Fatalf("default = %d, want %d", got, DefaultEventBufferSize)
+	}
+	if got := (Options{EventBufferSize: 7}).eventBufferSize(); got != 7 {
+		t.Fatalf("explicit = %d, want 7", got)
+	}
+}
+
+func TestDialFailsWhenEndpointUnreachable(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	_, err := Dial(ctx, Options{
+		Endpoint:    "bufnet",
+		Plaintext:   true,
+		DialTimeout: 300 * time.Millisecond,
+		DialOptions: []grpc.DialOption{grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
+			return nil, errors.New("refused")
+		})},
+	})
+	if err == nil {
+		t.Fatal("Dial() error = nil, want connect failure")
+	}
+	var gwErr *GatewayError
+	if !errors.As(err, &gwErr) || gwErr.Op != "dial" {
+		t.Fatalf("Dial() error = %v, want *GatewayError Op=dial", err)
+	}
+}
+
+func okStatus() *pb.ProtocolStatus {
+	return &pb.ProtocolStatus{Code: pb.ProtocolStatusCode_PROTOCOL_STATUS_CODE_OK}
+}
+
+func TestSessionPingReturnsEchoedMessage(t *testing.T) {
+	fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
+		Kind:              pb.MxCommandKind_MX_COMMAND_KIND_PING,
+		ProtocolStatus:    okStatus(),
+		DiagnosticMessage: "pong:hi",
+	}}
+	client, cleanup := newBufconnClient(t, fake)
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	got, err := session.Ping(context.Background(), "hi")
+	if err != nil {
+		t.Fatalf("Ping() error = %v", err)
+	}
+	if got != "pong:hi" {
+		t.Fatalf("Ping() = %q, want pong:hi", got)
+	}
+	if fake.invokeRequest.GetCommand().GetPing().GetMessage() != "hi" {
+		t.Fatalf("ping payload not sent: %v", fake.invokeRequest)
+	}
+}
+
+func TestSessionGetSessionState(t *testing.T) {
+	fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
+		Kind:           pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
+		ProtocolStatus: okStatus(),
+		Payload: &pb.MxCommandReply_SessionState{SessionState: &pb.SessionStateReply{
+			State: pb.SessionState_SESSION_STATE_CREATING,
+		}},
+	}}
+	client, cleanup := newBufconnClient(t, fake)
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	got, err := session.GetSessionState(context.Background())
+	if err != nil {
+		t.Fatalf("GetSessionState() error = %v", err)
+	}
+	if got != pb.SessionState_SESSION_STATE_CREATING {
+		t.Fatalf("state = %v", got)
+	}
+	if fake.invokeRequest.GetCommand().GetKind() != pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE {
+		t.Fatalf("kind = %v", fake.invokeRequest.GetCommand().GetKind())
+	}
+}
+
+func TestSessionGetWorkerInfo(t *testing.T) {
+	fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
+		Kind:           pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
+		ProtocolStatus: okStatus(),
+		Payload: &pb.MxCommandReply_WorkerInfo{WorkerInfo: &pb.WorkerInfoReply{
+			WorkerProcessId: 42, WorkerVersion: "1.2.3", MxaccessProgid: "prog", MxaccessClsid: "clsid",
+		}},
+	}}
+	client, cleanup := newBufconnClient(t, fake)
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	info, err := session.GetWorkerInfo(context.Background())
+	if err != nil {
+		t.Fatalf("GetWorkerInfo() error = %v", err)
+	}
+	if info.GetWorkerProcessId() != 42 || info.GetWorkerVersion() != "1.2.3" {
+		t.Fatalf("info = %v", info)
+	}
+}
+
+func TestSessionDrainEvents(t *testing.T) {
+	fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
+		Kind:           pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
+		ProtocolStatus: okStatus(),
+		Payload: &pb.MxCommandReply_DrainEvents{DrainEvents: &pb.DrainEventsReply{
+			Events: []*pb.MxEvent{{WorkerSequence: 1}, {WorkerSequence: 2}},
+		}},
+	}}
+	client, cleanup := newBufconnClient(t, fake)
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	events, err := session.DrainEvents(context.Background(), 10)
+	if err != nil {
+		t.Fatalf("DrainEvents() error = %v", err)
+	}
+	if len(events) != 2 || events[1].GetWorkerSequence() != 2 {
+		t.Fatalf("events = %v", events)
+	}
+	if fake.invokeRequest.GetCommand().GetDrainEvents().GetMaxEvents() != 10 {
+		t.Fatalf("max_events not sent: %v", fake.invokeRequest)
+	}
+}
+
+func TestSessionDiagnosticsErrorOnMissingPayload(t *testing.T) {
+	fake := &fakeGatewayServer{}
+	client, cleanup := newBufconnClient(t, fake)
+	defer cleanup()
+	session := NewSessionForID(client, "session-1")
+
+	if _, err := session.GetWorkerInfo(context.Background()); err == nil {
+		t.Fatal("GetWorkerInfo() with empty reply: want error")
+	}
+	if _, err := session.GetSessionState(context.Background()); err == nil {
+		t.Fatal("GetSessionState() with empty reply: want error")
+	}
+	if _, err := session.DrainEvents(context.Background(), 1); err == nil {
+		t.Fatal("DrainEvents() with empty reply: want error")
+	}
+}

clients/go/mxgateway/galaxy.go

@@ -99,11 +99,10 @@ func DialGalaxy(ctx context.Context, opts Options) (*GalaxyClient, error) {
 		grpc.WithTransportCredentials(transportCredentials),
 		grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
 		grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
-		grpc.WithBlock(),
 	}
 	dialOptions = append(dialOptions, opts.DialOptions...)
 
-	conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
+	conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
 	if err != nil {
 		return nil, &GatewayError{Op: "dial", Err: err}
 	}

clients/go/mxgateway/options.go

@@ -38,6 +38,21 @@ type Options struct {
 	// no CACertFile is pinned. Default false: the gateway's self-signed cert is
 	// accepted without verification (internal-tool posture).
 	RequireCertificateValidation bool
+	// EventBufferSize is the number of data results the channel returned by
+	// Session.Events/EventsAfter buffers before a slow consumer is cancelled with
+	// ErrSlowConsumer. Zero or negative applies DefaultEventBufferSize. It does not
+	// affect SubscribeEvents, which blocks instead of dropping.
+	EventBufferSize int
+}
+
+// DefaultEventBufferSize is the default Options.EventBufferSize.
+const DefaultEventBufferSize = 1024
+
+func (o Options) eventBufferSize() int {
+	if o.EventBufferSize <= 0 {
+		return DefaultEventBufferSize
+	}
+	return o.EventBufferSize
 }
 
 // BrowseChildrenOptions configures lazy Galaxy hierarchy walks performed by

clients/go/mxgateway/session.go

@@ -18,8 +18,9 @@ import (
 
 const maxBulkItems = 1000
 
-// eventBufferSize is the number of buffered event slots on the
-// Events/EventsAfter (cancel-when-full) results channel.
+// eventBufferSize is the data-slot count the unit tests use for the legacy
+// fixed-size assertions; the runtime value comes from Options.EventBufferSize
+// (default DefaultEventBufferSize).
 const eventBufferSize = 16
 
 // eventBufferReservedSlots is the extra capacity reserved beyond
@@ -548,8 +549,9 @@ func (s *Session) WriteSecured2Bulk(ctx context.Context, serverHandle int32, ent
 // MXAccess COM has no synchronous Read; the worker satisfies this by returning the
 // most recent cached OnDataChange value when the tag is already advised (WasCached=true),
 // or by taking a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
-// otherwise. timeout bounds the wait per tag in the snapshot case; pass zero to use the
-// worker default. Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
+// otherwise. timeout is one shared deadline for the whole command, not a per-tag budget:
+// the worker advises every uncached tag, waits once for all of them, then tears them down.
+// Pass zero to use the worker default (1000 ms). Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
 // with WasSuccessful=false; the call never returns an error for per-tag MXAccess failures.
 //
 // A non-nil but empty tagAddresses slice is treated as a no-op and returns an empty
@@ -666,6 +668,68 @@ func buildSparseArrayValue(elementDataType MxDataType, totalLength uint32, eleme
 	}
 }
 
+// Ping sends a diagnostic PING command and returns the message the gateway
+// echoed back in the reply's DiagnosticMessage field.
+func (s *Session) Ping(ctx context.Context, message string) (string, error) {
+	reply, err := s.PingRaw(ctx, message)
+	if err != nil {
+		return "", err
+	}
+	return reply.GetDiagnosticMessage(), nil
+}
+
+// GetSessionState returns the gateway's current lifecycle state for the session.
+func (s *Session) GetSessionState(ctx context.Context) (SessionState, error) {
+	reply, err := s.invokeCommand(ctx, &pb.MxCommand{
+		Kind:    pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
+		Payload: &pb.MxCommand_GetSessionState{GetSessionState: &pb.GetSessionStateCommand{}},
+	})
+	if err != nil {
+		return 0, err
+	}
+	payload := reply.GetSessionState()
+	if payload == nil {
+		return 0, errors.New("mxgateway: get session state reply carried no session_state payload")
+	}
+	return payload.GetState(), nil
+}
+
+// GetWorkerInfo returns diagnostic information about the session's worker
+// process (process id, version, MXAccess ProgID and CLSID).
+func (s *Session) GetWorkerInfo(ctx context.Context) (*WorkerInfoReply, error) {
+	reply, err := s.invokeCommand(ctx, &pb.MxCommand{
+		Kind:    pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
+		Payload: &pb.MxCommand_GetWorkerInfo{GetWorkerInfo: &pb.GetWorkerInfoCommand{}},
+	})
+	if err != nil {
+		return nil, err
+	}
+	payload := reply.GetWorkerInfo()
+	if payload == nil {
+		return nil, errors.New("mxgateway: get worker info reply carried no worker_info payload")
+	}
+	return payload, nil
+}
+
+// DrainEvents removes and returns up to maxEvents queued events through the
+// diagnostic DRAIN_EVENTS command. The reply is bounded server-side, so it can
+// carry fewer events than requested or queued; call repeatedly until it returns
+// an empty slice. It is a diagnostic tool, not a replacement for Events.
+func (s *Session) DrainEvents(ctx context.Context, maxEvents uint32) ([]*MxEvent, error) {
+	reply, err := s.invokeCommand(ctx, &pb.MxCommand{
+		Kind:    pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
+		Payload: &pb.MxCommand_DrainEvents{DrainEvents: &pb.DrainEventsCommand{MaxEvents: maxEvents}},
+	})
+	if err != nil {
+		return nil, err
+	}
+	payload := reply.GetDrainEvents()
+	if payload == nil {
+		return nil, errors.New("mxgateway: drain events reply carried no drain_events payload")
+	}
+	return payload.GetEvents(), nil
+}
+
 // PingRaw sends a diagnostic PING command and returns the raw reply.
 // The message is echoed back by the gateway in the reply's DiagnosticMessage field.
 func (s *Session) PingRaw(ctx context.Context, message string) (*MxCommandReply, error) {
@@ -1001,7 +1065,9 @@ func stringSecrets(values ...*MxValue) []string {
 // The returned channel is buffered. If the consumer falls behind and the buffer
 // overflows with data, the stream is terminated and a final EventResult carrying
 // a GatewayError that wraps ErrSlowConsumer is delivered before the channel
-// closes; match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
+// closes. Results already buffered are delivered first, in order, ahead of that
+// terminal result. The buffer holds Options.EventBufferSize data results
+// (default DefaultEventBufferSize); match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
 // slow-consumer drop from a graceful server end. A genuine stream error is
 // reported as itself even under overflow — it is never relabeled as
 // ErrSlowConsumer, so the underlying gRPC status stays inspectable. Use
@@ -1046,7 +1112,8 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
 		return nil, err
 	}
 
-	results := make(chan EventResult, eventBufferSize+eventBufferReservedSlots)
+	dataSlots := s.client.opts.eventBufferSize()
+	results := make(chan EventResult, dataSlots+eventBufferReservedSlots)
 	done := make(chan struct{})
 	go func() {
 		defer close(results)
@@ -1062,7 +1129,7 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
 				if gap := event.GetReplayGap(); gap != nil {
 					result = EventResult{ReplayGap: gap}
 				}
-				if !sendEventResult(streamCtx, results, result, cancelWhenResultBufferFull, cancel) {
+				if !sendEventResult(streamCtx, results, result, dataSlots, cancelWhenResultBufferFull, cancel) {
 					return
 				}
 				continue
@@ -1126,6 +1193,7 @@ func sendEventResult(
 	ctx context.Context,
 	results chan<- EventResult,
 	result EventResult,
+	dataSlots int,
 	cancelWhenBufferFull bool,
 	cancel context.CancelFunc,
 ) bool {
@@ -1136,7 +1204,7 @@ func sendEventResult(
 		// grows by our own sends and shrinks as the consumer reads; the reserve
 		// therefore always survives to carry ErrSlowConsumer. A plain buffered
 		// send would instead consume the reserved slot as ordinary data.
-		if len(results) >= eventBufferSize {
+		if len(results) >= dataSlots {
 			// The consumer fell behind and the data slots are full. Cancel the
 			// stream, then use the reserved terminal slot to enqueue a loud
 			// ErrSlowConsumer result so overflow is always observable rather

clients/go/mxgateway/types.go

@@ -28,6 +28,8 @@ type (
 	MxCommandReply = pb.MxCommandReply
 	// MxCommand is the discriminated union of MXAccess command payloads.
 	MxCommand = pb.MxCommand
+	// WorkerInfoReply is the GetWorkerInfo diagnostic reply payload.
+	WorkerInfoReply = pb.WorkerInfoReply
 	// MxEvent is one ordered event delivered on a session event stream.
 	MxEvent = pb.MxEvent
 	// ReplayGap is the gateway sentinel payload signalling that a resumed event

clients/go/mxgateway/types_alias_test.go

@@ -10,8 +10,9 @@ import (
 // the two status payloads carried by AlarmFeedMessage.
 func TestAlarmFeedStatusAliasesArePublic(t *testing.T) {
 	t.Helper()
-	var provider mxgateway.AlarmProviderStatus
-	var snapshot mxgateway.AlarmSnapshotStatus
+	// Pointers, not values: protobuf messages embed a mutex and must not be copied.
+	var provider *mxgateway.AlarmProviderStatus
+	var snapshot *mxgateway.AlarmSnapshotStatus
 	_ = provider
 	_ = snapshot
 }

clients/java/README.md

@@ -79,6 +79,30 @@ only stops the client from waiting; it does not abort an in-flight MXAccess COM
 call on the worker STA. It is a **single-consumer** surface: drive
 `hasNext()`/`next()` (or `nextItem()`) from one thread only.
 
+### Event buffer, overflow, and interrupts
+
+The iterator buffers events in a bounded queue whose size is
+`MxGatewayClientOptions.builder().eventQueueCapacity(n)` (default 1024; it was a
+hard-coded 16, which a short burst could overflow). If the consumer falls more
+than `n` events behind, the client cancels the call, keeps the events already
+buffered, and throws `MxGatewayException` only after you have drained them. Resume
+with `streamEventsAfter(lastSeenWorkerSequence)`. If the consuming thread is
+interrupted, the call is cancelled, the interrupt flag is restored, and the
+iterator throws a `CANCELLED` `StatusRuntimeException`.
+
+`streamEventsAsync(request, StreamObserver<MxEvent>)` is the non-dropping
+alternative: it delivers each event to your observer on the gRPC thread with no
+client-side queue, so nothing is discarded by the client (a slow observer
+instead applies normal gRPC flow control).
+
+### Session diagnostics
+
+`MxGatewaySession` exposes typed helpers for the worker diagnostic commands:
+`ping(message)` (returns the raw `MxCommandReply`), `getSessionState()`
+(`SessionState`), `getWorkerInfo()` (`WorkerInfoReply`: process id, version,
+ProgID, CLSID) and `drainEvents(maxEvents)` (`List<MxEvent>`; replies are
+bounded server-side, so loop until an empty list).
+
 ### Reconnect-replay gap signal
 
 When you resume a stream with `streamEventsAfter(afterWorkerSequence)` and the
@@ -114,8 +138,11 @@ try (MxEventStream events = session.streamEventsAfter(lastSeenSequence)) {
 For alarms, `MxGatewayClient` exposes `queryActiveAlarms` (one-shot snapshot),
 `streamAlarms` (returns an `MxGatewayAlarmFeedSubscription` whose iterator
 yields alarm-feed messages from the gateway's central monitor), and
-`acknowledgeAlarm` (ack by full alarm reference with an optional comment and
-ack target). Close the subscription to cancel the underlying gRPC stream.
+`acknowledgeAlarm` (ack by full alarm reference with an optional comment).
+The ack carries operator credentials: `operator_user` is required, and
+`operator_password` / `operator_domain` let the gateway authenticate as that
+operator so the Historian records the real operator. An empty password is
+allowed. The password is scrubbed from exceptions the client throws. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. Close the subscription to cancel the underlying gRPC stream.
 
 `ActiveAlarmSnapshot.getFromTruncatedSnapshot()` reports that the record came
 from a provider fetch which hit the per-fetch cap: the snapshot set may omit
@@ -417,7 +444,7 @@ gradle :zb-mom-ww-mxgateway-cli:run --args="authenticate-user --endpoint localho
 gradle :zb-mom-ww-mxgateway-cli:run --args="write-secured --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --current-user-id 100 --verifier-user-id 100 --json"
 gradle :zb-mom-ww-mxgateway-cli:run --args="stream-events --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --limit 1 --json"
 gradle :zb-mom-ww-mxgateway-cli:run --args="stream-alarms --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --filter-prefix Galaxy --limit 4 --json"
-gradle :zb-mom-ww-mxgateway-cli:run --args="acknowledge-alarm --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --reference \"\\Galaxy\Area001.Pump001.PumpFault\" --json"
+gradle :zb-mom-ww-mxgateway-cli:run --args="acknowledge-alarm --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --reference \"\\Galaxy\Area001.Pump001.PumpFault\" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json"
 gradle :zb-mom-ww-mxgateway-cli:run --args="smoke --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --item TestObject.TestInt --json"
 ```
 

clients/java/src/main/generated/main/java/mxaccess_worker/v1/MxaccessWorker.java

@@ -216,6 +216,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
      */
     WORKER_FAULT_CATEGORY_UNSPECIFIED(0),
     /**
+     * <pre>
+     * Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
+     * arguments, gateway authentication) exit via WorkerExitCode before the pipe
+     * exists, so there is no channel to carry a fault. Names kept so the numbers
+     * are not reused.
+     * </pre>
+     *
      * <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
      */
     WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS(1),
@@ -276,6 +283,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
      */
     public static final int WORKER_FAULT_CATEGORY_UNSPECIFIED_VALUE = 0;
     /**
+     * <pre>
+     * Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
+     * arguments, gateway authentication) exit via WorkerExitCode before the pipe
+     * exists, so there is no channel to carry a fault. Names kept so the numbers
+     * are not reused.
+     * </pre>
+     *
      * <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
      */
     public static final int WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS_VALUE = 1;

clients/java/zb-mom-ww-mxgateway-cli/src/main/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCli.java

@@ -1262,7 +1262,7 @@ public final class MxGatewayCli implements Callable<Integer> {
         @Option(
                 names = "--timeout-ms",
                 defaultValue = "0",
-                description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
+                description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
         int timeoutMs;
 
         ReadBulkCommand(MxGatewayCliClientFactory clientFactory) {
@@ -1510,7 +1510,7 @@ public final class MxGatewayCli implements Callable<Integer> {
         @Option(
                 names = "--timeout-ms",
                 defaultValue = "0",
-                description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
+                description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
         int timeoutMs;
 
         @Option(names = "--iterations", defaultValue = "10", description = "Number of ReadBulk calls to perform.")
@@ -1822,6 +1822,31 @@ public final class MxGatewayCli implements Callable<Integer> {
         @Option(names = "--operator", description = "Operator user performing the acknowledge.")
         String operator = "";
 
+        @Option(names = "--operator-domain", description = "Operator Galaxy domain (optional).")
+        String operatorDomain = "";
+
+        @Option(
+                names = "--operator-password",
+                description = "Operator Galaxy password (credential; prefer --operator-password-env). Never echoed.")
+        String operatorPassword;
+
+        @Option(
+                names = "--operator-password-env",
+                defaultValue = "MXGATEWAY_OPERATOR_PASSWORD",
+                description = "Environment variable holding the operator password when --operator-password is omitted.")
+        String operatorPasswordEnv;
+
+        private String resolveOperatorPassword() {
+            if (operatorPassword != null && !operatorPassword.isEmpty()) {
+                return operatorPassword;
+            }
+            String name = operatorPasswordEnv == null || operatorPasswordEnv.isBlank()
+                    ? "MXGATEWAY_OPERATOR_PASSWORD"
+                    : operatorPasswordEnv;
+            String value = System.getenv(name);
+            return value == null ? "" : value;
+        }
+
         AcknowledgeAlarmCommand(MxGatewayCliClientFactory clientFactory) {
             super(clientFactory);
         }
@@ -1833,6 +1858,8 @@ public final class MxGatewayCli implements Callable<Integer> {
                         .setAlarmFullReference(reference)
                         .setComment(comment)
                         .setOperatorUser(operator)
+                        .setOperatorDomain(operatorDomain == null ? "" : operatorDomain)
+                        .setOperatorPassword(resolveOperatorPassword())
                         .build());
                 writeOutput(
                         "acknowledge-alarm",

clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java

@@ -714,9 +714,16 @@ final class MxGatewayCliTests {
                 "checked",
                 "--operator",
                 "operator1",
+                "--operator-domain",
+                "CORP",
+                "--operator-password",
+                "pw-secret",
                 "--json");
 
         assertEquals(0, run.exitCode());
+        assertEquals("CORP", factory.client.lastAcknowledgeAlarmRequest.getOperatorDomain());
+        assertEquals("pw-secret", factory.client.lastAcknowledgeAlarmRequest.getOperatorPassword());
+        assertFalse(run.output().contains("pw-secret"), run.output());
         assertEquals("Tank01.Level.HiHi", factory.client.lastAcknowledgeAlarmRequest.getAlarmFullReference());
         assertEquals("checked", factory.client.lastAcknowledgeAlarmRequest.getComment());
         assertEquals("operator1", factory.client.lastAcknowledgeAlarmRequest.getOperatorUser());
@@ -770,6 +777,10 @@ final class MxGatewayCliTests {
                 "--plaintext",
                 "--reference",
                 "\\Galaxy\\Area001.Pump001.PumpFault",
+                "--operator",
+                "operator1",
+                "--operator-password-env",
+                "MXGATEWAY_OPERATOR_PASSWORD",
                 "--json"
         };
         assertReadmeExampleParses(args);

clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxEventStream.java

@@ -19,8 +19,12 @@ import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
  * <p>Events arrive on a background gRPC thread and are buffered in a bounded
  * blocking queue; the iterator drains them on the calling thread. Closing the
  * stream cancels the underlying gRPC call. If the queue overflows the call is
- * cancelled and a follow-up call to {@link #next()} throws
- * {@link MxGatewayException}.
+ * cancelled, the events already buffered remain readable, and the
+ * {@link MxGatewayException} is thrown only after they are drained (the consumer
+ * can then resume with {@code after_worker_sequence}). The capacity comes from
+ * {@link MxGatewayClientOptions#eventQueueCapacity()}. Callers that must never be
+ * dropped should use {@link MxGatewayClient#streamEventsAsync}, which delivers on
+ * the gRPC thread with the caller's own flow control.
  *
  * <p><strong>Single consumer.</strong> This stream is not safe to drain from
  * more than one thread. Interleave {@link #hasNext()}/{@link #next()} (or
@@ -46,10 +50,15 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
     private final BlockingQueue<Object> queue;
     private volatile ClientCallStreamObserver<StreamEventsRequest> requestStream;
     private volatile boolean closed;
+    private final int capacity;
+    private boolean overflowed;
     private Object next;
 
     MxEventStream(int capacity) {
-        queue = new ArrayBlockingQueue<>(capacity);
+        this.capacity = capacity;
+        // Two extra slots are reserved so the overflow error and END sentinel
+        // can always be enqueued behind a full buffer of received events.
+        queue = new ArrayBlockingQueue<>(capacity + 2);
     }
 
     ClientResponseObserver<StreamEventsRequest, MxEvent> observer() {
@@ -143,12 +152,16 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
                 return queue.take();
             } catch (InterruptedException error) {
                 Thread.currentThread().interrupt();
+                ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
+                if (stream != null) {
+                    stream.cancel("interrupted while reading events", error);
+                }
                 return new StatusRuntimeException(Status.CANCELLED.withDescription("interrupted while reading events"));
             }
         }
     }
 
-    private void offer(Object value) {
+    private synchronized void offer(Object value) {
         Objects.requireNonNull(value, "value");
         if (value == END) {
             if (!queue.offer(value)) {
@@ -157,14 +170,29 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
             }
             return;
         }
-        if (!queue.offer(value)) {
+        if (overflowed) {
+            return;
+        }
+        if (value instanceof MxEvent) {
+            if (queue.size() < capacity) {
+                queue.offer(value);
+                return;
+            }
+            // Keep the events already received so the consumer can drain them
+            // and resume from the last one via after_worker_sequence; the
+            // overflow error is delivered behind them.
+            overflowed = true;
             ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
             if (stream != null) {
                 stream.cancel("client event stream queue overflowed", null);
             }
-            queue.clear();
             queue.offer(new MxGatewayException("gateway stream events queue overflowed"));
             queue.offer(END);
+            return;
+        }
+        if (!queue.offer(value)) {
+            queue.clear();
+            queue.offer(value);
         }
     }
 }

clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxGatewayClient.java

@@ -245,7 +245,7 @@ public final class MxGatewayClient implements AutoCloseable {
      * @return an iterator-style stream of events
      */
     public MxEventStream streamEvents(StreamEventsRequest request) {
-        MxEventStream stream = new MxEventStream(16);
+        MxEventStream stream = new MxEventStream(options.eventQueueCapacity());
         withStreamDeadline(rawAsyncStub()).streamEvents(request, stream.observer());
         return stream;
     }
@@ -273,6 +273,11 @@ public final class MxGatewayClient implements AutoCloseable {
      * worker's MXAccess session; the resulting native MxStatus is returned
      * in the reply. Acks are idempotent at the MxAccess layer.
      *
+     * <p>{@code operator_user} is required; the gateway authenticates it with
+     * {@code operator_password} (and optional {@code operator_domain}) and the
+     * Historian records that operator. An empty password is allowed. The
+     * password is scrubbed from any exception this method throws.
+     *
      * @param request the {@code AcknowledgeAlarmRequest}
      * @return the raw acknowledge reply
      * @throws MxGatewayException on transport or protocol failure
@@ -283,10 +288,14 @@ public final class MxGatewayClient implements AutoCloseable {
             MxGatewayErrors.ensureProtocolSuccess("acknowledge alarm", reply.getProtocolStatus(), null);
             return reply;
         } catch (RuntimeException error) {
-            if (error instanceof MxGatewayException) {
-                throw error;
+            MxGatewayException mapped = error instanceof MxGatewayException gatewayError
+                    ? gatewayError
+                    : (MxGatewayException) MxGatewayErrors.fromGrpc("acknowledge alarm", error);
+            String secret = request.getOperatorPassword();
+            if (secret.isEmpty()) {
+                throw mapped;
             }
-            throw MxGatewayErrors.fromGrpc("acknowledge alarm", error);
+            throw MxGatewayErrors.redacted(mapped, secret);
         }
     }
 
@@ -299,10 +308,24 @@ public final class MxGatewayClient implements AutoCloseable {
      */
     public CompletableFuture<AcknowledgeAlarmReply> acknowledgeAlarmAsync(AcknowledgeAlarmRequest request) {
         CompletableFuture<AcknowledgeAlarmReply> future = toCompletable(rawFutureStub().acknowledgeAlarm(request));
-        return future.thenApply(reply -> {
+        String secret = request.getOperatorPassword();
+        CompletableFuture<AcknowledgeAlarmReply> checked = future.thenApply(reply -> {
             MxGatewayErrors.ensureProtocolSuccess("acknowledge alarm", reply.getProtocolStatus(), null);
             return reply;
         });
+        if (secret.isEmpty()) {
+            return checked;
+        }
+        // Scrub the operator password from any failure surfaced through the future.
+        return checked.exceptionally(error -> {
+            Throwable cause = error instanceof java.util.concurrent.CompletionException && error.getCause() != null
+                    ? error.getCause()
+                    : error;
+            if (cause instanceof MxGatewayException gatewayError) {
+                throw new java.util.concurrent.CompletionException(MxGatewayErrors.redacted(gatewayError, secret));
+            }
+            throw new java.util.concurrent.CompletionException(cause);
+        });
     }
 
     /**

clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxGatewayClientOptions.java

@@ -15,6 +15,8 @@ public final class MxGatewayClientOptions {
     private static final Duration DEFAULT_CONNECT_TIMEOUT = Duration.ofSeconds(10);
     private static final Duration DEFAULT_CALL_TIMEOUT = Duration.ofSeconds(30);
     private static final int DEFAULT_MAX_GRPC_MESSAGE_BYTES = 16 * 1024 * 1024;
+    /** Default {@code StreamEvents} iterator buffer; sized to absorb bursts without dropping a slow consumer. */
+    static final int DEFAULT_EVENT_QUEUE_CAPACITY = 1024;
 
     private final String endpoint;
     private final String apiKey;
@@ -26,6 +28,7 @@ public final class MxGatewayClientOptions {
     private final Duration callTimeout;
     private final Duration streamTimeout;
     private final int maxGrpcMessageBytes;
+    private final int eventQueueCapacity;
 
     private MxGatewayClientOptions(Builder builder) {
         endpoint = requireText(builder.endpoint, "endpoint");
@@ -40,6 +43,10 @@ public final class MxGatewayClientOptions {
         maxGrpcMessageBytes = builder.maxGrpcMessageBytes <= 0
                 ? DEFAULT_MAX_GRPC_MESSAGE_BYTES
                 : builder.maxGrpcMessageBytes;
+        if (builder.eventQueueCapacity <= 0) {
+            throw new IllegalArgumentException("eventQueueCapacity must be positive");
+        }
+        eventQueueCapacity = builder.eventQueueCapacity;
     }
 
     /**
@@ -149,6 +156,17 @@ public final class MxGatewayClientOptions {
         return maxGrpcMessageBytes;
     }
 
+    /**
+     * Returns the capacity of the buffer behind {@link MxGatewayClient#streamEvents}.
+     * A consumer that falls more than this many events behind has its stream
+     * cancelled with an overflow error after the buffered events are drained.
+     *
+     * @return the event buffer capacity (default 1024)
+     */
+    public int eventQueueCapacity() {
+        return eventQueueCapacity;
+    }
+
     @Override
     public String toString() {
         return "MxGatewayClientOptions{"
@@ -199,6 +217,7 @@ public final class MxGatewayClientOptions {
         private Duration callTimeout;
         private Duration streamTimeout;
         private int maxGrpcMessageBytes;
+        private int eventQueueCapacity = DEFAULT_EVENT_QUEUE_CAPACITY;
 
         private Builder() {
         }
@@ -314,6 +333,17 @@ public final class MxGatewayClientOptions {
             return this;
         }
 
+        /**
+         * Sets the capacity of the iterator-style event stream buffer. Defaults to 1024.
+         *
+         * @param value the buffer capacity, must be positive
+         * @return this builder
+         */
+        public Builder eventQueueCapacity(int value) {
+            eventQueueCapacity = value;
+            return this;
+        }
+
         /**
          * Builds an immutable {@link MxGatewayClientOptions} from the current state.
          *

clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxGatewayErrors.java

@@ -28,6 +28,43 @@ final class MxGatewayErrors {
         return new MxGatewayException("gateway " + operation + " failed: " + error.getMessage(), error);
     }
 
+    /**
+     * Rebuilds a gateway exception with an exact secret scrubbed from its
+     * message and protocol status. The original is not chained as a cause
+     * because its message and cause chain may carry the secret.
+     */
+    static MxGatewayException redacted(MxGatewayException error, String secret) {
+        String message = MxGatewaySecrets.redactExact(error.getMessage(), secret);
+        if (error instanceof MxAccessException ex) {
+            return new MxAccessException(message, scrub(ex.protocolStatus(), secret), ex.reply(), null);
+        }
+        if (error instanceof MxGatewayCommandException ex) {
+            return new MxGatewayCommandException(message, scrub(ex.protocolStatus(), secret), ex.reply(), null);
+        }
+        if (error instanceof MxGatewaySessionException ex) {
+            return new MxGatewaySessionException(message, scrub(ex.protocolStatus(), secret), null);
+        }
+        if (error instanceof MxGatewayWorkerException ex) {
+            return new MxGatewayWorkerException(message, scrub(ex.protocolStatus(), secret), null);
+        }
+        if (error instanceof MxGatewayAuthenticationException) {
+            return new MxGatewayAuthenticationException(message, null);
+        }
+        if (error instanceof MxGatewayAuthorizationException) {
+            return new MxGatewayAuthorizationException(message, null);
+        }
+        return new MxGatewayException(message);
+    }
+
+    private static ProtocolStatus scrub(ProtocolStatus status, String secret) {
+        if (status == null) {
+            return null;
+        }
+        return status.toBuilder()
+                .setMessage(MxGatewaySecrets.redactExact(status.getMessage(), secret))
+                .build();
+    }
+
     static void ensureProtocolSuccess(String operation, ProtocolStatus status, MxCommandReply reply) {
         if (status == null || status.getCode() == ProtocolStatusCode.PROTOCOL_STATUS_CODE_OK) {
             return;

clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxGatewaySession.java

@@ -37,6 +37,13 @@ import mxaccess_gateway.v1.MxaccessGateway.RegisterCommand;
 import mxaccess_gateway.v1.MxaccessGateway.RemoveItemBulkCommand;
 import mxaccess_gateway.v1.MxaccessGateway.RemoveItemCommand;
 import mxaccess_gateway.v1.MxaccessGateway.SetBufferedUpdateIntervalCommand;
+import mxaccess_gateway.v1.MxaccessGateway.DrainEventsCommand;
+import mxaccess_gateway.v1.MxaccessGateway.GetSessionStateCommand;
+import mxaccess_gateway.v1.MxaccessGateway.GetWorkerInfoCommand;
+import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
+import mxaccess_gateway.v1.MxaccessGateway.PingCommand;
+import mxaccess_gateway.v1.MxaccessGateway.SessionState;
+import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
 import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
 import mxaccess_gateway.v1.MxaccessGateway.SubscribeBulkCommand;
 import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -548,15 +555,17 @@ public final class MxGatewaySession implements AutoCloseable {
      * {@code OnDataChange} value for any tag that is already advised
      * ({@code wasCached == true}) without modifying the existing subscription,
      * and falls back to a full AddItem + Advise + wait + UnAdvise + RemoveItem
-     * snapshot lifecycle otherwise. The supplied {@code timeout} bounds the
-     * per-tag wait in the snapshot case; pass {@link Duration#ZERO} (or
-     * {@code null}) to use the worker default (1000 ms). Per-tag failures
+     * snapshot lifecycle otherwise. The supplied {@code timeout} is one shared
+     * deadline for the whole command, not a per-tag budget: the worker advises
+     * every uncached tag, waits once for all of them, then tears them down.
+     * Pass {@link Duration#ZERO} (or {@code null}) to use the worker default
+     * (1000 ms). Per-tag failures
      * appear as {@link BulkReadResult} entries with {@code wasSuccessful == false};
      * this method does not throw for per-tag MXAccess failures.
      *
      * @param serverHandle the {@code ServerHandle} owning the items
      * @param tagAddresses the tag addresses to read
-     * @param timeout per-tag snapshot timeout (zero or null = worker default)
+     * @param timeout snapshot deadline shared by all tags in the command (zero or null = worker default)
      * @return a per-tag {@link BulkReadResult} list
      * @throws MxGatewayException on transport or protocol failure
      * @throws NullPointerException if {@code tagAddresses} is {@code null}
@@ -1033,6 +1042,67 @@ public final class MxGatewaySession implements AutoCloseable {
                 .build());
     }
 
+    /**
+     * Sends a {@code Ping} diagnostic command to the session's worker.
+     *
+     * @param message free-form text carried in the command
+     * @return the raw command reply
+     * @throws MxGatewayException on transport or protocol failure
+     */
+    public MxCommandReply ping(String message) {
+        return invokeCommand(MxCommand.newBuilder()
+                .setKind(MxCommandKind.MX_COMMAND_KIND_PING)
+                .setPing(PingCommand.newBuilder().setMessage(message == null ? "" : message))
+                .build());
+    }
+
+    /**
+     * Queries the worker-side state of this session.
+     *
+     * @return the reported {@link SessionState}
+     * @throws MxGatewayException on transport or protocol failure
+     */
+    public SessionState getSessionState() {
+        return invokeCommand(MxCommand.newBuilder()
+                        .setKind(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE)
+                        .setGetSessionState(GetSessionStateCommand.getDefaultInstance())
+                        .build())
+                .getSessionState()
+                .getState();
+    }
+
+    /**
+     * Queries the worker process and MXAccess COM identity.
+     *
+     * @return the {@link WorkerInfoReply} (process id, version, ProgID, CLSID)
+     * @throws MxGatewayException on transport or protocol failure
+     */
+    public WorkerInfoReply getWorkerInfo() {
+        return invokeCommand(MxCommand.newBuilder()
+                        .setKind(MxCommandKind.MX_COMMAND_KIND_GET_WORKER_INFO)
+                        .setGetWorkerInfo(GetWorkerInfoCommand.getDefaultInstance())
+                        .build())
+                .getWorkerInfo();
+    }
+
+    /**
+     * Pulls queued events from the worker without opening a stream. The reply
+     * is bounded server-side and may carry fewer than {@code maxEvents}; drain
+     * iteratively until an empty list is returned.
+     *
+     * @param maxEvents the maximum number of events to return
+     * @return the drained events, possibly empty
+     * @throws MxGatewayException on transport or protocol failure
+     */
+    public List<MxEvent> drainEvents(int maxEvents) {
+        return invokeCommand(MxCommand.newBuilder()
+                        .setKind(MxCommandKind.MX_COMMAND_KIND_DRAIN_EVENTS)
+                        .setDrainEvents(DrainEventsCommand.newBuilder().setMaxEvents(maxEvents))
+                        .build())
+                .getDrainEvents()
+                .getEventsList();
+    }
+
     /**
      * Sends a pre-built {@link MxCommand} for this session and returns the raw
      * reply, attaching a freshly generated client correlation id.

clients/java/zb-mom-ww-mxgateway-client/src/test/java/com/zb/mom/ww/mxgateway/client/MxEventStreamTests.java

@@ -0,0 +1,93 @@
+package com.zb.mom.ww.mxgateway.client;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import io.grpc.stub.ClientCallStreamObserver;
+import io.grpc.stub.ClientResponseObserver;
+import java.util.concurrent.atomic.AtomicInteger;
+import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
+import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
+import org.junit.jupiter.api.Test;
+
+final class MxEventStreamTests {
+    @Test
+    void overflowKeepsRetainedEventsThenSurfacesError() {
+        MxEventStream stream = new MxEventStream(3);
+        RecordingCall call = new RecordingCall();
+        ClientResponseObserver<StreamEventsRequest, MxEvent> observer = stream.observer();
+        observer.beforeStart(call);
+
+        for (long i = 1; i <= 5; i++) {
+            observer.onNext(MxEvent.newBuilder().setWorkerSequence(i).build());
+        }
+
+        assertTrue(call.cancelCount.get() >= 1, "overflow cancels the call");
+        assertEquals(1L, stream.next().getWorkerSequence());
+        assertEquals(2L, stream.next().getWorkerSequence());
+        assertEquals(3L, stream.next().getWorkerSequence());
+        MxGatewayException error = assertThrows(MxGatewayException.class, stream::hasNext);
+        assertTrue(error.getMessage().contains("overflowed"));
+        assertFalse(stream.hasNext());
+    }
+
+    @Test
+    void interruptCancelsCallAndRestoresInterruptFlag() {
+        MxEventStream stream = new MxEventStream(4);
+        RecordingCall call = new RecordingCall();
+        stream.observer().beforeStart(call);
+
+        Thread.currentThread().interrupt();
+        try {
+            assertThrows(io.grpc.StatusRuntimeException.class, stream::hasNext);
+            assertTrue(Thread.currentThread().isInterrupted(), "interrupt flag restored");
+            assertEquals(1, call.cancelCount.get(), "call cancelled on interrupt");
+        } finally {
+            Thread.interrupted();
+        }
+    }
+
+    private static final class RecordingCall extends ClientCallStreamObserver<StreamEventsRequest> {
+        final AtomicInteger cancelCount = new AtomicInteger();
+
+        @Override
+        public void cancel(String message, Throwable cause) {
+            cancelCount.incrementAndGet();
+        }
+
+        @Override
+        public boolean isReady() {
+            return true;
+        }
+
+        @Override
+        public void setOnReadyHandler(Runnable onReadyHandler) {
+        }
+
+        @Override
+        public void request(int count) {
+        }
+
+        @Override
+        public void setMessageCompression(boolean enable) {
+        }
+
+        @Override
+        public void disableAutoInboundFlowControl() {
+        }
+
+        @Override
+        public void onNext(StreamEventsRequest value) {
+        }
+
+        @Override
+        public void onError(Throwable t) {
+        }
+
+        @Override
+        public void onCompleted() {
+        }
+    }
+}

clients/java/zb-mom-ww-mxgateway-client/src/test/java/com/zb/mom/ww/mxgateway/client/MxGatewayClientSessionTests.java

@@ -51,7 +51,10 @@ import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
 import mxaccess_gateway.v1.MxaccessGateway.QueryActiveAlarmsRequest;
 import mxaccess_gateway.v1.MxaccessGateway.RegisterReply;
 import mxaccess_gateway.v1.MxaccessGateway.ReplayGap;
+import mxaccess_gateway.v1.MxaccessGateway.DrainEventsReply;
 import mxaccess_gateway.v1.MxaccessGateway.SessionState;
+import mxaccess_gateway.v1.MxaccessGateway.SessionStateReply;
+import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
 import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest;
 import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
 import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -727,6 +730,63 @@ final class MxGatewayClientSessionTests {
         }
     }
 
+    @Test
+    void diagnosticsHelpersBuildTypedCommandsAndDecodeReplies() throws Exception {
+        AtomicReference<MxCommandRequest> last = new AtomicReference<>();
+        TestGatewayService service = new TestGatewayService() {
+            @Override
+            public void invoke(MxCommandRequest request, StreamObserver<MxCommandReply> responseObserver) {
+                last.set(request);
+                MxCommandReply.Builder reply = MxCommandReply.newBuilder()
+                        .setSessionId(request.getSessionId())
+                        .setKind(request.getCommand().getKind())
+                        .setProtocolStatus(ok());
+                switch (request.getCommand().getKind()) {
+                    case MX_COMMAND_KIND_GET_SESSION_STATE -> reply.setSessionState(
+                            SessionStateReply.newBuilder().setState(SessionState.SESSION_STATE_READY));
+                    case MX_COMMAND_KIND_GET_WORKER_INFO -> reply.setWorkerInfo(
+                            WorkerInfoReply.newBuilder().setWorkerProcessId(99).setWorkerVersion("1.2.3"));
+                    case MX_COMMAND_KIND_DRAIN_EVENTS -> reply.setDrainEvents(DrainEventsReply.newBuilder()
+                            .addEvents(MxEvent.newBuilder().setWorkerSequence(5)));
+                    default -> { }
+                }
+                responseObserver.onNext(reply.build());
+                responseObserver.onCompleted();
+            }
+        };
+
+        try (InProcessGateway gateway = InProcessGateway.start(service, new AtomicReference<>());
+                MxGatewayClient client = gateway.client("", Duration.ofSeconds(5))) {
+            MxGatewaySession session = MxGatewaySession.forSessionId(client, "existing-session");
+
+            MxCommandReply ping = session.ping("hello");
+            assertEquals(MxCommandKind.MX_COMMAND_KIND_PING, ping.getKind());
+            assertEquals("hello", last.get().getCommand().getPing().getMessage());
+
+            assertEquals(SessionState.SESSION_STATE_READY, session.getSessionState());
+            assertEquals(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE, last.get().getCommand().getKind());
+
+            WorkerInfoReply info = session.getWorkerInfo();
+            assertEquals(99, info.getWorkerProcessId());
+            assertEquals("1.2.3", info.getWorkerVersion());
+
+            List<MxEvent> events = session.drainEvents(10);
+            assertEquals(1, events.size());
+            assertEquals(5L, events.get(0).getWorkerSequence());
+            assertEquals(10, last.get().getCommand().getDrainEvents().getMaxEvents());
+        }
+    }
+
+    @Test
+    void eventQueueCapacityOptionDefaultsAndValidates() {
+        assertEquals(1024, MxGatewayClientOptions.builder().endpoint("x").build().eventQueueCapacity());
+        assertEquals(
+                7, MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(7).build().eventQueueCapacity());
+        assertThrows(
+                IllegalArgumentException.class,
+                () -> MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(0).build());
+    }
+
     private static TestGatewayService okInvokeService(AtomicReference<MxCommandRequest> commandRequest) {
         return new TestGatewayService() {
             @Override

clients/java/zb-mom-ww-mxgateway-client/src/test/java/com/zb/mom/ww/mxgateway/client/MxGatewayCredentialReplyTests.java

@@ -74,6 +74,52 @@ final class MxGatewayCredentialReplyTests {
         }
     }
 
+    @Test
+    void acknowledgeAlarmAsyncRedactsOperatorPasswordFromFailure() throws Exception {
+        String serverName = "mxgw-java-ack-" + UUID.randomUUID();
+        MxAccessGatewayGrpc.MxAccessGatewayImplBase service = new MxAccessGatewayGrpc.MxAccessGatewayImplBase() {
+            @Override
+            public void acknowledgeAlarm(
+                    mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmRequest request,
+                    StreamObserver<mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmReply> observer) {
+                observer.onNext(mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmReply.newBuilder()
+                        .setProtocolStatus(ProtocolStatus.newBuilder()
+                                .setCode(ProtocolStatusCode.PROTOCOL_STATUS_CODE_MXACCESS_FAILURE)
+                                .setMessage("auth failed for operator1 password " + CREDENTIAL))
+                        .build());
+                observer.onCompleted();
+            }
+        };
+        Server server = InProcessServerBuilder.forName(serverName)
+                .directExecutor().addService(service).build().start();
+        ManagedChannel channel = InProcessChannelBuilder.forName(serverName).directExecutor().build();
+        try (MxGatewayClient client = new MxGatewayClient(
+                channel,
+                MxGatewayClientOptions.builder()
+                        .endpoint("in-process").apiKey("").plaintext(true)
+                        .callTimeout(Duration.ofSeconds(5)).build())) {
+            var request = mxaccess_gateway.v1.MxaccessGateway.AcknowledgeAlarmRequest.newBuilder()
+                    .setAlarmFullReference("Tank01.Level.HiHi")
+                    .setOperatorUser("operator1")
+                    .setOperatorPassword(CREDENTIAL)
+                    .build();
+
+            java.util.concurrent.ExecutionException async = assertThrows(
+                    java.util.concurrent.ExecutionException.class,
+                    () -> client.acknowledgeAlarmAsync(request).get());
+            assertTrue(async.getCause() instanceof MxGatewayException, String.valueOf(async.getCause()));
+            assertFalse(async.getCause().getMessage().contains(CREDENTIAL));
+            assertFalse(((MxGatewayCommandException) async.getCause()).protocolStatus().getMessage()
+                    .contains(CREDENTIAL));
+
+            MxGatewayException sync = assertThrows(MxGatewayException.class, () -> client.acknowledgeAlarm(request));
+            assertFalse(sync.getMessage().contains(CREDENTIAL));
+        } finally {
+            channel.shutdownNow();
+            server.shutdownNow();
+        }
+    }
+
     @Test
     void authenticateUserMissingPayloadThrowsMalformedReply() throws Exception {
         MxCommandReply reply = loadReply("authenticate-user.missing-payload.reply.json");

clients/python/README.md

@@ -111,9 +111,15 @@ MXAccess commands and preserve raw replies on typed command exceptions.
 For alarms, the client exposes `GatewayClient.query_active_alarms` (one-shot
 snapshot), `GatewayClient.stream_alarms` (async generator yielding alarm-feed
 messages from the gateway's central monitor), and
-`GatewayClient.acknowledge_alarm` (ack by alarm reference, optional comment
-and ack target). Cancel the surrounding task or `aclose()` the iterator to
-terminate the stream.
+`GatewayClient.acknowledge_alarm` (ack by alarm reference with an optional
+comment). The ack is made as the operator: set `operator_user` (required) and
+`operator_password` (may be empty) and optionally `operator_domain` on
+`AcknowledgeAlarmRequest`. The gateway authenticates the operator and writes the
+comment to the alarm's `AckMsg`, so the alarm history records the operator. The
+password is redacted from any raised error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account.
+
+Cancel the surrounding task or `aclose()` an alarm iterator to terminate the
+stream.
 
 `ActiveAlarmSnapshot.from_truncated_snapshot` reports that the record came from
 a provider fetch which hit the per-fetch cap: the snapshot set may omit active
@@ -457,7 +463,7 @@ mxgw-py add-item --session-id <id> --server-handle 1 --item Object.Attribute --j
 mxgw-py advise --session-id <id> --server-handle 1 --item-handle 2 --json
 mxgw-py stream-events --session-id <id> --max-events 1 --json
 mxgw-py stream-alarms --max-messages 4 --json
-mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --json
+mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --operator operator1 --operator-password-env MXGATEWAY_OPERATOR_PASSWORD --json
 mxgw-py write --session-id <id> --server-handle 1 --item-handle 2 --type int32 --value 123 --json
 ```
 
@@ -517,3 +523,10 @@ a `~/.netrc` entry or `PIP_INDEX_URL=https://<user>:<token>@gitea.dohertylan.com
 - [Client Packaging](../../docs/ClientPackaging.md)
 - [Client Proto Generation](../../docs/ClientProtoGeneration.md)
 - [Python Client Detailed Design](./PythonClientDesign.md)
+
+## Typed diagnostics
+
+`Session.ping(message)`, `get_session_state()`, `get_worker_info()`, and
+`drain_events(max_events)` wrap the worker diagnostic commands. `drain_events`
+returns a list that may be shorter than `max_events`; call it until it returns
+an empty list.

clients/python/src/zb_mom_ww_mxgateway/client.py

@@ -9,7 +9,7 @@ from typing import Any
 import grpc
 
 from .auth import merge_metadata
-from .errors import ensure_protocol_success, map_rpc_error
+from .errors import MxGatewayError, ensure_protocol_success, map_rpc_error
 from .generated import mxaccess_gateway_pb2 as pb
 from .generated import mxaccess_gateway_pb2_grpc as pb_grpc
 from .options import ClientOptions, create_channel, resolve_channel_security
@@ -148,17 +148,23 @@ class GatewayClient:
         self,
         request: pb.AcknowledgeAlarmRequest,
     ) -> pb.AcknowledgeAlarmReply:
-        """Acknowledge an active MXAccess alarm condition through the gateway.
+        """Acknowledge an alarm as the operator.
 
-        The gateway authenticates the request against the API key's
-        ``invoke:alarm-ack`` scope and forwards the acknowledge to the worker's
-        MXAccess session; the resulting native ``MxStatus`` is returned in the
-        reply. Acks are idempotent — re-acking an already-acked condition is a
-        no-op at the MxAccess layer.
+        The gateway authenticates ``operator_user`` with ``operator_password`` and
+        writes the comment to the alarm's ``AckMsg``, so the alarm history records
+        the operator. The password is redacted from any raised error.
         """
-        reply = await self._unary("acknowledge alarm", self.raw_stub.AcknowledgeAlarm, request)
-        ensure_protocol_success("acknowledge alarm", reply.protocol_status, reply)
-        return reply
+        try:
+            reply = await self._unary("acknowledge alarm", self.raw_stub.AcknowledgeAlarm, request)
+            ensure_protocol_success("acknowledge alarm", reply.protocol_status, reply)
+            return reply
+        except MxGatewayError as error:
+            if request.operator_password:
+                # Imported lazily: session.py imports this module at its tail.
+                from .session import _redact_error
+
+                _redact_error(error, [request.operator_password])
+            raise
 
     def query_active_alarms(
         self,

clients/python/src/zb_mom_ww_mxgateway/session.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 from collections.abc import AsyncIterator, Sequence
+from typing import Any
 
 from .auth import redact_secret
 from .errors import MalformedReplyError, MxGatewayError, ensure_mxaccess_success
@@ -447,8 +448,10 @@ class Session:
         ``OnDataChange`` value for any tag that is already advised (``was_cached =
         True``) without modifying the existing subscription, and falls back to
         a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
-        otherwise. ``timeout_ms`` bounds the per-tag wait in the snapshot case;
-        pass ``0`` to use the worker default (1000 ms).
+        otherwise. ``timeout_ms`` is one shared deadline for the whole command,
+        not a per-tag budget: the worker advises every uncached tag, waits once
+        for all of them, then tears them down. Pass ``0`` to use the worker
+        default (1000 ms).
         """
         if tag_addresses is None:
             raise TypeError("tag_addresses is required")
@@ -837,6 +840,55 @@ class Session:
         )
         return reply.activate.status
 
+    async def ping(self, message: str = "", *, correlation_id: str = "") -> pb.MxCommandReply:
+        """Invoke the worker `Ping` diagnostic and return its reply."""
+        return await self.invoke(
+            pb.MxCommand(kind=pb.MX_COMMAND_KIND_PING, ping=pb.PingCommand(message=message)),
+            correlation_id=correlation_id,
+        )
+
+    async def get_session_state(self, *, correlation_id: str = "") -> int:
+        """Return the worker-reported `SessionState` enum value."""
+        reply = await self.invoke(
+            pb.MxCommand(
+                kind=pb.MX_COMMAND_KIND_GET_SESSION_STATE,
+                get_session_state=pb.GetSessionStateCommand(),
+            ),
+            correlation_id=correlation_id,
+        )
+        return reply.session_state.state
+
+    async def get_worker_info(self, *, correlation_id: str = "") -> pb.WorkerInfoReply:
+        """Return worker process id, version, and MXAccess ProgID/CLSID."""
+        reply = await self.invoke(
+            pb.MxCommand(
+                kind=pb.MX_COMMAND_KIND_GET_WORKER_INFO,
+                get_worker_info=pb.GetWorkerInfoCommand(),
+            ),
+            correlation_id=correlation_id,
+        )
+        return reply.worker_info
+
+    async def drain_events(
+        self,
+        max_events: int = 0,
+        *,
+        correlation_id: str = "",
+    ) -> list[pb.MxEvent]:
+        """Drain queued worker events (diagnostic).
+
+        A reply may carry fewer than *max_events* (server count and frame-size
+        caps); call repeatedly until an empty list is returned.
+        """
+        reply = await self.invoke(
+            pb.MxCommand(
+                kind=pb.MX_COMMAND_KIND_DRAIN_EVENTS,
+                drain_events=pb.DrainEventsCommand(max_events=max_events),
+            ),
+            correlation_id=correlation_id,
+        )
+        return list(reply.drain_events.events)
+
     def stream_events(
         self,
         *,
@@ -943,22 +995,26 @@ def _redact_error(error: MxGatewayError, secrets: Sequence[str | None]) -> None:
         error.raw_reply = _redact_reply(error.raw_reply, scrubbed)
 
 
-def _redact_reply(reply: pb.MxCommandReply, secrets: Sequence[str]) -> pb.MxCommandReply:
+def _redact_reply(reply: Any, secrets: Sequence[str]) -> Any:
     """Return a deep copy of *reply* with credential text scrubbed from diagnostics.
 
     Operates on a clone so the caller's original reply object is never mutated.
     Only the free-text diagnostic fields that can echo a client-supplied secret
     are scrubbed; the structured/enum fields the gateway itself sets are left as-is.
+    Works for any reply message (``MxCommandReply``, ``AcknowledgeAlarmReply``, ...):
+    each field is scrubbed only when the message type declares it.
     """
     clone = type(reply)()
     clone.CopyFrom(reply)
-    if clone.protocol_status.message:
+    fields = clone.DESCRIPTOR.fields_by_name
+    if "protocol_status" in fields and clone.protocol_status.message:
         clone.protocol_status.message = redact_secret(clone.protocol_status.message, secrets)
-    if clone.diagnostic_message:
+    if "diagnostic_message" in fields and clone.diagnostic_message:
         clone.diagnostic_message = redact_secret(clone.diagnostic_message, secrets)
-    for status in clone.statuses:
-        if status.diagnostic_text:
-            status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
+    if "statuses" in fields:
+        for status in clone.statuses:
+            if status.diagnostic_text:
+                status.diagnostic_text = redact_secret(status.diagnostic_text, secrets)
     return clone
 
 

clients/python/src/zb_mom_ww_mxgateway_cli/commands.py

@@ -36,6 +36,7 @@ MAX_AGGREGATE_EVENTS = 10_000
 #: CLI (CLI-45) so one exported variable drives the same operator workflow in all
 #: five languages.
 DEFAULT_VERIFY_PASSWORD_ENV = "MXGATEWAY_VERIFY_PASSWORD"
+DEFAULT_OPERATOR_PASSWORD_ENV = "MXGATEWAY_OPERATOR_PASSWORD"
 
 _BATCH_EOR = "__MXGW_BATCH_EOR__"
 
@@ -391,7 +392,7 @@ def unsubscribe_bulk(**kwargs: Any) -> None:
 @click.option("--server-handle", required=True, type=int, help="MXAccess server handle.")
 @click.option("--items", required=True, help="Comma-separated MXAccess tag addresses.")
 @click.option("--timeout-ms", default=0, type=int, show_default=True,
-              help="Per-tag snapshot timeout in milliseconds. 0 = worker default.")
+              help="Snapshot deadline in milliseconds, shared by all tags in the command. 0 = worker default (1000 ms).")
 @click.option("--correlation-id", default="", help="Client correlation id.")
 @click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
 def read_bulk(**kwargs: Any) -> None:
@@ -528,16 +529,30 @@ def stream_alarms(**kwargs: Any) -> None:
 @gateway_options
 @click.option("--reference", required=True, help="Alarm full reference to acknowledge.")
 @click.option("--comment", default="", help="Acknowledgement comment.")
-@click.option("--operator", default="", help="Operator user name.")
+@click.option("--operator", required=True, help="Operator user name (recorded in the alarm history).")
+@click.option("--operator-domain", default="", help="Operator Windows/Galaxy domain (optional).")
+@click.option(
+    "--operator-password",
+    default=None,
+    help="Operator password. Prefer --operator-password-env so the secret is not visible on the command line.",
+)
+@click.option(
+    "--operator-password-env",
+    default=DEFAULT_OPERATOR_PASSWORD_ENV,
+    show_default=True,
+    help="Environment variable holding the operator password (an unset variable means an empty password).",
+)
 @click.option("--correlation-id", default="", help="Client correlation id.")
 @click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
 def acknowledge_alarm(**kwargs: Any) -> None:
-    """Acknowledge an active MXAccess alarm condition (session-less)."""
+    """Acknowledge an active MXAccess alarm condition as the operator (session-less)."""
 
+    password = _resolve_operator_password(kwargs)
+    kwargs["operator_password"] = password
     _run(
         _acknowledge_alarm(**kwargs),
         output_json=kwargs["output_json"],
-        secrets=_secrets(kwargs),
+        secrets=_secrets(kwargs) + [password],
     )
 
 
@@ -836,6 +851,21 @@ async def _authenticate_user(**kwargs: Any) -> dict[str, Any]:
         return {"userId": user_id}
 
 
+def _resolve_operator_password(kwargs: dict[str, Any]) -> str:
+    """Resolve the alarm-ack operator password from --operator-password or its env var.
+
+    Unlike :func:`_resolve_password`, an unset or empty value is allowed: Galaxy
+    accounts may have empty passwords. The value is never echoed; callers add it
+    to the ``secrets`` redaction list.
+    """
+
+    env_name = kwargs.get("operator_password_env") or DEFAULT_OPERATOR_PASSWORD_ENV
+    password = kwargs.get("operator_password")
+    if password is None:
+        password = os.environ.get(env_name, "")
+    return password
+
+
 def _resolve_password(kwargs: dict[str, Any]) -> str:
     """Resolve the authenticate-user password from --password or --password-env.
 
@@ -1161,6 +1191,8 @@ async def _acknowledge_alarm(**kwargs: Any) -> dict[str, Any]:
                 alarm_full_reference=kwargs["reference"],
                 comment=kwargs["comment"],
                 operator_user=kwargs["operator"],
+                operator_domain=kwargs["operator_domain"],
+                operator_password=kwargs["operator_password"],
             ),
         )
         return _message_dict(reply)

clients/python/tests/test_alarms.py

@@ -34,6 +34,8 @@ async def test_acknowledge_alarm_sends_request_and_returns_reply() -> None:
             alarm_full_reference="Tank01.Level.HiHi",
             comment="investigating",
             operator_user="alice",
+            operator_password="pw-secret",
+            operator_domain="CORP",
         ),
     )
 
@@ -44,9 +46,68 @@ async def test_acknowledge_alarm_sends_request_and_returns_reply() -> None:
     assert captured_request.alarm_full_reference == "Tank01.Level.HiHi"
     assert captured_request.comment == "investigating"
     assert captured_request.operator_user == "alice"
+    assert captured_request.operator_password == "pw-secret"
+    assert captured_request.operator_domain == "CORP"
     assert stub.acknowledge_alarm.metadata == (("authorization", "Bearer mxgw_test_secret"),)
 
 
+@pytest.mark.asyncio
+async def test_acknowledge_alarm_error_redacts_operator_password() -> None:
+    stub = FakeGatewayStub()
+    stub.acknowledge_alarm.exception = FakeRpcError(
+        grpc.StatusCode.UNAUTHENTICATED, "bad credentials pw-secret",
+    )
+    client = await GatewayClient.connect(
+        ClientOptions(endpoint="fake", api_key="mxgw_test_secret", plaintext=True),
+        stub=stub,
+    )
+
+    with pytest.raises(MxGatewayAuthenticationError) as excinfo:
+        await client.acknowledge_alarm(
+            pb.AcknowledgeAlarmRequest(
+                alarm_full_reference="Tank01.Level.HiHi",
+                operator_user="alice",
+                operator_password="pw-secret",
+            ),
+        )
+
+    assert "pw-secret" not in str(excinfo.value)
+
+
+@pytest.mark.asyncio
+async def test_acknowledge_alarm_protocol_failure_redacts_operator_password() -> None:
+    from zb_mom_ww_mxgateway.errors import MxGatewayError
+
+    stub = FakeGatewayStub()
+    stub.acknowledge_alarm.replies = [
+        pb.AcknowledgeAlarmReply(
+            protocol_status=pb.ProtocolStatus(
+                code=pb.PROTOCOL_STATUS_CODE_INVALID_REQUEST,
+                message="rejected credential pw-secret",
+            ),
+        ),
+    ]
+    client = await GatewayClient.connect(
+        ClientOptions(endpoint="fake", api_key="mxgw_test_secret", plaintext=True),
+        stub=stub,
+    )
+
+    with pytest.raises(MxGatewayError) as excinfo:
+        await client.acknowledge_alarm(
+            pb.AcknowledgeAlarmRequest(
+                alarm_full_reference="Tank01.Level.HiHi",
+                operator_user="alice",
+                operator_password="pw-secret",
+            ),
+        )
+
+    assert "pw-secret" not in str(excinfo.value)
+    raw = excinfo.value.raw_reply
+    assert isinstance(raw, pb.AcknowledgeAlarmReply)
+    assert "pw-secret" not in raw.protocol_status.message
+    assert "pw-secret" not in str(raw)
+
+
 @pytest.mark.asyncio
 async def test_acknowledge_alarm_unauthenticated_raises_typed_error() -> None:
     stub = FakeGatewayStub()

clients/python/tests/test_review_findings_022_to_026.py

@@ -771,6 +771,7 @@ def test_cli_stream_alarms_positive_max_messages_remains_finite(monkeypatch) ->
 
 
 def test_cli_acknowledge_alarm_happy_path(monkeypatch) -> None:
+    monkeypatch.setenv("MXGW_TEST_OPPW", "pw-secret")
     stub = _BulkFakeStub()
     stub.acknowledge_alarm = _BulkFakeUnary(
         [
@@ -798,12 +799,19 @@ def test_cli_acknowledge_alarm_happy_path(monkeypatch) -> None:
             "investigating",
             "--operator",
             "alice",
+            "--operator-domain",
+            "CORP",
+            "--operator-password-env",
+            "MXGW_TEST_OPPW",
             "--json",
         ],
     )
 
     assert result.exit_code == 0, result.output
+    assert "pw-secret" not in result.output
     captured_request = stub.acknowledge_alarm.requests[0]
+    assert captured_request.operator_domain == "CORP"
+    assert captured_request.operator_password == "pw-secret"
     assert captured_request.alarm_full_reference == "Tank01.Level.HiHi"
     assert captured_request.comment == "investigating"
     assert captured_request.operator_user == "alice"

clients/python/tests/test_typed_command_helpers.py

@@ -234,3 +234,61 @@ async def test_suspend_and_activate_return_status() -> None:
     )
     status = await session.activate(12, 34)
     assert status.category == pb.MX_STATUS_CATEGORY_OK
+
+
+@pytest.mark.asyncio
+async def test_ping_sends_message_and_returns_reply() -> None:
+    session, stub = await _session_with([_ok(pb.MX_COMMAND_KIND_PING)])
+    reply = await session.ping("hello")
+    command = stub.invoke.requests[0].command
+    assert command.kind == pb.MX_COMMAND_KIND_PING
+    assert command.ping.message == "hello"
+    assert reply.kind == pb.MX_COMMAND_KIND_PING
+
+
+@pytest.mark.asyncio
+async def test_get_session_state_returns_state() -> None:
+    session, stub = await _session_with(
+        [
+            _ok(
+                pb.MX_COMMAND_KIND_GET_SESSION_STATE,
+                session_state=pb.SessionStateReply(state=pb.SESSION_STATE_CLOSED),
+            ),
+        ],
+    )
+    state = await session.get_session_state()
+    assert state == pb.SESSION_STATE_CLOSED
+    assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_SESSION_STATE
+
+
+@pytest.mark.asyncio
+async def test_get_worker_info_returns_info() -> None:
+    session, stub = await _session_with(
+        [
+            _ok(
+                pb.MX_COMMAND_KIND_GET_WORKER_INFO,
+                worker_info=pb.WorkerInfoReply(worker_process_id=42, worker_version="1.2"),
+            ),
+        ],
+    )
+    info = await session.get_worker_info()
+    assert info.worker_process_id == 42
+    assert info.worker_version == "1.2"
+    assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_WORKER_INFO
+
+
+@pytest.mark.asyncio
+async def test_drain_events_sends_max_and_returns_events() -> None:
+    session, stub = await _session_with(
+        [
+            _ok(
+                pb.MX_COMMAND_KIND_DRAIN_EVENTS,
+                drain_events=pb.DrainEventsReply(events=[pb.MxEvent(), pb.MxEvent()]),
+            ),
+        ],
+    )
+    events = await session.drain_events(max_events=5)
+    assert len(events) == 2
+    command = stub.invoke.requests[0].command
+    assert command.kind == pb.MX_COMMAND_KIND_DRAIN_EVENTS
+    assert command.drain_events.max_events == 5

clients/rust/Cargo.toml

@@ -40,7 +40,7 @@ prost-types = "0.13.5"
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = "1.0.145"
 thiserror = "2.0.17"
-tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time"] }
+tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time", "fs"] }
 tokio-stream = { version = "0.1.17", features = ["net"] }
 tonic = { version = "0.13.1", features = ["transport", "tls-ring", "tls-native-roots"] }
 tonic-build = "0.13.1"

clients/rust/README.md

@@ -78,7 +78,8 @@ cargo run -p mxgw-cli -- add-item --session-id <session-id> --server-handle 1 --
 cargo run -p mxgw-cli -- advise --session-id <session-id> --server-handle 1 --item-handle 1 --json
 cargo run -p mxgw-cli -- stream-events --session-id <session-id> --max-events 1 --json
 cargo run -p mxgw-cli -- stream-alarms --session-id <session-id> --max-messages 4 --json
-cargo run -p mxgw-cli -- acknowledge-alarm --session-id <session-id> --alarm-reference "\\Galaxy\Area001.Pump001.PumpFault" --json
+cargo run -p mxgw-cli -- acknowledge-alarm --reference "Area001.Pump001.PumpFault" --operator <user> [--operator-domain <domain>] --json
+# operator password: MXGATEWAY_OPERATOR_PASSWORD, or --operator-password-env / --operator-password
 cargo run -p mxgw-cli -- write --session-id <session-id> --server-handle 1 --item-handle 1 --value-type int32 --value 123 --json
 ```
 
@@ -124,6 +125,9 @@ creates an authenticated `tonic` client and attaches `authorization: Bearer
 `stream_alarms`, `acknowledge_alarm`, and `raw_client`. `stream_alarms`
 returns an `AlarmFeedStream` async stream of alarm-feed messages and
 shares the gateway's central alarm monitor with every other client.
+`acknowledge_alarm` requires `operator_user` and carries `operator_password` (may be
+empty) and `operator_domain`; the gateway authenticates as that operator so the Historian
+records them. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The password is scrubbed from any returned error's `Display`/`Debug`.
 
 The `from_truncated_snapshot` field on `ActiveAlarmSnapshot` reports that the record came from
 a provider fetch which hit the per-fetch cap: the snapshot set may omit active
@@ -174,6 +178,18 @@ redaction. Per-item bulk failures are reported inside each result entry
 `invoke_raw` / `client.invoke_raw` escape hatch performs neither check and
 returns the unvalidated reply.
 
+### Diagnostic commands
+
+`Session` wraps the worker diagnostic commands as typed calls: `ping(message)`
+(returns `()` on success), `get_session_state()` (returns `SessionState`),
+`get_worker_info()` (returns `WorkerInfoReply`: pid, version, MXAccess
+ProgID/CLSID) and `drain_events(max_events)` (returns `Vec<MxEvent>`; the reply
+is bounded server-side, so drain until an empty result). A reply missing its
+expected payload surfaces as `Error::MalformedReply`.
+
+TLS note: `ClientOptions::with_ca_file` is read asynchronously (`tokio::fs`)
+during `connect`, so connecting no longer blocks the executor thread.
+
 ## Event Streaming And Reconnect-Replay Gaps
 
 `session.events()` / `session.events_after(after_worker_sequence)` (and the

clients/rust/RustClientDesign.md

@@ -174,6 +174,8 @@ alarm monitor — no worker session is opened — so any number of clients may
 attach. Dropping the stream cancels the gRPC call cooperatively.
 `acknowledge_alarm` is idempotent at the MxAccess layer; the returned
 `AcknowledgeAlarmReply` carries the native MxStatus from the worker.
+`operator_user` is required; `operator_password` may be empty and is scrubbed from any
+returned error via `Error::redact_secrets`.
 
 ## Galaxy Repository
 
@@ -346,7 +348,7 @@ mxgw write-secured-bulk ...
 mxgw write-secured2-bulk ...
 mxgw stream-events --session-id <id> --json
 mxgw stream-alarms [--filter-prefix <prefix>] [--max-events <n>]
-mxgw acknowledge-alarm --reference <full-ref> [--comment <txt>] [--operator <user>]
+mxgw acknowledge-alarm --reference <full-ref> [--comment <txt>] --operator <user> [--operator-domain <domain>] [--operator-password-env <VAR>=MXGATEWAY_OPERATOR_PASSWORD] [--operator-password <pw>]
 mxgw bench-read-bulk [--duration-seconds <n>] [--warmup-seconds <n>] [--bulk-size <n>]
 mxgw smoke --endpoint http://localhost:5000 --api-key-env MXGATEWAY_API_KEY --item TestChildObject.TestInt
 mxgw batch

clients/rust/crates/mxgw-cli/src/main.rs

@@ -215,7 +215,7 @@ enum Command {
         server_handle: i32,
         #[arg(long, value_delimiter = ',')]
         items: Vec<String>,
-        /// Per-tag snapshot timeout in milliseconds. `0` uses the worker default (1000 ms).
+        /// Snapshot deadline in milliseconds, shared by all tags in the command. `0` uses the worker default (1000 ms).
         #[arg(long, default_value_t = 0)]
         timeout_ms: u32,
         #[arg(long)]
@@ -374,8 +374,19 @@ enum Command {
         reference: String,
         #[arg(long, default_value = "")]
         comment: String,
-        #[arg(long, default_value = "")]
+        /// Galaxy account of the acknowledging operator (required; the gateway
+        /// rejects an empty operator so the real operator is recorded).
+        #[arg(long)]
         operator: String,
+        /// Galaxy domain of the operator; empty authenticates the bare user name.
+        #[arg(long, default_value = "")]
+        operator_domain: String,
+        /// Operator password (prefer `--operator-password-env`; empty is allowed).
+        #[arg(long)]
+        operator_password: Option<String>,
+        /// Environment variable holding the operator password.
+        #[arg(long, default_value = "MXGATEWAY_OPERATOR_PASSWORD")]
+        operator_password_env: String,
         #[arg(long)]
         json: bool,
     },
@@ -1081,8 +1092,16 @@ async fn dispatch(command: Command) -> Result<(), Error> {
             reference,
             comment,
             operator,
+            operator_domain,
+            operator_password,
+            operator_password_env,
             json,
         } => {
+            // An empty password is allowed (Galaxy accounts may have one), so
+            // an unset flag and env resolve to an empty string.
+            let operator_password = operator_password
+                .or_else(|| env::var(&operator_password_env).ok())
+                .unwrap_or_default();
             let client = connect(connection).await?;
             let reply = client
                 .acknowledge_alarm(AcknowledgeAlarmRequest {
@@ -1090,6 +1109,8 @@ async fn dispatch(command: Command) -> Result<(), Error> {
                     alarm_full_reference: reference,
                     comment,
                     operator_user: operator,
+                    operator_password,
+                    operator_domain,
                 })
                 .await?;
             print_acknowledge_alarm_reply(&reply, json);
@@ -2771,6 +2792,17 @@ mod tests {
         assert!(parsed.is_ok(), "parse failed: {parsed:?}");
     }
 
+    #[test]
+    fn acknowledge_alarm_requires_operator() {
+        let parsed = Cli::try_parse_from([
+            "mxgw",
+            "acknowledge-alarm",
+            "--reference",
+            "Tank01.Level.HiHi",
+        ]);
+        assert!(parsed.is_err());
+    }
+
     #[test]
     fn acknowledge_alarm_requires_reference() {
         let parsed = Cli::try_parse_from(["mxgw", "acknowledge-alarm"]);

clients/rust/protos/mxaccess_gateway.proto

@@ -391,10 +391,13 @@ message AlarmFailoverConfig {
   int32 failback_stable_probes = 3;          // clean probes before switching back (>=1)
 }
 
-// Acknowledge a single alarm by its GUID. Operator identity fields are
-// recorded atomically with the ack transition in the alarm-history log.
-// The reply's hresult / native_status surfaces AVEVA's
-// AlarmAckByGUID return code.
+// Acknowledge a single alarm by its GUID. The worker resolves the GUID to the
+// alarm's Galaxy reference from its active-alarm snapshot and acknowledges it
+// by writing `comment` to `<TAGNAME>.AckMsg` over MXAccess after
+// AuthenticateUser as the operator, so the alarm history records the real
+// operator. The reply's hresult / native_status is 0 when the write was
+// applied (or sent but unconfirmed), otherwise the AuthenticateUser HRESULT or
+// the OnWriteComplete status detail.
 message AcknowledgeAlarmCommand {
   // Canonical 8-4-4-4-12 GUID string (e.g. "BCC47053-9542-4D65-BDAA-BCDEA6A32A73").
   string alarm_guid = 1;
@@ -403,6 +406,10 @@ message AcknowledgeAlarmCommand {
   string operator_node = 4;
   string operator_domain = 5;
   string operator_full_name = 6;
+  // Operator credential used for AuthenticateUser before the AckMsg write.
+  // Implementations must keep this field out of logs, metrics labels, command
+  // lines, and diagnostics.
+  string operator_password = 7;
 }
 
 // Snapshot the currently-active alarm set. Optional filter prefix scopes
@@ -414,9 +421,7 @@ message QueryActiveAlarmsCommand {
 
 // Acknowledge a single alarm by its (name, provider, group) tuple. Used
 // when the public RPC's AlarmFullReference (Provider!Group.Tag) cannot
-// be resolved to a GUID directly. The worker invokes
-// wwAlarmConsumerClass.AlarmAckByName which reaches the same alarm
-// history path as AlarmAckByGUID.
+// be resolved to a GUID directly. The worker acknowledges by writing `comment` to `<alarm_name>.AckMsg` (Galaxy provider only) after AuthenticateUser as the operator.
 message AcknowledgeAlarmByNameCommand {
   // Tag/alarm name (e.g. "TestMachine_001.TestAlarm001"). Tag itself
   // may contain dots; the gateway-side parser splits on the first dot
@@ -431,6 +436,10 @@ message AcknowledgeAlarmByNameCommand {
   string operator_node = 6;
   string operator_domain = 7;
   string operator_full_name = 8;
+  // Operator credential used for AuthenticateUser before the AckMsg write.
+  // Implementations must keep this field out of logs, metrics labels, command
+  // lines, and diagnostics.
+  string operator_password = 9;
 }
 
 message UnsubscribeBulkCommand {
@@ -725,11 +734,11 @@ message DrainEventsReply {
 // Reply payload for AcknowledgeAlarmCommand AND
 // AcknowledgeAlarmByNameCommand — both ack command kinds reuse this
 // payload case (`MxCommandReply.acknowledge_alarm`); there is no
-// dedicated by-name reply case. Surfaces AVEVA's native ack return
-// code (AlarmAckByGUID for the GUID arm, AlarmAckByName for the
-// by-name arm); 0 means success. The MxCommandReply's hresult field
+// dedicated by-name reply case. Surfaces the ack outcome: 0 when the
+// AckMsg write was applied or sent but unconfirmed; otherwise the
+// AuthenticateUser HRESULT or the OnWriteComplete status detail. The MxCommandReply's hresult field
 // carries the same value and is preferred for protocol consumers —
-// this payload exists so the gateway-side WorkerAlarmRpcDispatcher
+// this payload exists so GatewayAlarmMonitor
 // can echo native_status into AcknowledgeAlarmReply.hresult without
 // unpacking the outer envelope.
 message AcknowledgeAlarmReplyPayload {
@@ -987,6 +996,15 @@ message AcknowledgeAlarmRequest {
   // Operator principal performing the acknowledgement. The lmxopcua side
   // resolves this from the OPC UA session prior to invoking the RPC.
   string operator_user = 5;
+  // Operator's Galaxy (or OS) password. The gateway authenticates as
+  // operator_user with it before writing the acknowledgement so the alarm
+  // history records the real operator. May be empty for accounts without a
+  // password. Credential-sensitive: implementations must keep this field out
+  // of logs, metrics labels, command lines, and diagnostics.
+  string operator_password = 6;
+  // Optional domain for OS-authenticated galaxies. When set the gateway
+  // authenticates as "<operator_domain>\<operator_user>".
+  string operator_domain = 7;
 }
 
 message AcknowledgeAlarmReply {
@@ -997,8 +1015,7 @@ message AcknowledgeAlarmReply {
   ProtocolStatus protocol_status = 3;
   // Native ack return code echoed from the worker. The worker carries the
   // ack outcome as a single int32 (AcknowledgeAlarmReplyPayload.native_status,
-  // = AlarmAckByName / AlarmAckByGUID return code; 0 = success); the gateway's
-  // WorkerAlarmRpcDispatcher copies that value here. This is the authoritative
+  // 0 = applied or sent; otherwise the AuthenticateUser HRESULT or OnWriteComplete detail); GatewayAlarmMonitor copies that value here. This is the authoritative
   // ack-outcome field for the public RPC. Absent only when the worker reply
   // omitted the value entirely (a protocol violation).
   optional int32 hresult = 4;

clients/rust/src/client.rs

@@ -187,7 +187,7 @@ impl GatewayClient {
             })?;
         endpoint = endpoint.connect_timeout(options.connect_timeout());
 
-        if let Some(tls) = build_tls_config(&options)? {
+        if let Some(tls) = build_tls_config(&options).await? {
             endpoint = endpoint.tls_config(tls)?;
         }
 
@@ -327,6 +327,9 @@ impl GatewayClient {
     /// MXAccess session; the resulting native MxStatus is returned in the
     /// reply. Acks are idempotent at the MxAccess layer.
     ///
+    /// `operator_user` is required; `operator_password` may be empty. The
+    /// password is scrubbed from any returned error's `Display`/`Debug`.
+    ///
     /// # Errors
     ///
     /// Returns [`Error::ProtocolStatus`] when the gateway accepts the call but
@@ -336,13 +339,19 @@ impl GatewayClient {
         &self,
         request: AcknowledgeAlarmRequest,
     ) -> Result<AcknowledgeAlarmReply, Error> {
+        let secret = request.operator_password.clone();
         let mut client = self.inner.clone();
-        let response = client
-            .acknowledge_alarm(self.unary_request(request))
-            .await?;
-        let reply = response.into_inner();
-        ensure_protocol_success("acknowledge alarm", reply.protocol_status.as_ref())?;
-        Ok(reply)
+        let result = async {
+            let response = client
+                .acknowledge_alarm(self.unary_request(request))
+                .await?;
+            let reply = response.into_inner();
+            ensure_protocol_success("acknowledge alarm", reply.protocol_status.as_ref())?;
+            Ok(reply)
+        }
+        .await;
+        // The operator password must never surface in a returned error.
+        result.map_err(|error: Error| error.redact_secrets(&[secret.as_str()]))
     }
 
     /// Open the server-streaming `QueryActiveAlarms` RPC — the gateway's

clients/rust/src/error.rs

@@ -314,6 +314,68 @@ impl std::fmt::Display for MxAccessError {
 
 impl std::error::Error for MxAccessError {}
 
+impl Error {
+    /// Scrub exact caller-supplied secrets (for example an operator password)
+    /// from the error's rendered text and any wrapped `tonic::Status`, so that
+    /// neither `Display` nor `Debug` can reveal them. Empty secrets are ignored.
+    #[must_use]
+    pub fn redact_secrets(self, secrets: &[&str]) -> Self {
+        let scrub = |text: &str| {
+            let mut out = text.to_owned();
+            for secret in secrets.iter().filter(|secret| !secret.is_empty()) {
+                out = out.replace(*secret, "<redacted>");
+            }
+            out
+        };
+        let scrub_status = |status: Box<tonic::Status>| {
+            Box::new(tonic::Status::new(status.code(), scrub(status.message())))
+        };
+        match self {
+            Self::Authentication { message, status } => Self::Authentication {
+                message: scrub(&message),
+                status: scrub_status(status),
+            },
+            Self::Authorization { message, status } => Self::Authorization {
+                message: scrub(&message),
+                status: scrub_status(status),
+            },
+            Self::Timeout { message, status } => Self::Timeout {
+                message: scrub(&message),
+                status: scrub_status(status),
+            },
+            Self::Cancelled { message, status } => Self::Cancelled {
+                message: scrub(&message),
+                status: scrub_status(status),
+            },
+            Self::Unavailable { message, status } => Self::Unavailable {
+                message: scrub(&message),
+                status: scrub_status(status),
+            },
+            Self::Status(status) => Self::Status(scrub_status(status)),
+            Self::ProtocolStatus {
+                operation,
+                code,
+                message,
+            } => Self::ProtocolStatus {
+                operation,
+                code,
+                message: scrub(&message),
+            },
+            Self::MalformedReply { detail } => Self::MalformedReply {
+                detail: scrub(&detail),
+            },
+            Self::InvalidArgument { name, detail } => Self::InvalidArgument {
+                name,
+                detail: scrub(&detail),
+            },
+            Self::MxAccess(error) => Self::MxAccess(Box::new(
+                error.with_secrets(secrets.iter().map(|secret| (*secret).to_owned()).collect()),
+            )),
+            other => other,
+        }
+    }
+}
+
 impl From<tonic::Status> for Error {
     fn from(status: tonic::Status) -> Self {
         let message = redact_credentials(status.message());
@@ -461,6 +523,14 @@ fn redact_credentials(message: &str) -> String {
 
 #[cfg(test)]
 mod tests {
+    #[test]
+    fn redact_secrets_scrubs_status_display_and_debug() {
+        let error: super::Error = tonic::Status::internal("boom pw-secret").into();
+        let error = error.redact_secrets(&["pw-secret", ""]);
+        assert!(!format!("{error:?}").contains("pw-secret"));
+        assert!(!error.to_string().contains("pw-secret"));
+    }
+
     use tonic::{Code, Status};
 
     use super::{ensure_mxaccess_success, Error};

clients/rust/src/galaxy.rs

@@ -182,7 +182,7 @@ impl GalaxyClient {
             })?;
         endpoint = endpoint.connect_timeout(options.connect_timeout());
 
-        if let Some(tls) = build_tls_config(&options)? {
+        if let Some(tls) = build_tls_config(&options).await? {
             endpoint = endpoint.tls_config(tls)?;
         }
 

clients/rust/src/options.rs

@@ -3,7 +3,6 @@
 //! chain of `with_*` setters; the `Debug` impl redacts the API key.
 
 use std::fmt;
-use std::fs;
 use std::path::PathBuf;
 use std::time::Duration;
 
@@ -235,7 +234,9 @@ pub(crate) fn tls_trust_decision(options: &ClientOptions) -> TlsTrustDecision {
 /// rejects the configuration early with an actionable error. The strict opt-in
 /// instead loads the system trust roots so a certificate chaining to an
 /// already-trusted root validates.
-pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientTlsConfig>, Error> {
+pub(crate) async fn build_tls_config(
+    options: &ClientOptions,
+) -> Result<Option<ClientTlsConfig>, Error> {
     let decision = tls_trust_decision(options);
     if decision == TlsTrustDecision::None {
         return Ok(None);
@@ -248,10 +249,13 @@ pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientT
     match decision {
         TlsTrustDecision::PinnedCa => {
             let ca_file = options.ca_file().expect("PinnedCa implies a CA file");
-            let certificate = fs::read(ca_file).map_err(|source| Error::InvalidEndpoint {
-                endpoint: options.endpoint().to_owned(),
-                detail: format!("failed to read CA file {}: {source}", ca_file.display()),
-            })?;
+            let certificate =
+                tokio::fs::read(ca_file)
+                    .await
+                    .map_err(|source| Error::InvalidEndpoint {
+                        endpoint: options.endpoint().to_owned(),
+                        detail: format!("failed to read CA file {}: {source}", ca_file.display()),
+                    })?;
             tls = tls.ca_certificate(Certificate::from_pem(certificate));
         }
         TlsTrustDecision::SystemRoots => {
@@ -334,11 +338,11 @@ mod tests {
         assert!(!debug.contains("mxgw_secret"));
     }
 
-    #[test]
-    fn plaintext_needs_no_tls() {
+    #[tokio::test]
+    async fn plaintext_needs_no_tls() {
         let options = ClientOptions::new("http://127.0.0.1:5000").with_plaintext(true);
         assert_eq!(tls_trust_decision(&options), TlsTrustDecision::None);
-        assert!(build_tls_config(&options).unwrap().is_none());
+        assert!(build_tls_config(&options).await.unwrap().is_none());
     }
 
     #[test]
@@ -349,8 +353,8 @@ mod tests {
         assert_eq!(tls_trust_decision(&options), TlsTrustDecision::PinnedCa);
     }
 
-    #[test]
-    fn strict_without_ca_uses_system_roots() {
+    #[tokio::test]
+    async fn strict_without_ca_uses_system_roots() {
         // Regression for Client.Rust-031: strict verification with no pinned CA
         // must verify against the system trust roots, not produce a config with
         // zero trust anchors. The trust decision proves roots are consulted; the
@@ -365,15 +369,24 @@ mod tests {
             "strict-no-CA must request the system trust roots"
         );
         assert!(
-            build_tls_config(&options).unwrap().is_some(),
+            build_tls_config(&options).await.unwrap().is_some(),
             "strict-no-CA must build a usable TLS config"
         );
     }
 
-    #[test]
-    fn lenient_without_ca_is_rejected() {
+    #[tokio::test]
+    async fn lenient_without_ca_is_rejected() {
         let options = ClientOptions::new("https://127.0.0.1:5000").with_plaintext(false);
         assert_eq!(tls_trust_decision(&options), TlsTrustDecision::RejectNoCa);
-        assert!(build_tls_config(&options).is_err());
+        assert!(build_tls_config(&options).await.is_err());
+    }
+
+    #[tokio::test]
+    async fn missing_pinned_ca_file_errors_without_blocking_read() {
+        let options = ClientOptions::new("https://127.0.0.1:5000")
+            .with_plaintext(false)
+            .with_ca_file("/nonexistent/definitely-missing-ca.pem");
+        let error = build_tls_config(&options).await.unwrap_err();
+        assert!(error.to_string().contains("failed to read CA file"));
     }
 }

clients/rust/src/session.rs

@@ -17,14 +17,15 @@ use crate::generated::mxaccess_gateway::v1::mx_command_reply;
 use crate::generated::mxaccess_gateway::v1::{
     ActivateCommand, AddBufferedItemCommand, AddItem2Command, AddItemBulkCommand, AddItemCommand,
     AdviseCommand, AdviseItemBulkCommand, AdviseSupervisoryCommand, ArchestrAUserToIdCommand,
-    AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest, MxCommand,
-    MxCommandKind, MxCommandReply, MxCommandRequest, MxDataType, MxSparseArray, MxSparseElement,
-    MxValue as ProtoMxValue, OpenSessionRequest, ReadBulkCommand, RegisterCommand,
-    RemoveItemBulkCommand, RemoveItemCommand, SetBufferedUpdateIntervalCommand,
+    AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest,
+    DrainEventsCommand, GetSessionStateCommand, GetWorkerInfoCommand, MxCommand, MxCommandKind,
+    MxCommandReply, MxCommandRequest, MxDataType, MxEvent, MxSparseArray, MxSparseElement,
+    MxValue as ProtoMxValue, OpenSessionRequest, PingCommand, ReadBulkCommand, RegisterCommand,
+    RemoveItemBulkCommand, RemoveItemCommand, SessionState, SetBufferedUpdateIntervalCommand,
     StreamEventsRequest, SubscribeBulkCommand, SubscribeResult, SuspendCommand, UnAdviseCommand,
-    UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, Write2BulkCommand,
-    Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry, WriteCommand,
-    WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
+    UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, WorkerInfoReply,
+    Write2BulkCommand, Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry,
+    WriteCommand, WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
     WriteSecuredBulkCommand, WriteSecuredBulkEntry, WriteSecuredCommand,
 };
 use crate::value::{MxStatus, MxValue, MxValueProjection};
@@ -532,7 +533,10 @@ impl Session {
     /// returning the most recent cached `OnDataChange` value when the tag is
     /// already advised (`was_cached = true`), or by taking a full AddItem +
     /// Advise + wait + UnAdvise + RemoveItem snapshot lifecycle otherwise.
-    /// `timeout_ms == 0` lets the worker pick its default (1000 ms).
+    /// `timeout_ms` is one shared deadline for the whole command, not a
+    /// per-tag budget: the worker advises every uncached tag, waits once for
+    /// all of them, then tears them down. `timeout_ms == 0` lets the worker
+    /// pick its default (1000 ms).
     /// Per-tag failures appear as `BulkReadResult` entries with
     /// `was_successful = false`; the call never errors on per-tag failure.
     ///
@@ -960,6 +964,90 @@ impl Session {
             .await
     }
 
+    /// Send a `Ping` diagnostic command to the worker. A successful return
+    /// means the worker answered; `message` is forwarded for log correlation.
+    ///
+    /// # Errors
+    ///
+    /// Returns [`Error::Command`] for non-OK statuses plus transport errors.
+    pub async fn ping(&self, message: &str) -> Result<(), Error> {
+        self.invoke(
+            MxCommandKind::Ping,
+            Payload::Ping(PingCommand {
+                message: message.to_owned(),
+            }),
+        )
+        .await?;
+        Ok(())
+    }
+
+    /// Fetch the gateway-side [`SessionState`] of this session.
+    ///
+    /// # Errors
+    ///
+    /// Returns [`Error::MalformedReply`] when the reply lacks a session-state
+    /// payload, plus the errors of [`invoke`](Self::invoke).
+    pub async fn get_session_state(&self) -> Result<SessionState, Error> {
+        let reply = self
+            .invoke(
+                MxCommandKind::GetSessionState,
+                Payload::GetSessionState(GetSessionStateCommand {}),
+            )
+            .await?;
+        match reply.payload {
+            Some(mx_command_reply::Payload::SessionState(state)) => {
+                Ok(SessionState::try_from(state.state).unwrap_or(SessionState::Unspecified))
+            }
+            _ => Err(Error::MalformedReply {
+                detail: "get_session_state reply did not carry a SessionState payload".to_owned(),
+            }),
+        }
+    }
+
+    /// Fetch worker process details (pid, version, MXAccess ProgID/CLSID).
+    ///
+    /// # Errors
+    ///
+    /// Returns [`Error::MalformedReply`] when the reply lacks a worker-info
+    /// payload, plus the errors of [`invoke`](Self::invoke).
+    pub async fn get_worker_info(&self) -> Result<WorkerInfoReply, Error> {
+        let reply = self
+            .invoke(
+                MxCommandKind::GetWorkerInfo,
+                Payload::GetWorkerInfo(GetWorkerInfoCommand {}),
+            )
+            .await?;
+        match reply.payload {
+            Some(mx_command_reply::Payload::WorkerInfo(info)) => Ok(info),
+            _ => Err(Error::MalformedReply {
+                detail: "get_worker_info reply did not carry a WorkerInfo payload".to_owned(),
+            }),
+        }
+    }
+
+    /// Drain up to `max_events` queued events through the diagnostic command
+    /// (distinct from the streaming [`events`](Self::events) surface). The
+    /// reply is bounded server-side, so call repeatedly until it is empty.
+    ///
+    /// # Errors
+    ///
+    /// Returns [`Error::MalformedReply`] when the reply lacks a drain-events
+    /// payload, plus the errors of [`invoke`](Self::invoke).
+    pub async fn drain_events(&self, max_events: u32) -> Result<Vec<MxEvent>, Error> {
+        let reply = self
+            .invoke(
+                MxCommandKind::DrainEvents,
+                Payload::DrainEvents(DrainEventsCommand { max_events }),
+            )
+            .await?;
+        match reply.payload {
+            Some(mx_command_reply::Payload::DrainEvents(drain)) => Ok(drain.events),
+            _ => Err(Error::MalformedReply {
+                detail: "drain_events reply did not carry a DrainEvents payload".to_owned(),
+            }),
+        }
+    }
+
     /// Issue a raw `Invoke` for an arbitrary command, without filtering on
     /// the protocol status. Useful when callers need the full reply for
     /// commands not yet wrapped by `Session`.

clients/rust/tests/client_behavior.rs

@@ -26,12 +26,12 @@ use zb_mom_ww_mxgateway_client::generated::mxaccess_gateway::v1::{
     AddItem2Reply, AddItemReply, AlarmConditionState, AlarmFeedMessage, AlarmTransitionKind,
     AuthenticateUserCommand, AuthenticateUserReply, BulkReadReply, BulkReadResult,
     BulkSubscribeReply, BulkWriteReply, BulkWriteResult, CloseSessionReply, CloseSessionRequest,
-    MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily, MxSparseArray,
-    MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
+    DrainEventsReply, MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily,
+    MxSparseArray, MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
     OnAlarmTransitionEvent, OpenSessionReply, OpenSessionRequest, ProtocolStatus,
     ProtocolStatusCode, QueryActiveAlarmsRequest, RegisterReply, ReplayGap, SessionState,
-    StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, Write2BulkEntry, WriteBulkEntry,
-    WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
+    SessionStateReply, StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, WorkerInfoReply,
+    Write2BulkEntry, WriteBulkEntry, WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
 };
 use zb_mom_ww_mxgateway_client::{
     next_correlation_id, ApiKey, ClientOptions, CommandError, Error, EventItem, GatewayClient,
@@ -229,6 +229,8 @@ async fn acknowledge_alarm_returns_reply_with_native_status() {
             alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
             comment: "investigating".to_owned(),
             operator_user: "alice".to_owned(),
+            operator_password: String::new(),
+            operator_domain: String::new(),
         })
         .await
         .unwrap();
@@ -548,6 +550,81 @@ async fn unary_invoke_maps_status_unavailable_to_error_unavailable() {
     );
 }
 
+#[tokio::test]
+async fn ping_round_trips_through_the_fake_gateway() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+    let session = client.session("session-fixture");
+
+    session.ping("hello").await.unwrap();
+
+    assert_eq!(
+        *state.last_command_kind.lock().await,
+        Some(MxCommandKind::Ping as i32)
+    );
+}
+
+#[tokio::test]
+async fn get_session_state_returns_typed_state() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+    let session = client.session("session-fixture");
+
+    let session_state = session.get_session_state().await.unwrap();
+
+    assert_eq!(session_state, SessionState::Ready);
+}
+
+#[tokio::test]
+async fn get_worker_info_returns_typed_reply() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+    let session = client.session("session-fixture");
+
+    let info = session.get_worker_info().await.unwrap();
+
+    assert_eq!(info.worker_process_id, 4242);
+    assert_eq!(info.worker_version, "1.2.3");
+}
+
+#[tokio::test]
+async fn drain_events_returns_events() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+    let session = client.session("session-fixture");
+
+    let events = session.drain_events(10).await.unwrap();
+
+    assert_eq!(events.len(), 1);
+}
+
+#[tokio::test]
+async fn get_session_state_malformed_without_payload() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+    let session = client.session("session-fixture");
+    *state.invoke_override.lock().await = Some(InvokeOverride::OkWithoutPayload);
+
+    let error = session.get_session_state().await.unwrap_err();
+
+    assert!(matches!(error, Error::MalformedReply { .. }));
+}
+
 #[tokio::test]
 async fn read_bulk_round_trips_through_the_fake_gateway() {
     let state = Arc::new(FakeState::default());
@@ -1036,6 +1113,30 @@ async fn stream_alarms_emits_snapshot_then_complete_then_transition_in_order() {
     ));
 }
 
+#[tokio::test]
+async fn acknowledge_alarm_error_does_not_leak_operator_password() {
+    let state = Arc::new(FakeState::default());
+    let endpoint = spawn_fake_gateway(state.clone()).await;
+    let client = GatewayClient::connect(ClientOptions::new(endpoint))
+        .await
+        .unwrap();
+
+    let err = client
+        .acknowledge_alarm(AcknowledgeAlarmRequest {
+            client_correlation_id: "corr-pw".to_owned(),
+            alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
+            comment: String::new(),
+            operator_user: "alice".to_owned(),
+            operator_password: "pw-secret".to_owned(),
+            operator_domain: String::new(),
+        })
+        .await
+        .unwrap_err();
+
+    assert!(!format!("{err:?}").contains("pw-secret"));
+    assert!(!err.to_string().contains("pw-secret"));
+}
+
 #[tokio::test]
 async fn cli_subcommands_propagate_unique_correlation_ids_from_next_correlation_id() {
     // The CLI's `stream-alarms` and `acknowledge-alarm` paths used to
@@ -1073,6 +1174,8 @@ async fn cli_subcommands_propagate_unique_correlation_ids_from_next_correlation_
             alarm_full_reference: "Tank01.Level.HiHi".to_owned(),
             comment: String::new(),
             operator_user: String::new(),
+            operator_password: String::new(),
+            operator_domain: String::new(),
         })
         .await
         .unwrap();
@@ -1309,6 +1412,57 @@ impl MxAccessGateway for FakeGateway {
             }));
         }
 
+        if kind == MxCommandKind::Ping as i32 {
+            return Ok(Response::new(MxCommandReply {
+                session_id: request.session_id,
+                correlation_id: "fake-correlation".to_owned(),
+                kind,
+                protocol_status: Some(ok_status("command ok")),
+                ..MxCommandReply::default()
+            }));
+        }
+
+        if kind == MxCommandKind::GetSessionState as i32 {
+            return Ok(Response::new(MxCommandReply {
+                session_id: request.session_id,
+                correlation_id: "fake-correlation".to_owned(),
+                kind,
+                protocol_status: Some(ok_status("command ok")),
+                payload: Some(mx_command_reply::Payload::SessionState(SessionStateReply {
+                    state: SessionState::Ready as i32,
+                })),
+                ..MxCommandReply::default()
+            }));
+        }
+
+        if kind == MxCommandKind::GetWorkerInfo as i32 {
+            return Ok(Response::new(MxCommandReply {
+                session_id: request.session_id,
+                correlation_id: "fake-correlation".to_owned(),
+                kind,
+                protocol_status: Some(ok_status("command ok")),
+                payload: Some(mx_command_reply::Payload::WorkerInfo(WorkerInfoReply {
+                    worker_process_id: 4242,
+                    worker_version: "1.2.3".to_owned(),
+                    ..WorkerInfoReply::default()
+                })),
+                ..MxCommandReply::default()
+            }));
+        }
+
+        if kind == MxCommandKind::DrainEvents as i32 {
+            return Ok(Response::new(MxCommandReply {
+                session_id: request.session_id,
+                correlation_id: "fake-correlation".to_owned(),
+                kind,
+                protocol_status: Some(ok_status("command ok")),
+                payload: Some(mx_command_reply::Payload::DrainEvents(DrainEventsReply {
+                    events: vec![MxEvent::default()],
+                })),
+                ..MxCommandReply::default()
+            }));
+        }
+
         if kind == MxCommandKind::ReadBulk as i32 {
             return Ok(Response::new(MxCommandReply {
                 session_id: request.session_id,
@@ -1424,8 +1578,11 @@ impl MxAccessGateway for FakeGateway {
         &self,
         request: Request<AcknowledgeAlarmRequest>,
     ) -> Result<Response<AcknowledgeAlarmReply>, Status> {
-        *self.state.last_correlation_id.lock().await =
-            Some(request.into_inner().client_correlation_id);
+        let request = request.into_inner();
+        if request.operator_password == "pw-secret" {
+            return Err(Status::internal("boom pw-secret"));
+        }
+        *self.state.last_correlation_id.lock().await = Some(request.client_correlation_id);
         Ok(Response::new(AcknowledgeAlarmReply {
             correlation_id: "corr-1".to_owned(),
             protocol_status: Some(ok_status("ack ok")),

docs/AlarmClientDiscovery.md

@@ -621,8 +621,9 @@ returned to normal but is unacknowledged — i.e., visible in the
 "current alarms" list because operator hasn't acked it yet) and
 `UNACK_ALM` (the alarm is currently active and unacknowledged).
 The other states from `eAlmState` (`ACK_RTN`, `ACK_ALM`) would
-appear when an ack is performed — `wwAlarmConsumerClass.AlarmAckByGUID`
-is the method to call.
+appear when an ack is performed. The alarm-client ack methods
+(`AlarmAckByGUID` and friends) do not produce them headless; an `AckMsg`
+write does (`AlarmProbeFindings.md`).
 
 ### `GetStatistics` AV — unrelated quirk
 
@@ -660,7 +661,9 @@ alarm-consumer surface unblocks A.2 fully. Outline:
      events for added/changed/removed records.
    - `AlarmAckByGUID(VBGUID, comment, oprName, node, domain,
      fullName)` for client-driven acknowledgements (matches
-     PR A.5's `AlarmAckCommand` payload).
+     PR A.5's `AlarmAckCommand` payload). *Ack step superseded
+     2026-10-19: acks write `AckMsg` as the authenticated operator
+     (see "Acknowledge in subtag mode" and `AlarmProbeFindings.md`).*
    - Lifecycle teardown: `DeregisterConsumer` +
      `UninitializeConsumer` + `Marshal.FinalReleaseComObject`.
 3. **Conversion layer:** map XML record fields to
@@ -701,8 +704,9 @@ alarm-consumer surface unblocks A.2 fully. Outline:
   `bRetainHiddenAlarms` parameter at all.
 - Subscription expression format: `\\<machine>\Galaxy!<area>`
   (literal `Galaxy` provider) for both libraries.
-- Native ack: `AlarmAckByGUID(VBGUID guid, comment, oprName,
-  node, domain, fullName)` on the v2 surface; ID 5-arg
+- Native ack surface (not used by the gateway; superseded 2026-10-19 by
+  the authenticated `AckMsg` write): `AlarmAckByGUID(VBGUID guid,
+  comment, oprName, node, domain, fullName)` on the v2 surface; ID 5-arg
   variant on the legacy `IwwAlarmConsumer` interface.
 
 These findings retire the open follow-up probes from the
@@ -735,25 +739,32 @@ for `SetXmlAlarmQuery`.
 the canonical `\\<machine>\Galaxy!<area>` form into the XML's
 NODE/PROVIDER/GROUP fields. Mangled or not, the call enables reads.
 
-### 2. Two consumers required: read-side vs. ack-side
+### 2. Two consumers required: read-side vs. ack-side (historical)
+
+> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
+> see `AlarmProbeFindings.md`. The worker no longer provisions an ack-only
+> wnwrap consumer. The text below records why the split existed.
 
 `SetXmlAlarmQuery` enables reads but **breaks `AlarmAckByName` on
 the same consumer instance**. With SetXml applied, AlarmAckByName
 returns -55 even with valid name+provider+group+operator. Without
 SetXml, AlarmAckByName succeeds with rc=0.
 
-The production consumer therefore provisions **two** wnwrap COM
+The consumer at the time provisioned **two** wnwrap COM
 instances:
 - Primary consumer (`client`): runs full lifecycle including
   `SetXmlAlarmQuery` for `GetXmlCurrentAlarms2` polls.
 - Ack-only consumer (`ackClient`): runs Initialize → Register →
   Subscribe via the v1-prefixed methods, **no SetXmlAlarmQuery**.
-  All `AcknowledgeByName` calls dispatch through this instance.
+  All by-name ack calls dispatched through this instance.
 
 Both consumers subscribe to the same expression. Disposal cleans up
 both via a shared `ReleaseConsumerCom` helper.
 
-### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg
+### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg (historical)
+
+> Superseded 2026-10-19: the gateway no longer calls `AlarmAckByName`; see
+> `AlarmProbeFindings.md`.
 
 `wwAlarmConsumerClass` exposes two `AlarmAckByName` overloads:
 - `IwwAlarmConsumer2` v2: 8 args (`name, provider, group, comment,
@@ -762,8 +773,8 @@ both via a shared `ReleaseConsumerCom` helper.
 
 The v2 8-arg method returns -55 on this AVEVA build regardless of
 operator-identity inputs — looks like a stub. The v1 6-arg method
-works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the
-6-arg overload and discards the proto's `domain` + `full_name` fields.
+works. `WnWrapAlarmConsumer.AcknowledgeByName` called the
+6-arg overload and discarded the proto's `domain` + `full_name` fields.
 
 **Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves
 of the paragraph above are narrower than they read. The -55 tracks the
@@ -775,23 +786,58 @@ snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm
 extension's own `.Acked` attribute stay `False` for 16 s after each
 call. On this rig the wnwrap ack is accepted and then inert, so a zero
 return code must not be read as an applied acknowledgement.
-The proto contract keeps the 8 fields for forward compatibility if
-AVEVA fixes the v2 method later.
 
-### 4. `AlarmAckByGUID` is not implemented
+### 4. `AlarmAckByGUID` is not implemented (historical)
+
+> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
+> see `AlarmProbeFindings.md`.
 
 The v2 `AlarmAckByGUID(VBGUID, …)` throws `NotImplementedException`
 (COM `E_NOTIMPL`) on `wwAlarmConsumerClass` against this AVEVA
 build. The reference→GUID lookup that we initially planned to wire
-through `AlarmAckByGUID` is therefore not viable on wnwrap; all acks
-must go through `AlarmAckByName`.
+through `AlarmAckByGUID` is therefore not viable on wnwrap.
 
-The proto `AcknowledgeAlarmCommand` (GUID-based) and the worker's
-`MxAccessCommandExecutor.ExecuteAcknowledgeAlarm` switch arm remain
-in the codebase for the forward-compat shape, but the gateway-side
-`WorkerAlarmRpcDispatcher.AcknowledgeAsync` now always routes through
-`AcknowledgeAlarmByName` when the public RPC supplies a recognizable
-`Provider!Group.Tag` reference.
+#### Managed GUID ack experiment — did not work (live-tested 2026-10-18, removed 2026-10-19)
+
+An opt-in experiment drove `aaAlarmManagedClient.AlarmClient` by reflection
+(its members are non-public) to take GUID-keyed acks in place of the wnwrap
+method. The live test showed it could not work, and the experiment and its
+enabling flag were removed once the `AckMsg` write replaced it. The findings
+are kept because they explain why the alarm-client route was abandoned.
+
+The test raised `TestMachine_001.TestAlarm001` with a secured write,
+read its GUID from the wnwrap XML snapshot (the public API exposes no
+alarm GUIDs), and acknowledged it through the `AcknowledgeAlarm` RPC
+with the experiment enabled:
+
+| Step | Result |
+|---|---|
+| RPC ack by GUID, experiment off (control) | `NotImplementedException`; `.Acked` stays `False` |
+| RPC ack by GUID, experiment on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty |
+| `AlarmClient.RegisterConsumer` in a process with a registered wnwrap consumer | `-65` — and wnwrap gets `-65` if the managed client registered first; a real `hWnd` does not help |
+| Managed client alone in its own process, records loaded | `AlarmAckByGUID` → `-1`, `SelectByGUID` → `-1` for the wnwrap GUID in both byte orders, `AlarmAckSelected` → `-57` |
+
+Three separate problems follow:
+
+1. **The two libraries cannot share a process.** The worker always runs
+   a wnwrap reader, so the managed `RegisterConsumer` fails with `-65`.
+   The dispatcher is built to drop back to the wnwrap ack path on a
+   lifecycle failure (so alarm reads keep working), and it does so
+   silently — which is why the experiment-on RPC was indistinguishable from the
+   control.
+2. **The managed client does not recognise the wnwrap GUID.** `-1` from
+   `SelectByGUID` is "not found", not a security refusal, even with the
+   client holding records. Getting the client's own identifier for the
+   alarm is not reliable either: `GetHighPriAlarm` mostly throws the
+   FILETIME `ArgumentOutOfRangeException` described above. The
+   `MxSecurityOperate` question was therefore never reached.
+3. **The gateway does not send the identity fields.** Its GUID arm fills
+   `OperatorNode`, `OperatorDomain` and `OperatorFullName` with empty
+   strings, so the domain and full name the managed method carries
+   would not arrive through the public RPC even if 1 and 2 were solved.
+
+The route was dropped in favour of the authenticated `AckMsg` write
+(`AlarmProbeFindings.md`, "Fourth attempt").
 
 ### 5. STA / threading — resolved
 
@@ -947,20 +993,24 @@ AVEVA-internal GUID.
 
 ### Acknowledge in subtag mode
 
-`AlarmDispatcher` routes ack calls by active provider mode:
+Both provider modes acknowledge through the worker's `AlarmAckWriter`, which
+calls `AuthenticateUser` as the requesting operator and writes the comment to
+the ack item over MXAccess on its own `LMXProxyServer`:
 
-- **Alarm-manager mode:** `AlarmAckByName` on `wwAlarmConsumerClass` (unchanged).
-- **Subtag mode:** `SubtagAlarmConsumer.AcknowledgeByName` resolves the
-  watch-list entry's `ack_comment_subtag` and issues a `Write(comment)` on
-  the STA via `LmxSubtagAlarmSource`. Writing the `AckMsg` subtag performs
-  the acknowledge in AVEVA (`AckMsg` is the confirmed `AlarmExtension` ack-comment
-  write target).
+- **Alarm-manager mode:** the item is `<TagName>.AckMsg` for Galaxy-provider
+  alarms. An alarm from another provider is not ackable.
+- **Subtag mode:** `SubtagAlarmConsumer` resolves the watch-list entry's
+  `ack_comment_subtag` and the writer writes that item verbatim. Writing the
+  `AckMsg` subtag performs the acknowledge in AVEVA (`AckMsg` is the confirmed
+  `AlarmExtension` ack-comment write target).
 
-If the alarm has no writable ack-comment subtag (`AckComment` config key is
-empty, or the entry's `ack_comment_subtag` field is empty), the ack call
-returns a failure code that the gateway surfaces as `FailedPrecondition`.
-`AcknowledgeByGuid` maps the synthetic GUID back to its reference via an
-internal dictionary, then calls the same write path.
+Because the write is authenticated, the Historian records the operator's name,
+domain and full name, and the writer's MXAccess client name
+(`MxGateway:Alarms:AckClientName`) as OperatorNode. If the alarm has no ack
+target (`AckComment` config key is empty, the entry's `ack_comment_subtag` is
+empty, or the reference or synthetic GUID is unknown), the worker reports it
+as not found or not ackable and the gateway surfaces `InvalidRequest` with
+`hresult` -1.
 
 `SubtagAlarmConsumer.Subscribe` advises the ack-comment subtag alongside the
 observed ones (active/acked/priority). This is required: MXAccess rejects a
@@ -978,7 +1028,7 @@ The subtag path was validated against live MXAccess on the dev rig
   `…​.Priority` → `500` (Int32), `…​.AckMsg` → string — confirming the field
   names **and** the runtime reference shape `<Object>.<AlarmAttr>.<field>`
   with **no** intermediate alarm-condition segment.
-- `AcknowledgeByName` (AckMsg write) returned `0` once the ack-comment subtag
+- The `AckMsg` write returned `0` once the ack-comment subtag
   was advised — confirming the ack-by-comment-write mechanism end to end.
 
 ### Fidelity limitations
@@ -1001,6 +1051,18 @@ Every transition and snapshot record carries `degraded = true` and
 `source_provider = ALARM_PROVIDER_MODE_SUBTAG`. Clients that require full
 fidelity must wait for failback to the alarm manager.
 
+### Retrigger transitions
+
+`ALARM_TRANSITION_KIND_RETRIGGER` is reserved in the contract and the worker
+never emits it, in either provider mode. The alarmmgr XML reports only the
+four states `UNACK_ALM`, `ACK_ALM`, `UNACK_RTN`, and `ACK_RTN`, so a condition
+that returns to alarm after clearing (`*_RTN` → `*_ALM`) looks the same as a
+new raise, and `AlarmRecordTransitionMapper.MapTransition` reports it as
+`ALARM_TRANSITION_KIND_RAISE`. The subtag source has no re-alarm counter to
+observe either. The worker does not infer a retrigger from timing or
+from a raise that follows a clear, because the gateway forwards only
+transitions its source reports.
+
 ### Provider mode reflection
 
 When `FailoverAlarmConsumer` switches between providers, it raises

docs/AlarmProbeFindings.md

@@ -24,13 +24,15 @@ the blocker rather than rediscovering it.
 |---|---|
 | GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) |
 | GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID |
-| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state; 2026-08-18 (third attempt) identifies the reason as the test attribute's `MxSecurityOperate` classification, which no non-interactive ack surface on the rig can satisfy |
+| GUID stable across `UNACK → ACK` | Blocker resolved 2026-10-19 — the rig is driven to `ACK_ALM` by an `AckMsg` write, and the `MxSecurityOperate` classification was not the cause (see "Fourth attempt"); GUID survival across that transition has not been recorded yet |
+| Acknowledge an alarm headless | Answered 2026-10-19 — **an `AckMsg` write as the authenticated operator**; every alarm-client ack is inert (see "Fourth attempt") |
 | `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** |
 
 The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The
 2026-08-18 run cleared that blocker with `AuthenticateUser` + `WriteSecured` and answered
 two of the three questions; the acknowledge leg is now blocked on something narrower and
-different, described in "Second attempt" and diagnosed in "Third attempt".
+different, described in "Second attempt" and diagnosed in "Third attempt"
+(resolved in the Fourth attempt, 2026-10-19).
 
 ## First attempt (2026-08-17): plain `Write`
 
@@ -420,3 +422,126 @@ The second attempt's inventory, unchanged since: the three
 snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only
 `UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on
 the next `alarmmgr` restart.
+
+## Fourth attempt (2026-10-19): authenticated AckMsg write
+
+The third attempt left the acknowledge leg blocked. This run found a route that
+acknowledges and identifies the operator, and ruled out the alarm-client routes.
+It drove `TestMachine_001.TestAlarm001`, raised with
+`mxa write TestMachine_001.TestAlarm001 true --type bool --secured -u Administrator`.
+
+### Writing `AckMsg` acknowledges
+
+Writing the comment to `TestMachine_001.TestAlarm001.AckMsg` over MXAccess
+acknowledges the alarm: `.Acked` becomes `True` and the wnwrap record moves to
+`STATE=ACK_ALM` with an `ACK_COMMENT`. `AckMsg` is `MxSecurityFreeAccess` even
+though the alarm UDA itself is `MxSecurityOperate`, so no Galaxy change is
+needed.
+
+What the Historian records, read from `Runtime.dbo.v_AlarmEventHistory2`:
+
+| Write | Operator | DomainName | UserFullName | OperatorNode | Description |
+|---|---|---|---|---|---|
+| Unauthenticated `Write` | `NULL` | `NULL` | `NULL` | `NULL` | the comment |
+| `AuthenticateUser` + `Write(value, userId)` | `Administrator` | `ArchestrA` | `Administrator` | `mxa` (the MXAccess `Register()` client name) | the comment |
+
+The wnwrap XML also gains `ACK_OPERATOR_NAME`, `ACK_DOMAIN_NAME`,
+`ACK_OPERATOR_NODE` and `ACK_COMMENT` on the acknowledged record; the worker's
+snapshot parser reads them for Acknowledge transitions. An unauthenticated write
+therefore acknowledges but leaves no operator in the Historian, which is why the
+gateway requires `operator_user` and authenticates it.
+
+Another actor's `AckMsg` writes on the same day (authenticated as another
+operator account) logged that account as `Operator`, `Domain="."` and that machine's name as the node,
+which matches the client-name rule. By this date unauthenticated writes to
+`MxSecurityOperate` attributes were also accepted Galaxy-wide on the rig, where
+August refused them with 1008.
+
+### Alarm-client acks are inert in every variant
+
+All of these returned `rc=0` or `-1` and changed neither the snapshot, `.Acked`
+nor the Historian:
+
+- wnwrap `AlarmAckByName` and `AlarmAckByGUID` (`E_NOTIMPL`) on the ack-only and
+  reader consumers.
+- wnwrap `IwwAlarmConsumer2` on a sole consumer with `SetXmlAlarmQuery` applied
+  (list populated, `GetAlarm(0)` returns the row): `AlarmAck` with ids
+  0/1/2/3/-1/65536, and `Select(0)` + `AlarmAckSelected`, with operators
+  `TestOperator`, `Administrator` and a third account. `SelectByGUID` is `E_NOTIMPL`.
+- The same calls with `TestAlarm001` temporarily reclassified to
+  `MxSecurityFreeAccess` (template and instance, restored afterwards), so the
+  security classification is not the cause.
+- Managed `aaAlarmManagedClient`: `AlarmAckByGUID` and `SelectByGUID` return
+  `-1` (the client's list never fills headless, even with a real `hWnd` and a
+  message pump); `AckSelectedTagAlarms_Ex` returns `0` even for a
+  `NOSUCHPROVIDER` provider and writes no Historian row. The managed and wnwrap
+  consumers cannot share a process (`RegisterConsumer` returns `-65`).
+- A 20 s continuous message pump after `AlarmAck`, and registering as `InTouch`,
+  `WindowViewer` or `Wonderware InTouch`.
+
+Static reverse engineering matched: the five-string COM acks forward domain and
+full name with an empty password, and InTouch's `AlarmViewer.ocx` also sends an
+empty password. The user-request call is synchronous request-to-return-code with
+no completion channel, so the drop happens provider-side. Hooking the user
+request in a live WindowViewer that acknowledges successfully and diffing it
+against the headless probe is the one definitive step not taken.
+
+### Consequence
+
+The gateway acknowledges by authenticating the operator and writing `AckMsg`
+(`docs/DesignDecisions.md`, "Alarm acknowledgement writes AckMsg as the
+authenticated operator"). The managed ack consumer and the second wnwrap ack
+consumer were removed.
+
+### Gateway-level verification
+
+The same result holds end to end through the gateway. With the feature build deployed on
+the windev rig and `TestAlarm001` raised, an `AcknowledgeAlarm` RPC for
+`Galaxy!TestArea.TestMachine_001.TestAlarm001` with `operator_user=Administrator` and an
+empty password returned `PROTOCOL_STATUS_CODE_OK`, `hresult` 0. `.Acked` turned True,
+`.AckMsg` held the comment, and the Historian logged the `ACK_ALM` row with
+`Operator=Administrator`, `DomainName=ArchestrA`, `UserFullName=Administrator`,
+`OperatorNode=MxGateway` (the default `AckClientName`), and `Description` = the comment.
+
+A wrong password, or a user Galaxy security does not know, makes `AuthenticateUser`
+throw `ArgumentException` (`E_INVALIDARG`, 0x80070057) rather than a `COMException`.
+That is why the writer treats `ArgumentException` from the MXAccess call itself as an
+interop failure alongside `COMException`, and reports the HRESULT the exception carries.
+Such an ack returns `PROTOCOL_STATUS_CODE_MXACCESS_FAILURE`, the alarm stays unacked,
+the Historian gains no ack row, and the password string does not appear in the gateway
+log, `stdout.log`, or `stderr.log`. A non-Galaxy provider reference returns
+`PROTOCOL_STATUS_CODE_INVALID_REQUEST` with `hresult` -1, and an empty `operator_user`
+returns gRPC `InvalidArgument`.
+
+The GUID arm of the ack command was not driven live. The public alarm RPCs expose only
+the full reference, so clients reach the by-name arm; worker unit tests cover the GUID
+arm.
+
+## Item handle reuse
+
+`AlarmAckWriter` correlates an ack with its `OnWriteComplete` by the
+(server handle, item handle) pair, because the callback carries no per-write id.
+After an unconfirmed ack it drops the `AckMsg` item and re-adds it so the next
+ack waits on a new handle that a late completion for the old one cannot match.
+That only works if MXAccess does not hand the same number out again.
+
+`ItemHandleReuseProbeTests` (Skip-gated, worker tests) checked this on windev
+(`10.100.0.48`, Galaxy `DEV`) on 2026-10-20 against
+`TestMachine_001.TestAlarm001.AckMsg`, without writing:
+
+| Sequence on one registration (server handle 1) | Item handles returned |
+|---|---|
+| `AddItem` + `Advise`, then `UnAdvise` + `RemoveItem`, five times | 1, 2, 3, 4, 5 |
+| `AddItem`, then `RemoveItem`, five times | 6, 7, 8, 9, 10 |
+| Add and remove `AckMsg`, add `Acked`, add `AckMsg` again | 11, then 12 (`Acked`), 13 |
+| Add `AckMsg` twice while both are held | 14, 15 |
+
+Within a registration the numbers increase and a freed number is never reused,
+so the drop-and-re-add mitigation holds. Numbering is per registration:
+`Unregister` + `Register` on the same COM object returned server handle 2 and
+item handle 1, and a new `LMXProxyServerClass` returned server handle 1 and
+item handle 1 again. The writer is not exposed to that restart because its
+teardown detaches its `OnWriteComplete` handler from the old COM object and
+clears that server handle's completion entries before it creates a new one.
+No per-handle generation counter is needed.
+

docs/Authentication.md

@@ -113,6 +113,16 @@ library:
   library returns carries no expiry timestamp, so the cache cannot cap an entry at the
   key's expiry (capping it needs the donor library to surface expiry on the
   verification identity). The default 15 s TTL bounds both windows.
+  Misses are **single-flight**: concurrent misses for the same token share one inner
+  verification, so a TTL expiry under load costs one store read instead of one per
+  in-flight RPC. The flight is keyed on the token hash *and* the key's generation, so
+  a miss that starts after an `Invalidate` never joins a verification that started
+  before it and cannot receive the pre-revoke identity. The shared inner call is not
+  bound to any one caller's cancellation token; a caller that gives up stops waiting
+  without cancelling the others. The per-key index `Invalidate` uses to find every
+  cached secret variant is pruned by a post-eviction callback when the cache drops an
+  entry on its own (TTL expiry, compaction), so it tracks the live cache instead of
+  growing with every token ever verified.
 - **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
   most one `MarkUsed` write per key per
   `MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a

docs/Authorization.md

@@ -10,7 +10,7 @@ Authorization runs as a single gRPC server interceptor registered for every call
 
 The participating types live under `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/`:
 
-- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for unary and server-streaming calls.
+- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for every call shape: unary, server-streaming, client-streaming, and duplex-streaming.
 - `GatewayGrpcScopeResolver` maps a request message (and, for `MxCommandRequest`, the inner `MxCommandKind`) to the scope string that must be present on the caller.
 - `GatewayScopes` exposes the canonical scope constants used by the resolver and any downstream consumer.
 - `GatewayRequestIdentityAccessor` and `IGatewayRequestIdentityAccessor` expose the verified identity to handlers and any service code that runs inside the call.
@@ -32,7 +32,7 @@ Centralizing the policy in `GatewayGrpcAuthorizationInterceptor` produces three
 
 ## Interceptor Flow
 
-`GatewayGrpcAuthorizationInterceptor` overrides both `UnaryServerHandler` and `ServerStreamingServerHandler`. Both call the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then push the resolved identity onto the accessor for the duration of the call.
+`GatewayGrpcAuthorizationInterceptor` overrides all four server handlers: `UnaryServerHandler`, `ServerStreamingServerHandler`, `ClientStreamingServerHandler`, and `DuplexStreamingServerHandler`. Each calls the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then pushes the resolved identity onto the accessor for the duration of the call. The base `Interceptor` passes any handler it does not override straight through, so a client- or duplex-streaming RPC added to a mapped service would otherwise run with no API key check at all. The unary and server-streaming handlers resolve the required scope from the request message; the client- and duplex-streaming handlers have no single request to inspect before the handler reads the stream, so they fail closed to `admin` — the same scope the resolver assigns an unrecognized request type. No public RPC uses either shape today.
 
 ```csharp
 public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
@@ -40,7 +40,10 @@ public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
     ServerCallContext context,
     UnaryServerMethod<TRequest, TResponse> continuation)
 {
-    ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
+    ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
+            scopeResolver.ResolveRequiredScope(request),
+            context)
+        .ConfigureAwait(false);
     IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
     using (identityScope)
     {
@@ -96,7 +99,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
 - **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
 - **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
 
-An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
+An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is anonymous by default (`MxGateway:Metrics:RequireAuthentication` is opt-in), so neither key ids nor peer addresses may appear there.
 
 The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
 
@@ -125,7 +128,7 @@ public string ResolveRequiredScope(object request)
 }
 ```
 
-The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
+The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*`. It stays `invoke:write` rather than `invoke:secure` because the request carries the operator's own Galaxy credential, which the worker authenticates before the `AckMsg` write; the credential, not the API-key scope, authorizes the identity the Historian records. The trade-off is that an `invoke:write` key can tell `AuthenticationFailed` (an `MXACCESS_FAILURE` whose diagnostic reports operator authentication failure) apart from every other outcome, which makes `AcknowledgeAlarm` a Galaxy-password oracle for that key — whereas the `AuthenticateUser` command itself requires `invoke:secure`. What bounds it: the caller must already hold a valid, admin-issued, revocable `invoke:write` key, and every acknowledgement is serialized through the single monitor worker session's STA, so guesses run one at a time. The gateway does **not** throttle failed operator authentications: the `ApiKeyFailureLimiter` counts only failed API-key verifications, and the ASP.NET Core rate limiter covers only the dashboard `/login` endpoint. Any lockout after repeated bad passwords comes from the Galaxy security configuration, not from the gateway. Grant `invoke:write` only to clients trusted with that exposure. `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
 
 `MxCommandRequest` is special because it multiplexes many MxAccess operations through a single RPC. The resolver inspects the embedded `MxCommandKind` so each operation gets its own scope:
 
@@ -262,9 +265,17 @@ the gateway fails closed.
 
 Non-bulk constraint failures return gRPC `PermissionDenied`. Bulk read
 commands preserve input order and return a failed `SubscribeResult` for each
-denied item while still forwarding allowed items to the worker. Every denial
-adds an `api_key_audit` entry with the key id, command kind, target, and
-blocking constraint; secured values and raw credentials are never logged.
+denied item while still forwarding allowed items to the worker. A non-bulk
+denial adds one `constraint-denied` audit event (canonical `audit_event` table)
+with the key id, command kind, target, and blocking constraint. A bulk command
+adds **one aggregated** `constraint-denied` event per command, not one per
+denied item: its target is `<commandKind>:bulk` and its details carry the
+denied and total item counts, the distinct blocking constraint names, a sample
+of at most 20 denied targets (`sampleTruncated` says whether more were
+denied), and the client correlation id. The audit writer is a bounded channel
+that drops on full, so per-item events let a single low-privilege key flush
+every other pending audit record with one large bulk command. Secured values
+and raw credentials are never logged.
 
 ## Scope Catalog
 

docs/Contracts.md

@@ -49,6 +49,17 @@ worker prepares the replacement on its STA before swapping it into the existing
 failover consumer and returns a normal `MxCommandReply`; callers must treat only
 `protocol_status = OK` as acknowledgement of the new list.
 
+`AcknowledgeAlarmRequest` carries the operator's identity: `operator_user`
+(required), `operator_password` (field 6) and `operator_domain` (field 7). The
+gateway copies them onto `AcknowledgeAlarmCommand` (`operator_password` = 7) and
+`AcknowledgeAlarmByNameCommand` (`operator_password` = 9); the worker
+authenticates the operator and writes `AckMsg` as that user. The fields are
+additive, so `GatewayProtocolVersion` stays `3`. `operator_password` is a
+credential and follows the same never-log rule as
+`AuthenticateUserCommand.verify_user_password`: it must not reach logs, metric
+labels, `DiagnosticMessage`, `ProtocolStatus.Message`, or exception text. A
+not-found or not-ackable alarm is reported with `hresult` -1.
+
 The command model also includes bulk write/read command kinds:
 `WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
 `ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`

docs/DesignDecisions.md

@@ -430,6 +430,44 @@ Suggested mapping:
 - Galaxy SQL metadata endpoints if added: `metadata:read`
 - worker shutdown diagnostics and key management: `admin`
 
+## Metrics Endpoint Authentication (SEC-14)
+
+Decision: `/metrics` stays anonymous by default and can be put behind
+authentication with `MxGateway:Metrics:RequireAuthentication=true`; `/health`
+stays anonymous unconditionally.
+
+- **Opt-in rather than on by default.** Existing Prometheus scrapers on the
+  deployed hosts scrape anonymously; flipping the default would silently break
+  them on upgrade. Hosts whose network exposure makes the counters sensitive
+  enable it explicitly.
+- **Health is never gated.** The family Overview dashboard (another repo) polls
+  `/health/active` without credentials, and liveness/readiness probes are
+  conventionally anonymous. Health responses carry status only.
+- **Credentials reuse existing plumbing.** One `MxGateway.Metrics` scheme
+  accepts an API key with the `admin` scope (verified through the same cached
+  `IApiKeyVerifier` and the same per-peer failure limiter as gRPC, so `/metrics`
+  is not a way around the gRPC brute-force throttle) or, when no authorization
+  header is sent, the dashboard login cookie with a real dashboard role. No new
+  credential type or store is introduced.
+- **`admin` scope, any dashboard role.** Metrics are an operator surface, so a
+  data-plane client key (`invoke:*`, `events:read`) is not enough. A dashboard
+  `Viewer` qualifies because every dashboard role already sees the same aggregate
+  counters on the dashboard; the exported metrics carry no session ids or tag
+  values.
+- **No environment bypass.** The requirement reads no options, so anonymous
+  localhost and `Authentication:Mode=Disabled` — both of which satisfy the
+  dashboard Viewer policy — do not satisfy it. Nor does dev-only
+  `Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
+  name, so the metrics scheme skips the cookie fallback while it is on rather than
+  letting a dev flag silently open an endpoint the operator asked to protect.
+- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
+  the remote IP, while the gRPC one is the gRPC peer string (per connection), so
+  only the per-key-id aggregate is shared between the two paths.
+- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
+  would challenge an anonymous scrape with a 302 redirect to `/login`; a single
+  scheme that falls back to the cookie internally keeps the challenge a `401`
+  with `WWW-Authenticate: Bearer`, which scrapers understand.
+
 ## Worker Process Identity
 
 Decision: run workers as the gateway service identity for v1.
@@ -703,14 +741,15 @@ latch in subtag mode.
 
 ### Acknowledge via ack-comment write
 
-In subtag mode, `AcknowledgeAlarm` writes the operator comment to the alarm
-attribute's ack-comment subtag (`Fallback:Subtags:AckComment`). The write
-performs the native ack in AVEVA. This differs from alarmmgr mode, where
-`AlarmAckByName` on `wwAlarmConsumerClass` is called directly. The `AckComment`
-subtag name is empty by default; configuring it is required for ack to work in
-subtag mode. The exact AVEVA subtag names are not hard-coded — the `Subtags`
-config block exists precisely so names are not guessed without validation
-against the live MXAccess attribute set.
+Both provider modes acknowledge by writing the operator comment to `AckMsg` as
+the authenticated operator (see "Alarm acknowledgement writes AckMsg as the
+authenticated operator" below). In alarm-manager mode the target is
+`<TagName>.AckMsg`; in subtag mode it is the watch-list entry's ack-comment
+subtag (`Fallback:Subtags:AckComment`, default `AckMsg`). The write performs
+the ack in AVEVA. An entry with an empty ack-comment subtag is not ackable and
+the RPC reports `InvalidRequest`. The exact AVEVA subtag names are not
+hard-coded — the `Subtags` config block exists so names are not guessed
+without validation against the live MXAccess attribute set.
 
 ### Related documentation
 
@@ -784,6 +823,47 @@ The following items were previously listed here and have since shipped:
   `MxGateway:Sessions:MaxEventSubscribersPerSession`.
   See [Event Subscribers](#event-subscribers) above and [Sessions](./Sessions.md).
 
+## Alarm acknowledgement writes AckMsg as the authenticated operator
+
+**Decision (2026-10-19).** `AcknowledgeAlarm` authenticates the requesting
+operator with `AuthenticateUser` (`operator_user`, `operator_password`,
+optional `operator_domain`) and writes the comment to `<TagName>.AckMsg` over
+MXAccess as that user, on a dedicated `LMXProxyServer` registered as
+`MxGateway:Alarms:AckClientName`. Both provider modes use this path.
+
+**Why.** Live tests (2026-10-18/19) showed every alarm-client ack is inert on a
+headless consumer: wnwrap `AlarmAckByName`/`AlarmAckByGUID`/`AlarmAck`/
+`AlarmAckSelected`, and the managed `aaAlarmManagedClient` GUID ack, return
+rc=0 or -1 and change nothing, regardless of security classification,
+operator name or consumer name. An MXAccess write to `AckMsg` does acknowledge
+the alarm, and when the write is made under an authenticated user the AVEVA
+runtime historizes that identity (`Operator`, `DomainName`, `UserFullName`),
+with the MXAccess client name as `OperatorNode`. The requirement is that the
+real operator appears in the Historian, as with InTouch, so a shared gateway
+service account was rejected. Operators have Galaxy accounts, so the
+per-request credential is workable. See
+[Alarm Probe Findings](./AlarmProbeFindings.md).
+
+**Consequences.**
+
+- `operator_user` is required; `operator_password` may be empty.
+- `operator_password` is a credential. It is forwarded to the worker over the
+  pipe but never logged: `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are
+  credential-bearing commands and any property named like `password` is
+  redacted.
+- `Applied` and `Unconfirmed` (write sent, no `OnWriteComplete` in the bounded
+  wait) both report `OK`; clients that need certainty watch for the transition.
+- The managed ack consumer, the WNAL native ack consumer, the second wnwrap
+  ack-only consumer and the opt-in managed-ack environment flag are removed.
+
+**Upgrade note.** This is a breaking change for callers. They must now send
+`operator_user` (required; empty is rejected with `InvalidArgument`) and the
+operator's Galaxy password; a caller that sends no password for a
+password-protected account gets `MXACCESS_FAILURE` from `AuthenticateUser`
+instead of an acknowledgement. OtOpcUa's `GalaxyDriver` alarm acknowledger is a
+known caller that must be updated to forward the operator's credentials before
+the gateway is upgraded.
+
 ## Related Documentation
 
 - [Gateway Process Detailed Design](./GatewayProcessDesign.md)

docs/Diagnostics.md

@@ -4,7 +4,7 @@ The diagnostics subsystem provides structured logging, credential redaction, and
 
 ## Goals
 
-The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
+The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, `AcknowledgeAlarm` (its `operator_password`), or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
 
 A second goal is parity-test diagnosability. Because MXAccess sessions, workers, correlation ids, and command methods are the units of comparison, every log entry produced inside a request scope must carry those identifiers without each call site having to format them.
 
@@ -70,7 +70,9 @@ private static readonly HashSet<string> SensitiveCommandMethods = new(StringComp
 {
     "AuthenticateUser",
     "WriteSecured",
-    "WriteSecured2"
+    "WriteSecured2",
+    "AcknowledgeAlarm",
+    "AcknowledgeAlarmByName"
 };
 
 public static bool IsCredentialBearingCommand(string? commandMethod)
@@ -80,7 +82,7 @@ public static bool IsCredentialBearingCommand(string? commandMethod)
 }
 ```
 
-The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below.
+The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below. `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are in the set because their commands carry the operator's `operator_password`. Independently of the method set, the name-based seam redacts any property whose name contains `password`, so a credential field added to a payload later is redacted without a code change.
 
 ### API key redaction
 

docs/GatewayConfiguration.md

@@ -38,6 +38,7 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
       "LeaseSweepIntervalSeconds": 30,
       "DetachGraceSeconds": 30,
       "FaultedGraceSeconds": 0,
+      "UnattachedSessionGraceSeconds": 0,
       "AllowMultipleEventSubscribers": false,
       "MaxEventSubscribersPerSession": 8,
       "WorkerReadyWaitTimeoutMs": 0
@@ -153,7 +154,8 @@ validation fails fast when either half of the headroom rule is violated.
 | `MxGateway:Sessions:LeaseSweepIntervalSeconds` | `30` | Hosted monitor interval for closing expired leases. Active event-stream subscribers keep a session from expiring while the stream remains attached. |
 | `MxGateway:Sessions:DetachGraceSeconds` | `30` | Detach-grace retention window. When positive, a session whose last external (gRPC) event-stream subscriber drops is retained in `Ready` for this many seconds so a client can reconnect; if no external subscriber re-attaches within the window, the lease monitor closes it with `detach-grace-expired`. The internal dashboard mirror does not count as an external subscriber, so a dashboard-only session still enters detach-grace. `0` disables retention and reverts to closing only on normal lease expiry. Must be zero or greater. Reconnect/replay itself is implemented separately (Task 12); this option controls retention and expiry only. The effective close happens within the next sweep cycle after the window elapses — up to `LeaseSweepIntervalSeconds` after expiry. Operators wanting a firm minimum retention bound should set `DetachGraceSeconds` greater than `LeaseSweepIntervalSeconds`. |
 | `MxGateway:Sessions:FaultedGraceSeconds` | `0` | Grace window before the lease monitor reaps a faulted session (killing its worker and freeing the slot). A faulted session is permanently unusable yet otherwise pins a session slot and a live x86 worker until its normal lease expires. `0` (the default) reaps it on the next sweep cycle, bounding blast radius; a positive value keeps the faulted session observable via `GetSessionStatus` for that window before it is reclaimed, with the close reason `faulted-reaped`. Must be zero or greater. |
-| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `AlreadyExists`. Set to `true` to enable fan-out via the `SessionEventDistributor`. |
+| `MxGateway:Sessions:UnattachedSessionGraceSeconds` | `0` | Idle window after which the lease monitor reaps a `Ready` session that has **never** attached a `StreamEvents` subscriber, with the close reason `unattached-grace-expired`. Exists because a client that opens a session and vanishes before it ever streams — OtOpcUa's `GalaxyDriver` failing part-way through start-up, for example — otherwise pins a session slot and a live x86 worker for the full `DefaultLeaseSeconds` (30 minutes). The window is measured from the later of the session becoming `Ready` and its last unary command, so a slow worker start does not eat it and a unary-only client that keeps calling is never reaped by this rule; it stops applying once the session's first subscriber attaches (`DetachGraceSeconds` and the normal lease govern it from then on), and the gateway's own alarm-monitor session, whose subscriber is internal, is exempt the same way. The dashboard Browse live-data session (`DashboardLiveDataService`) is not exempt: it is unary-only, so it is reaped once idle for the window, and it opens a new session through `EnsureReadyAsync` on its next use. `0` (the default) disables it. Must be zero or greater. Set it comfortably above the time a well-behaved client needs between `OpenSession` and its first `StreamEvents`; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. |
+| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `ResourceExhausted` (`EventSubscriberAlreadyActive`). Set to `true` to enable fan-out via the `SessionEventDistributor`. |
 | `MxGateway:Sessions:MaxEventSubscribersPerSession` | `8` | Maximum number of concurrent `StreamEvents` subscribers per session when `AllowMultipleEventSubscribers` is `true`. Effectively 1 when `AllowMultipleEventSubscribers` is `false`. Must be greater than zero. |
 | `MxGateway:Sessions:WorkerReadyWaitTimeoutMs` | `0` | Bounded time, in milliseconds, the gateway will wait for a worker to reach `Ready` when the session is already `Ready` but the worker state has transiently diverged (e.g. `Handshaking` after a heartbeat blip). Applies only to transient worker states; terminal states (`Faulted`/`Closing`/`Closed`/no worker) fail fast immediately regardless of this setting. `0` (the default) disables the wait and preserves the original fail-fast behavior. Must be greater than or equal to zero. |
 
@@ -419,6 +421,45 @@ model requires otherwise.
 | `MxGateway:Security:AuditRetentionDays` | `90` | Days of canonical audit history kept in the `audit_event` table. The audit drain sweeps once at startup and hourly thereafter, deleting rows older than this window; without it the table grows without bound inside the same SQLite file the authentication hot path reads. Rows whose timestamp SQLite cannot parse are never swept. Must be greater than zero — retention can be widened but not switched off. |
 | `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
 
+## Metrics Options
+
+Bound from `MxGateway:Metrics`. Governs the Prometheus `/metrics` scrape endpoint
+(mounted by the shared `ZB.MOM.WW.Telemetry` package). The health endpoints
+(`/healthz`, `/health/ready`, `/health/active`) are not affected by this section and
+always stay anonymous: the family Overview dashboard polls `/health/active` from
+another repo without credentials, so gating it would break that consumer.
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access, `Authentication:Mode=Disabled`, and `Dashboard:DisableLogin` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
+
+Configure a Prometheus scraper against an authenticated endpoint with a dedicated
+admin-scoped key, for example:
+
+```yaml
+scrape_configs:
+  - job_name: mxgateway
+    authorization:
+      type: Bearer
+      credentials_file: /etc/prometheus/mxgateway.key   # contains mxgw_<keyId>_<secret>
+    static_configs:
+      - targets: ["gateway-host:5130"]
+```
+
+The setting is read from configuration when endpoints are mapped, so changing it
+requires a restart. Dev-only `Dashboard:DisableLogin=true` registers an
+always-succeeding auto-login handler under the dashboard cookie scheme; the
+metrics scheme deliberately skips its cookie fallback while that flag is on, so
+`DisableLogin` never silently opens a `/metrics` endpoint that
+`RequireAuthentication` protects — only an admin-scoped API key is accepted then.
+
+The `/metrics` path reuses the gRPC API-key failure limiter
+(`MxGateway:Security:ApiKeyFailure*`), but its per-peer partition is keyed on the
+remote **IP address**, whereas the gRPC interceptor's is the gRPC peer string
+(address and port — effectively per connection). The two paths therefore share
+only the per-key-id aggregate layer (`ApiKeyFailureAggregateLimit`); per-peer
+failure counts on one path do not throttle the other.
+
 ## Galaxy Options
 
 | Option | Default | Description |
@@ -442,10 +483,19 @@ behavior.
 | `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor re-resolves and, on material change, hot-swaps the subtag fallback watch-list, then reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot. The acknowledged watch-list fingerprint advances only after an OK worker reply. Floored at 5 seconds. |
 | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
 | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
+| `MxGateway:Alarms:AckClientName` | `MxGateway` | MXAccess client name the alarm worker's ack writer registers with; the Historian records it as each acknowledgement's OperatorNode. 1-64 printable characters. Conveyed via `MXGATEWAY_ALARM_ACK_CLIENT_NAME`. |
 
 The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
 `StreamAlarms` are session-less RPCs served by the monitor.
 
+`AckClientName` is the MXAccess client name the alarm worker's ack writer
+registers its dedicated `LMXProxyServer` with. The Historian records it as the
+OperatorNode of each acknowledgement, so it identifies the gateway as the
+acknowledging node. The gateway passes it to each worker through the
+`MXGATEWAY_ALARM_ACK_CLIENT_NAME` environment variable. Startup validation
+rejects an empty value, a value over 64 characters, or one containing control
+characters.
+
 ### Alarm fallback options
 
 The `Fallback` sub-section controls how the alarm feed selects between the
@@ -463,7 +513,7 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback.
 | `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. |
 | `MxGateway:Alarms:Fallback:Subtags:Active` | `InAlarm` | Subtag name for the in-alarm boolean. Confirmed AVEVA `AlarmExtension` field name. |
 | `MxGateway:Alarms:Fallback:Subtags:Acked` | `Acked` | Subtag name for the acknowledged boolean. Confirmed AVEVA `AlarmExtension` field name. |
-| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the ack-comment write path is disabled. |
+| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag, as the authenticated operator, performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the alarm is not ackable and the RPC returns `InvalidRequest`. |
 | `MxGateway:Alarms:Fallback:Subtags:Priority` | `Priority` | Subtag name for the alarm priority / severity value. Confirmed AVEVA `AlarmExtension` field name. |
 
 Validation rules:

docs/GatewayDashboardDesign.md

@@ -273,9 +273,14 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
 
 | Hub | Path | Producer | Payload | Routing |
 |---|---|---|---|---|
-| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
-| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
-| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
+| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `SnapshotUpdated`: `DashboardSnapshot` **without** `configuration`; `ConfigurationUpdated`: `EffectiveGatewayConfiguration` | Ticks go to all connected clients on every snapshot interval, but only while at least one client is connected (see "Idle gating" below). The effective configuration is startup-static and the largest part of the record, so it is `[JsonIgnore]`d on `DashboardSnapshot` and sent once per connection instead: `OnConnectedAsync` pushes `ConfigurationUpdated` and then the current `SnapshotUpdated`, and the hub method `GetConfiguration()` returns it on demand. Each tick broadcast is bounded to 1 s; a send that misses the bound is abandoned, so one slow client cannot delay the next tick for everyone. |
+| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. Every subscription to the monitor — the first and each resubscribe — is preceded by an `AlarmFeedReset` frame (`{ generation }`, monotonic per process): the monitor primes each new subscription with a full `active_alarm` snapshot ending in `snapshot_complete`, so a client must discard its accumulated alarm state on the marker and rebuild from that snapshot rather than double-counting it. Each broadcast is bounded to 2 s and abandoned on timeout: awaiting it unbounded let one slow client stall the drain until the monitor dropped the publisher as a slow subscriber, and the resubscribe then replayed the whole snapshot to every client. The tradeoff: a client whose send is abandoned misses that one frame and receives no reset for it, so its alarm view can be stale until the next resubscribe (or until SignalR drops the stalled connection); there is no in-family consumer of `/hubs/alarms` today. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
+| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry`. Delivery to the group goes through one bounded per-session pump (see "Mirror gating"), so a slow viewer loses its oldest events instead of accumulating pending sends — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
+
+No hub client ships in this repository: the server-rendered pages read the
+in-process seams above, and the hubs serve external consumers. Those consumers
+must honour `AlarmFeedReset` and keep the configuration from
+`ConfigurationUpdated`, since ticks no longer carry it.
 
 ### Default cadences
 
@@ -343,12 +348,17 @@ subscribers. Two races motivate the extra state:
   gate open, and cancelling on their behalf would stop a live pump that other
   viewers depend on.
 
-Two per-tick costs inside the snapshot itself are bounded independently of either gate:
+Three per-tick costs inside the snapshot itself are bounded independently of either gate:
 
 - the effective configuration (`EffectiveGatewayConfiguration`) is built once and
   cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
   at startup and never reloads, so rebuilding the whole option tree every tick
-  produced an identical object;
+  produced an identical object. For the same reason it is not serialized on the
+  `/hubs/snapshot` tick (see the hub table above);
+- the gateway status is one `HealthCheckService.CheckHealthAsync` evaluation per
+  tick, bounded to 2 s and behind a zero-wait gate so concurrent enumerations (the
+  hub publisher and the page feed) do not both evaluate; `GetSnapshot()` reads the
+  cached result;
 - the API key summaries are refreshed at most once every 15 seconds
   (`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
   read whose content changes only when an operator creates, rotates, or revokes a
@@ -359,7 +369,7 @@ Two per-tick costs inside the snapshot itself are bounded independently of eithe
 Avoid pushing every MXAccess data-change event into a wider broadcast group.
 Events are routed strictly per session (`session:{id}` groups on the hub,
 per-session subscriber lists in process); the snapshot seams continue to carry
-aggregate event counters and rates.
+aggregate event counters.
 
 ### Mirror gating
 
@@ -394,10 +404,21 @@ group. With `ShowTagValues` true there is no clone at all — the original `MxEv
 instance is handed to both audiences — so the "one clone per event" cost holds only
 in the redacting configuration, and in the value-showing one both audiences share a
 reference to the session pipeline's own event object. In-process delivery runs first
-and synchronously — it cannot
-throw, and it must not be skipped by the guard clause around the hub send — into
-per-subscriber bounded drop-oldest channels, so a page that falls behind loses its
-oldest queued events rather than blocking the session's event pipeline.
+and synchronously — it cannot throw — into per-subscriber bounded drop-oldest
+channels, so a page that falls behind loses its oldest queued events rather than
+blocking the session's event pipeline.
+
+Hub delivery has the same shape. `EventsHubViewerRegistry` counts remote hub
+connections separately from `inproc-` subscribers (`HasHubViewers`), and while a
+session has at least one, `Publish` writes the event into that session's
+`HubPump`: a 256-deep drop-oldest channel drained by a single task that sends one
+event at a time to `session:{id}`, each send bounded to 5 s. A session watched only
+in process starts no pump. The pump starts on the first event published while a
+hub viewer is present and is torn down when the registry raises
+`HubViewersReleased` for the session's last hub viewer (unsubscribe or disconnect).
+Before this, `Publish` fired a group `SendAsync` per event and returned; a slow
+remote viewer made each send stay pending in the SignalR lifetime manager, and the
+pending sends — each holding a cloned event — grew without bound.
 
 The mirror subscriber itself is still registered on the `SessionEventDistributor`
 for the session's whole lifetime; only the per-event work is gated. Starting and
@@ -411,55 +432,67 @@ clone and send this gate already removes.
 
 ### Dashboard home
 
-Show top-level status:
+The Overview page (`/`) shows, from the snapshot:
 
-- gateway status,
-- gateway version,
-- uptime,
-- open sessions,
-- workers running,
-- sessions faulted,
-- command rate,
-- command failure count,
-- event rate,
-- event queue depth,
-- worker restart/kill count.
+- gateway status badge — the aggregate status of the registered ASP.NET Core
+  health checks (`Healthy` / `Degraded` / `Unhealthy`), evaluated once per snapshot
+  tick with a 2 s bound and cached for the synchronous `GetSnapshot()` seed path. It
+  reads `Unknown` before the first evaluation, when an evaluation throws or times
+  out, and when no `HealthCheckService` is registered — never a hardcoded `Healthy`
+  (SEC-27);
+- the snapshot's generation time;
+- metric cards: uptime (with the gateway version as detail), open sessions, workers
+  running, worker event queue depth, failed commands, events received, faults, and
+  queue overflows — all cumulative counters or point-in-time gauges, not rates;
+- a Galaxy Repository section: cache status, last deploy time, object count (with
+  area count), attribute, historized-attribute and alarm-attribute counts, and the
+  last refresh time;
+- the recent faults list (`Dashboard:RecentFaultLimit`).
 
-Use Bootstrap cards for individual metric summaries. Keep the layout compact
-and operational.
+There is no per-family command or event rate, no faulted-session counter, and no
+worker restart/kill card on this page; `mxgateway.workers.killed` and
+`mxgateway.workers.exited` are in the snapshot's metric list but not rendered.
+Metric summaries use Bootstrap cards; the layout stays compact and operational.
 
 ### Sessions page
 
-Show active and recent sessions in a table:
+Shows the most recent sessions (`Dashboard:RecentSessionLimit`, newest first) in a
+table with these columns:
 
-- session id,
-- client identity or API key display name,
-- state,
+- session id (links to the session details page),
+- session state,
+- client identity (redacted),
 - backend,
-- worker process id,
+- worker process id and worker state,
+- events received for the session (the `mxgateway.events.received` per-session
+  counter),
 - open time,
 - last client activity,
 - last worker heartbeat,
-- active event subscribers,
-- pending commands,
-- event queue depth,
-- last fault summary.
+- last fault summary (abbreviated, full text on hover),
+- Close / Kill actions for Admin principals.
 
-Rows should link to session details.
+Event subscriber counts, pending command counts, and per-session queue depth are
+not tracked per session and are not shown.
 
 ### Session details page
 
-Show:
+Shows:
 
-- session metadata,
-- worker metadata,
-- command counters by method,
-- event counters by family,
-- active server handles and item counts if gateway shadow state has them,
-- latest faults,
-- last heartbeat payload,
+- a session section: state badge, backend, client identity, client session name,
+  client correlation id (all redacted), open time, last activity, lease expiry,
+  events received, and last fault;
+- a worker section: process id, worker state, and last heartbeat time;
+- a "Recent events" table — the newest 50 events from the in-process mirror, with
+  worker sequence, family, server handle, item handle, and status (no values), plus
+  a live/offline pill; gated by the session ACL, so a caller who may not observe the
+  session sees "Not authorized" instead;
 - admin Close session / Kill worker controls (Admin role only).
 
+Per-method command counters, per-family event counters for the session, server
+handle / item counts, and the heartbeat payload are not shown — the gateway keeps
+no per-session shadow state for them.
+
 The Sessions list, the Workers list, and this details page all render the same
 admin controls when the signed-in principal carries the `Admin` role; viewers
 and the localhost-anonymous bypass see no action affordances and the server
@@ -484,33 +517,31 @@ management audit path.
 
 ### Workers page
 
-Show:
+Shows one row per worker that is not closed, with:
 
 - worker process id,
-- session id,
-- executable path/version,
-- state,
-- startup duration,
-- memory and CPU if available,
-- last heartbeat,
-- current command correlation id,
-- pending command count,
-- event queue depth,
-- restart/kill reason if terminal.
+- worker state,
+- owning session id (links to the session details page),
+- last heartbeat time,
+- last fault summary (the owning session's final fault),
+- a Kill worker action for Admin principals.
+
+Executable path and version, startup duration, memory/CPU, in-flight command
+correlation, pending command count, and per-worker queue depth are not collected
+and are not shown; the worker executable path is on the Settings page.
 
 ### Events page
 
-Show aggregate event diagnostics:
+Shows aggregate event diagnostics from the snapshot:
 
-- event rate by session,
-- event rate by event family,
-- total events since start,
+- events received since start,
+- worker event queue depth and gRPC stream queue depth (gauges),
 - queue overflow count,
-- stream disconnect count,
-- recent terminal faults.
+- gRPC stream disconnect count,
+- an "Event Families" table: cumulative events received per event family.
 
-Do not display full tag values by default. If value display is later added, make
-it opt-in and redacted.
+The figures are counters since start, not rates; per-session event counts are on
+the Sessions page. No tag values are shown here.
 
 ### Browse page
 
@@ -537,6 +568,33 @@ at the service, not in the page: one decision point, and the value never enters
 the render tree. Set the flag true to make the panel the explicit opt-in
 tag-value surface it was originally designed as.
 
+The filter box is debounced: keystrokes within 150 ms coalesce into one search,
+which runs on the thread pool with a cancellation token that the next keystroke
+cancels, and only the latest query may publish its results. The search reads
+`DashboardBrowseSearchIndex`, a singleton holding every attribute's full reference
+and name lowercased once, keyed on the Galaxy cache `Sequence` and rebuilt only when
+a deploy moves it. The previous filter rescanned the whole cached Galaxy with
+case-insensitive comparisons, synchronously on the render dispatcher, once per
+keystroke per circuit.
+
+The page's two background loops — the 2 s live-read poll and the Galaxy
+deploy-notification subscription — handle faults inside the loop and retry; only
+cancellation ends them. A poll fault surfaces in the "Live read failed" banner; a
+notifier fault surfaces as a "redeploy notifications failed (retrying)" warning and
+resubscribes after 5 s; the tree rebuild after a redeploy runs on the renderer
+dispatcher. `DisposeAsync` cancels the loops and any in-flight search, drains them
+concurrently with the same 5 s bound `DashboardPageBase` uses, and disposes its
+cancellation sources in a `finally`.
+
+A closed or backgrounded tab leaves its circuit alive, disconnected, for
+`DisconnectedCircuitRetentionPeriod`, which the dashboard sets to **30 seconds**
+(`DashboardServiceCollectionExtensions.DisconnectedCircuitRetention`) instead of
+the framework's three minutes. Within that window the poll does not read: a scoped
+`DashboardCircuitConnectionTracker`, registered as the circuit's `CircuitHandler`,
+reports the browser connection down and the poll skips its ticks until it comes
+back. Every tick it does run is a `ReadBulk` against the shared dashboard worker,
+so a closed tab previously kept the worker busy for three minutes.
+
 ### Alarms page
 
 `/dashboard/alarms` lists the alarms the gateway's central alarm monitor
@@ -552,7 +610,10 @@ client, no loopback socket, and no hub token — and the same subscription carri
 `snapshot_status` frame behind the truncated-snapshot banner ("Alarm snapshot may be
 incomplete"), so a capped provider fetch is caveated the moment the monitor decides
 it. The alarm rows themselves still come from the three-second poll, which also
-re-asserts the truncation verdict as the reconcile baseline. If `MxGateway:Alarms:Enabled` is
+re-asserts the truncation verdict as the reconcile baseline. The filtered rows and
+the area list are memoized in `DashboardAlarmView` on the poll's alarm generation and
+the filter inputs, so the "Showing" card and the table share one filter-and-sort per
+change rather than recomputing it several times per render. If `MxGateway:Alarms:Enabled` is
 false the central monitor never starts, and the page says so instead of showing
 an empty list with no explanation.
 
@@ -599,6 +660,19 @@ overflow against the oversized set and evicts the whole excess in one pass (a
 subscribes nothing new evicts nothing, but neither can it grow the set. A browse
 page requests far fewer tags than the cap, so in practice the set settles at 256.
 
+The 256 is shared between viewers rather than one global LRU. Each Browse page
+reads under its own viewer id, every advised tag records the viewer that read it
+last, and each active viewer (one that read within the last 30 s) owns a share of
+`256 / active viewers`, floored at 32. When a read needs room, eviction first takes
+the least-recently-read tags of *other* viewers that are over their share — an
+inactive viewer's share is zero, so a closed page's tags go first — and then the
+reader's own tags that this read did not touch. A viewer within its share is never
+evicted by someone else's read, so with several viewers the cap is soft: the set
+can sit above 256 rather than strip a neighbour. Under the global LRU, two circuits
+browsing different areas evicted each other's whole panels on every poll and
+re-advised them two seconds later. With a single viewer the behaviour is exactly
+the global LRU described above.
+
 The Alarms page does **not** use the dashboard session: alarm data comes from
 the gateway's always-on central monitor. `QueryAlarmsAsync` reads
 `IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the
@@ -665,20 +739,24 @@ Secrets and pepper values are never logged.
 
 ### Settings page
 
-Show read-only effective configuration:
+Shows the read-only effective configuration:
 
-- worker executable path,
-- configured timeouts,
-- queue capacities,
-- auth mode,
-- SQLite auth database path with sensitive parts redacted if needed,
-- dashboard enabled state,
+- authentication: mode, SQLite auth database path, pepper secret name, and whether
+  migrations run on startup;
+- LDAP: enabled, server and port, fallback servers, transport, search base,
+  service account DN, service password (redacted), and the username and group
+  attributes;
 - the LDAP group mappings that decide what a signed-in user may see —
   `Dashboard:GroupToRole` (group → `Administrator` / `Viewer`) and
-  `Dashboard:GroupToTag` (group → the session tags it grants),
-- `Dashboard:UntaggedSessionVisibility`, which decides who sees a session whose
-  owning API key carries no dashboard tags,
-- protocol version.
+  `Dashboard:GroupToTag` (group → the session tags it grants);
+- worker: executable path, required architecture, and the startup, shutdown, and
+  heartbeat-grace timeouts;
+- sessions: default command timeout and max sessions;
+- events: queue capacity and backpressure policy;
+- dashboard: enabled, anonymous localhost, snapshot interval, recent fault and
+  session limits, `ShowTagValues`, and `Dashboard:UntaggedSessionVisibility`
+  (who sees a session whose owning API key carries no dashboard tags);
+- the worker protocol version.
 
 Do not show API key secrets or pepper values.
 

docs/GatewayProcessDesign.md

@@ -177,9 +177,17 @@ return a successful close result with the final known state.
 
 `WorkerClient.ShutdownAsync` sends `WorkerShutdown`, waits for the worker read,
 write, and heartbeat loops to stop, and waits for the launched worker process to
-exit within the same shutdown timeout. If the pipe loops or process exit exceed
-the timeout, the close operation fails with `ShutdownTimeout`; `GatewaySession`
-then kills the worker process tree before surfacing the close failure.
+exit within the same shutdown timeout. The timeout starts before the
+`WorkerShutdown` envelope is queued, not after: the outbound channel is bounded,
+and a worker that has stopped reading the pipe parks the write loop in a pipe
+write while the channel behind it fills with commands and their cancels. Because
+the heartbeat watchdog only watches a `Ready` client, nothing else would end that
+stall once the client is `Closing`, so an enqueue bound to the caller's token
+alone waited for that token — for the lease sweep, until host stop — with the
+session's close lock held. If queuing the envelope, the pipe loops, or process
+exit exceed the timeout, the close operation fails with `ShutdownTimeout` (the
+fault cancels the loops and kills the worker); `GatewaySession` then runs its
+kill fallback before surfacing the close failure.
 
 ### Invoke
 

docs/Grpc.md

@@ -82,6 +82,8 @@ return mapper.MapCommandReply(workerReply);
 
 Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
 
+Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies. To keep the two ids joinable, `Invoke` logs the pair (command kind, session id, client correlation id, worker envelope correlation id — no values) once per call at `Debug`, so a client error report quoting its correlation id can be matched to gateway and worker log lines.
+
 An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
 
 Selected control commands have additional public behavior beyond their protobuf shape:
@@ -94,16 +96,42 @@ Selected control commands have additional public behavior beyond their protobuf
 
 `StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
 
+Each response write receives the call's cancellation token, so a write parked on a slow or vanished client is released when the call is cancelled rather than waiting on the transport. A write that then fails with an `IOException` or `OperationCanceledException` while the call token is cancelled is a routine client disconnect: it maps to `Cancelled` and logs nothing. The same `IOException` on a live call keeps the generic `Unavailable` + warning mapping, because that is a real transport failure. `StreamAlarms` applies the same rule.
+
 ### `AcknowledgeAlarm`
 
-`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
+`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
 
-An `OK` response means the alarm provider accepted the acknowledgement, not that it applied it: the gateway forwards the ack and reports what the provider returned, in keeping with the parity rule. On galaxies whose alarmed attributes carry a non-free-access security classification the by-name ack is accepted and inert — the underlying `AlarmAckByName` conveys an operator name rather than an authenticated identity, observed on the probe rig with the mechanism inferred rather than confirmed — so a client that needs to confirm an acknowledgement should watch for the resulting transition on `StreamAlarms` instead of treating `OK` as proof. See [Alarm Probe Findings](./AlarmProbeFindings.md).
+Request fields that identify the acknowledging operator:
+
+- `operator_user` (required): the Galaxy account name. An empty value is rejected with `InvalidArgument` so the real operator is always recorded.
+- `operator_password` (field 6): the operator's Galaxy password. May be empty because Galaxy accounts can have empty passwords. It is a credential and is never logged.
+- `operator_domain` (field 7): optional. When set, the worker authenticates `domain\user`; when empty it authenticates `user`.
+- `operator_full_name`: not a field on `AcknowledgeAlarmRequest`. The worker commands (`AcknowledgeAlarmCommand` field 6, `AcknowledgeAlarmByNameCommand` field 8) carry one, but it is accepted and **not propagated** to MXAccess: the 8-argument `AlarmAckByName` that carried it returns -55 (see [Alarm Client Discovery](./AlarmClientDiscovery.md)). `operator_domain` is used.
+
+The worker calls `AuthenticateUser` as the operator, then writes `operator_comment` to `<TagName>.AckMsg` (subtag mode: the configured ack-comment subtag) on a dedicated MXAccess server registered as `MxGateway:Alarms:AckClientName`, and waits a bounded time for `OnWriteComplete`. The AVEVA runtime treats that write as the acknowledgement, and the Historian (`Runtime.dbo.v_AlarmEventHistory2`) records the operator name, domain and full name from the authenticated account, with the ack client name as `OperatorNode`. The alarm-client ack calls (`AlarmAckByName`, `AlarmAckByGUID`) are not used because a headless consumer's acks are accepted and dropped; see [Alarm Probe Findings](./AlarmProbeFindings.md).
+
+| Worker outcome | Response status | `hresult` |
+|---|---|---|
+| Write applied, or sent but no `OnWriteComplete` within the wait (unconfirmed, with a diagnostic) | `OK` | `0` |
+| Unknown alarm, or no ack target (non-Galaxy provider, no ack-comment subtag) | `INVALID_REQUEST` | `-1` |
+| `AuthenticateUser` raised an MXAccess error (bad user or wrong password surfaces as `E_INVALIDARG` `0x80070057`, raised by the interop as `ArgumentException`; `COMException` and `UnauthorizedAccessException` are classified the same way) | `MXACCESS_FAILURE` | the exception's HRESULT |
+| `AuthenticateUser` returned user id `0` | `MXACCESS_FAILURE` | `-1` |
+| `AuthenticateUser` failed with a server-gone HRESULT (`RPC_E_DISCONNECTED`, `RPC_S_SERVER_UNAVAILABLE`, `RPC_E_SERVERFAULT`); the writer tears down its MXAccess registration and re-registers on the next ack | `MXACCESS_FAILURE` | that HRESULT |
+| `AddItem`, `Advise` or `Write` on the `AckMsg` item raised an MXAccess error (`COMException`, or `ArgumentException` for `E_INVALIDARG`); the item is dropped and re-added on the next ack, and a server-gone HRESULT also tears down the registration | `MXACCESS_FAILURE` | the exception's HRESULT |
+| Write rejected (`OnWriteComplete` non-OK) | `MXACCESS_FAILURE` | the `detail` of the `OnWriteComplete` `MXSTATUS_PROXY`, or `-1` when `detail` is `0` |
+| Any other exception escaping the ack (e.g. the ack writer's MXAccess `Register` failing, the monitor's worker session having no active alarm subscription — `InvalidOperationException` — or an exception from the worker's own code rather than from an MXAccess call: interop failures are classified only when the `AuthenticateUser`, `AddItem`, `Advise` or `Write` call itself raises them, and an `ArgumentNullException` or `ArgumentOutOfRangeException` is never classified because the interop does not raise them for an HRESULT) | `MXACCESS_FAILURE` | the exception's `HResult`; the diagnostic names only the exception type and HRESULT |
+
+`reply.Statuses` carries the `OnWriteComplete` `MxStatusProxy` rows when they were observed. Because an unconfirmed write also returns `OK`, a client that must be certain should watch for the resulting transition on `StreamAlarms`. `AcknowledgeAlarm` stays under the `invoke:write` scope; see [Authorization](./Authorization.md).
+
+**Upgrade note.** Callers must now send `operator_user` (required) and the operator's Galaxy password in `operator_password`. A caller that sends no `operator_user` gets `InvalidArgument`; a caller that sends no password for a password-protected Galaxy account gets `MXACCESS_FAILURE` from `AuthenticateUser` instead of an acknowledgement. Every existing caller must be updated before the gateway is upgraded — known callers include OtOpcUa's `GalaxyDriver` alarm acknowledger, which must supply the operator's credentials.
 
 ### `StreamAlarms`
 
 `StreamAlarms` is a server-streaming, **session-less** RPC that attaches to the gateway's central alarm feed. The handler delegates to `IGatewayAlarmService.StreamAsync`. The stream opens with a `provider_status` and a `snapshot_status` `AlarmFeedMessage` (the current provider mode and snapshot-completeness verdict), then one `AlarmFeedMessage` carrying an `active_alarm` per currently-active alarm (the ConditionRefresh snapshot), then a single `snapshot_complete`, then a `transition` for every subsequent raise / acknowledge / clear — interleaved with a further `provider_status` on each failover/failback and a further `snapshot_status` on each change of the truncation verdict. It is served by the always-on `GatewayAlarmMonitor`, which owns a single gateway-managed worker session and fans out to every attached client — clients no longer open a session of their own. `alarm_filter_prefix`, when set, scopes the stream to a sub-tree.
 
+A subscriber's opening snapshot is taken from the monitor's memoized alarm-set projection (the same one `QueryActiveAlarms` and the dashboard read), captured under the broadcast lock in the same critical section that registers the subscriber, so no transition can fall between the snapshot and the live stream; the prefix filter runs afterwards, outside the lock. Each subscriber has a bounded queue (2048 messages). A subscriber that falls that far behind is dropped and its stream ends with `ResourceExhausted` ("Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."), the same status a `StreamEvents` overflow carries; the remedy is to reconnect, which re-sends the snapshot. The drop is not logged as a gateway failure.
+
 ### `QueryActiveAlarms`
 
 `QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `events:read` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
@@ -217,6 +245,7 @@ receive this event directly.
 | `StreamEvents` | `session_id` must be non-empty. | `InvalidArgument` |
 | `Invoke` | `session_id` non-empty, `command` present, `kind` not `Unspecified`, payload oneof must match `kind`. | `InvalidArgument` |
 | `AcknowledgeAlarm` | `alarm_full_reference` must be non-empty. Validated inline in the handler, not by `MxAccessGrpcRequestValidator`. | `InvalidArgument` |
+| `AcknowledgeAlarm` | `operator_user` must be non-empty. Validated inline in the handler. | `InvalidArgument` |
 | `StreamAlarms` | No required fields — `alarm_filter_prefix` is optional. | — |
 | `QueryActiveAlarms` | No required fields — `alarm_filter_prefix` is optional. Not routed through `MxAccessGrpcRequestValidator`; the handler only null-checks the request. | — |
 
@@ -272,6 +301,8 @@ Every handler wraps its body in `try { ... } catch (Exception exception) when (e
 - `SessionManagerException` is mapped by its `ErrorCode`.
 - `WorkerClientException` is mapped by its `ErrorCode`.
 
+Two stream-specific cases are handled before `MapException`: on `StreamEvents` and `StreamAlarms` a write failure after the client cancelled becomes `Cancelled` without a log entry, and on `StreamAlarms` an `AlarmFeedSubscriberOverflowException` (slow-subscriber drop) becomes `ResourceExhausted` without a log entry.
+
 Anything else is logged at warning and surfaced as `Unavailable` with a generic message — clients see a retryable status, and the unexpected exception is captured in gateway logs rather than leaked over the wire.
 
 ```csharp
@@ -280,74 +311,43 @@ StatusCode statusCode = exception.ErrorCode switch
     SessionManagerErrorCode.SessionNotFound => StatusCode.NotFound,
     SessionManagerErrorCode.SessionNotReady => StatusCode.FailedPrecondition,
     SessionManagerErrorCode.EventSubscriberAlreadyActive => StatusCode.ResourceExhausted,
+    SessionManagerErrorCode.EventSubscriberLimitReached => StatusCode.ResourceExhausted,
     SessionManagerErrorCode.EventQueueOverflow => StatusCode.ResourceExhausted,
     SessionManagerErrorCode.SessionLimitExceeded => StatusCode.ResourceExhausted,
     SessionManagerErrorCode.OpenFailed => StatusCode.Unavailable,
     SessionManagerErrorCode.CloseFailed => StatusCode.Unavailable,
+    SessionManagerErrorCode.PermissionDenied => StatusCode.PermissionDenied,
+    SessionManagerErrorCode.GatewayStopping => StatusCode.Unavailable,
     _ => StatusCode.Unavailable,
 };
 ```
 
 `WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same.
 
+While the gateway is stopping, clients see three different outcomes. An in-flight `Invoke` gets `Cancelled` (`GatewayShutdown`). An active `StreamEvents` ends cleanly with `OK`, because the drain closes its session and completes the subscriber channel. A new `OpenSession` gets `Unavailable` (`GatewayStopping`), including an open whose session the drain closed while its worker was still starting. An `OpenSession` the caller cancels while its worker is starting gets `Cancelled`, not `Unavailable`, and is not counted as an open fault.
+
 ## Event Streaming Model
 
-`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the channel, backpressure policy, and metric bookkeeping can be unit-tested without spinning up a Kestrel host.
+`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the attach, replay, filtering and metric bookkeeping can be unit-tested without spinning up a Kestrel host. It does not read the worker itself: each session owns one `SessionEventDistributor` whose single pump drains the worker event channel, maps each `WorkerEvent` to the public `MxEvent`, appends it to the session's replay ring, and fans it to every registered subscriber. A `StreamEvents` call is one of those subscribers. [Sessions](./Sessions.md) describes the distributor, the replay ring and detach-grace in full; this section covers the per-RPC boundary.
 
-The pipeline has three stages:
+The pipeline has four stages:
 
 1. The session is resolved via `sessionManager.TryGetSession`. A miss raises `SessionManagerException(SessionNotFound)` so the handler reports `StatusCode.NotFound`.
-2. `session.AttachEventSubscriber(...)` enforces the single-subscriber-per-session rule (or allows multiple subscribers if `Sessions:AllowMultipleEventSubscribers` is enabled). The returned `IDisposable` is released in the `finally` block, ensuring the subscriber slot is freed even when the client cancels mid-stream.
-3. A `Channel<MxEvent>` decouples the worker-side producer from the gRPC writer. The channel is bounded by `Events:QueueCapacity` and configured for a single reader and writer:
+2. The attach is owner-scoped: the caller's API key id must match the key that opened the session, or the call fails with `PermissionDenied` before anything is attached or delivered.
+3. The call registers a subscriber on the distributor. A fresh stream (`after_worker_sequence == 0`) uses `session.AttachEventSubscriber(...)`. A resume uses `session.AttachEventSubscriberWithReplay(...)`, which snapshots the replay ring for events newer than the cursor and registers the live subscriber under one lock, so the replay-to-live handoff has no gap and no duplicate. In single-subscriber mode (the default) a second attach fails with `EventSubscriberAlreadyActive`; with `Sessions:AllowMultipleEventSubscribers` enabled, an attach beyond `MaxEventSubscribersPerSession` fails with `EventSubscriberLimitReached`. Both map to `ResourceExhausted`. The lease is released in the `finally` block, so the subscriber slot is freed even when the client cancels mid-stream.
+4. The call yields events from the lease's bounded channel (capacity `Events:QueueCapacity`). On a resume it first yields a `ReplayGap` sentinel when the cursor predates the oldest retained event, then the replayed batch, then live events filtered to sequences above the last replayed one. Per-session ordering is preserved end-to-end, because the pump is the single writer to each subscriber channel and writes in `WorkerSequence` order.
 
-```csharp
-Channel<MxEvent> eventQueue = Channel.CreateBounded<MxEvent>(
-    new BoundedChannelOptions(options.Value.Events.QueueCapacity)
-    {
-        SingleReader = true,
-        SingleWriter = true,
-        FullMode = BoundedChannelFullMode.Wait,
-        AllowSynchronousContinuations = false,
-    });
-```
-
-The producer reads `WorkerEvent`s from the session, maps them, and writes to the channel. Per-session ordering is preserved end-to-end: events arrive from the worker in `WorkerSequence` order, the producer is the single writer, and the consumer drains FIFO. The producer also honours the request's `AfterWorkerSequence` cursor by skipping any event whose `WorkerSequence` is at or before the requested cutoff, which lets clients resume after a disconnect without server-side replay state.
+The replay ring is server-side state that outlives a single RPC: with the default `DetachGraceSeconds` and `ReplayRetentionSeconds`, a client that drops can reconnect with its last `worker_sequence` and receive what it missed, as long as the events are still retained.
 
 ### Backpressure
 
-When `TryWrite` fails the queue is full. The handling depends on `Events:BackpressurePolicy`:
-
-```csharp
-if (!writer.TryWrite(publicEvent))
-{
-    string message = $"Session {session.SessionId} event stream queue overflowed.";
-    metrics.QueueOverflow("grpc-event-stream");
-    if (options.Value.Events.BackpressurePolicy == EventBackpressurePolicy.FailFast)
-    {
-        session.MarkFaulted(message);
-        metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
-    }
-    else
-    {
-        logger.LogDebug(
-            "Disconnecting event stream for session {SessionId} after queue overflow.",
-            session.SessionId);
-    }
-
-    writer.TryComplete(new SessionManagerException(
-        SessionManagerErrorCode.EventQueueOverflow,
-        message));
-    return;
-}
-```
-
-Under `FailFast` the session is faulted so subsequent commands return `FailedPrecondition`; the client must reopen. Under the default policy only the stream is dropped and the session continues to accept commands, leaving recovery to the client (typically a fresh `StreamEvents` call with an updated `AfterWorkerSequence`). Either way, the consumer side observes `StatusCode.ResourceExhausted` via the `EventQueueOverflow` mapping above.
+The pump never blocks on a slow subscriber: it writes to each subscriber channel without waiting. When one subscriber's channel is full, the distributor's overflow handler records `QueueOverflow("grpc-event-stream")` and completes that subscriber's channel with `SessionManagerException(EventQueueOverflow)`, which the client observes as `ResourceExhausted`. Under `Events:BackpressurePolicy = FailFast` (the default) in single-subscriber mode the session is also faulted (and `Fault(EventQueueOverflow)` recorded), so subsequent commands return `FailedPrecondition` and the client must reopen. In multi-subscriber mode, or under `DisconnectSubscriber`, only that stream is dropped and the session keeps serving commands and other subscribers; the client recovers with a fresh `StreamEvents` that resumes from its last `worker_sequence`. The internal dashboard mirror is registered as an internal subscriber, so a slow dashboard only disconnects itself and never faults the session.
 
 ### Cancellation and cleanup
 
-The handler creates a linked cancellation token (`streamCts`) so that completing the consumer (client disconnect, error, or graceful end-of-stream) also cancels the producer. The `finally` block cancels the source, disposes the subscriber slot, awaits the producer (swallowing the expected cancellation), and emits `StreamDisconnected("Detached")` so dashboards see the disconnection regardless of cause.
+The `finally` block removes the subscriber's backlog-gauge contribution, disposes the lease, and emits `StreamDisconnected(reason)` so dashboards see every disconnection. The reason is the bounded constant (`EventStreamDisconnectReasons`) for how the stream ended — `SessionClosed`, `Overflow`, `WorkerFault`, `ClientCancelled`, `Error`, or `Detached` — rather than one constant for every cause, so an operator can tell slow consumers from worker faults from routine client disconnects.
 
-`WorkerClientException` thrown by the producer marks the session as faulted before completing the channel — the worker is presumed gone, and any subsequent command on that session must observe the fault rather than silently retry.
+When the worker event stream ends with a `WorkerClientException`, the pump completes every subscriber channel with it. Each subscriber marks the session faulted, so any subsequent command observes the fault rather than silently retrying, and records the `WorkerFault` disconnect reason. The fault itself is counted once in `mxgateway.faults`, by `WorkerClient` when it faults, however many subscribers observe it.
 
 ## Authorization Interceptor Integration
 

docs/Metrics.md

@@ -45,7 +45,7 @@ All counters are `Counter<long>`. Tag values come from the call sites listed und
 | `mxgateway.workers.killed` | `reason` | Forced terminations of worker processes. |
 | `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
 | `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
-| `mxgateway.grpc.streams.disconnected` | `reason` | Detachments of the dashboard or client gRPC event stream. |
+| `mxgateway.grpc.streams.disconnected` | `reason` | Ends of a client `StreamEvents` stream. `reason` is one of a fixed set (`EventStreamDisconnectReasons`): `SessionClosed` (the session event stream completed), `Overflow` (subscriber fell behind its bounded queue), `WorkerFault` (the worker event stream faulted), `ClientCancelled` (the client cancelled or disconnected), `Error` (unclassified), `Detached` (the consumer stopped reading without cancelling). |
 | `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |
 
 ### Histograms
@@ -124,7 +124,7 @@ The recording call sites describe the code paths that write into each instrument
 ```csharp
 _metrics.SessionOpened();
 ...
-_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
+_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString()); // not for a cancelled open or GatewayStopping
 ...
 _metrics.SessionClosed();
 ...
@@ -181,14 +181,14 @@ IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
     () => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
 ...
 backlogRegistration.Dispose();
-metrics.StreamDisconnected("Detached");
+metrics.StreamDisconnected(disconnectReason); // an EventStreamDisconnectReasons constant
 ...
 metrics.QueueOverflow("grpc-event-stream");
 metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
-...
-metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
 ```
 
+A worker fault seen by a subscriber records only the `WorkerFault` disconnect reason, not a fault. `WorkerClient` records the fault once when it faults, and every subscriber of the session observes that same fault, so counting it per subscriber made one worker fault read as 1+K faults for K subscribers.
+
 Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.
 
 `Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):

docs/MxAccessWorkerInstanceDesign.md

@@ -133,6 +133,9 @@ starts the STA, or touches MXAccess:
 | `2` | `InvalidArguments` | Required arguments are missing or unknown arguments are present. |
 | `3` | `InvalidProtocolVersion` | `--protocol-version` is not numeric or does not match the supported worker protocol. |
 | `4` | `MissingNonce` | `MXGATEWAY_WORKER_NONCE` is absent or empty. |
+| `5` | `PipeConnectionFailed` | An `IOException` or `TimeoutException` escapes the pipe client. |
+| `6` | `ProtocolViolation` | A `WorkerFrameProtocolException` escapes the pipe client. |
+| `7` | `UnhandledException` | An exception reaches the `AppDomain.UnhandledException` hook (see `docs/WorkerBootstrap.md`). |
 
 Bootstrap logs use `WorkerConsoleLogger` key/value output. `WorkerLogRedactor`
 redacts fields whose names indicate nonce, secret, password, token,
@@ -270,12 +273,33 @@ is still responsive. Shutdown marks the runtime as closing, wakes the pump,
 rejects new commands, cancels queued work, uninitializes COM on the STA, and
 waits for the thread to exit.
 
+`ProcessQueuedCommands` pumps pending Windows messages between queued work
+items whenever more work remains, not only after the whole queue drains. A
+burst of queued commands would otherwise hold every inbound COM callback
+(`OnDataChange`, `OnWriteComplete`) behind the entire backlog.
+
+If the STA loop itself throws after `Start` has returned (a failed
+`MsgWaitForMultipleObjectsEx`, or an exception escaping a dispatched message),
+`StaRuntime` records the exception as `LoopFault`, fails every later
+`InvokeAsync` with an `InvalidOperationException` wrapping it (never the
+graceful `StaRuntimeShutdownException`), cancels queued work, and invokes the
+handler registered through `RegisterLoopFaultHandler` — unless a requested
+shutdown is already in progress, in which case the death is part of a
+deliberate teardown and is not reported. `MxAccessStaSession` registers a
+handler that records a `StaHung` `WorkerFault` on the event queue (diagnostic
+text carries the exception type and HRESULT only, never the message),
+so the IPC drain loop sends the fault to the gateway and ends the pipe session
+and the worker process. Before this, a post-start loop death was stored in the
+startup-exception field that only `Start` reads, leaving a live worker whose
+commands queued forever. Startup failures still surface from `Start` itself.
+
 ### Inner Completion Waits
 
 Two commands hold the STA while waiting for a COM event they just provoked: the
 unary write path waits for its `OnWriteComplete`
-(`MxAccessWriteCompletionCache.TryWaitForCompletion`), and `ReadBulk` waits per
-tag for the first `OnDataChange` (`MxAccessValueCache.TryWaitForUpdate`). Both
+(`MxAccessWriteCompletionCache.TryWaitForCompletion`), and `ReadBulk` waits for
+the first `OnDataChange` of every snapshot tag
+(`MxAccessValueCache.TryWaitForUpdates`). Both
 run the same loop shape as the outer pump, for the same reason — the event they
 are waiting for *is* a Windows message, so the thread must keep dispatching to
 receive it:
@@ -297,7 +321,7 @@ The idle slice is a Win32 wait (`StaWaitHelper.WaitForSignalOrMessages`), never
 `Thread.Sleep`. A sleeping STA pumps no messages, so a sleep-polled loop could
 only dispatch the awaited COM event at poll-tick granularity while stalling
 *every other* event for the same tick — up to 1.5 s for a write completion and
-up to `timeout_ms` per tag for `ReadBulk`. The Win32 wait returns the instant a
+up to `timeout_ms` for `ReadBulk`. The Win32 wait returns the instant a
 message needs pumping, so the apartment dispatches continuously for the whole
 wait. Each cache also sets an `AutoResetEvent` from its update path (outside the
 cache lock) so a cross-thread producer wakes the waiter immediately; in the live
@@ -317,8 +341,29 @@ nothing wakes the wait — a process with no STA message queue (unit tests drive
 these caches from ordinary threads, standing in for the STA by updating the
 cache from a fake `pumpStep`) must not block for a full poll interval. Timeouts,
 deadline math, and return values are unchanged by the wait mechanism: an expired
-write wait still yields the empty-`statuses` unconfirmed reply, and an expired
-per-tag `ReadBulk` wait still reports its own timeout.
+write wait still yields the empty-`statuses` unconfirmed reply, and every
+`ReadBulk` tag that did not deliver before the deadline still reports its own
+timeout.
+
+`ReadBulk` runs its snapshot lifecycle in phases so the STA is held for at most
+one `timeout_ms` per command, not one per tag: (1) every tag is validated and
+probed against the cached-and-advised fast path; (2) every remaining tag is
+added and advised; (3) one wait with a shared deadline collects the first
+`OnDataChange` per snapshot handle and returns as soon as all have delivered;
+(4) every snapshot handle is unadvised and removed, best-effort. Per-tag
+results are unchanged: request order, the blank-address failure, the cached
+result, an `AddItem` failure (item handle 0) or `Advise` failure (its own item
+handle) failing only that tag, and the `ReadBulk timed out after <n> ms waiting
+for first OnDataChange.` message for each tag that never delivered. The cache
+probes all run before any snapshot `AddItem`, so a later tag in the same
+request can never be served from an earlier tag's snapshot subscription.
+When MXAccess returns the same item handle for two tags in one batch (a
+duplicate address), the handle is advised, unadvised, and removed once, and
+every duplicate receives the shared outcome. Phases 2 and 4 make one or two
+COM calls per tag without waiting, so they call the pump step every
+`ReadBulkPumpInterval` (16) tags; a large batch therefore keeps dispatching
+inbound callbacks and refreshing `LastStaActivityUtc` instead of drifting
+toward `HeartbeatStuckCeiling`.
 
 The write wait's budget is `MxGateway:Worker:WriteCompletionWaitMilliseconds`
 (default 1500). It applies to `Write`, `Write2`, `WriteSecured`, and
@@ -351,6 +396,23 @@ native client correlating by handle has the identical ambiguity). The exposure
 is bounded: `statuses` is an enrichment field, and `Hresult`/`ProtocolStatus`
 stay acceptance-only regardless of which completion (if any) was attributed.
 
+**The alarm ack writer narrows the window for acknowledgements.**
+`AlarmAckWriter` registers its own MXAccess server with its own
+`MxAccessWriteCompletionCache` and reuses one advised `AckMsg` item per alarm
+(least-recently-used, capped at 256). After an ack ends unconfirmed it marks the
+item dirty, and the next ack to that alarm removes and re-adds the item so the
+write waits on a fresh item handle the stale completion cannot match. This
+relies on MXAccess never reusing an item handle number within one
+registration, which a live probe confirmed (see "Item handle reuse" in
+[Alarm Probe Findings](./AlarmProbeFindings.md)). Every
+drop of an item (dirty re-add, eviction, or a failed `AddItem`/`Advise`/`Write`)
+also removes that handle's completion entry, and tearing down the registration
+removes every entry for its server handle, so the cache does not grow by one
+entry per dropped handle for the life of the worker. The writer records a
+completion only for an item handle it currently holds on its live
+registration, so a late `OnWriteComplete` for a dropped handle cannot recreate
+its entry.
+
 ## Live alarm watch-list replacement
 
 An auto-mode `SubscribeAlarms` with `fallback_enabled = true` always builds a
@@ -654,7 +716,20 @@ The worker records plain and supervisory advice separately only after the COM
 call returns normally. Successful `UnAdvise` removes all tracked advice for the
 server and item pair because the public MXAccess cleanup method has no plain
 versus supervisory selector. Successful `RemoveItem` and `Unregister` also clear
-related advice state from the worker registry. Failed advice and cleanup calls
+related advice state from the worker registry, and evict the per-handle
+caches the event sink fills: `RemoveItem` drops the handle pair's
+`MxAccessValueCache` and `MxAccessWriteCompletionCache` entries, and
+`Unregister` drops every entry recorded under the server handle (MXAccess
+removes the server's items with it, so the per-item eviction never runs for
+them). Without this, an `AddItem` → `Write` → `RemoveItem` churn grew the
+completion cache by one entry per cycle for the life of the worker.
+`AddItem`, `AddItem2`, and `AddBufferedItem` also drop both cache entries for
+the handle they receive when the registry does not already track it. An
+`OnDataChange` or `OnWriteComplete` that was queued when `RemoveItem` or
+`Unregister` ran is dispatched after the eviction and recreates the entry;
+when MXAccess later reuses that item id, `ReadBulk`'s cached path would
+otherwise serve the stale value as `was_cached = true`. A handle the
+registry still tracks belongs to a live item, so its entries are kept. Failed advice and cleanup calls
 leave registry state unchanged so diagnostics continue to reflect the last
 successful MXAccess-owned state transition.
 
@@ -888,6 +963,23 @@ the STA more than 75 s without pumping *should* fault — that is the ceiling's
 contract). The flag is named generically so any future non-dispatcher STA work
 reuses it.
 
+A `PollOnce` that throws does not fault the session on the first failure. The
+poll loop counts consecutive failures and records the alarm-poll fault
+(`MxaccessEventConversionFailed`) only when the count reaches
+`MxAccessStaSession.DefaultAlarmPollFailureThreshold` (3, the same default as
+`AlarmFailoverConfig.consecutive_failure_threshold`); a successful poll resets
+the count. Each failure below the threshold writes one
+`level=Warning event=AlarmPollFailed` line to stderr carrying the exception
+type, HRESULT, and counts only — the exception message is left out because it
+can echo alarm or tag names. A single transient COM failure in
+`GetXmlCurrentAlarms2` used to fault the whole session, taking every data
+subscription with it; a persistent failure, including an STA-affinity
+regression that fails every tick, still faults within three poll intervals.
+A poll failure seen after the STA loop itself has died (`StaRuntime.LoopFault`
+is set) ends the poll loop without recording anything: the loop-fault handler
+has already recorded the `StaHung` fault, and the failure is only the
+fast-fail `InvokeAsync` returns afterwards.
+
 The in-flight-command suppression itself is bounded by
 `WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
 `HeartbeatGrace`). The motivating case for the suppression is a legitimately
@@ -904,7 +996,7 @@ has been stale for longer than `HeartbeatStuckCeiling`, the watchdog fires
 healthy long-running commands: `StaRuntime.PumpPendingMessages()` refreshes
 `LastStaActivityUtc` (via `MarkActivity()`) every time it runs, and long-hold
 STA commands invoke it on every wait iteration (`ReadBulk` routes its
-per-tag wait through the `pumpStep` wired from `StaRuntime.PumpPendingMessages`).
+shared snapshot wait through the `pumpStep` wired from `StaRuntime.PumpPendingMessages`).
 A command that keeps pumping therefore keeps its activity timestamp fresh and
 never reaches the ceiling, while a genuinely stuck STA — one that has stopped
 pumping — accrues staleness and faults correctly. The ceiling is thus the
@@ -1002,10 +1094,10 @@ and relies on the gateway to kill the process.
 
 ## Fault Handling
 
-Worker fault categories:
+Worker fault categories (`InvalidArguments` and `GatewayAuthenticationFailed`
+are reserved and never emitted: bootstrap failures exit via `WorkerExitCode`
+before the pipe exists):
 
-- `InvalidArguments`
-- `GatewayAuthenticationFailed`
 - `ProtocolMismatch`
 - `ProtocolViolation`
 - `PipeDisconnected`

docs/Sessions.md

@@ -80,7 +80,7 @@ private void EnsureSessionCapacity()
 }
 ```
 
-`SessionManager` also defines five close-reason constants — `DefaultCloseReason` (`"client-close"`), `GatewayShutdownReason` (`"gateway-shutdown"`), `LeaseExpiredReason` (`"lease-expired"`), `FaultedReason` (`"faulted-reaped"`), and `DetachGraceExpiredReason` (`"detach-grace-expired"`) — so that the metrics and worker shutdown paths agree on a fixed vocabulary.
+`SessionManager` also defines six close-reason constants — `DefaultCloseReason` (`"client-close"`), `GatewayShutdownReason` (`"gateway-shutdown"`), `LeaseExpiredReason` (`"lease-expired"`), `FaultedReason` (`"faulted-reaped"`), `DetachGraceExpiredReason` (`"detach-grace-expired"`), and `UnattachedGraceExpiredReason` (`"unattached-grace-expired"`) — so that the metrics and worker shutdown paths agree on a fixed vocabulary.
 
 ### SessionRegistry (ISessionRegistry)
 
@@ -137,18 +137,19 @@ public sealed record SessionOpenRequest(
 
 ### SessionCloseResult
 
-`SessionCloseResult` is the record returned from a successful close. `AlreadyClosed` distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting.
+`SessionCloseResult` is the record returned from a successful close. `AlreadyClosed` distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting. `WorkerShutdownFailure` is set when the worker refused a graceful shutdown and was killed instead — still a successful close (see [Close](#close)); the manager logs it and the gRPC reply does not expose it.
 
 ```csharp
 public sealed record SessionCloseResult(
     string SessionId,
     SessionState FinalState,
-    bool AlreadyClosed);
+    bool AlreadyClosed,
+    Exception? WorkerShutdownFailure = null);
 ```
 
 ### SessionCloseStartedException
 
-`SessionCloseStartedException` is `internal` and is only thrown from inside `GatewaySession.CloseAsync` when the close path has already begun mutating worker state and a subsequent step fails. `SessionManager.CloseSessionCoreAsync` catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as `SessionManagerException` with `CloseFailed`. The intermediate type exists so the public API surface only ever exposes `SessionManagerException`.
+`SessionCloseStartedException` is `internal` and is only thrown from inside `GatewaySession.CloseAsync` when the close path has already begun mutating worker state and a subsequent step fails — in practice, when the worker's graceful shutdown failed *and* the kill fall-back failed too, so a worker may still be running. `SessionManager.CloseSessionCoreAsync` catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as `SessionManagerException` with `CloseFailed`. The intermediate type exists so the public API surface only ever exposes `SessionManagerException`.
 
 ### SessionManagerException and SessionManagerErrorCode
 
@@ -162,26 +163,26 @@ public sealed record SessionCloseResult(
 | `EventSubscriberLimitReached` | In multi-subscriber mode, an attach exceeded `MaxEventSubscribersPerSession` concurrent external subscribers. |
 | `EventQueueOverflow` | Reserved for the worker event channel overflow path. |
 | `SessionLimitExceeded` | `MaxSessions` is in use. |
-| `OpenFailed` | `OpenSessionAsync` failed; the inner exception carries the cause. |
-| `CloseFailed` | A close started but did not complete cleanly; the session is removed and faulted. |
+| `OpenFailed` | `OpenSessionAsync` failed; the inner exception carries the cause. A caller-cancelled open and an open refused because the gateway is stopping do not use this code. |
+| `CloseFailed` | A close started but did not complete cleanly — the worker's graceful shutdown and the kill fall-back both failed (or a kill threw); the session is removed and faulted. A shutdown failure whose kill succeeded is a successful close, not this. |
+| `PermissionDenied` | The caller may not perform the operation on this session (for example, attaching `StreamEvents` to a session opened by a different API key). |
+| `GatewayStopping` | The gateway is stopping and refuses new sessions (see [Shutdown Coordination](#shutdown-coordination)). Maps to gRPC `Unavailable`; not recorded as a fault. |
 
 ## Lifecycle
 
 ### Open
 
-`SessionManager.OpenSessionAsync` allocates a session slot, builds the `GatewaySession`, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step:
+`SessionManager.OpenSessionAsync` allocates a session slot, builds the `GatewaySession`, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step. Two outcomes are not open failures and skip the fault metric and the warning: `GatewayStopping` (see [Stopping gate](#stopping-gate)) and the caller's own cancellation, which is rethrown as the `OperationCanceledException` so the gRPC layer maps it to `Cancelled` rather than `Unavailable`:
 
 ```csharp
+catch (OperationCanceledException exception) when (cancellationToken.IsCancellationRequested)
+{
+    await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
+    throw;
+}
 catch (Exception exception)
 {
-    session?.MarkFaulted(exception.Message);
-    if (session is not null)
-    {
-        _registry.TryRemove(session.SessionId, out _);
-        await session.DisposeAsync().ConfigureAwait(false);
-    }
-
-    ReleaseSessionSlot();
+    await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
     _metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
     _logger.LogWarning(
         exception,
@@ -195,7 +196,19 @@ catch (Exception exception)
 }
 ```
 
-The order — fault, deregister, dispose, release slot, record metric, log, rethrow — matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine.
+`RollBackOpenAsync` faults the session and, when it had been registered, routes it through `RemoveSessionAsync` — the single removal path every teardown uses, which deregisters, drops the session's per-session event metrics, releases the slot and disposes, in that order. The order matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine. A session that was never registered (id collision) releases the slot and disposes directly. The open-session gauge is not touched by the rollback: `SessionOpened` is recorded inside the attach gate described next, after which nothing in the open sequence can fail.
+
+#### Close or kill during worker startup
+
+The session is registered — and therefore visible to the dashboard's Kill button, the lease sweep and the host drain — for the whole worker startup, which can last up to `MxGateway:Worker:StartupTimeoutSeconds`. A kill or drain that lands in that window finds no worker client to stop, flips the session to `Closed`, removes it and disposes it. The worker client that `SessionWorkerClientFactory.CreateAsync` then returns must not be stored on that dead session and handed back to the caller as a successful open: nothing would ever shut the worker down (the close already ran) and the client would hold a `Closed` session.
+
+`GatewaySession.AttachWorkerClient` therefore returns `false`, under `_syncRoot`, when the session is already `Closing`, `Closed` or `Faulted` or has been disposed. The manager runs the end of the open sequence — attach, `MarkReady`, `SessionOpened` — through `TryAttachWorkerClientAndMarkReadyAsync`, which holds the per-session close gate (`_closeLock`) so a concurrent `CloseAsync` / `KillWorkerWithCloseGateAsync` runs either entirely before it (the attach is refused) or entirely after it (an ordinary close of a Ready session). `TryAttachWorkerClientAndMarkReadyAsync` returns a `SessionAttachOutcome` rather than a bool, because the open reports each refusal differently. On a refused attach the manager kills and disposes the late worker client; the registry is already empty and the slot was released by whichever teardown closed the session. A refusal caused by the host drain fails the open with `GatewayStopping`, the same code as an open refused at the door, so the client retries against the restarted gateway. A refusal caused by an admin kill fails it with `OpenFailed`.
+
+The attach also refuses a worker client that is already `Faulted` or `Closed` (`SessionAttachOutcome.WorkerTerminal`). A worker can die right after its handshake, for example from an STA loop fault or exit code 7. Attaching it would hand the caller a `Ready` session over a dead worker, and nothing would notice until the lease expired. The client is discarded and the open fails with `OpenFailed`. A worker that dies just after this check is caught by the event pump instead (see [Worker exit outside a gateway close](#worker-exit-outside-a-gateway-close)).
+
+Once `CreateAsync` has returned, a live worker exists that only the open path knows about, so the gate wait is **not cancellable** (it waits on `CancellationToken.None`; the gate is only ever held for the duration of one close or kill) and the manager discards the client on *any* exception out of the attach step — the caller's own cancellation (checked before the attach, so an abandoned open never yields a session), a disposed gate, anything thrown while becoming Ready. A cancellation surfaces as the `OperationCanceledException` itself; any other failure surfaces as `OpenFailed` with the cause as inner exception. An exception that escaped before the discard would be an orphaned x86 worker, which a restarted gateway terminates rather than adopts.
+
+Because such a session was never counted as opened, it must not be counted as closed either. `GatewaySession.OpenRecorded` is set inside the same gate right after `SessionOpened` is recorded, and every teardown path (`KillWorkerAsync`, `CloseSessionCoreAsync`, the close-failed branch) records `SessionClosed` / `SessionRemoved` only when it is set — so `sessions.opened`, `sessions.closed` and the `sessions.open` gauge stay consistent across the startup window.
 
 ### Run
 
@@ -215,11 +228,13 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
 
 The same one-distributor-per-session rule is enforced at the other end of the lifecycle. Session disposal disposes the distributor and latches the fact, so an attach that passed its readiness gate and then lost the CPU until the session was torn down throws `SessionNotReady` rather than constructing a replacement distributor. A replacement can never be pumped — the pump-start flag is a one-way latch — so its subscriber would block on an empty channel that is never completed until the client's own deadline expired, with no error, log or metric to explain it. Every attach entry point (`AttachEventSubscriber`, `AttachEventSubscriberWithReplay`, `AttachInternalEventSubscriber`) fails this way, and each rolls back the subscriber-count slot it took, so a caller that loses the race to teardown sees the same failure it would have seen a moment later from the readiness gate.
 
-Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
+Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration (`TouchClientActivity` is the only path that moves the lease). `IsLeaseExpired` cooperates with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
 
 #### Teardown parallelism
 
-A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses`, a compile-time constant of `4` in `SessionManager`. Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
+A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace, then unattached-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses`, a compile-time constant of `4` in `SessionManager`. Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
+
+`TryBeginCloseIfExpired` is a *reservation*, not the close: it writes `Closing` but leaves the session's close-started flag alone, so the `CloseAsync` the sweep issues next is the first close to run and reports `AlreadyClosed: false`. That is what lets `CloseSessionCoreAsync` account `SessionClosed` for lease-expired, faulted-reaped and detach-grace-expired closes exactly once; a reservation that also set the flag made every sweep-driven close look like a second concurrent close, and swept sessions never left the `sessions.open` gauge. A client close that races the reservation still counts once: whichever `CloseAsync` takes the close lock first does the accounting, and the other finds the session `Closed`.
 
 Splitting the phases moves the TOCTOU re-check earlier, and that is an accepted trade rather than an unchanged behavior: selection now flips **every** chosen session to `Closing` up front, before any teardown runs, whereas the sequential sweep re-checked session *N* only after sessions *1..N-1* had finished closing. A client that re-attaches a subscriber while the close phase is running therefore loses a race it could previously win — the eligibility snapshot is taken at one instant for the whole pass. Expiry evaluation itself is unaffected, because `now` is a parameter and is not re-read per session.
 
@@ -239,9 +254,21 @@ Mechanically: when the last external subscriber detaches and `DetachGraceSeconds
 
 `DetachGraceSeconds` controls retention and expiry only; the reconnect/replay path that re-attaches a dropped client to a retained session is described in [Reconnect and replay](#reconnect-and-replay).
 
+#### Worker exit outside a gateway close
+
+`IWorkerClient` has no state-change callback, so a worker that reached `Closed` or `Faulted` *without the gateway closing it* — the client sent the `ShutdownWorker` control command and the worker exited cleanly, the worker crashed, its pipe broke, a heartbeat timed out — used to leave the session reporting `Ready` until its lease expired (a live finding: `WorkerClient` was `Closed` on the shutdown ack, the session still `Ready`, even with an event stream attached). The session now observes it from its own worker-event pump source (`MapWorkerEventsAsync`): the single distributor pump is the session's one always-on view of the worker (it runs from `MarkReady` via the dashboard mirror), and `WorkerClient` enters its terminal state *before* it completes the event channel, so when the enumeration ends the terminal state is visible. `ObserveWorkerTerminalState` then faults a `Ready` session whose worker is `Closed`/`Faulted`. The pump also calls it when its first readiness check (`GetReadyWorkerClientAsync`) fails. A worker that dies between the open's attach and the pump's first read never reaches the event enumeration, so its `finally` never runs. Without that second call the session would stay `Ready` over the dead worker until its lease expired.
+
+The choice is **`Faulted`, not `Closed`**: `Closed` means the gateway closed the session, and a `Closed` session left in the registry would linger until its lease expired with nothing reaping it. A faulted session is permanently unusable (every command fails the readiness check, as the worker is gone), stays observable via `GetSessionStatus` with `FinalFault` naming the worker state, and is reaped by the faulted-session sweep below after `FaultedGraceSeconds` — which is where `SessionClosed` is accounted, exactly once, through the ordinary `CloseSessionCoreAsync` path (`WorkerClient.ShutdownAsync` on a `Closed` client returns immediately, so no kill fallback or warning). Only a `Ready` session follows its worker: during a gateway-initiated close the worker closing is the expected outcome, not a fault.
+
 #### Faulted-session reaping
 
-A session that faults (for example, a slow single-subscriber client that overflows its event queue under the `FailFast` backpressure policy) is left permanently unusable — every command fails `EvaluateReadyUnderLock` — but its worker keeps running and it still pins one of `MxGateway:Sessions:MaxSessions` slots. Rather than waiting up to `DefaultLeaseSeconds` for the lease to expire, `SessionManager.CloseExpiredLeasesAsync` also checks `IsFaultedReapable(now)` and reaps a faulted session through the same `TryBeginCloseIfExpired` → `CloseSessionCoreAsync` teardown, with the distinct `FaultedReason` (`"faulted-reaped"`). `MarkFaulted` stamps a fault timestamp from the session's `TimeProvider`; `MxGateway:Sessions:FaultedGraceSeconds` (default `0`) bounds how long the faulted session is retained before the next sweep reaps it — `0` reaps it on the next sweep, a positive value keeps it observable via `GetSessionStatus` for that window first. Sweep precedence when several conditions fire at once is lease-expiry, then faulted, then detach-grace.
+A session that faults (for example, a slow single-subscriber client that overflows its event queue under the `FailFast` backpressure policy) is left permanently unusable — every command fails `EvaluateReadyUnderLock` — but its worker keeps running and it still pins one of `MxGateway:Sessions:MaxSessions` slots. Rather than waiting up to `DefaultLeaseSeconds` for the lease to expire, `SessionManager.CloseExpiredLeasesAsync` also checks `IsFaultedReapable(now)` and reaps a faulted session through the same `TryBeginCloseIfExpired` → `CloseSessionCoreAsync` teardown, with the distinct `FaultedReason` (`"faulted-reaped"`). `MarkFaulted` stamps a fault timestamp from the session's `TimeProvider`; `MxGateway:Sessions:FaultedGraceSeconds` (default `0`) bounds how long the faulted session is retained before the next sweep reaps it — `0` reaps it on the next sweep, a positive value keeps it observable via `GetSessionStatus` for that window first. Sweep precedence when several conditions fire at once is lease-expiry, then faulted, then detach-grace, then unattached-grace.
+
+#### Never-subscribed session reaping
+
+Detach-grace only helps a session that *had* a subscriber. A client that opens a session and then disappears before it ever calls `StreamEvents` — OtOpcUa's `GalaxyDriver` failing part-way through its start-up is the motivating case — leaves a `Ready` session with no subscriber, no detach timestamp and a lease 30 minutes out, pinning one of `MaxSessions` and a live x86 worker for all of `DefaultLeaseSeconds`. `MxGateway:Sessions:UnattachedSessionGraceSeconds` (default `0`, off) closes that gap: when positive, `CloseExpiredLeasesAsync` also checks `IsUnattachedGraceExpired(now)` and reaps, with the distinct `UnattachedGraceExpiredReason` (`"unattached-grace-expired"`), any `Ready` session that has never had a subscriber of either kind and has been idle for the window.
+
+Four deliberate limits keep the rule from catching legitimate sessions. It applies only to `Ready` sessions, and its idle clock starts at `Ready` (`MarkReady` stamps `_readyAtUtc`), so a slow worker start neither reaps the session mid-startup nor eats the window before the client even has the session id. Idle is measured from the later of that Ready stamp and the session's *last client activity* (its most recent unary command), so a unary-only client (reads and writes, never streams) is not reaped while it keeps calling; the rule is an idle timeout for sessions that never streamed, not a deadline to stream. It stops applying the moment the first subscriber attaches, after which detach-grace and the normal lease govern the session. And an *internal* in-process subscriber counts as attached for this rule (`AttachInternalEventSubscriber` sets a separate `_everHadInternalEventSubscriber` latch): the gateway's own `GatewayAlarmMonitor` session only ever attaches that way, and it must not be reaped by its own gateway every cycle. The latch is separate from `_everHadEventSubscriber` because that one drives detach-grace, which is keyed on external subscribers only. The dashboard mirror, which every session registers at `MarkReady`, is deliberately *not* a subscriber for this rule. The alarm monitor is the only gateway-owned session exempted. The dashboard Browse page's live-data session (`DashboardLiveDataService`) is unary-only: it registers and reads but never attaches a subscriber, so once it has been idle for the window the rule reaps it. That is expected. The service checks its cached session on every call (`EnsureReadyAsync`) and opens and registers a new one when the old one is gone. Like the other sweep conditions it is re-verified inside `TryBeginCloseIfExpired`, so a subscriber that attaches between selection and the close wins the race. Operators should set the window comfortably above a well-behaved client's open-to-first-`StreamEvents` time; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later.
 
 #### Reconnect and replay
 
@@ -267,9 +294,10 @@ The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream a
 
 ### Close
 
-`GatewaySession.CloseAsync` is serialized by a per-session `SemaphoreSlim` (`_closeLock`) so only one close runs at a time, but every read/write of `_state` still passes through `_syncRoot` (via `TryBeginClose` and `MarkClosed`). The close path therefore obeys the same lock discipline as `TransitionTo` / `MarkFaulted`: it transitions to `Closing`, asks the worker client to shut down within `ShutdownTimeout`, and on success transitions to `Closed`. `DisposeAsync` waits on `_closeLock` once before disposing the semaphore so an in-flight close's `Release()` cannot race against the dispose. If `WorkerClient.ShutdownAsync` throws, the session falls back to `IWorkerClient.Kill` (forced close):
+`GatewaySession.CloseAsync` is serialized by a per-session `SemaphoreSlim` (`_closeLock`) so only one close runs at a time, but every read/write of `_state` still passes through `_syncRoot` (via `TryBeginClose` and `MarkClosed`). The close path therefore obeys the same lock discipline as `TransitionTo` / `MarkFaulted`: it transitions to `Closing`, asks the worker client to shut down within `ShutdownTimeout`, and on success transitions to `Closed`. `DisposeAsync` waits on `_closeLock` once before disposing the semaphore so an in-flight close's `Release()` cannot race against the dispose. `WorkerClient.ShutdownAsync` is bounded by `ShutdownTimeout` end to end — queuing the `WorkerShutdown` envelope included, since a worker that stopped reading the pipe can leave the bounded outbound channel full with no watchdog left to clear it once the client is `Closing` — so a close never holds `_closeLock` longer than the timeout before reaching the fallback. If `WorkerClient.ShutdownAsync` throws, the session falls back to `IWorkerClient.Kill` (forced close):
 
 ```csharp
+Exception? workerShutdownFailure = null;
 if (_workerClient is not null)
 {
     try
@@ -289,54 +317,76 @@ if (_workerClient is not null)
                 new AggregateException(exception, killException));
         }
 
-        throw;
+        workerShutdownFailure = exception;
     }
 }
+
+MarkClosed();
+return new SessionCloseResult(SessionId, SessionState.Closed, alreadyClosing, workerShutdownFailure);
 ```
 
-If both graceful shutdown and the kill fall-back fail, the original and kill exceptions are bundled into an `AggregateException` and surfaced as `SessionCloseStartedException`. `SessionManager.CloseSessionCoreAsync` then translates that into a `SessionManagerException` with `CloseFailed` and removes the session.
+A kill that succeeds completes the close **successfully**: the worker is gone and the session is `Closed`, exactly as if the shutdown had worked, and the caller's `CloseSession` reply says so. The graceful failure is not lost — it travels back as `SessionCloseResult.WorkerShutdownFailure`, and `SessionManager.CloseSessionCoreAsync` logs it at warning level ("did not shut down gracefully and was killed"), which is the operator signal that workers are hanging on shutdown. The kill itself is counted by `WorkerClient.Kill` (`mxgateway.workers.killed`). Nothing is faulted and no `CloseFailed` fault is recorded, because nothing failed from the session's point of view; reporting the common hung-worker case that way had made every such close look like a teardown failure.
 
-`GatewaySession.KillWorker` is the unconditional forced-close path used by shutdown when graceful close itself throws, and also by `SessionManager.KillWorkerAsync` — the explicit kill path that the dashboard's admin Kill button invokes. `KillWorkerAsync` skips `WorkerClient.ShutdownAsync` entirely, so `KillCount` increments while `ShutdownCount` does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful `CloseSessionAsync`.
+Only if both the graceful shutdown and the kill fall-back fail are the original and kill exceptions bundled into an `AggregateException` and surfaced as `SessionCloseStartedException`. `SessionManager.CloseSessionCoreAsync` then translates that into a `SessionManagerException` with `CloseFailed`, faults the session, records the fault metric and removes the session — a worker may still be running, which is what `CloseFailed` is reserved for.
+
+**Losing a race to a concurrent close.** A caller that looked a session up (a kill, a client close, a sweep's snapshot) and then found its close gate already disposed — because a concurrent close finished, removed the session, accounted `SessionClosed` and disposed it in between — sees `ObjectDisposedException` from the gate. Both `KillWorkerAsync` and `CloseSessionCoreAsync` catch that case *when the session is no longer registered* and return `AlreadyClosed: true` with no accounting, no fault and no log: the close that won already did all of it. (The old kill catch recorded a fault, faulted the session and decremented the open gauge a second time; the sweep rethrew it as a sweep failure.) A sweep does not count such a session among the sessions it closed. A disposed gate on a session that *is* still registered is a real failure and takes the normal `CloseFailed` path.
+
+`SessionManager.KillWorkerAsync` is the explicit kill path that the dashboard's admin Kill button invokes and that `ShutdownAsync` falls back to when a graceful close throws. It goes through `GatewaySession.KillWorkerWithCloseGateAsync`, which skips `WorkerClient.ShutdownAsync` entirely, so `KillCount` increments while `ShutdownCount` does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful `CloseSessionAsync`.
 
 ## Shutdown Coordination
 
 `SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. Sessions are drained with the same bounded fan-out the lease sweep uses (`MaxParallelSessionCloses`), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions — *every* exception, including from the fallback — calls `KillWorkerAsync` on an uncancellable token, and removes the session, so that neither one stuck worker nor one failing teardown can block or abort the rest of the host's drain:
 
 ```csharp
-await Parallel.ForEachAsync(
-    _registry.Snapshot(),
-    new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
-    async (session, _) =>
+MarkStopping();
+for (int pass = 0; pass < MaxShutdownDrainPasses; pass++)
+{
+    if (_registry.Count == 0)
     {
-        try
+        return;
+    }
+
+    await Parallel.ForEachAsync(
+        _registry.Snapshot(),
+        new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
+        async (session, _) =>
         {
-            await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
-        }
-        catch (Exception exception)
-        {
-            _logger.LogWarning(
-                exception,
-                "Graceful shutdown failed for session {SessionId}; killing worker.",
-                session.SessionId);
-            if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
-                && registeredSession is not null)
+            try
             {
-                try
+                await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
+            }
+            catch (Exception exception)
+            {
+                _logger.LogWarning(
+                    exception,
+                    "Graceful shutdown failed for session {SessionId}; killing worker.",
+                    session.SessionId);
+                if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
+                    && registeredSession is not null)
                 {
-                    // Not the caller's token: the kill is the last-resort orphan preventer.
-                    await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
-                }
-                catch (Exception killException)
-                {
-                    _logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
+                    try
+                    {
+                        // Not the caller's token: the kill is the last-resort orphan preventer.
+                        await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
+                    }
+                    catch (Exception killException)
+                    {
+                        _logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
+                    }
                 }
             }
-        }
-    }).ConfigureAwait(false);
+        }).ConfigureAwait(false);
+}
 ```
 
 Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.
 
+### Stopping gate
+
+A drain that only ever looked at one snapshot could be outrun by `OpenSession`: the host raises `ApplicationStopping` *before* hosted services stop, and Kestrel keeps accepting gRPC calls until its own `StopAsync` runs, so a session opened behind the snapshot would have nobody left to close it and its x86 worker would survive the gateway as an orphan. `SessionManager` therefore keeps a stopping flag that is set by the host's `IHostApplicationLifetime.ApplicationStopping` (the manager registers for it in its constructor when the lifetime is injected) and, as a fallback for hosts without one, at the start of `ShutdownAsync`. Once set it is never cleared. `OpenSessionAsync` checks it twice: on entry, before a slot is taken, and again right after the session is registered. The second check is what closes the race — the flag is written before the drain takes its registry snapshot, so an open that does not observe it there registered its session before that snapshot and is drained, while one that does observe it rolls its own session back. Either way the caller receives `SessionManagerException` with `GatewayStopping`, which the gRPC layer maps to `Unavailable` so the client retries against the restarted gateway; it is not an open failure and records no fault metric. An open that passed both checks but whose session the drain closed while its worker was still starting gets the same code: the attach is refused, the late worker is discarded, and the manager rechecks the flag before it reports the refusal.
+
+The drain loop above is the backstop to that gate: it re-snapshots until the registry is empty, bounded at `MaxShutdownDrainPasses` (`3`) so a session that somehow cannot be removed cannot hold the host stop forever. Anything still registered after the last pass is logged and left to the orphan terminator on the next start.
+
 ## Dependency Injection
 
 `SessionServiceCollectionExtensions.AddGatewaySessions` registers the four singletons and the hosted service:

docs/WorkerBootstrap.md

@@ -117,8 +117,13 @@ public static WorkerBootstrapResult Failure(WorkerExitCode exitCode, IEnumerable
 | `MissingNonce` | 4 | The `MXGATEWAY_WORKER_NONCE` environment variable is null, empty, or whitespace. |
 | `PipeConnectionFailed` | 5 | An `IOException` or `TimeoutException` escapes the pipe client. |
 | `ProtocolViolation` | 6 | A `WorkerFrameProtocolException` escapes the pipe client. |
+| `UnhandledException` | 7 | An exception reaches `AppDomain.CurrentDomain.UnhandledException` — thrown on a thread outside `WorkerApplication.Run`'s `try/catch` (a background task continuation, a timer, a COM callback thread). |
 
-`InvalidArguments`, `InvalidProtocolVersion`, and `MissingNonce` originate in the parser; the others originate in `WorkerApplication.Run`'s `try/catch` around the pipe client.
+`InvalidArguments`, `InvalidProtocolVersion`, and `MissingNonce` originate in the parser; `UnhandledException` originates in the process-wide hook described below; the others originate in `WorkerApplication.Run`'s `try/catch` around the pipe client.
+
+### Unhandled-exception hook
+
+The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because they can carry tag values or credentials passed through a command, the same rule every other worker failure line follows. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
 
 ## Environment Abstraction
 

docs/WorkerSta.md

@@ -86,10 +86,16 @@ StaWorkItem<T> workItem = new(command, cancellationToken);
 
 lock (gate)
 {
+    if (loopFault is not null)
+    {
+        return Task.FromException<T>(new InvalidOperationException(
+            "The worker STA loop terminated unexpectedly.",
+            loopFault));
+    }
+
     if (shutdownRequested)
     {
-        return Task.FromException<T>(
-            new InvalidOperationException("The worker STA runtime is shutting down."));
+        return Task.FromException<T>(new StaRuntimeShutdownException());
     }
 
     commandQueue.Enqueue(workItem);
@@ -101,7 +107,11 @@ return workItem.Task;
 
 `StaWorkItem<T>` uses an `Interlocked.CompareExchange` on `started` so that exactly one of three outcomes happens: the STA thread executes the delegate, an external `CancellationToken` cancellation fires first, or `CancelBeforeExecution` runs during shutdown. `Execute` runs on the STA thread, sets `Completion` from `command()`, and propagates exceptions through `TrySetException` so the awaiting caller observes them.
 
-`ProcessQueuedCommands` is the only consumer of the queue and runs on the STA thread, so each work item is guaranteed to execute on the apartment that owns the COM object.
+`ProcessQueuedCommands` is the only consumer of the queue and runs on the STA thread, so each work item is guaranteed to execute on the apartment that owns the COM object. It calls `PumpPendingMessages` between work items whenever more work is queued, so a backlog of commands cannot hold inbound COM callbacks until the whole queue drains.
+
+### Loop failure after start
+
+An exception thrown by the loop after `Start()` returned — `WaitForWorkOrMessages` reporting `WAIT_FAILED`, or an exception escaping a dispatched message — is not a startup failure, and nothing reads `startupException` after `Start()`. `ThreadMain` instead stores it as `LoopFault` under `gate`, so every later `InvokeAsync` fails immediately with an `InvalidOperationException` whose inner exception is the loop fault, and work already queued is cancelled by the `finally` block. It then invokes the handler registered with `RegisterLoopFaultHandler`; a handler registered after the loop died runs immediately on the registering thread. Neither happens when `Shutdown` was already requested, because a loop that throws during a deliberate teardown is not a worker fault. `MxAccessStaSession` registers a handler that records a `StaHung` `WorkerFault` on the event queue, which the IPC drain loop sends to the gateway before ending the pipe session and the worker process.
 
 ## Command Dispatch
 

docs/plans/2026-06-14-deferred-followups.md

@@ -1,7 +1,7 @@
 # Deferred Follow-ups Implementation Plan
 
 **Date:** 2026-06-14
-**Status:** D1 executed (commit 4af24b9 — `mxgateway.alarms.provider_switches` emitted in `DashboardSnapshotService.cs:198`). D2 resolved as no-op (see resolution section below). D3–D5 remain pending (ops/validation, no code).
+**Status:** D1 executed (commit 4af24b9 — `mxgateway.alarms.provider_switches` emitted in `DashboardSnapshotService.cs:198`). D2 resolved as no-op (see resolution section below). D3 (ForceSubtag validation) was exercised through the forced-subtag work (`2026-06-15-forced-subtag-mode-fix.md`); D4 (windev backup prune) has no completion record in the repo and is treated as pending; D5 (wonder redeploy) was superseded by later wonder deployments. D3–D5 are ops/validation with no code.
 **Context:** After the alarm-subtag-fallback cleanup (merged `5976770`) and its redeploy to
 windev (10.100.0.48), five items remain deferred. This plan handles all five. They are
 independent — execute in any order, or cherry-pick. Items D1–D2 are code (branch off `main`);

gateway.md

@@ -140,8 +140,9 @@ one gateway-owned worker session subscribes the configured AVEVA alarm
 provider, caches the active-alarm set (reconciled periodically against the
 worker's snapshot), and fans it out to every client through the session-less
 `StreamAlarms` RPC — the stream opens with the current active-alarm snapshot,
-then streams live transitions. `AcknowledgeAlarm` is session-less and routes
-through the monitor. Clients never open a worker session to see alarms, and
+then streams live transitions. `AcknowledgeAlarm` is session-less, routes
+through the monitor, and carries the operator's credentials (see
+"Acknowledge" below). Clients never open a worker session to see alarms, and
 alarm monitoring is independent of client lifecycle; the monitor re-opens its
 session if the worker faults. Gated by `MxGateway:Alarms:Enabled` — see
 `docs/DesignDecisions.md` for why this reverses the v1 single-subscriber rule
@@ -221,20 +222,22 @@ forwards only events the worker emits and never synthesizes transitions
 itself. The synthesis rules are documented in
 `docs/AlarmClientDiscovery.md`.
 
-**Acknowledge in subtag mode:** the ack-by-name path writes the operator
-comment to the alarm attribute's ack-comment subtag. The write performs the
-ack. If the attribute has no writable ack-comment subtag configured, the RPC
-returns `FailedPrecondition`. In alarm-manager mode, `AlarmAckByName` is
-used as before.
-
-**Acceptance is not application:** the ack is forwarded to the provider, and a
-successful return means the provider accepted the call, not that the
-acknowledgement was applied. On galaxies whose alarmed attributes carry a
-non-free-access security classification, `AlarmAckByName` returns `rc=0` and the
-alarm stays `UNACK_ALM` — it carries an operator name, not an authenticated
-identity. (Observed on the probe rig; the mechanism is inferred.) Confirm an ack
-by the resulting transition, never by the return code. See
-`docs/AlarmProbeFindings.md`.
+**Acknowledge:** the gateway acknowledges an alarm by writing the operator's
+comment to the alarm attribute's `AckMsg` (subtag mode: the configured
+ack-comment subtag) over MXAccess, after `AuthenticateUser` as the requesting
+operator (`operator_user`, `operator_password`, optional `operator_domain`).
+The AVEVA runtime treats that write as the acknowledgement and historizes it
+with the authenticated account's name, domain and full name; the Historian's
+OperatorNode is the ack writer's MXAccess client name
+(`MxGateway:Alarms:AckClientName`). The worker waits a bounded time for
+`OnWriteComplete`: an OK status is reported as success, a non-OK status as
+`MxaccessFailure` with the status detail, and no completion as success with a
+diagnostic noting the write was sent but unconfirmed. An unknown alarm or one
+with no ack target (non-Galaxy provider, no ack-comment subtag) is reported as
+`InvalidRequest` with `hresult` -1. Alarm-client ack calls
+(`AlarmAckByName`/`AlarmAckByGUID`, the managed `AlarmClient`, WNAL) are not
+used: live tests showed a headless consumer's acks are accepted and silently
+dropped (see `docs/AlarmProbeFindings.md`).
 
 **Degraded state visibility:** every subtag-mode transition carries
 `degraded = true` and `source_provider = ALARM_PROVIDER_MODE_SUBTAG` on the
@@ -443,7 +446,9 @@ Rules:
   enforcement is ever added, this is the property it will rely on.
 - `correlation_id` links a command to its reply; it is authoritative on the
   envelope, and the inner `MxCommandReply.correlation_id` echoes it for
-  MXAccess parity.
+  MXAccess parity. The envelope id never reaches public clients: the gateway
+  overwrites the public `MxCommandReply.correlation_id` with the request's
+  `client_correlation_id` (empty when the client sent none).
 - Events carry their own correlation id or leave it empty.
 - Replies must preserve MXAccess HRESULT/status information even when the
   command is also represented as a protocol-level failure.
@@ -566,8 +571,15 @@ returns one result per requested entry — per-entry failures populate
   advised (`was_cached = true`, no subscription side-effects), or by
   taking a full `AddItem` + `Advise` + wait-for-first-OnDataChange +
   `UnAdvise` + `RemoveItem` snapshot lifecycle when no live subscription
-  exists (`was_cached = false`). Per-tag timeouts surface as
-  `was_successful = false` rather than throwing. The cache lives on the
+  exists (`was_cached = false`). `timeout_ms` is one shared deadline for
+  the whole command, not a per-tag budget: the worker adds and advises
+  every uncached tag first, waits once for all of them, then tears them
+  all down, so the STA is held for at most one `timeout_ms` (plus the
+  add/advise and teardown COM calls, during which it pumps every 16 tags).
+  Each tag that has not delivered by the deadline surfaces its own
+  `was_successful = false` timeout rather than throwing. Duplicate
+  addresses that MXAccess resolves to the same item handle share one
+  subscription and receive the same result. The cache lives on the
   worker's `MxAccessValueCache`, populated by `MxAccessBaseEventSink` on
   every `OnDataChange` after the event clears the outbound queue.
   Required scope: `invoke:read`. `timeout_ms == 0` uses the worker's
@@ -1028,8 +1040,9 @@ Internal worker IPC:
 - gateway validates worker executable path and version,
 - no secrets in command line when avoidable.
 
-Credential-sensitive commands such as `AuthenticateUser` and `WriteSecured`
-must not log passwords or raw credential values.
+Credential-sensitive commands such as `AuthenticateUser`, `WriteSecured` and
+`AcknowledgeAlarm` (its `operator_password`) must not log passwords or raw
+credential values.
 
 ## Observability
 

src/ZB.MOM.WW.MxGateway.Contracts/Generated/MxaccessWorker.cs

@@ -132,6 +132,12 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
 
   public enum WorkerFaultCategory {
     [pbr::OriginalName("WORKER_FAULT_CATEGORY_UNSPECIFIED")] Unspecified = 0,
+    /// <summary>
+    /// Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
+    /// arguments, gateway authentication) exit via WorkerExitCode before the pipe
+    /// exists, so there is no channel to carry a fault. Names kept so the numbers
+    /// are not reused.
+    /// </summary>
     [pbr::OriginalName("WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS")] InvalidArguments = 1,
     [pbr::OriginalName("WORKER_FAULT_CATEGORY_GATEWAY_AUTHENTICATION_FAILED")] GatewayAuthenticationFailed = 2,
     [pbr::OriginalName("WORKER_FAULT_CATEGORY_PROTOCOL_MISMATCH")] ProtocolMismatch = 3,

src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto

@@ -391,10 +391,13 @@ message AlarmFailoverConfig {
   int32 failback_stable_probes = 3;          // clean probes before switching back (>=1)
 }
 
-// Acknowledge a single alarm by its GUID. Operator identity fields are
-// recorded atomically with the ack transition in the alarm-history log.
-// The reply's hresult / native_status surfaces AVEVA's
-// AlarmAckByGUID return code.
+// Acknowledge a single alarm by its GUID. The worker resolves the GUID to the
+// alarm's Galaxy reference from its active-alarm snapshot and acknowledges it
+// by writing `comment` to `<TAGNAME>.AckMsg` over MXAccess after
+// AuthenticateUser as the operator, so the alarm history records the real
+// operator. The reply's hresult / native_status is 0 when the write was
+// applied (or sent but unconfirmed), otherwise the AuthenticateUser HRESULT or
+// the OnWriteComplete status detail.
 message AcknowledgeAlarmCommand {
   // Canonical 8-4-4-4-12 GUID string (e.g. "BCC47053-9542-4D65-BDAA-BCDEA6A32A73").
   string alarm_guid = 1;
@@ -402,7 +405,14 @@ message AcknowledgeAlarmCommand {
   string operator_user = 3;
   string operator_node = 4;
   string operator_domain = 5;
+  // Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
+  // AlarmAckByName that carried it returns -55. operator_domain IS used (the
+  // AuthenticateUser identity is "<operator_domain>\<operator_user>").
   string operator_full_name = 6;
+  // Operator credential used for AuthenticateUser before the AckMsg write.
+  // Implementations must keep this field out of logs, metrics labels, command
+  // lines, and diagnostics.
+  string operator_password = 7;
 }
 
 // Snapshot the currently-active alarm set. Optional filter prefix scopes
@@ -414,9 +424,7 @@ message QueryActiveAlarmsCommand {
 
 // Acknowledge a single alarm by its (name, provider, group) tuple. Used
 // when the public RPC's AlarmFullReference (Provider!Group.Tag) cannot
-// be resolved to a GUID directly. The worker invokes
-// wwAlarmConsumerClass.AlarmAckByName which reaches the same alarm
-// history path as AlarmAckByGUID.
+// be resolved to a GUID directly. The worker acknowledges by writing `comment` to `<alarm_name>.AckMsg` (Galaxy provider only) after AuthenticateUser as the operator.
 message AcknowledgeAlarmByNameCommand {
   // Tag/alarm name (e.g. "TestMachine_001.TestAlarm001"). Tag itself
   // may contain dots; the gateway-side parser splits on the first dot
@@ -430,7 +438,14 @@ message AcknowledgeAlarmByNameCommand {
   string operator_user = 5;
   string operator_node = 6;
   string operator_domain = 7;
+  // Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
+  // AlarmAckByName that carried it returns -55. operator_domain IS used (the
+  // AuthenticateUser identity is "<operator_domain>\<operator_user>").
   string operator_full_name = 8;
+  // Operator credential used for AuthenticateUser before the AckMsg write.
+  // Implementations must keep this field out of logs, metrics labels, command
+  // lines, and diagnostics.
+  string operator_password = 9;
 }
 
 message UnsubscribeBulkCommand {
@@ -509,11 +524,15 @@ message WriteSecured2BulkEntry {
 //     reply returns the cached value WITHOUT modifying the existing
 //     subscription (was_cached = true).
 //   - Otherwise the worker takes the snapshot lifecycle itself: AddItem +
-//     Advise, wait up to `timeout_ms` for the first OnDataChange, then
-//     UnAdvise + RemoveItem before returning. The session is left exactly
-//     as it was before the call (was_cached = false).
+//     Advise, wait for the first OnDataChange, then UnAdvise + RemoveItem
+//     before returning. The session is left exactly as it was before the
+//     call (was_cached = false).
 //
-// `timeout_ms == 0` uses the gateway-configured default (1000 ms).
+// `timeout_ms` is one deadline shared by the whole command, not a per-tag
+// budget: the worker adds and advises every uncached tag, waits once for all
+// of them, then tears them all down. A tag that has not delivered by the
+// deadline gets its own was_successful = false result.
+// `timeout_ms == 0` uses the worker's default (1000 ms).
 message ReadBulkCommand {
   int32 server_handle = 1;
   repeated string tag_addresses = 2;
@@ -540,6 +559,7 @@ message ShutdownWorkerCommand {
 
 message MxCommandReply {
   string session_id = 1;
+  // Echoes the request's client_correlation_id.
   string correlation_id = 2;
   MxCommandKind kind = 3;
   ProtocolStatus protocol_status = 4;
@@ -725,11 +745,11 @@ message DrainEventsReply {
 // Reply payload for AcknowledgeAlarmCommand AND
 // AcknowledgeAlarmByNameCommand — both ack command kinds reuse this
 // payload case (`MxCommandReply.acknowledge_alarm`); there is no
-// dedicated by-name reply case. Surfaces AVEVA's native ack return
-// code (AlarmAckByGUID for the GUID arm, AlarmAckByName for the
-// by-name arm); 0 means success. The MxCommandReply's hresult field
+// dedicated by-name reply case. Surfaces the ack outcome: 0 when the
+// AckMsg write was applied or sent but unconfirmed; otherwise the
+// AuthenticateUser HRESULT or the OnWriteComplete status detail. The MxCommandReply's hresult field
 // carries the same value and is preferred for protocol consumers —
-// this payload exists so the gateway-side WorkerAlarmRpcDispatcher
+// this payload exists so GatewayAlarmMonitor
 // can echo native_status into AcknowledgeAlarmReply.hresult without
 // unpacking the outer envelope.
 message AcknowledgeAlarmReplyPayload {
@@ -871,7 +891,7 @@ message OnAlarmTransitionEvent {
 
   // When the alarm originally entered the active state. Preserved across
   // acknowledge transitions so the Part 9 condition keeps the original raise
-  // time. Unset on retrigger from a previously-cleared condition.
+  // time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
   google.protobuf.Timestamp original_raise_timestamp = 6;
 
   // When this specific transition occurred (raise time on Raise, ack time on
@@ -921,6 +941,8 @@ enum AlarmTransitionKind {
   ALARM_TRANSITION_KIND_RAISE = 1;
   ALARM_TRANSITION_KIND_ACKNOWLEDGE = 2;
   ALARM_TRANSITION_KIND_CLEAR = 3;
+  // Reserved: never produced by the gateway or worker. Retained so the numeric
+  // value is not reused.
   ALARM_TRANSITION_KIND_RETRIGGER = 4;
 }
 
@@ -987,6 +1009,15 @@ message AcknowledgeAlarmRequest {
   // Operator principal performing the acknowledgement. The lmxopcua side
   // resolves this from the OPC UA session prior to invoking the RPC.
   string operator_user = 5;
+  // Operator's Galaxy (or OS) password. The gateway authenticates as
+  // operator_user with it before writing the acknowledgement so the alarm
+  // history records the real operator. May be empty for accounts without a
+  // password. Credential-sensitive: implementations must keep this field out
+  // of logs, metrics labels, command lines, and diagnostics.
+  string operator_password = 6;
+  // Optional domain for OS-authenticated galaxies. When set the gateway
+  // authenticates as "<operator_domain>\<operator_user>".
+  string operator_domain = 7;
 }
 
 message AcknowledgeAlarmReply {
@@ -997,8 +1028,7 @@ message AcknowledgeAlarmReply {
   ProtocolStatus protocol_status = 3;
   // Native ack return code echoed from the worker. The worker carries the
   // ack outcome as a single int32 (AcknowledgeAlarmReplyPayload.native_status,
-  // = AlarmAckByName / AlarmAckByGUID return code; 0 = success); the gateway's
-  // WorkerAlarmRpcDispatcher copies that value here. This is the authoritative
+  // 0 = applied or sent; otherwise the AuthenticateUser HRESULT or OnWriteComplete detail); GatewayAlarmMonitor copies that value here. This is the authoritative
   // ack-outcome field for the public RPC. Absent only when the worker reply
   // omitted the value entirely (a protocol violation).
   optional int32 hresult = 4;

src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto

@@ -127,6 +127,10 @@ enum WorkerState {
 
 enum WorkerFaultCategory {
   WORKER_FAULT_CATEGORY_UNSPECIFIED = 0;
+  // Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
+  // arguments, gateway authentication) exit via WorkerExitCode before the pipe
+  // exists, so there is no channel to carry a fault. Names kept so the numbers
+  // are not reused.
   WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;
   WORKER_FAULT_CATEGORY_GATEWAY_AUTHENTICATION_FAILED = 2;
   WORKER_FAULT_CATEGORY_PROTOCOL_MISMATCH = 3;

src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs

@@ -22,4 +22,10 @@ public static class WorkerLaunchEnvironment
 
     /// <summary>Maximum alarm records requested in one current-alarm fetch.</summary>
     public const string MaxAlarmsPerFetch = "MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH";
+
+    /// <summary>
+    ///     MXAccess client name the worker's alarm-ack writer registers with. The
+    ///     Historian records it as the acknowledgement's OperatorNode.
+    /// </summary>
+    public const string AlarmAckClientName = "MXGATEWAY_ALARM_ACK_CLIENT_NAME";
 }

src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs

@@ -1052,6 +1052,231 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
         }
     }
 
+    /// <summary>
+    /// TST-05: the real worker answers the four non-terminating control commands off-STA in
+    /// <c>WorkerPipeSession.HandleControlCommandAsync</c>. The default suite only ever sees the
+    /// FakeWorkerHarness canned replies, so this drives <c>Ping</c>, <c>GetSessionState</c>,
+    /// <c>GetWorkerInfo</c>, and <c>DrainEvents</c> through the gateway against a live worker and
+    /// asserts each reply carries real worker state rather than an <c>INVALID_REQUEST</c>.
+    /// </summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [LiveMxAccessFact]
+    public async Task GatewaySession_WithLiveWorker_ControlCommands_ReturnRealWorkerState()
+    {
+        string workerExecutablePath = IntegrationTestEnvironment.ResolveLiveMxAccessWorkerExecutablePath();
+        Assert.True(
+            File.Exists(workerExecutablePath),
+            $"Live MXAccess worker executable was not found at {workerExecutablePath}. Build the worker or set {IntegrationTestEnvironment.LiveMxAccessWorkerExecutableVariableName}.");
+
+        TestWorkerProcessFactory processFactory = new(output);
+        await using GatewayServiceFixture fixture = new(workerExecutablePath, processFactory, output);
+
+        string? sessionId = null;
+
+        try
+        {
+            OpenSessionReply openReply = await fixture.Service.OpenSession(
+                new OpenSessionRequest
+                {
+                    ClientSessionName = "live-mxaccess-control",
+                    ClientCorrelationId = "live-open-control",
+                    CommandTimeout = Duration.FromTimeSpan(CommandTimeout),
+                },
+                new TestServerCallContext()).ConfigureAwait(false);
+
+            sessionId = openReply.SessionId;
+            Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
+            Assert.True(openReply.WorkerProcessId > 0);
+
+            MxCommandReply pingReply = await fixture.Service.Invoke(
+                CreateControlRequest(sessionId, "live-ping", new MxCommand
+                {
+                    Kind = MxCommandKind.Ping,
+                    Ping = new PingCommand { Message = "live-ping-echo" },
+                }),
+                new TestServerCallContext()).ConfigureAwait(false);
+            LogReply("Ping", pingReply);
+            Assert.Equal(ProtocolStatusCode.Ok, pingReply.ProtocolStatus.Code);
+            Assert.Equal(MxCommandKind.Ping, pingReply.Kind);
+
+            MxCommandReply stateReply = await fixture.Service.Invoke(
+                CreateControlRequest(sessionId, "live-get-session-state", new MxCommand
+                {
+                    Kind = MxCommandKind.GetSessionState,
+                    GetSessionState = new GetSessionStateCommand(),
+                }),
+                new TestServerCallContext()).ConfigureAwait(false);
+            LogReply("GetSessionState", stateReply);
+            Assert.Equal(ProtocolStatusCode.Ok, stateReply.ProtocolStatus.Code);
+            Assert.Equal(MxCommandKind.GetSessionState, stateReply.Kind);
+            Assert.NotNull(stateReply.SessionState);
+            Assert.Equal(SessionState.Ready, stateReply.SessionState.State);
+
+            MxCommandReply infoReply = await fixture.Service.Invoke(
+                CreateControlRequest(sessionId, "live-get-worker-info", new MxCommand
+                {
+                    Kind = MxCommandKind.GetWorkerInfo,
+                    GetWorkerInfo = new GetWorkerInfoCommand(),
+                }),
+                new TestServerCallContext()).ConfigureAwait(false);
+            LogReply("GetWorkerInfo", infoReply);
+            Assert.Equal(ProtocolStatusCode.Ok, infoReply.ProtocolStatus.Code);
+            Assert.Equal(MxCommandKind.GetWorkerInfo, infoReply.Kind);
+            Assert.NotNull(infoReply.WorkerInfo);
+            output.WriteLine(
+                $"GetWorkerInfo pid={infoReply.WorkerInfo.WorkerProcessId} version={infoReply.WorkerInfo.WorkerVersion} "
+                + $"progid={infoReply.WorkerInfo.MxaccessProgid} clsid={infoReply.WorkerInfo.MxaccessClsid}");
+            Assert.Equal(openReply.WorkerProcessId, infoReply.WorkerInfo.WorkerProcessId);
+            Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.WorkerVersion));
+            Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.MxaccessProgid));
+            Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.MxaccessClsid));
+
+            // No StreamEvents subscriber is attached, so whatever the worker queued since start
+            // is still drainable; the reply must carry the DrainEvents payload (possibly empty).
+            MxCommandReply drainReply = await fixture.Service.Invoke(
+                CreateControlRequest(sessionId, "live-drain-events", new MxCommand
+                {
+                    Kind = MxCommandKind.DrainEvents,
+                    DrainEvents = new DrainEventsCommand { MaxEvents = 16 },
+                }),
+                new TestServerCallContext()).ConfigureAwait(false);
+            LogReply("DrainEvents", drainReply);
+            Assert.Equal(ProtocolStatusCode.Ok, drainReply.ProtocolStatus.Code);
+            Assert.Equal(MxCommandKind.DrainEvents, drainReply.Kind);
+            Assert.Equal(MxCommandReply.PayloadOneofCase.DrainEvents, drainReply.PayloadCase);
+            Assert.True(drainReply.DrainEvents.Events.Count <= 16);
+            output.WriteLine($"DrainEvents drained={drainReply.DrainEvents.Events.Count}");
+
+            // A control command must not disturb the session: it is still Ready afterwards.
+            Assert.True(fixture.TryGetSession(sessionId, out GatewaySession? session));
+            Assert.Equal(SessionState.Ready, session!.State);
+        }
+        finally
+        {
+            await ShutDownAsync(fixture, processFactory, sessionId, streamTask: null).ConfigureAwait(false);
+        }
+    }
+
+    /// <summary>
+    /// TST-05: <c>ShutdownWorker</c> writes its OK reply before the worker shuts down, so the
+    /// reply reaches the caller (a reply written after the pipe closed would surface as a
+    /// pipe-disconnected failure instead), the worker process then exits on its own, and a later
+    /// command on the session does not succeed. Uses its own fixture and worker because it
+    /// terminates the worker.
+    /// </summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [LiveMxAccessFact]
+    public async Task GatewaySession_WithLiveWorker_ShutdownWorker_RepliesOkThenWorkerExits()
+    {
+        string workerExecutablePath = IntegrationTestEnvironment.ResolveLiveMxAccessWorkerExecutablePath();
+        Assert.True(
+            File.Exists(workerExecutablePath),
+            $"Live MXAccess worker executable was not found at {workerExecutablePath}. Build the worker or set {IntegrationTestEnvironment.LiveMxAccessWorkerExecutableVariableName}.");
+
+        TestWorkerProcessFactory processFactory = new(output);
+        await using GatewayServiceFixture fixture = new(workerExecutablePath, processFactory, output);
+        using RecordingServerStreamWriter<MxEvent> eventWriter = new();
+
+        string? sessionId = null;
+        Task? streamTask = null;
+        using CancellationTokenSource streamCancellation = new();
+
+        try
+        {
+            OpenSessionReply openReply = await fixture.Service.OpenSession(
+                new OpenSessionRequest
+                {
+                    ClientSessionName = "live-mxaccess-shutdown-worker",
+                    ClientCorrelationId = "live-open-shutdown-worker",
+                    CommandTimeout = Duration.FromTimeSpan(CommandTimeout),
+                },
+                new TestServerCallContext()).ConfigureAwait(false);
+
+            sessionId = openReply.SessionId;
+            Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
+
+            // The gateway observes a worker's pipe closing through the event-stream read loop
+            // (the same path the abnormal-exit fact relies on); without a subscriber the session
+            // would sit in Ready until the heartbeat grace expired.
+            streamTask = fixture.Service.StreamEvents(
+                new StreamEventsRequest { SessionId = sessionId },
+                eventWriter,
+                new TestServerCallContext(streamCancellation.Token));
+
+            MxCommandReply shutdownReply = await fixture.Service.Invoke(
+                CreateControlRequest(sessionId, "live-shutdown-worker", new MxCommand
+                {
+                    Kind = MxCommandKind.ShutdownWorker,
+                    ShutdownWorker = new ShutdownWorkerCommand
+                    {
+                        GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(5)),
+                    },
+                }),
+                new TestServerCallContext()).ConfigureAwait(false);
+            LogReply("ShutdownWorker", shutdownReply);
+            Assert.Equal(ProtocolStatusCode.Ok, shutdownReply.ProtocolStatus.Code);
+            Assert.Equal(MxCommandKind.ShutdownWorker, shutdownReply.Kind);
+
+            // The worker exits on its own — no CloseSession, no kill. WaitForProcessesAsync
+            // throws on timeout, which fails the test.
+            await processFactory.WaitForProcessesAsync(StreamShutdownTimeout).ConfigureAwait(false);
+
+            // The session can no longer serve commands once its worker is gone: either the
+            // gateway rejects the call outright or the reply is not Ok.
+            MxCommandReply? afterReply = null;
+            Exception? afterException = null;
+            try
+            {
+                afterReply = await fixture.Service.Invoke(
+                    CreateControlRequest(sessionId, "live-ping-after-shutdown", new MxCommand
+                    {
+                        Kind = MxCommandKind.Ping,
+                        Ping = new PingCommand(),
+                    }),
+                    new TestServerCallContext()).ConfigureAwait(false);
+                LogReply("PingAfterShutdown", afterReply);
+            }
+            catch (Exception exception)
+            {
+                afterException = exception;
+                output.WriteLine($"PingAfterShutdown rejected: {exception.GetType().Name}: {exception.Message}");
+            }
+
+            Assert.True(
+                afterException is not null || afterReply!.ProtocolStatus.Code != ProtocolStatusCode.Ok,
+                "A command sent after ShutdownWorker must not succeed against the exited worker.");
+
+            // Logged, not asserted: the gateway's WorkerClient moves to Closed on the shutdown
+            // ack, but GatewaySession.State is not driven from that transition today, so the
+            // public session state still reads Ready here (observed live on windev). That is a
+            // gateway-side gap, not a worker one — this fact pins the worker half of the
+            // contract: OK reply first, then a clean exit.
+            string observedState = fixture.TryGetSession(sessionId, out GatewaySession? session)
+                ? session.State.ToString()
+                : "<removed>";
+            output.WriteLine($"ShutdownWorker observed_session_state={observedState}");
+        }
+        finally
+        {
+            // The worker is gone; a CloseSession round-trip would only log a cleanup failure.
+            streamCancellation.Cancel();
+            await ShutDownAsync(fixture, processFactory, sessionId: null, streamTask).ConfigureAwait(false);
+        }
+    }
+
+    private static MxCommandRequest CreateControlRequest(
+        string sessionId,
+        string correlationId,
+        MxCommand command)
+    {
+        return new MxCommandRequest
+        {
+            SessionId = sessionId,
+            ClientCorrelationId = correlationId,
+            Command = command,
+        };
+    }
+
     /// <summary>
     /// Closes the session and drains the event stream / worker processes without letting a
     /// cleanup timeout mask the original failure from the test body.
@@ -1704,6 +1929,20 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
         /// </summary>
         public WriteOptions? WriteOptions { get; set; }
 
+        /// <summary>
+        /// Records the message unless the token is already cancelled.
+        /// </summary>
+        /// <param name="message">The message to write.</param>
+        /// <param name="cancellationToken">Token that cancels the write.</param>
+        /// <returns>A task that represents the asynchronous operation.</returns>
+        public Task WriteAsync(T message, CancellationToken cancellationToken)
+        {
+            // The interface default throws NotSupportedException for a cancelable token; the
+            // gateway passes the call's token on every streamed write.
+            cancellationToken.ThrowIfCancellationRequested();
+            return WriteAsync(message);
+        }
+
         /// <summary>
         /// Records the message and signals any pending waiter.
         /// </summary>
@@ -2112,26 +2351,26 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
     private sealed class AllowAllConstraintEnforcer : IConstraintEnforcer
     {
         /// <inheritdoc />
-        public Task<ConstraintFailure?> CheckReadTagAsync(
+        public ValueTask<ConstraintFailure?> CheckReadTagAsync(
             ApiKeyIdentity? identity,
             string tagAddress,
-            CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
+            CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
 
         /// <inheritdoc />
-        public Task<ConstraintFailure?> CheckReadHandleAsync(
+        public ValueTask<ConstraintFailure?> CheckReadHandleAsync(
             ApiKeyIdentity? identity,
             GatewaySession session,
             int serverHandle,
             int itemHandle,
-            CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
+            CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
 
         /// <inheritdoc />
-        public Task<ConstraintFailure?> CheckWriteHandleAsync(
+        public ValueTask<ConstraintFailure?> CheckWriteHandleAsync(
             ApiKeyIdentity? identity,
             GatewaySession session,
             int serverHandle,
             int itemHandle,
-            CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
+            CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
 
         /// <inheritdoc />
         public Task RecordDenialAsync(
@@ -2141,6 +2380,14 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
             ConstraintFailure failure,
             string? correlationId,
             CancellationToken cancellationToken) => Task.CompletedTask;
+
+        /// <inheritdoc />
+        public Task RecordBulkDenialAsync(
+            ApiKeyIdentity? identity,
+            string commandKind,
+            ConstraintDenialSummary summary,
+            string? correlationId,
+            CancellationToken cancellationToken) => Task.CompletedTask;
     }
 
 }

src/ZB.MOM.WW.MxGateway.Server/Alarms/AlarmFeedSubscriberOverflowException.cs

@@ -0,0 +1,39 @@
+namespace ZB.MOM.WW.MxGateway.Server.Alarms;
+
+/// <summary>
+///     Completes an alarm-feed subscriber that fell behind its bounded queue and was dropped.
+/// </summary>
+/// <remarks>
+///     A typed exception rather than a bare <see cref="InvalidOperationException"/> so the gRPC
+///     layer can map it to <c>ResourceExhausted</c> — the status a <c>StreamEvents</c> overflow
+///     carries — without treating a slow client as a gateway failure (an unrecognized exception is
+///     logged as a warning with its stack and surfaced as <c>Unavailable</c>). The remedy is the
+///     same as for an event-stream overflow: reconnect, which re-snapshots the alarm set.
+/// </remarks>
+public sealed class AlarmFeedSubscriberOverflowException : Exception
+{
+    /// <summary>The message every dropped subscriber is completed with.</summary>
+    public const string DefaultMessage =
+        "Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot.";
+
+    /// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
+    public AlarmFeedSubscriberOverflowException()
+        : base(DefaultMessage)
+    {
+    }
+
+    /// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
+    /// <param name="message">The error message.</param>
+    public AlarmFeedSubscriberOverflowException(string message)
+        : base(message)
+    {
+    }
+
+    /// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
+    /// <param name="message">The error message.</param>
+    /// <param name="innerException">The exception that caused this one.</param>
+    public AlarmFeedSubscriberOverflowException(string message, Exception innerException)
+        : base(message, innerException)
+    {
+    }
+}

src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs

@@ -967,7 +967,9 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
     }
 
     // Caller holds _sync. Pushes a feed message to every matching subscriber;
-    // a subscriber that has fallen behind is completed with an error and dropped.
+    // a subscriber that has fallen behind is completed with an AlarmFeedSubscriberOverflowException
+    // and dropped. The typed fault is what lets StreamAlarms answer ResourceExhausted instead of
+    // treating a slow client as a gateway failure.
     private void Broadcast(AlarmFeedMessage message, string reference)
     {
         for (int index = _subscribers.Count - 1; index >= 0; index--)
@@ -980,8 +982,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
 
             if (!subscriber.Channel.Writer.TryWrite(message))
             {
-                subscriber.Channel.Writer.TryComplete(new InvalidOperationException(
-                    "Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."));
+                subscriber.Channel.Writer.TryComplete(new AlarmFeedSubscriberOverflowException());
                 _subscribers.RemoveAt(index);
             }
         }
@@ -997,8 +998,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
             Subscriber subscriber = _subscribers[index];
             if (!subscriber.Channel.Writer.TryWrite(message))
             {
-                subscriber.Channel.Writer.TryComplete(new InvalidOperationException(
-                    "Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."));
+                subscriber.Channel.Writer.TryComplete(new AlarmFeedSubscriberOverflowException());
                 _subscribers.RemoveAt(index);
             }
         }
@@ -1046,9 +1046,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
             _subscribers.Add(subscriber);
             providerStatus = BuildProviderStatus();
             snapshotTruncated = _snapshotTruncated;
-            snapshot = _alarms.Values
-                .Where(alarm => prefix.Length == 0
-                    || alarm.AlarmFullReference.StartsWith(prefix, StringComparison.Ordinal))
+
+            // Take the memoized projection rather than cloning the whole set per attach: the
+            // clone runs at most once per alarm-set generation (shared with CurrentAlarms), so a
+            // burst of reconnecting subscribers no longer holds the broadcast lock for one full
+            // copy each. The array is never mutated once built — a write to _alarms replaces it
+            // with null — so capturing the reference here pins exactly the generation this
+            // subscriber registered against, which is what keeps snapshot-then-live ordering.
+            snapshot = _currentAlarmsProjection ??= _alarms.Values
                 .Select(alarm => alarm.Clone())
                 .ToArray();
         }
@@ -1068,8 +1073,15 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
                 SnapshotStatus = new AlarmSnapshotStatus { Truncated = snapshotTruncated },
             };
 
+            // The prefix filter runs here, outside _sync, over the immutable projection.
             foreach (ActiveAlarmSnapshot alarm in snapshot)
             {
+                if (prefix.Length != 0
+                    && !alarm.AlarmFullReference.StartsWith(prefix, StringComparison.Ordinal))
+                {
+                    continue;
+                }
+
                 yield return new AlarmFeedMessage { ActiveAlarm = alarm };
             }
 
@@ -1169,7 +1181,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
         return string.Empty;
     }
 
-    private static MxCommand? BuildAcknowledgeCommand(AcknowledgeAlarmRequest request, out string? parseError)
+    internal static MxCommand? BuildAcknowledgeCommand(AcknowledgeAlarmRequest request, out string? parseError)
     {
         parseError = null;
         if (string.IsNullOrWhiteSpace(request.AlarmFullReference))
@@ -1180,6 +1192,8 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
 
         string comment = request.Comment ?? string.Empty;
         string operatorUser = request.OperatorUser ?? string.Empty;
+        string operatorPassword = request.OperatorPassword ?? string.Empty;
+        string operatorDomain = request.OperatorDomain ?? string.Empty;
 
         if (Guid.TryParse(request.AlarmFullReference, out Guid guid))
         {
@@ -1192,7 +1206,8 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
                     Comment = comment,
                     OperatorUser = operatorUser,
                     OperatorNode = string.Empty,
-                    OperatorDomain = string.Empty,
+                    OperatorDomain = operatorDomain,
+                    OperatorPassword = operatorPassword,
                     OperatorFullName = string.Empty,
                 },
             };
@@ -1211,7 +1226,8 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
                     Comment = comment,
                     OperatorUser = operatorUser,
                     OperatorNode = string.Empty,
-                    OperatorDomain = string.Empty,
+                    OperatorDomain = operatorDomain,
+                    OperatorPassword = operatorPassword,
                     OperatorFullName = string.Empty,
                 },
             };

src/ZB.MOM.WW.MxGateway.Server/Configuration/AlarmsOptions.cs

@@ -73,6 +73,13 @@ public sealed class AlarmsOptions
     /// </summary>
     public int MaxAlarmsPerFetch { get; init; } = 1024;
 
+    /// <summary>
+    ///     MXAccess client name the alarm worker's ack writer registers with. The
+    ///     AVEVA Historian records it as each acknowledgement's OperatorNode.
+    ///     Conveyed to workers via MXGATEWAY_ALARM_ACK_CLIENT_NAME.
+    /// </summary>
+    public string AckClientName { get; init; } = "MxGateway";
+
     /// <summary>
     ///     Configuration for the alarm-manager ↔ subtag fallback mechanism:
     ///     operating mode, failure-detection thresholds, discovery, and subtag

src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptions.cs

@@ -52,4 +52,7 @@ public sealed class GatewayOptions
     /// login / API-key rate limiting).
     /// </summary>
     public SecurityOptions Security { get; init; } = new();
+
+    /// <summary>Gets Prometheus <c>/metrics</c> endpoint options.</summary>
+    public MetricsOptions Metrics { get; init; } = new();
 }

src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs

@@ -325,6 +325,10 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
             options.FaultedGraceSeconds,
             "MxGateway:Sessions:FaultedGraceSeconds must be zero or greater (0 reaps a faulted session on the next sweep).",
             builder);
+        AddIfNegative(
+            options.UnattachedSessionGraceSeconds,
+            "MxGateway:Sessions:UnattachedSessionGraceSeconds must be zero or greater (0 disables reaping of never-subscribed sessions).",
+            builder);
         AddIfNegative(
             options.WorkerReadyWaitTimeoutMs,
             "MxGateway:Sessions:WorkerReadyWaitTimeoutMs must be greater than or equal to zero.",
@@ -488,6 +492,13 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
                 $"MxGateway:Alarms:MaxAlarmsPerFetch must be between {MinimumMaxAlarmsPerFetch} and {MaximumMaxAlarmsPerFetch}.");
         }
 
+        if (string.IsNullOrWhiteSpace(options.AckClientName)
+            || options.AckClientName.Length > 64
+            || options.AckClientName.Any(char.IsControl))
+        {
+            builder.Add("MxGateway:Alarms:AckClientName must be 1-64 printable characters.");
+        }
+
         if (!options.Enabled)
         {
             return;

src/ZB.MOM.WW.MxGateway.Server/Configuration/MetricsOptions.cs

@@ -0,0 +1,15 @@
+namespace ZB.MOM.WW.MxGateway.Server.Configuration;
+
+/// <summary>Options for the Prometheus <c>/metrics</c> scrape endpoint (<c>MxGateway:Metrics</c>).</summary>
+public sealed class MetricsOptions
+{
+    /// <summary>
+    /// Gets a value indicating whether <c>/metrics</c> requires an authenticated caller. When
+    /// <see langword="true"/>, a scrape must present either an API key carrying the <c>admin</c>
+    /// scope (<c>Authorization: Bearer mxgw_…</c>) or a dashboard login cookie holding a real
+    /// dashboard role; anonymous localhost access does not qualify. Default <see langword="false"/>
+    /// keeps the endpoint anonymous for existing scrapers. The <c>/health</c> endpoints stay
+    /// anonymous either way.
+    /// </summary>
+    public bool RequireAuthentication { get; init; }
+}

src/ZB.MOM.WW.MxGateway.Server/Configuration/SessionOptions.cs

@@ -57,6 +57,28 @@ public sealed class SessionOptions
     /// </summary>
     public int FaultedGraceSeconds { get; init; }
 
+    /// <summary>
+    ///     Gets the idle window, in seconds, after which the lease monitor reaps a session that
+    ///     has <em>never</em> attached an event-stream subscriber. A client that opens a session
+    ///     and vanishes before it ever streams (for example OtOpcUa's <c>GalaxyDriver</c> failing
+    ///     part-way through its start-up) otherwise pins a session slot and a live x86 worker for
+    ///     the full <see cref="DefaultLeaseSeconds"/> (30 minutes by default). The window is
+    ///     measured from the later of the session becoming Ready and its most recent unary
+    ///     command, so a slow worker start does not eat it and a client that only ever reads and
+    ///     writes is not reaped while it keeps calling; it stops applying the moment the session's
+    ///     first subscriber — external, or the gateway's own internal alarm-monitor subscriber —
+    ///     attaches (from then on <see cref="DetachGraceSeconds"/> and the normal lease govern
+    ///     it). A value of <c>0</c> (the default) disables the sweep. Must be greater than or
+    ///     equal to zero.
+    /// </summary>
+    /// <remarks>
+    ///     Set it comfortably above the time a well-behaved client needs between
+    ///     <c>OpenSession</c> and its first <c>StreamEvents</c>, and remember the close lands on
+    ///     the next sweep after the window elapses (up to <see cref="LeaseSweepIntervalSeconds"/>
+    ///     later).
+    /// </remarks>
+    public int UnattachedSessionGraceSeconds { get; init; }
+
     /// <summary>
     /// Gets a value indicating whether multiple event subscribers are allowed per session.
     /// </summary>

src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/AlarmsPage.razor

@@ -58,6 +58,8 @@
     <MetricCard Label="Active (unacked)" Value="@_unackedCount.ToString("N0")" />
     <MetricCard Label="Acknowledged" Value="@_ackedCount.ToString("N0")" />
     <MetricCard Label="Total Active" Value="@_alarms.Count.ToString("N0")" />
+    @* FilteredAlarms() is memoized on (alarm generation, filter inputs), so this card and the
+       table below share one computation per change instead of filtering twice per render. *@
     <MetricCard Label="Showing" Value="@FilteredAlarms().Count.ToString("N0")" />
 </section>
 
@@ -189,6 +191,10 @@
     private int _maxSeverity = 1000;
     private string _search = string.Empty;
 
+    // Bumped whenever _alarms is replaced, so the memoized view recomputes exactly then.
+    private readonly DashboardAlarmView _view = new();
+    private long _alarmGeneration;
+
     // Upper bound on waiting for either background loop while disposing, mirroring
     // DashboardPageBase's snapshot-watch drain: both loops marshal renders through the
     // renderer's dispatcher and disposal can run on that same dispatcher, so the wait is
@@ -295,42 +301,13 @@
         return _workerPid is int pid ? $"{refreshed} · worker pid {pid}" : refreshed;
     }
 
-    private IReadOnlyList<string> Areas()
-    {
-        return _alarms
-            .Select(alarm => alarm.Area)
-            .Where(area => !string.IsNullOrWhiteSpace(area))
-            .Distinct(StringComparer.OrdinalIgnoreCase)
-            .OrderBy(area => area, StringComparer.OrdinalIgnoreCase)
-            .ToArray();
-    }
+    private IReadOnlyList<string> Areas() => _view.Areas(_alarmGeneration, _alarms);
 
-    private IReadOnlyList<DashboardActiveAlarm> FilteredAlarms()
-    {
-        string query = _search.Trim();
-        return _alarms
-            .Where(MatchesState)
-            .Where(alarm => _areaFilter.Length == 0
-                || string.Equals(alarm.Area, _areaFilter, StringComparison.OrdinalIgnoreCase))
-            .Where(alarm => alarm.Severity >= _minSeverity && alarm.Severity <= _maxSeverity)
-            .Where(alarm => query.Length == 0
-                || alarm.Reference.Contains(query, StringComparison.OrdinalIgnoreCase)
-                || alarm.Source.Contains(query, StringComparison.OrdinalIgnoreCase)
-                || alarm.Description.Contains(query, StringComparison.OrdinalIgnoreCase))
-            .OrderByDescending(alarm => alarm.Severity)
-            .ThenByDescending(alarm => alarm.LastTransition ?? DateTimeOffset.MinValue)
-            .ToArray();
-    }
-
-    private bool MatchesState(DashboardActiveAlarm alarm)
-    {
-        return alarm.State switch
-        {
-            AlarmConditionState.Active => _showActive,
-            AlarmConditionState.ActiveAcked => _showAcked,
-            _ => true,
-        };
-    }
+    private IReadOnlyList<DashboardActiveAlarm> FilteredAlarms() =>
+        _view.Filtered(
+            _alarmGeneration,
+            new DashboardAlarmFilterCriteria(_showActive, _showAcked, _areaFilter, _minSeverity, _maxSeverity, _search),
+            _alarms);
 
     private static string StateText(AlarmConditionState state)
     {
@@ -439,6 +416,7 @@
         _lastRefresh = DateTimeOffset.UtcNow;
         _alarms.Clear();
         _alarms.AddRange(result.Alarms);
+        _alarmGeneration++;
         _unackedCount = _alarms.Count(alarm => alarm.State == AlarmConditionState.Active);
         _ackedCount = _alarms.Count(alarm => alarm.State == AlarmConditionState.ActiveAcked);
         StateHasChanged();

src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/BrowsePage.razor

@@ -4,6 +4,8 @@
 @inject IDashboardLiveDataService LiveData
 @inject IDashboardBrowseService BrowseService
 @inject IGalaxyDeployNotifier DeployNotifier
+@inject DashboardBrowseSearchIndex SearchIndex
+@inject DashboardCircuitConnectionTracker Circuit
 @using ZB.MOM.WW.GalaxyRepository
 @using ZB.MOM.WW.GalaxyRepository.Grpc
 
@@ -36,7 +38,15 @@
         }
         else if (!string.IsNullOrWhiteSpace(Search))
         {
-            @if (_searchMatches.Count == 0)
+            @if (!string.IsNullOrWhiteSpace(_searchError))
+            {
+                <div class="alert alert-danger">Search failed: @_searchError</div>
+            }
+            else if (_searchPending && _searchMatches.Count == 0)
+            {
+                <div class="empty-state">Searching&hellip;</div>
+            }
+            else if (_searchMatches.Count == 0)
             {
                 <div class="empty-state">No attributes match &ldquo;@Search&rdquo;.</div>
             }
@@ -115,6 +125,13 @@
             <div class="alert alert-danger">Live read failed: @_readError</div>
         }
 
+        @if (!string.IsNullOrWhiteSpace(_deployError))
+        {
+            <div class="alert alert-warning">
+                Galaxy redeploy notifications failed (retrying): @_deployError
+            </div>
+        }
+
         @if (_subscribed.Count == 0)
         {
             <div class="empty-state">
@@ -194,13 +211,35 @@
 @code {
     private const int SearchResultLimit = 300;
 
+    // Keystrokes inside this window coalesce into one search: typing a tag name no
+    // longer rescans the Galaxy once per character.
+    private static readonly TimeSpan SearchDebounce = TimeSpan.FromMilliseconds(150);
+
+    // Delay before resubscribing to deploy notifications after the stream ends or faults.
+    private static readonly TimeSpan DeployResubscribeDelay = TimeSpan.FromSeconds(5);
+
+    // Upper bound on waiting for the page's background loops while disposing, the same
+    // bound DashboardPageBase and AlarmsPage use: the loops marshal renders through the
+    // renderer's dispatcher and disposal can run on that same dispatcher, so the wait is
+    // bounded rather than unconditional.
+    private static readonly TimeSpan LoopDrainTimeout = TimeSpan.FromSeconds(5);
+
+    // Identifies this page to the shared live-data service, which budgets the advise set
+    // per viewer so another circuit's reads cannot evict this page's whole panel.
+    private readonly string _viewerId = "browse-" + Guid.NewGuid().ToString("N");
+
     private List<DashboardBrowseNode> _roots = [];
     private ulong _cacheSequence;
     private string? _staleBanner;
-    private CancellationTokenSource _deployCts = new();
+    private readonly CancellationTokenSource _deployCts = new();
     private Task? _deployTask;
+    private string? _deployError;
     private string _search = string.Empty;
     private IReadOnlyList<GalaxyAttribute> _searchMatches = [];
+    private bool _searchPending;
+    private string? _searchError;
+    private CancellationTokenSource? _searchCts;
+    private Task? _searchTask;
     private readonly List<string> _subscribed = [];
     private readonly Dictionary<string, DashboardTagValue> _values = new(StringComparer.Ordinal);
     // Per-tag bookkeeping for the Updated column: the signature of the value
@@ -225,8 +264,8 @@
         BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
         _roots = [.. roots.Nodes];
         _cacheSequence = roots.CacheSequence;
-        _pollTask = PollLoopAsync();
-        _deployTask = SubscribeToDeployEventsAsync();
+        _pollTask = PollLoopAsync(_cts.Token);
+        _deployTask = SubscribeToDeployEventsAsync(_deployCts.Token);
     }
 
     private async Task LoadChildrenAsync(DashboardBrowseNode node)
@@ -249,30 +288,55 @@
         await InvokeAsync(StateHasChanged);
     }
 
-    private async Task SubscribeToDeployEventsAsync()
+    // Faults are handled inside the loop: a notifier stream that faults or ends is
+    // surfaced on the page and resubscribed after a delay, so the tree does not silently
+    // stop tracking redeploys. Only cancellation ends the loop, so DrainAsync never
+    // observes a faulted task.
+    private async Task SubscribeToDeployEventsAsync(CancellationToken cancellationToken)
     {
-        try
+        while (!cancellationToken.IsCancellationRequested)
         {
-            await foreach (GalaxyDeployEventInfo info in DeployNotifier
-                .SubscribeAsync(_deployCts.Token)
-                .ConfigureAwait(false))
+            try
             {
-                // First Latest replay echoes the sequence we already projected
-                // from — skip those to avoid a spurious "redeployed" banner.
-                if (info.Sequence == 0 || (ulong)info.Sequence == _cacheSequence)
+                await foreach (GalaxyDeployEventInfo info in DeployNotifier
+                    .SubscribeAsync(cancellationToken)
+                    .ConfigureAwait(false))
                 {
-                    continue;
-                }
+                    // First Latest replay echoes the sequence we already projected
+                    // from — skip those to avoid a spurious "redeployed" banner.
+                    if (info.Sequence == 0 || (ulong)info.Sequence == _cacheSequence)
+                    {
+                        continue;
+                    }
 
-                BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
-                _roots = [.. roots.Nodes];
-                _cacheSequence = roots.CacheSequence;
-                _staleBanner = "Galaxy redeployed — tree refreshed.";
-                await InvokeAsync(StateHasChanged);
+                    await InvokeAsync(() =>
+                    {
+                        BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
+                        _roots = [.. roots.Nodes];
+                        _cacheSequence = roots.CacheSequence;
+                        _staleBanner = "Galaxy redeployed — tree refreshed.";
+                        _deployError = null;
+                        StateHasChanged();
+                    }).ConfigureAwait(false);
+                }
+            }
+            catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
+            {
+                return;
+            }
+            catch (Exception ex)
+            {
+                await ReportAsync(() => _deployError = ex.Message).ConfigureAwait(false);
+            }
+
+            try
+            {
+                await Task.Delay(DeployResubscribeDelay, cancellationToken).ConfigureAwait(false);
+            }
+            catch (OperationCanceledException)
+            {
+                return;
             }
-        }
-        catch (OperationCanceledException)
-        {
         }
     }
 
@@ -288,42 +352,101 @@
             + $"{entry.AlarmAttributeCount:N0} alarm attributes";
     }
 
+    // Bound to the filter box with @bind:event="oninput", so the setter runs once per
+    // keystroke on the render dispatcher. It only schedules the search; the scan itself
+    // runs after the debounce, off the dispatcher, and a newer keystroke cancels it.
     private string Search
     {
         get => _search;
         set
         {
             _search = value ?? string.Empty;
-            _searchMatches = ComputeSearch(_search);
+            ScheduleSearch(_search);
         }
     }
 
-    private IReadOnlyList<GalaxyAttribute> ComputeSearch(string rawQuery)
+    private void ScheduleSearch(string query)
     {
-        string query = rawQuery.Trim();
-        if (query.Length == 0)
+        CancelQuietly(_searchCts);
+        _searchCts = null;
+        _searchError = null;
+
+        if (query.Trim().Length == 0)
         {
-            return [];
+            _searchMatches = [];
+            _searchPending = false;
+            return;
         }
 
-        List<GalaxyAttribute> matches = [];
-        foreach (GalaxyObject galaxyObject in GalaxyCache.Current.Objects)
+        // Linked to the page token, so disposal cancels an in-flight search too. The
+        // search task owns the source and disposes it when it finishes.
+        CancellationTokenSource searchCts = CancellationTokenSource.CreateLinkedTokenSource(_cts.Token);
+        _searchCts = searchCts;
+        _searchPending = true;
+        _searchTask = RunSearchAsync(query, searchCts);
+    }
+
+    private async Task RunSearchAsync(string query, CancellationTokenSource searchCts)
+    {
+        CancellationToken cancellationToken = searchCts.Token;
+        try
         {
-            foreach (GalaxyAttribute attr in galaxyObject.Attributes)
+            await Task.Delay(SearchDebounce, cancellationToken).ConfigureAwait(false);
+            GalaxyHierarchyCacheEntry entry = GalaxyCache.Current;
+            IReadOnlyList<GalaxyAttribute> matches = await Task.Run(
+                    () => SearchIndex.Search(entry, query, SearchResultLimit, cancellationToken),
+                    cancellationToken)
+                .ConfigureAwait(false);
+
+            await InvokeAsync(() =>
             {
-                if (attr.FullTagReference.Contains(query, StringComparison.OrdinalIgnoreCase)
-                    || attr.AttributeName.Contains(query, StringComparison.OrdinalIgnoreCase))
+                // Only the latest query may publish: a slower, superseded scan that
+                // finishes late must not overwrite newer results.
+                if (ReferenceEquals(_searchCts, searchCts) && !cancellationToken.IsCancellationRequested)
                 {
-                    matches.Add(attr);
-                    if (matches.Count >= SearchResultLimit)
-                    {
-                        return matches;
-                    }
+                    _searchMatches = matches;
+                    _searchPending = false;
+                    StateHasChanged();
                 }
-            }
+            }).ConfigureAwait(false);
+        }
+        catch (OperationCanceledException)
+        {
+            // Superseded by a newer keystroke, or the page is going away.
+        }
+        catch (Exception ex)
+        {
+            await ReportAsync(() =>
+            {
+                if (ReferenceEquals(_searchCts, searchCts))
+                {
+                    _searchPending = false;
+                    _searchError = ex.Message;
+                }
+            }).ConfigureAwait(false);
+        }
+        finally
+        {
+            searchCts.Dispose();
+        }
+    }
+
+    // The search task disposes its own source, so a cancel can race that disposal.
+    private static void CancelQuietly(CancellationTokenSource? source)
+    {
+        if (source is null)
+        {
+            return;
         }
 
-        return matches;
+        try
+        {
+            source.Cancel();
+        }
+        catch (ObjectDisposedException)
+        {
+            // The search already finished; there is nothing left to cancel.
+        }
     }
 
     private static string FormatType(GalaxyAttribute attr)
@@ -422,17 +545,66 @@
             : "When the dashboard first observed this value — MXAccess did not supply a source timestamp for this tag.";
     }
 
-    private async Task PollLoopAsync()
+    // Fault handling sits inside the loop, matching AlarmsPage: a fault on one tick is
+    // surfaced in the "Live read failed" banner and retried on the next tick rather than
+    // ending polling for the life of the page. Cancellation is the only exit, so the loop
+    // task never faults — which is what DrainAsync in DisposeAsync relies on.
+    //
+    // A tick is skipped while the circuit's browser is disconnected: a closed tab keeps
+    // its circuit for the retention period, and polling ReadBulk on the shared dashboard
+    // worker for a page nobody can see is pure cost. The first tick after a reconnect
+    // refreshes as normal.
+    private async Task PollLoopAsync(CancellationToken cancellationToken)
     {
-        try
+        using PeriodicTimer timer = new(TimeSpan.FromSeconds(2));
+        while (true)
         {
-            using PeriodicTimer timer = new(TimeSpan.FromSeconds(2));
-            while (await timer.WaitForNextTickAsync(_cts.Token).ConfigureAwait(false))
+            try
+            {
+                if (!await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false))
+                {
+                    return;
+                }
+            }
+            catch (OperationCanceledException)
+            {
+                return;
+            }
+
+            if (!Circuit.IsConnected)
+            {
+                continue;
+            }
+
+            try
             {
                 await InvokeAsync(RefreshValuesAsync).ConfigureAwait(false);
             }
+            catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
+            {
+                return;
+            }
+            catch (Exception ex)
+            {
+                await ReportAsync(() => _readError = ex.Message).ConfigureAwait(false);
+            }
         }
-        catch (OperationCanceledException)
+    }
+
+    // Best-effort: the fault being reported may itself be an InvokeAsync against a
+    // disposed renderer, in which case the dispatch fails the same way and there is no
+    // page left to show it on.
+    private async Task ReportAsync(Action update)
+    {
+        try
+        {
+            await InvokeAsync(() =>
+            {
+                update();
+                StateHasChanged();
+            }).ConfigureAwait(false);
+        }
+        catch
         {
         }
     }
@@ -445,7 +617,7 @@
         }
 
         string[] tags = [.. _subscribed];
-        DashboardLiveReadResult result = await LiveData.ReadAsync(tags, _cts.Token);
+        DashboardLiveReadResult result = await LiveData.ReadAsync(tags, _viewerId, _cts.Token);
         _readError = result.Error;
         _workerPid = result.WorkerProcessId;
         DateTimeOffset now = DateTimeOffset.UtcNow;
@@ -454,7 +626,7 @@
             // Stamp the observed-change time only when the value/quality
             // signature actually changes, so the Updated column does not
             // tick on every poll for a static tag.
-            string signature = $"{value.ValueText}{value.Quality}{value.Ok}";
+            string signature = $"{value.ValueText}{value.Quality}{value.Ok}";
             if (!_valueSignature.TryGetValue(value.TagAddress, out string? previous)
                 || previous != signature)
             {
@@ -471,31 +643,49 @@
     /// <inheritdoc />
     public async ValueTask DisposeAsync()
     {
-        await _cts.CancelAsync();
-        await _deployCts.CancelAsync();
-        if (_pollTask is not null)
+        try
         {
-            try
-            {
-                await _pollTask;
-            }
-            catch (OperationCanceledException)
-            {
-            }
+            await _cts.CancelAsync().ConfigureAwait(false);
+            await _deployCts.CancelAsync().ConfigureAwait(false);
+
+            // Drained together, not in sequence: a wedged dispatcher blocks every loop at
+            // once, so sequential drains would multiply the bound. DrainAsync tolerates null.
+            await Task.WhenAll(DrainAsync(_pollTask), DrainAsync(_deployTask), DrainAsync(_searchTask))
+                .ConfigureAwait(false);
         }
-        if (_deployTask is not null)
+        finally
         {
-            try
-            {
-                await _deployTask;
-            }
-            catch (OperationCanceledException)
-            {
-            }
+            // Always released, even if cancellation or a drain throws. A search still in
+            // flight disposes its own linked source.
+            _cts.Dispose();
+            _deployCts.Dispose();
+            GC.SuppressFinalize(this);
+        }
+    }
+
+    // The accepted cost of the bound is an abandoned loop that keeps its timer or
+    // notifier subscription until it does unwind; waiting forever on a dispatcher that
+    // is not draining would wedge the circuit teardown itself.
+    private static async Task DrainAsync(Task? loop)
+    {
+        if (loop is null)
+        {
+            return;
         }
 
-        _cts.Dispose();
-        _deployCts.Dispose();
-        GC.SuppressFinalize(this);
+        try
+        {
+            await loop.WaitAsync(LoopDrainTimeout).ConfigureAwait(false);
+        }
+        catch (TimeoutException)
+        {
+        }
+        catch (OperationCanceledException)
+        {
+        }
+        catch
+        {
+            // Other disposal-time errors are best-effort.
+        }
     }
 }

src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Shared/StatusBadge.razor

@@ -10,7 +10,7 @@
         "Ready" or "Healthy" or "Active" => StatusState.Ok,
         "Creating" or "StartingWorker" or "WaitingForPipe" or "InitializingWorker" or "Closing"
             or "Stale" or "Degraded" or "Expiring" => StatusState.Warn,
-        "Faulted" or "Unavailable" or "Expired" => StatusState.Bad,
+        "Faulted" or "Unavailable" or "Expired" or "Unhealthy" => StatusState.Bad,
         _ => StatusState.Idle,
     };
 }

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardAlarmView.cs

@@ -0,0 +1,125 @@
+using ZB.MOM.WW.MxGateway.Contracts.Proto;
+
+namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
+
+/// <summary>The Alarms page's filter inputs, compared by value to decide whether a filtered view is stale.</summary>
+/// <param name="ShowActive">Include unacknowledged <c>Active</c> alarms.</param>
+/// <param name="ShowAcked">Include <c>ActiveAcked</c> alarms.</param>
+/// <param name="Area">Area to match (case-insensitive), or empty for all.</param>
+/// <param name="MinSeverity">Lowest severity shown.</param>
+/// <param name="MaxSeverity">Highest severity shown.</param>
+/// <param name="Search">Reference/source/description text search; trimmed before matching.</param>
+public sealed record DashboardAlarmFilterCriteria(
+    bool ShowActive,
+    bool ShowAcked,
+    string Area,
+    int MinSeverity,
+    int MaxSeverity,
+    string Search);
+
+/// <summary>
+/// Memoized projections of the Alarms page's alarm list: the filtered, sorted rows and
+/// the distinct area list. A render used to run the full filter-and-sort twice (the
+/// "Showing" card and the table) and the area projection once more, on every render —
+/// including the renders that typing in the search box triggers per keystroke. Each
+/// projection is now recomputed only when the alarm list (its generation) or, for the
+/// rows, the filter criteria change.
+/// </summary>
+/// <remarks>
+/// Owned by one page instance and touched only from its render dispatcher, so it is not
+/// synchronized.
+/// </remarks>
+public sealed class DashboardAlarmView
+{
+    private long _filteredGeneration = -1;
+    private DashboardAlarmFilterCriteria? _filteredCriteria;
+    private IReadOnlyList<DashboardActiveAlarm> _filtered = [];
+    private long _areasGeneration = -1;
+    private IReadOnlyList<string> _areas = [];
+
+    /// <summary>Gets how many times the filtered rows were actually computed (diagnostic, for tests).</summary>
+    internal int FilterComputations { get; private set; }
+
+    /// <summary>Gets how many times the area list was actually computed (diagnostic, for tests).</summary>
+    internal int AreaComputations { get; private set; }
+
+    /// <summary>Returns the alarms matching <paramref name="criteria"/>, highest severity then newest first.</summary>
+    /// <param name="generation">
+    /// Version of <paramref name="alarms"/>; the caller bumps it whenever it replaces the list's contents.
+    /// </param>
+    /// <param name="criteria">The current filter inputs.</param>
+    /// <param name="alarms">The full active-alarm list.</param>
+    /// <returns>The filtered rows; the same instance while generation and criteria are unchanged.</returns>
+    public IReadOnlyList<DashboardActiveAlarm> Filtered(
+        long generation,
+        DashboardAlarmFilterCriteria criteria,
+        IReadOnlyList<DashboardActiveAlarm> alarms)
+    {
+        ArgumentNullException.ThrowIfNull(criteria);
+        ArgumentNullException.ThrowIfNull(alarms);
+
+        if (generation == _filteredGeneration && criteria == _filteredCriteria)
+        {
+            return _filtered;
+        }
+
+        _filtered = Apply(criteria, alarms);
+        _filteredGeneration = generation;
+        _filteredCriteria = criteria;
+        FilterComputations++;
+        return _filtered;
+    }
+
+    /// <summary>Returns the distinct non-blank areas of <paramref name="alarms"/>, sorted case-insensitively.</summary>
+    /// <param name="generation">Version of <paramref name="alarms"/>.</param>
+    /// <param name="alarms">The full active-alarm list.</param>
+    /// <returns>The area list; the same instance while the generation is unchanged.</returns>
+    public IReadOnlyList<string> Areas(long generation, IReadOnlyList<DashboardActiveAlarm> alarms)
+    {
+        ArgumentNullException.ThrowIfNull(alarms);
+
+        if (generation == _areasGeneration)
+        {
+            return _areas;
+        }
+
+        _areas = alarms
+            .Select(alarm => alarm.Area)
+            .Where(area => !string.IsNullOrWhiteSpace(area))
+            .Distinct(StringComparer.OrdinalIgnoreCase)
+            .OrderBy(area => area, StringComparer.OrdinalIgnoreCase)
+            .ToArray();
+        _areasGeneration = generation;
+        AreaComputations++;
+        return _areas;
+    }
+
+    private static DashboardActiveAlarm[] Apply(
+        DashboardAlarmFilterCriteria criteria,
+        IReadOnlyList<DashboardActiveAlarm> alarms)
+    {
+        string query = criteria.Search.Trim();
+        return alarms
+            .Where(alarm => MatchesState(criteria, alarm))
+            .Where(alarm => criteria.Area.Length == 0
+                || string.Equals(alarm.Area, criteria.Area, StringComparison.OrdinalIgnoreCase))
+            .Where(alarm => alarm.Severity >= criteria.MinSeverity && alarm.Severity <= criteria.MaxSeverity)
+            .Where(alarm => query.Length == 0
+                || alarm.Reference.Contains(query, StringComparison.OrdinalIgnoreCase)
+                || alarm.Source.Contains(query, StringComparison.OrdinalIgnoreCase)
+                || alarm.Description.Contains(query, StringComparison.OrdinalIgnoreCase))
+            .OrderByDescending(alarm => alarm.Severity)
+            .ThenByDescending(alarm => alarm.LastTransition ?? DateTimeOffset.MinValue)
+            .ToArray();
+    }
+
+    private static bool MatchesState(DashboardAlarmFilterCriteria criteria, DashboardActiveAlarm alarm)
+    {
+        return alarm.State switch
+        {
+            AlarmConditionState.Active => criteria.ShowActive,
+            AlarmConditionState.ActiveAcked => criteria.ShowAcked,
+            _ => true,
+        };
+    }
+}

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardBrowseSearchIndex.cs

@@ -0,0 +1,117 @@
+using ZB.MOM.WW.GalaxyRepository;
+using ZB.MOM.WW.GalaxyRepository.Grpc;
+
+namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
+
+/// <summary>
+/// Flat, pre-lowercased attribute index behind the Browse page's filter box, shared by
+/// every circuit and rebuilt only when the Galaxy cache sequence changes.
+/// </summary>
+/// <remarks>
+/// The filter used to walk every object and attribute of the cached Galaxy, doing two
+/// case-insensitive <c>Contains</c> per attribute, synchronously in the property setter
+/// bound to <c>oninput</c> — so a large Galaxy was rescanned on the render dispatcher
+/// once per keystroke, per circuit. The page now debounces and runs the search off the
+/// dispatcher; this index makes each search a single pass of ordinal substring checks
+/// over strings lowercased once per Galaxy deploy rather than once per comparison.
+/// Lowercasing is invariant-culture, which matches the previous
+/// <see cref="StringComparison.OrdinalIgnoreCase"/> behaviour for the ASCII tag names
+/// Galaxy uses.
+/// </remarks>
+public sealed class DashboardBrowseSearchIndex
+{
+    // How often a scan checks its cancellation token. A superseded keystroke stops
+    // within this many attributes rather than finishing the whole Galaxy.
+    private const int CancellationCheckInterval = 1024;
+
+    private IndexSnapshot? _snapshot;
+    private int _buildCount;
+
+    /// <summary>Gets how many times the index has been (re)built (diagnostic, for tests).</summary>
+    internal int BuildCount => Volatile.Read(ref _buildCount);
+
+    /// <summary>
+    /// Returns up to <paramref name="limit"/> attributes whose full tag reference or
+    /// attribute name contains <paramref name="rawQuery"/>, case-insensitively, in
+    /// Galaxy object order.
+    /// </summary>
+    /// <param name="entry">The current Galaxy cache entry; its sequence keys the index.</param>
+    /// <param name="rawQuery">The filter text as typed; it is trimmed.</param>
+    /// <param name="limit">Maximum number of matches.</param>
+    /// <param name="cancellationToken">Cancelled when a newer query supersedes this one.</param>
+    /// <returns>The matches; empty for a blank query.</returns>
+    /// <exception cref="OperationCanceledException"><paramref name="cancellationToken"/> was cancelled.</exception>
+    public IReadOnlyList<GalaxyAttribute> Search(
+        GalaxyHierarchyCacheEntry entry,
+        string rawQuery,
+        int limit,
+        CancellationToken cancellationToken)
+    {
+        ArgumentNullException.ThrowIfNull(entry);
+
+        string query = (rawQuery ?? string.Empty).Trim().ToLowerInvariant();
+        if (query.Length == 0 || limit <= 0)
+        {
+            return [];
+        }
+
+        IndexedAttribute[] attributes = GetOrBuild(entry, cancellationToken);
+        List<GalaxyAttribute> matches = [];
+        for (int i = 0; i < attributes.Length; i++)
+        {
+            if (i % CancellationCheckInterval == 0)
+            {
+                cancellationToken.ThrowIfCancellationRequested();
+            }
+
+            IndexedAttribute candidate = attributes[i];
+            if (candidate.FullTagReference.Contains(query, StringComparison.Ordinal)
+                || candidate.AttributeName.Contains(query, StringComparison.Ordinal))
+            {
+                matches.Add(candidate.Attribute);
+                if (matches.Count >= limit)
+                {
+                    break;
+                }
+            }
+        }
+
+        return matches;
+    }
+
+    private IndexedAttribute[] GetOrBuild(GalaxyHierarchyCacheEntry entry, CancellationToken cancellationToken)
+    {
+        IndexSnapshot? current = Volatile.Read(ref _snapshot);
+        if (current is not null && current.Sequence == entry.Sequence)
+        {
+            return current.Attributes;
+        }
+
+        // A racing build for the same sequence is harmless: the projection is pure, so
+        // either winner stores equivalent content.
+        List<IndexedAttribute> attributes = [];
+        foreach (GalaxyObject galaxyObject in entry.Objects)
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            foreach (GalaxyAttribute attribute in galaxyObject.Attributes)
+            {
+                attributes.Add(new IndexedAttribute(
+                    attribute.FullTagReference.ToLowerInvariant(),
+                    attribute.AttributeName.ToLowerInvariant(),
+                    attribute));
+            }
+        }
+
+        IndexSnapshot built = new(entry.Sequence, [.. attributes]);
+        Volatile.Write(ref _snapshot, built);
+        Interlocked.Increment(ref _buildCount);
+        return built.Attributes;
+    }
+
+    private sealed record IndexSnapshot(long Sequence, IndexedAttribute[] Attributes);
+
+    private readonly record struct IndexedAttribute(
+        string FullTagReference,
+        string AttributeName,
+        GalaxyAttribute Attribute);
+}

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardCircuitConnectionTracker.cs

@@ -0,0 +1,42 @@
+using Microsoft.AspNetCore.Components.Server.Circuits;
+
+namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
+
+/// <summary>
+/// Per-circuit view of whether the browser is still attached. Registered scoped, so a
+/// page injecting it sees its own circuit, and also registered as that circuit's
+/// <see cref="CircuitHandler"/> so Blazor drives it.
+/// </summary>
+/// <remarks>
+/// A closed or backgrounded browser tab leaves its circuit alive, disconnected, for
+/// <see cref="DashboardServiceCollectionExtensions.DisconnectedCircuitRetention"/>
+/// before Blazor disposes it, and every timer on the page keeps running in that
+/// window. The Browse page's two-second <c>ReadBulk</c> poll is the costly one — it
+/// drives the shared dashboard worker — so it skips its tick while
+/// <see cref="IsConnected"/> is false and resumes on reconnect.
+/// </remarks>
+public sealed class DashboardCircuitConnectionTracker : CircuitHandler
+{
+    private volatile bool _connected = true;
+
+    /// <summary>
+    /// Gets a value indicating whether the circuit's browser connection is up. True
+    /// until Blazor reports the connection down, and also outside a circuit (prerender),
+    /// where no handler callbacks ever arrive.
+    /// </summary>
+    public bool IsConnected => _connected;
+
+    /// <inheritdoc />
+    public override Task OnConnectionUpAsync(Circuit circuit, CancellationToken cancellationToken)
+    {
+        _connected = true;
+        return Task.CompletedTask;
+    }
+
+    /// <inheritdoc />
+    public override Task OnConnectionDownAsync(Circuit circuit, CancellationToken cancellationToken)
+    {
+        _connected = false;
+        return Task.CompletedTask;
+    }
+}

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardLiveDataService.cs

@@ -34,20 +34,38 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
     //
     // The bound is per-read, not absolute: a read may never evict a tag it is itself
     // about to return, so a single read of more distinct tags than the cap leaves the
-    // set that large. The invariant EvictForAsync actually maintains is
+    // set that large. With a single viewer the invariant EvictForAsync maintains is
     //
     //     |advise set| after a read <= max(MaxSubscribedTags, distinct tags in that read)
     //
-    // and any overshoot is squeezed back out by the next read that subscribes a tag
+    // (with several active viewers the per-viewer share below can hold it higher, by
+    // at most what the other viewers hold within their shares) and any overshoot is squeezed back out by the next read that subscribes a tag
     // (see EvictForAsync). A browse page requests far fewer tags than the cap, so in
     // practice the set settles at MaxSubscribedTags.
     private const int MaxSubscribedTags = 256;
 
+    // Per-viewer soft cap: MaxSubscribedTags is shared by every active viewer (Browse
+    // circuit), so each owns at most MaxSubscribedTags / activeViewers tags before its
+    // own least-recently-read tags become the first eviction candidates — never fewer
+    // than this floor. Without the split, two circuits browsing different areas
+    // thrashed the single global LRU: each read evicted the other circuit's whole set,
+    // which then re-advised it two seconds later.
+    private const int MinTagsPerViewer = 32;
+
+    // A viewer that has not read within this window no longer counts as active: its
+    // share is released and its tags become the first eviction candidates. Generous
+    // against the Browse page's 2 s poll, so a briefly paused circuit keeps its share.
+    private static readonly TimeSpan ActiveViewerWindow = TimeSpan.FromSeconds(30);
+
+    // Viewer id for callers that do not identify themselves.
+    private const string SharedViewerId = "";
+
     private static readonly TimeSpan ReadTimeout = TimeSpan.FromSeconds(5);
 
     private readonly ISessionManager _sessionManager;
     private readonly IGatewayAlarmService _alarmService;
     private readonly ILogger<DashboardLiveDataService> _logger;
+    private readonly TimeProvider _timeProvider;
     private readonly SemaphoreSlim _gate = new(1, 1);
 
     /// <summary>
@@ -70,6 +88,10 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
 
     private readonly LinkedList<SubscribedTag> _recency = new();
 
+    // Per-viewer ownership and activity. A tag is owned by the viewer that read it
+    // last; only touched under _gate.
+    private readonly Dictionary<string, ViewerState> _viewers = new(StringComparer.Ordinal);
+
     private GatewaySession? _session;
     private int _serverHandle;
     private bool _disposed;
@@ -79,22 +101,31 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
     /// <param name="alarmService">Gateway central alarm service.</param>
     /// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
     /// <param name="logger">Diagnostic logger.</param>
+    /// <param name="timeProvider">Clock for viewer activity; defaults to system time.</param>
     public DashboardLiveDataService(
         ISessionManager sessionManager,
         IGatewayAlarmService alarmService,
         IOptions<GatewayOptions> options,
-        ILogger<DashboardLiveDataService> logger)
+        ILogger<DashboardLiveDataService> logger,
+        TimeProvider? timeProvider = null)
     {
         ArgumentNullException.ThrowIfNull(options);
         _sessionManager = sessionManager ?? throw new ArgumentNullException(nameof(sessionManager));
         _alarmService = alarmService ?? throw new ArgumentNullException(nameof(alarmService));
         _logger = logger ?? throw new ArgumentNullException(nameof(logger));
         _showTagValues = options.Value.Dashboard.ShowTagValues;
+        _timeProvider = timeProvider ?? TimeProvider.System;
     }
 
+    /// <inheritdoc />
+    public Task<DashboardLiveReadResult> ReadAsync(
+        IReadOnlyCollection<string> tagAddresses,
+        CancellationToken cancellationToken) => ReadAsync(tagAddresses, viewerId: null, cancellationToken);
+
     /// <inheritdoc />
     public async Task<DashboardLiveReadResult> ReadAsync(
         IReadOnlyCollection<string> tagAddresses,
+        string? viewerId,
         CancellationToken cancellationToken)
     {
         ArgumentNullException.ThrowIfNull(tagAddresses);
@@ -109,15 +140,19 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
             (GatewaySession session, int serverHandle) = await EnsureReadyAsync(cancellationToken)
                 .ConfigureAwait(false);
 
-            string[] toSubscribe = TouchAndCollectNewTags(tagAddresses, out int justReadCount);
+            string viewer = viewerId ?? SharedViewerId;
+            DateTimeOffset now = _timeProvider.GetUtcNow();
+            TouchViewer(viewer, now);
+
+            string[] toSubscribe = TouchAndCollectNewTags(tagAddresses, viewer, out HashSet<string> justRead);
             if (toSubscribe.Length > 0)
             {
-                await EvictForAsync(session, serverHandle, toSubscribe.Length, justReadCount, cancellationToken)
+                await EvictForAsync(session, serverHandle, toSubscribe.Length, justRead, viewer, now, cancellationToken)
                     .ConfigureAwait(false);
                 IReadOnlyList<SubscribeResult> subscribeResults = await session
                     .SubscribeBulkAsync(serverHandle, toSubscribe, cancellationToken)
                     .ConfigureAwait(false);
-                TrackSubscribed(toSubscribe, subscribeResults);
+                TrackSubscribed(toSubscribe, subscribeResults, viewer);
             }
 
             IReadOnlyList<BulkReadResult> results = await session
@@ -175,18 +210,20 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
     }
 
     // Promotes every already-advised tag in this read to the front of the recency
-    // list and returns the tags that still need subscribing (distinct, in request
-    // order). `justReadCount` is how many distinct tags of this read were already
-    // advised — they now occupy the front of the list and must never be evicted to
-    // make room for the same read's new tags. Callers must hold _gate.
+    // list, hands its ownership to the reading viewer, and returns the tags that
+    // still need subscribing (distinct, in request order). `justRead` is every
+    // distinct tag of this read — none of them may be evicted to make room for the
+    // same read's new tags. Callers must hold _gate.
     //
     // Every tag of one read is equally recently read; the recency list needs a total
     // order anyway, so the whole service uses one tie-break: later in the request wins.
     // Promoting in request order gives that here, and TrackSubscribed inserts new tags
     // the same way.
-    private string[] TouchAndCollectNewTags(IReadOnlyCollection<string> tagAddresses, out int justReadCount)
+    private string[] TouchAndCollectNewTags(
+        IReadOnlyCollection<string> tagAddresses,
+        string viewer,
+        out HashSet<string> justRead)
     {
-        int touched = 0;
         List<string> toSubscribe = [];
         HashSet<string> seen = new(StringComparer.OrdinalIgnoreCase);
 
@@ -200,10 +237,14 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
                     _recency.AddFirst(node);
                 }
 
-                if (seen.Add(tag))
+                if (!string.Equals(node.Value.Owner, viewer, StringComparison.Ordinal))
                 {
-                    touched++;
+                    AdjustOwned(node.Value.Owner, -1);
+                    AdjustOwned(viewer, +1);
+                    node.Value = node.Value with { Owner = viewer };
                 }
+
+                seen.Add(tag);
             }
             else if (seen.Add(tag))
             {
@@ -211,7 +252,7 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
             }
         }
 
-        justReadCount = touched;
+        justRead = seen;
         return [.. toSubscribe];
     }
 
@@ -221,7 +262,17 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
     // from tracking regardless, and the session-invalidation path already handles
     // gateway/worker drift. Callers must hold _gate.
     //
-    // Eviction stops at the tags this read just touched (`justReadCount`), so a read
+    // Victims are chosen in two passes, each least-recently-read first:
+    //   1. tags of OTHER viewers that are over their share (PerViewerCap; an
+    //      inactive viewer's share is zero);
+    //   2. the reading viewer's own tags that this read did not touch.
+    // A viewer within its share is never evicted by someone else's read, so the
+    // cap is soft: when every other viewer sits within its share and the reader's
+    // own older tags run out, the set ends over MaxSubscribedTags rather than
+    // stripping a neighbour. With one viewer, pass 1 is empty and pass 2 is the
+    // plain global LRU.
+    //
+    // Eviction never touches the tags this read just touched (`justRead`), so a read
     // whose own distinct tags outnumber the cap ends over it — see MaxSubscribedTags
     // for the exact invariant. That overshoot is not sticky: the next read that
     // subscribes anything computes `overflow` against the oversized set and evicts the
@@ -236,26 +287,56 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
         GatewaySession session,
         int serverHandle,
         int incomingCount,
-        int justReadCount,
+        HashSet<string> justRead,
+        string reader,
+        DateTimeOffset now,
         CancellationToken cancellationToken)
     {
         int overflow = _subscribed.Count + incomingCount - MaxSubscribedTags;
-        int evictable = _subscribed.Count - justReadCount;
-        int evictCount = Math.Min(overflow, evictable);
-        if (evictCount <= 0)
+        if (overflow <= 0)
         {
             return;
         }
 
-        List<int> evictedHandles = new(evictCount);
-        for (int i = 0; i < evictCount && _recency.Last is { } oldest; i++)
+        int share = PerViewerCap(now);
+        List<int> evictedHandles = [];
+        int evictCount = 0;
+
+        // Pass 1: other viewers over their share.
+        for (LinkedListNode<SubscribedTag>? node = _recency.Last; node is not null && overflow > 0;)
         {
-            _recency.RemoveLast();
-            _subscribed.Remove(oldest.Value.TagAddress);
-            if (oldest.Value.ItemHandle != 0)
+            LinkedListNode<SubscribedTag>? previous = node.Previous;
+            string owner = node.Value.Owner;
+            if (!string.Equals(owner, reader, StringComparison.Ordinal)
+                && !justRead.Contains(node.Value.TagAddress)
+                && OwnedCount(owner) > (IsActive(owner, now) ? share : 0))
             {
-                evictedHandles.Add(oldest.Value.ItemHandle);
+                Evict(node, evictedHandles);
+                overflow--;
+                evictCount++;
             }
+
+            node = previous;
+        }
+
+        // Pass 2: the reader's own tags this read did not touch.
+        for (LinkedListNode<SubscribedTag>? node = _recency.Last; node is not null && overflow > 0;)
+        {
+            LinkedListNode<SubscribedTag>? previous = node.Previous;
+            if (string.Equals(node.Value.Owner, reader, StringComparison.Ordinal)
+                && !justRead.Contains(node.Value.TagAddress))
+            {
+                Evict(node, evictedHandles);
+                overflow--;
+                evictCount++;
+            }
+
+            node = previous;
+        }
+
+        if (evictCount == 0)
+        {
+            return;
         }
 
         _logger.LogDebug(
@@ -282,12 +363,81 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
         }
     }
 
+    // Removes one tag from tracking, releasing its owner's count and collecting its
+    // item handle for the batch unadvise. Callers must hold _gate.
+    private void Evict(LinkedListNode<SubscribedTag> node, List<int> evictedHandles)
+    {
+        _recency.Remove(node);
+        _subscribed.Remove(node.Value.TagAddress);
+        AdjustOwned(node.Value.Owner, -1);
+        if (node.Value.ItemHandle != 0)
+        {
+            evictedHandles.Add(node.Value.ItemHandle);
+        }
+    }
+
+    // Each active viewer's share of MaxSubscribedTags, floored at MinTagsPerViewer.
+    // Callers must hold _gate.
+    private int PerViewerCap(DateTimeOffset now)
+    {
+        int active = _viewers.Values.Count(state => now - state.LastReadAt <= ActiveViewerWindow);
+        return Math.Max(MinTagsPerViewer, MaxSubscribedTags / Math.Max(1, active));
+    }
+
+    private bool IsActive(string viewer, DateTimeOffset now) =>
+        _viewers.TryGetValue(viewer, out ViewerState? state) && now - state.LastReadAt <= ActiveViewerWindow;
+
+    private int OwnedCount(string viewer) =>
+        _viewers.TryGetValue(viewer, out ViewerState? state) ? state.OwnedCount : 0;
+
+    // Records a read by `viewer` and forgets viewers that are both inactive and own
+    // nothing, so the map does not grow one entry per Browse page ever opened.
+    // Callers must hold _gate.
+    private void TouchViewer(string viewer, DateTimeOffset now)
+    {
+        if (!_viewers.TryGetValue(viewer, out ViewerState? state))
+        {
+            state = new ViewerState();
+            _viewers[viewer] = state;
+        }
+
+        state.LastReadAt = now;
+
+        foreach (KeyValuePair<string, ViewerState> entry in _viewers.ToArray())
+        {
+            if (entry.Value.OwnedCount <= 0 && now - entry.Value.LastReadAt > ActiveViewerWindow)
+            {
+                _viewers.Remove(entry.Key);
+            }
+        }
+    }
+
+    // Callers must hold _gate.
+    private void AdjustOwned(string viewer, int delta)
+    {
+        if (!_viewers.TryGetValue(viewer, out ViewerState? state))
+        {
+            if (delta <= 0)
+            {
+                return;
+            }
+
+            state = new ViewerState { LastReadAt = DateTimeOffset.MinValue };
+            _viewers[viewer] = state;
+        }
+
+        state.OwnedCount = Math.Max(0, state.OwnedCount + delta);
+    }
+
     // Records the freshly advised tags as the most recently read, keeping each
     // tag's item handle so eviction can unadvise it. Tags the worker failed to
     // advise are still tracked (matching the pre-cap behaviour of not retrying
     // them on every read) but carry no handle, so eviction just forgets them.
     // Callers must hold _gate.
-    private void TrackSubscribed(IReadOnlyList<string> tagAddresses, IReadOnlyList<SubscribeResult> results)
+    private void TrackSubscribed(
+        IReadOnlyList<string> tagAddresses,
+        IReadOnlyList<SubscribeResult> results,
+        string viewer)
     {
         Dictionary<string, int> handles = new(results.Count, StringComparer.OrdinalIgnoreCase);
         foreach (SubscribeResult result in results)
@@ -303,7 +453,8 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
         foreach (string tag in tagAddresses)
         {
             handles.TryGetValue(tag, out int itemHandle);
-            _subscribed[tag] = _recency.AddFirst(new SubscribedTag(tag, itemHandle));
+            _subscribed[tag] = _recency.AddFirst(new SubscribedTag(tag, itemHandle, viewer));
+            AdjustOwned(viewer, +1);
         }
     }
 
@@ -314,6 +465,10 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
     {
         _subscribed.Clear();
         _recency.Clear();
+        foreach (ViewerState state in _viewers.Values)
+        {
+            state.OwnedCount = 0;
+        }
     }
 
     // Returns a Ready session + its Register server handle, opening a fresh
@@ -424,5 +579,14 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
 
     // One entry of the advise set. ItemHandle is the handle the worker bound for
     // the tag, or 0 when the subscribe failed and there is nothing to unadvise.
-    private readonly record struct SubscribedTag(string TagAddress, int ItemHandle);
+    // Owner is the viewer that read the tag most recently.
+    private readonly record struct SubscribedTag(string TagAddress, int ItemHandle, string Owner);
+
+    // One viewer's slice of the advise set.
+    private sealed class ViewerState
+    {
+        public int OwnedCount { get; set; }
+
+        public DateTimeOffset LastReadAt { get; set; }
+    }
 }

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardServiceCollectionExtensions.cs

@@ -1,6 +1,7 @@
 using Microsoft.AspNetCore.Authentication;
 using Microsoft.AspNetCore.Authentication.Cookies;
 using Microsoft.AspNetCore.Authorization;
+using Microsoft.AspNetCore.Components.Server.Circuits;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
@@ -16,6 +17,16 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
 /// </summary>
 public static class DashboardServiceCollectionExtensions
 {
+    /// <summary>
+    /// How long a disconnected dashboard circuit is kept for reconnect before Blazor
+    /// disposes it. The framework default is three minutes, during which a closed Browse
+    /// tab's page — and its two-second live-read poll against the shared dashboard
+    /// worker — keeps running. Thirty seconds still rides out a network blip or a
+    /// laptop lid; a longer outage reloads the page, which costs nothing here because
+    /// every page re-seeds from in-process state.
+    /// </summary>
+    public static readonly TimeSpan DisconnectedCircuitRetention = TimeSpan.FromSeconds(30);
+
     /// <summary>
     /// Registers all dashboard services, authentication, and Razor components.
     /// </summary>
@@ -54,6 +65,14 @@ public static class DashboardServiceCollectionExtensions
         // nothing in this process builds a hub connection or needs a token for one.
         services.AddSingleton<HubTokenService>();
         services.AddScoped<IDashboardBrowseService, DashboardBrowseService>();
+        // Singleton: the lowercased attribute index is keyed on the Galaxy cache sequence
+        // and shared by every circuit's Browse filter, so it is built once per deploy.
+        services.AddSingleton<DashboardBrowseSearchIndex>();
+        // Scoped (one per circuit) and registered as that circuit's CircuitHandler, so the
+        // Browse page can pause its live-read poll while its browser is disconnected.
+        services.AddScoped<DashboardCircuitConnectionTracker>();
+        services.AddScoped<CircuitHandler>(
+            static provider => provider.GetRequiredService<DashboardCircuitConnectionTracker>());
         // Singleton: EventsHub instances are transient (one per hub invocation), so the
         // subscriber bookkeeping they share with the broadcaster must outlive them.
         services.AddSingleton<Hubs.EventsHubViewerRegistry>();
@@ -76,7 +95,8 @@ public static class DashboardServiceCollectionExtensions
         services.AddAntiforgery();
         services.AddCascadingAuthenticationState();
         services.AddRazorComponents()
-            .AddInteractiveServerComponents();
+            .AddInteractiveServerComponents(circuit =>
+                circuit.DisconnectedCircuitRetentionPeriod = DisconnectedCircuitRetention);
         services.AddSignalR();
 
         // DEV/TEST ONLY. Read directly from configuration here because authentication scheme

src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardSnapshot.cs

@@ -1,7 +1,33 @@
+using System.Text.Json.Serialization;
 using ZB.MOM.WW.MxGateway.Server.Configuration;
 
 namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
 
+/// <summary>
+/// One dashboard refresh: gateway status, sessions, workers, metrics, faults, API
+/// keys, the effective configuration, and the Galaxy summary.
+/// </summary>
+/// <remarks>
+/// <see cref="Configuration"/> is excluded from JSON serialization. It is
+/// startup-static, yet it is the largest part of the record, and the only wire this
+/// record crosses is the <c>/hubs/snapshot</c> tick sent every snapshot interval.
+/// Remote clients receive it once per connection instead, as
+/// <see cref="Hubs.DashboardSnapshotHub.ConfigurationMessage"/> (or on demand via
+/// <see cref="Hubs.DashboardSnapshotHub.GetConfiguration"/>). In-process pages read the
+/// property directly and are unaffected.
+/// </remarks>
+/// <param name="GeneratedAt">When the snapshot was built.</param>
+/// <param name="GatewayStartedAt">When the gateway process started.</param>
+/// <param name="GatewayUptime">Elapsed time since start.</param>
+/// <param name="GatewayStatus">Aggregate health-check status, or <c>Unknown</c>.</param>
+/// <param name="GatewayVersion">Server assembly version.</param>
+/// <param name="Sessions">Most recent sessions, newest first.</param>
+/// <param name="Workers">Workers that are not closed.</param>
+/// <param name="Metrics">Gateway counters and gauges.</param>
+/// <param name="Faults">Recent session and worker faults.</param>
+/// <param name="ApiKeys">API key summaries (no secrets).</param>
+/// <param name="Configuration">Effective configuration; not serialized on the hub tick.</param>
+/// <param name="Galaxy">Galaxy hierarchy cache summary.</param>
 public sealed record DashboardSnapshot(
     DateTimeOffset GeneratedAt,
     DateTimeOffset GatewayStartedAt,
@@ -13,5 +39,5 @@ public sealed record DashboardSnapshot(
     IReadOnlyList<DashboardMetricSummary> Metrics,
     IReadOnlyList<DashboardFaultSummary> Faults,
     IReadOnlyList<DashboardApiKeySummary> ApiKeys,
-    EffectiveGatewayConfiguration Configuration,
+    [property: JsonIgnore] EffectiveGatewayConfiguration Configuration,
     DashboardGalaxySummary Galaxy);