Compare commits

...

60 Commits

Author SHA1 Message Date
Joseph Doherty b2bce9c58b fix(go-client): go vet copylocks in types_alias_test (declare pointer types)
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / portable (push) Failing after 4m8s
ci / windows-x86 (push) Successful in 6m28s
2026-10-22 10:15:58 -04:00
Joseph Doherty c9597f7adc docs(archreview): record stream M final-review fixes 2026-10-22 08:48:09 -04:00
Joseph Doherty abd135a220 Merge stream M (cross-stream review fixes: terminal-worker attach, fault double-count, cancelled/stopping opens, docs) 2026-10-22 08:47:59 -04:00
Joseph Doherty dd623356f4 docs(clients): ReadBulk timeout is one shared deadline for the command, not per tag
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
2026-10-22 08:46:29 -04:00
Joseph Doherty 3ffba83723 docs: dashboard live-data session reaping, subscriber-refusal code, distributor-based StreamEvents model
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
  unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
  EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
  replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
  clients see while the gateway is stopping.
2026-10-22 08:43:41 -04:00
Joseph Doherty a0b5e91227 fix(grpc): count a worker fault once, not once per StreamEvents subscriber
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
2026-10-22 08:42:35 -04:00
Joseph Doherty a178d68b60 fix(sessions): refuse a dead worker at attach, fault on pump readiness failure, keep cancel/stopping out of OpenFailed
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
  worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
  fails, so a worker that dies between attach and the first pump read faults the session
  and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
  (gRPC Cancelled); an attach refused because the drain closed the session reports
  GatewayStopping.
- docs/Sessions.md updated.
2026-10-22 08:41:02 -04:00
Joseph Doherty 2375df5df3 chore(java-client): regenerate protos for stream L comment-only proto edits 2026-10-22 08:30:34 -04:00
Joseph Doherty 4962fa57cb docs(archreview): mark stream E/L rows done after merge; defer MxStatusDetailText test 2026-10-22 08:29:14 -04:00
Joseph Doherty ad67c486e8 Merge stream K (2026-10-21 tracker, cycle-1 re-triage, 2026-08-19 header fix) 2026-10-22 08:29:03 -04:00
Joseph Doherty 20135734b3 Merge stream E (worker: cache eviction, ReadBulk shared deadline + dedupe, alarm poll threshold, STA loop fault, unhandled-exception hook, live control-command facts) 2026-10-22 08:28:55 -04:00
Joseph Doherty ac4edc6e0a docs(archreview): 2026-10-21 remediation tracker, cycle-1 re-triage, fix 2026-08-19 header
Add archreview/2026-10-21 (overall summary + tracker with R21-* rows).
Re-triage the 68 cycle-1 Not-started rows in place: 21 Done, 13 Won't fix,
34 Open (promoted as R21-O1..O34). Correct the 2026-08-19 tracker header
so CLI-47..50, IPC-36/37, TST-35 read Done like their rows.
2026-10-22 08:28:44 -04:00
Joseph Doherty 2fad124bf8 Merge stream L (proto reserved/propagation comments + regen, stale docs, gateway test gaps) 2026-10-22 08:25:25 -04:00
Joseph Doherty 908cf7ccca test(server): cover GatewayConfigPathRules, DashboardRedactor, EventSubscriberReplayAttachment 2026-10-22 08:25:11 -04:00
Joseph Doherty dd3ea056b0 docs: note operator_full_name not propagated; refresh stillpending and deferred-followups status 2026-10-22 08:24:04 -04:00
Joseph Doherty 7f665bf25b docs(contracts): document unpropagated operator_full_name, reserved enum values, correlation echo 2026-10-22 08:23:26 -04:00
Joseph Doherty 38fbbde191 fix(worker): review follow-up — ghost cache entries, ReadBulk dedupe/pumping, fault hygiene
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m19s
ci / windows-x86 (push) Successful in 6m16s
- AddItem/AddItem2/AddBufferedItem drop stale cache entries for a reused handle id
- ReadBulk: dedupe handles within a batch (advise/teardown once, fan out result);
  pump every 16 items in add/advise and teardown phases
- alarm poll stops without recording a fault once StaRuntime.LoopFault is set
- StaRuntime skips the loop-fault handler during a requested shutdown
- StaHung loop fault diagnostic carries type + HRESULT only
- gateway.md documents ReadBulk's shared deadline
2026-10-22 08:23:05 -04:00
Joseph Doherty 7f0720f956 Merge stream A (session lifecycle: startup orphan, sweep metrics, stopping gate, kill-fallback Closed, unattached grace, worker-exit faults session) 2026-10-22 08:21:47 -04:00
Joseph Doherty cc1e895474 fix(sessions): fault a Ready session whose worker reached a terminal state on its own
Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.

IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.

FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
2026-10-22 06:48:31 -04:00
Joseph Doherty 3f9448f619 Merge stream C (gRPC surface: audit aggregation, streaming auth, alarm overflow/snapshot, write cancellation, correlation echo, disconnect reasons, verifier single-flight, opt-in /metrics auth) 2026-10-22 06:47:49 -04:00
Joseph Doherty 82a856b61e fix(metrics-auth): DisableLogin no longer opens /metrics; log correlation pair; host-level tests
- The metrics scheme skips its dashboard-cookie fallback while
  Dashboard:DisableLogin is on, so the dev auto-login handler cannot
  satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
  loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
  non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
  vs gRPC per connection, so only the per-key aggregate is shared.
2026-10-22 06:47:36 -04:00
Joseph Doherty ec9f26400a fix(sessions): review follow-up — never orphan a late worker, exempt internal subscribers from unattached grace, tolerate lost close races
- Open path: the attach gate now waits on CancellationToken.None and the
  manager discards (kills + disposes) the late worker client on ANY
  exception out of the attach step, including the caller's own cancellation
  (checked first, so an abandoned open never yields a session). Previously
  an OperationCanceledException from the gate wait escaped before the
  discard and orphaned the x86 worker. The gate Release is guarded against
  ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
  the gateway's own alarm-monitor session (internal subscriber only) is
  never reaped; the idle clock starts at Ready (MarkReady stamps
  _readyAtUtc) instead of at open, so a slow worker start does not eat the
  window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
  the close gate on a session that is no longer registered means a
  concurrent close already removed, accounted and disposed it; both now
  return AlreadyClosed with no fault, no second gauge decrement and no sweep
  failure. The general kill catch only decrements the gauge when the
  session is still registered. The sweep no longer counts AlreadyClosed
  results among the sessions it closed.
2026-10-22 06:43:32 -04:00
Joseph Doherty 4ea3672b1e test(integration): ShutdownWorker live fact pins reply-then-exit; log gateway session state
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m10s
ci / portable (push) Failing after 4m15s
ci / windows-x86 (push) Successful in 6m31s
2026-10-22 06:37:55 -04:00
Joseph Doherty 65bca804bb feat(security): opt-in authentication for /metrics (SEC-14)
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.

MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
2026-10-22 06:37:31 -04:00
Joseph Doherty ed3a2c9c8b fix(auth): single-flight API-key verification misses; prune key-id index on eviction
CachingApiKeyVerifier's key-id index was only pruned by Invalidate, so it
grew with every token ever verified. A post-eviction callback now drops the
slot (under the same lock as Set + index, so a late callback cannot
unindex a re-cached entry). Concurrent misses for the same token and key
generation now share one inner verification, so a TTL expiry under load no
longer stampedes SQLite; keying the flight on the generation keeps the
revoke guarantee for joiners.
2026-10-22 06:33:18 -04:00
Joseph Doherty 8212b0e550 Merge stream D (dashboard: bounded hub fan-out, alarm reset marker, browse debounce, snapshot config once, health status, per-viewer LRU, circuit retention, doc rewrite) 2026-10-22 06:32:17 -04:00
Joseph Doherty bd7abf4e35 fix(dashboard): re-check hub viewers under lock before starting a pump; document alarm send-timeout tradeoff 2026-10-22 06:32:17 -04:00
Joseph Doherty 285ef64160 feat(sessions): add UnattachedSessionGraceSeconds to reap never-subscribed idle sessions
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.

MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.

GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
2026-10-22 06:31:39 -04:00
Joseph Doherty 069972aff0 test(integration): attach event stream in ShutdownWorker live fact so the gateway observes the worker exit
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 4m16s
ci / windows-x86 (push) Successful in 6m30s
2026-10-22 06:31:36 -04:00
Joseph Doherty 68d6b5fd6f perf(metrics): cache MxEventFamily names for per-event metric tags
WorkerClient.StageWorkerEvent and StreamEvents called Enum.ToString per
event (per subscriber on the stream path). MxEventFamilyNames serves the
contract-fixed names from a frozen lookup, falling back to ToString for
undefined values so exported tag values are unchanged.
2026-10-22 06:30:57 -04:00
Joseph Doherty 6ae5ed905f fix(grpc): record the real StreamEvents disconnect reason
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
2026-10-22 06:29:31 -04:00
Joseph Doherty 0b4a1682c8 fix(grpc): typed alarm-feed overflow, memoized alarm snapshot, cancellable stream writes, uniform reply correlation id
- StreamAlarms: slow subscribers are completed with
  AlarmFeedSubscriberOverflowException and surface as ResourceExhausted
  (like a StreamEvents overflow) instead of warning-with-stack + Unavailable.
- GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection
  instead of cloning the whole set under the broadcast lock per attach; the
  prefix filter runs outside the lock.
- StreamEvents/StreamAlarms pass the call token to response writes; a write
  failure after client cancel maps to Cancelled without a warning.
- Every MxCommandReply.correlation_id now echoes the request's
  client_correlation_id (previously only constraint-denied replies did;
  normal replies leaked the internal envelope id).
2026-10-22 06:27:36 -04:00
Joseph Doherty 2c963b7230 docs(dashboard): describe the pages as shipped and the bounded hub/feed behaviour
Rewrite the Pages 'Show:' lists to what each page actually renders (no
per-session subscriber/pending-command/queue counters, no worker memory/CPU,
no per-family command counters, counters not rates). Document the per-session
EventsHub pump, alarm-hub send bound and AlarmFeedReset marker, configuration
sent once per snapshot-hub connection, health-derived gateway status, Browse
search debounce/index, bounded disposal, 30s circuit retention with
connection-gated polling, per-viewer advise-set share, and AlarmsPage memo.
2026-10-22 06:27:14 -04:00
Joseph Doherty 2b165cd3a4 fix(sessions): report a close whose kill fallback succeeded as Closed, not CloseFailed
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.

A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
2026-10-22 06:26:33 -04:00
Joseph Doherty db7833609d fix(dashboard): debounced off-dispatcher Browse search, bounded disposal, circuit-aware polling
- Search: the oninput setter scanned the whole Galaxy synchronously per
  keystroke. It now debounces 150ms, runs on the thread pool with
  cancellation (newer keystroke supersedes), and reads a shared singleton
  DashboardBrowseSearchIndex of pre-lowercased references keyed on the
  Galaxy cache Sequence.
- Disposal: DisposeAsync awaited both loops unbounded and the loops caught
  only OCE. Loops now handle faults per iteration (poll faults surface in
  _readError, deploy-notifier faults in a retrying warning), mutate the tree
  on the dispatcher, drain concurrently with a 5s bound, and the CTSs are
  disposed in finally.
- Circuits: DisconnectedCircuitRetentionPeriod is 30s (was the 3 min
  default) and a scoped DashboardCircuitConnectionTracker (CircuitHandler)
  lets the 2s ReadBulk poll skip ticks while the browser is disconnected.
- Reads pass a per-page viewer id to the live-data per-viewer share.
2026-10-22 06:25:46 -04:00
Joseph Doherty 37ab629d70 fix(sessions): refuse OpenSession while the gateway is stopping; drain until the registry is empty
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.

SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
2026-10-22 06:23:40 -04:00
Joseph Doherty 5d4aefc99c fix(dashboard): per-viewer share of the live-data advise set
The 256-tag LRU was global, so two Browse circuits on different areas evicted
each other's whole sets every poll. Tags now record the viewer that read them
last; each active viewer (read within 30s) owns MaxSubscribedTags/active
(floor 32). Eviction takes other viewers' excess over their share first
(inactive viewers' share is zero), then the reader's own untouched tags, and
never strips a neighbour within its share (soft cap). Single-viewer behaviour
is unchanged. IDashboardLiveDataService gains a viewerId ReadAsync overload.
2026-10-22 06:22:17 -04:00
Joseph Doherty 6c2d238e20 test(worker): add TestSupport using to StaRuntimeTests
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m16s
ci / java (push) Successful in 2m32s
ci / portable (push) Failing after 4m45s
2026-10-22 06:21:34 -04:00
Joseph Doherty 2a7d6facce perf(dashboard): memoize AlarmsPage filtered rows and area list
FilteredAlarms() ran the full filter+sort twice per render (Showing card and
table) and Areas() once more, including every search keystroke render. Both
now go through DashboardAlarmView, recomputed only when the alarm generation
(bumped per poll) or the filter criteria change.
2026-10-22 06:19:34 -04:00
Joseph Doherty 7618cafb6c fix(worker): review 2026-10-21 stream E — cache eviction, phased ReadBulk, poll/STA fault handling
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m3s
ci / java (push) Successful in 2m17s
ci / portable (push) Failing after 4m42s
- RemoveItem/Unregister evict value + write-completion caches (leak under churn)
- ReadBulk snapshot path: add/advise all, one shared deadline, then tear down
- alarm poll faults only after 3 consecutive failures; transient ones logged
- AlarmCommandHandler disposes primary when standby build throws
- StaRuntime pumps between queued work items
- WRK-02: post-start STA loop death surfaces as StaHung WorkerFault
- WRK-09: AppDomain unhandled-exception hook, exit code 7
- RETRIGGER documented as reserved, never synthesized
- TST-05: live facts for Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker
2026-10-22 06:19:32 -04:00
Joseph Doherty 09666cc719 fix(sessions): account SessionClosed for sweep-driven closes
TryBeginCloseIfExpired set _closeStarted when it reserved a session, so the
sweep's own CloseAsync reported AlreadyClosed: true and CloseSessionCoreAsync
skipped SessionClosed: lease-expired, faulted-reaped and detach-grace-expired
closes never left the sessions.open gauge and under-counted sessions.closed.
The reservation now writes Closing only; the first CloseAsync to take the
close lock does the accounting exactly once.
2026-10-22 06:19:20 -04:00
Joseph Doherty 6733519ae5 fix(dashboard): derive gateway status from health checks; send config once per hub connection
SEC-27: DashboardSnapshot.GatewayStatus was hardcoded Healthy. It is now the
aggregate HealthCheckService status, evaluated once per snapshot tick (2s
bound, zero-wait gate) and cached for GetSnapshot; Unknown before the first
evaluation, on failure, or without a health service.

The startup-static EffectiveGatewayConfiguration is JsonIgnore'd on the
snapshot, so /hubs/snapshot ticks carry only volatile state. The hub sends it
once per connection (ConfigurationUpdated) and serves GetConfiguration().
Snapshot broadcasts are time-bounded (1s) so one slow client cannot delay
ticks. StatusBadge maps Unhealthy to Bad.
2026-10-22 06:18:02 -04:00
Joseph Doherty 87f23d9712 fix(sessions): refuse late worker attach after close during startup; route open rollback through removal path
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.

AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.

The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
2026-10-22 06:16:53 -04:00
Joseph Doherty 3688fa19e6 fix(dashboard): time-bound alarm hub sends and mark each resubscribe
AlarmsHubPublisher awaited every group send unbounded inside the StreamAsync
loop, so one slow client stalled the drain until the monitor dropped the
subscriber and the resubscribe replayed the full snapshot to everyone. Each
broadcast is now bounded (2s) and abandoned on timeout, and every
(re)subscription is preceded by an AlarmFeedReset frame so clients discard
state before the replayed snapshot.
2026-10-22 06:14:53 -04:00
Joseph Doherty 19cc74f21a fix(auth): authenticate client- and duplex-streaming gRPC calls
GatewayGrpcAuthorizationInterceptor only overrode the unary and
server-streaming handlers; the base Interceptor passes client- and
duplex-streaming calls through unauthenticated. Both now run the same
API-key pipeline and fail closed to the admin scope, since there is no
single request message to resolve a scope from.
2026-10-22 06:14:33 -04:00
Joseph Doherty f1db002b87 fix(dashboard): bound EventsHub delivery with one drop-oldest pump per session
Publish fired a group SendAsync per event and returned, so a slow remote
viewer accumulated unbounded pending sends plus cloned MxEvents. Hub delivery
now goes through a 256-deep DropOldest channel per hub-watched session with a
single drain task and time-bounded sends. The registry counts hub viewers
separately and raises HubViewersReleased so the pump is torn down with the
last remote viewer; in-process-only sessions start no pump.
2026-10-22 06:13:05 -04:00
Joseph Doherty ca0cf19a1f fix(auth): aggregate bulk constraint denials into one audit event; ValueTask constraint checks
A bulk command with N denied items wrote N audit events into the bounded,
drop-on-full audit channel, so a low-privilege key could evict every other
pending audit record. Bulk filters now accumulate a ConstraintDenialSummary
(count, constraint names, <=20 sample targets) and write one aggregated
constraint-denied event per command.

The per-item constraint checks are synchronous, so IConstraintEnforcer's
Check* methods now return ValueTask instead of allocating a Task per item.
2026-10-22 06:12:02 -04:00
Joseph Doherty 81d4cd8644 Merge stream B (bound WorkerClient.ShutdownAsync enqueue by shutdown timeout) 2026-10-22 06:11:17 -04:00
Joseph Doherty b0100e738a fix(worker): bound ShutdownAsync's envelope enqueue by the shutdown timeout
WorkerClient.ShutdownAsync marked the client Closing and then queued the
WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS
was only created afterwards. The outbound channel is bounded, and a worker that
has stopped reading the pipe parks the write loop in a pipe write it can only
leave through _stopCts while the channel behind it fills with commands and
their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing
cancelled _stopCts and the enqueue waited for the caller's token - the lease
sweep's stopping token, which fires at host stop and never before - with
GatewaySession.CloseAsync holding its close lock throughout.

Create the linked timeout CTS before the enqueue and run the enqueue, the loop
waits and the process-exit wait on it, so the whole call is bounded by the
shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout
fault, which cancels the loops, kills the worker process tree and hands
GatewaySession its kill fallback.

Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker
stalls the gateway-side stream (a worker that stopped reading), fills the
outbound channel with abandoned commands and their cancels, and asserts that
ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake
worker. It hung past the bound before the fix.
2026-10-22 06:10:52 -04:00
Joseph Doherty b2a7fc525d Merge stream I (Rust client: async CA read, typed diagnostics) 2026-10-22 06:10:07 -04:00
Joseph Doherty 3a1ee28729 Merge stream F (.NET client: no ack retry, shared channel, safe dispose, typed diagnostics, AttachSession) 2026-10-22 06:09:55 -04:00
Joseph Doherty c18566bcf2 Merge stream G (Java client: configurable event queue, overflow retention, interrupt cancel, typed diagnostics) 2026-10-22 06:09:55 -04:00
Joseph Doherty 0345726a79 fix(rust-client): async CA file read; typed ping/session-state/worker-info/drain-events 2026-10-22 06:09:51 -04:00
Joseph Doherty 6f0b408833 fix(dotnet-client): no ack retry, caller-owned channels, safe session dispose, typed diagnostics, AttachSession 2026-10-22 06:09:25 -04:00
Joseph Doherty 48c9323d4a fix(java-client): configurable event queue, keep events on overflow, cancel on interrupt, typed diagnostics
- eventQueueCapacity option (default 1024, was hard-coded 16)
- overflow retains buffered events and delivers the error after them
- interrupt cancels the gRPC call and restores the interrupt flag
- ping/getSessionState/getWorkerInfo/drainEvents on MxGatewaySession
2026-10-22 06:09:25 -04:00
Joseph Doherty 7636573ebb Merge stream H (Go client: event buffer option, grpc.NewClient, typed diagnostics) 2026-10-22 06:08:48 -04:00
Joseph Doherty a3a7bd2170 feat(go-client): configurable event buffer, grpc.NewClient dial, typed diagnostics
- Options.EventBufferSize (default 1024) replaces the fixed 16-slot Events buffer
- Dial/DialGalaxy use grpc.NewClient + explicit READY wait (replaces DialContext+WithBlock)
- Session.Ping/GetSessionState/GetWorkerInfo/DrainEvents typed diagnostics
2026-10-22 06:08:31 -04:00
Joseph Doherty 29d4e51c7b Merge stream J (Python typed diagnostics) into fix/review-2026-10-21 2026-10-22 06:07:17 -04:00
Joseph Doherty d716dec160 docs(python): trim diagnostics note 2026-10-22 06:06:56 -04:00
Joseph Doherty 99347d3f00 feat(python): typed ping/get_session_state/get_worker_info/drain_events session helpers 2026-10-22 06:06:54 -04:00
154 changed files with 11841 additions and 1037 deletions
@@ -27,7 +27,7 @@ This plan is executed as **seven parallel batches**, each by one subagent workin
|---|---|---|
| TST-32 | Blocked (external) | Gitea runner registration-token reset is a web-UI action on the Gitea admin page; no repo change. Operator to-do. |
| SEC-37 | Blocked (external) | Family-wide: TLS on the shared GLAuth lives in `scadaproj/infra/glauth/` (separate repo + docker host). Plan separately; after it lands, move the deployed service to `Production`. |
| CLI-47/48/49/50, IPC-36/37, TST-35 | Not started | Activated by the 2026-08-25 P2 continuation below. |
| CLI-47/48/49/50, IPC-36/37, TST-35 | Done | Activated by the 2026-08-25 P2 continuation below and closed there: CLI-47/48/49/50 in batch K (`5cdb3da`), IPC-36/37 in batch J (`d6c5a11`), TST-35 in batch M (`cc143b5`). |
| GWC-33/35, SEC-38/39 | Done | Closed by continuation batch I; regression counts and commits are recorded below. |
| GWC-36, IPC-38, TST-33/34/36 | Done | Closed by continuation batch H; verification and commit are recorded below. |
| WRK-34 | Done | Completed opportunistically in batch F (`3679e59`): watchdog faults use `CommandMethod=Watchdog` and carry the stuck correlation ID in the diagnostic message. Tracker correction recorded by the continuation audit. |
+27
View File
@@ -0,0 +1,27 @@
# MxAccessGateway — System Review (2026-10-21)
Summary of the fourth review cycle. Per-finding status is in [`remediation/00-tracking.md`](remediation/00-tracking.md); the per-domain finding text is not kept in this directory, so the tracker rows carry the description.
## Dimensions
Six reviewers each took one dimension of the system.
| Dimension | Reviewed | Outcome |
|---|---|---|
| Sessions | Session lifecycle, sweep, shutdown, lease, metrics accounting (`GatewaySession`, `SessionManager`) | One High: a kill or shutdown during worker startup orphaned the worker. Nine findings, all fixed in stream A. Replay-ring byte cap remains open (low) |
| Worker | STA runtime, alarm polling, `ReadBulk`, caches, process exit paths (`ZB.MOM.WW.MxGateway.Worker`) | Eight findings fixed on `fix/review-2026-10-21-e`, not yet merged; four deferred (one needs a parity probe, one is a security tradeoff) |
| gRPC | Interceptor coverage, audit, alarm feed, correlation ids, `/metrics` | Eleven findings fixed. `/metrics` authentication is opt-in |
| Dashboard | Hub fan-out, browse, snapshot feed, health status, design doc | Ten findings fixed; three low races left open |
| Clients | Five language clients: ack retry, buffers, dispose, typed diagnostics | All fixed. The Go `go vet` copylocks error in a test predates the review and remains |
| Docs vs code | Proto comments, `stillpending.md`, tracker headers, untested helpers | In progress in stream L |
## Cross-cutting observations
- The two-process lifecycle is the most failure-prone area: most of the nine session findings are races between open, close, kill and shutdown. Each fix added a regression test; the pattern to keep is that every exit path accounts for the session exactly once.
- Silent failure was the recurring theme in the worker (STA death, alarm-poll failure, missing unhandled-exception hook). Each now ends in a typed fault or a distinct exit code.
- Bounded fan-out was the recurring theme in the dashboard: every hub send now has a bound, a timeout or a drop policy.
- Three external items are outside this repository: shared GLAuth TLS, the Gitea runner token, and an OtOpcUa session leak.
## Cycle-1 backlog
The review also re-triaged 68 first-cycle findings that no later tracker had re-listed: 21 are done, 13 are closed as won't fix with a reason, and 34 remain open and are promoted as `R21-O1` to `R21-O34`. The highest-weight open items are the missing reply to a command that races worker shutdown, the worker pipe without `CurrentUserOnly`, the no-op startup probe with retry options that cannot retry, the redactor gap for `WriteSecuredBulk`/`WriteSecured2Bulk`, the .NET retry budget that a `DeadlineExceeded` consumes, and the Go client's missing typed auth errors and strict-TLS flag.
@@ -0,0 +1,224 @@
# MxAccessGateway — Remediation Tracking (2026-10-21 system review)
Source review: [`../00-overall.md`](../00-overall.md) (summary of the six review dimensions). Generated 2026-10-22 from the review outcomes and the integration branch `fix/review-2026-10-21`.
Tracks the findings of the 2026-10-21 system review and the cycle-1 findings that no later tracker had re-listed. The earlier trackers remain the record for their own IDs: [cycle 1](../../remediation/00-tracking.md), [cycle 2](../../2026-07-12/remediation/00-tracking.md), [cycle 3](../../2026-08-19/remediation/00-tracking.md).
## How to use this document
- IDs in this cycle use the prefix `R21-` so they cannot collide with the `GWC`/`WRK`/`IPC`/`SEC`/`CLI`/`TST` registers of earlier cycles. `R21-S` is sessions, `R21-W` is the worker client, `R21-G` is the gRPC surface, `R21-D` is the dashboard, `R21-K` is the Windows worker, `R21-C*` are the client libraries, `R21-L` is docs and test gaps, `R21-X` is external, and `R21-O` is a cycle-1 finding re-triaged as still open.
- Where a finding is the same as an earlier-cycle ID, the description names it (for example `R21-G11` is `SEC-14`).
- The review's own severity table is not in the repository. Only `R21-S1` carries a severity from the review (High); every other severity here is the triage estimate from the finding's impact and is open to correction.
- The **Status** column is the single source of truth. Update it in the same change that lands the fix, and cite the finding ID in commit messages.
**Status legend:** `Done` · `Done (merge 2013573)` · `Done (merge 2fad124)` · `Deferred` · `Won't fix` · `Open` · `Blocked (external)`.
## Execution model
The review was remediated by ten parallel streams on branches `fix/review-2026-10-21-{a..j}`, merged into the integration branch `fix/review-2026-10-21`, plus stream K (this tracker triage) and stream L (docs and test gaps).
| Stream | Branch | Scope | Status |
|---|---|---|---|
| A | `fix/review-2026-10-21-a` | Sessions (`R21-S*`) | Done, merged (`7f0720f`) |
| B | `fix/review-2026-10-21-b` | WorkerClient (`R21-W1`) | Done, merged (`81d4cd8`) |
| C | `fix/review-2026-10-21-c` | gRPC surface (`R21-G*`) | Done, merged (`3f9448f`) |
| D | `fix/review-2026-10-21-d` | Dashboard (`R21-D*`) | Done, merged (`8212b0e`) |
| E | `fix/review-2026-10-21-e` | Worker (`R21-K*`) | Done (merge 2013573) — `7618caf`, follow-up `38fbbde` |
| F | `fix/review-2026-10-21-f` | .NET client | Done, merged (`3a1ee28`) |
| G | `fix/review-2026-10-21-g` | Java client | Done, merged (`c18566b`) |
| H | `fix/review-2026-10-21-h` | Go client | Done, merged (`7636573`) |
| I | `fix/review-2026-10-21-i` | Rust client | Done, merged (`b2a7fc5`) |
| J | `fix/review-2026-10-21-j` | Python client | Done, merged (`29d4e51`) |
| K | `fix/review-2026-10-21-k` | Tracker triage (this document) | Done |
| L | (stream L branch) | Proto comments, docs, test gaps | Done (merge 2fad124) |
Worker changes (stream E) need the windev x86 build and `Worker.Tests` run after merge; stream E's own live facts passed on windev.
## Sessions (stream A)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-S1 | High | Sessions | A kill or shutdown during worker startup orphaned the worker | Done | A: `87f23d9`, `ec9f264` | Gated attach, discard on any exception, `OpenRecorded`; a late worker is never orphaned |
| R21-S2 | Medium | Sessions | Sweep-driven closes skipped `SessionClosed()`, so the open-sessions gauge drifted | Done | A: `09666cc` | |
| R21-S3 | Medium | Sessions | `OpenSession` was still accepted while the host was stopping | Done | A: `37ab629` | `GatewayStopping` returns `Unavailable`; `ShutdownAsync` re-snapshots until the registry is empty |
| R21-S4 | Low | Sessions | A shutdown timeout whose kill succeeded was reported as `CloseFailed` | Done | A: `2b165cd` | Now `Closed`, with `WorkerShutdownFailure` recorded |
| R21-S5 | Low | Sessions | A failed open left a metrics entry behind | Done | A: `87f23d9` | Open rollback goes through the removal path |
| R21-S6 | Medium | Sessions | Sessions that never stream pinned a worker for the whole lease | Done | A: `285ef64`, `ec9f264` | Config-gated `UnattachedSessionGraceSeconds` (default 0 = off); `ExtendLease` deleted; internal subscribers count as attached; idle clock starts at Ready |
| R21-S7 | Low | Sessions | A kill racing a close double-decremented the gauge | Done | A: `ec9f264` | Found in review of stream A |
| R21-S8 | Low | Sessions | A sweep that lost the race to a close was reported as a failure | Done | A: `ec9f264` | Found in review of stream A |
| R21-S9 | Medium | Sessions | After `ShutdownWorker` the session stayed `Ready` | Done | A: `cc1e895` | The session faults and the sweep reaps it. Found live |
| R21-S10 | Low | Sessions | The replay ring has a count cap but no byte cap | Open | — | Documented as bounded by count. Next step: add a byte cap if large-array events make the count cap insufficient |
## WorkerClient (stream B)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-W1 | Medium | WorkerClient | `ShutdownAsync` could block past `ShutdownTimeout` | Done | B: `b0100e7` | The timeout CTS now covers the envelope enqueue |
## gRPC surface (stream C)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-G1 | Medium | gRPC | Denied bulk commands flooded the audit channel | Done | C: `ca0cf19` | One aggregated event per bulk call |
| R21-G2 | Medium | gRPC | The authorization interceptor did not cover client-streaming or duplex calls (cycle-1 `SEC-17`) | Done | C: `19cc74f` | Both fail closed to the admin scope |
| R21-G3 | Low | gRPC | `StreamAlarms` overflow returned `Unavailable` | Done | C: `0b4a168` | Now `ResourceExhausted` |
| R21-G4 | Low | gRPC | `StreamAlarms` cloned the snapshot under the lock on every attach | Done | C: `0b4a168` | Memoized projection |
| R21-G5 | Low | gRPC | `WriteAsync` was not given the call's cancellation token | Done | C: `0b4a168` | |
| R21-G6 | Low | gRPC | `correlation_id` handling was inconsistent across replies | Done | C: `0b4a168`, `82a856b` | `client_correlation_id` is echoed; a Debug log records the client/envelope pair |
| R21-G7 | Low | gRPC | The `StreamDisconnected` reason was a constant (cycle-1 `GWC-22`) | Done | C: `6ae5ed9` | |
| R21-G8 | Low | Security | `CachingApiKeyVerifier` index pruning and single-flight misses | Done | C: `ed3a2c9` | |
| R21-G9 | Low | gRPC | `ConstraintEnforcer` allocated a `Task` per item | Done | C: `ca0cf19` | `ValueTask` |
| R21-G10 | Low | Metrics | `Enum.ToString` was called per event | Done | C: `68d6b5f` | `MxEventFamilyNames` cache |
| R21-G11 | Medium | Security | `/metrics` was unauthenticated (cycle-1 `SEC-14`) | Done | C: `65bca80`, `82a856b` | Opt-in `Metrics:RequireAuthentication` (admin key, or dashboard Admin/Viewer cookie); `DisableLogin` does not bypass it; `/health` stays open by design |
| R21-G12 | Low | Metrics | The `/metrics` per-peer limiter is keyed by IP while the gRPC limiter is keyed by connection | Won't fix | — | Documented difference; the two surfaces have different peer identities |
| R21-G13 | Low | Security | The verifier generation guard depends on key-id case | Open | — | Latent only. Next step: normalize the key id (ordinal-ignore-case) at the guard and add a test |
## Dashboard (stream D)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-D1 | Medium | Dashboard | Unbounded events-hub sends | Done | D: `f1db002`, `bd7abf4` | One drop-oldest pump per session; a lock re-check race was fixed in review |
| R21-D2 | Medium | Dashboard | One slow client stalled the alarms hub | Done | D: `3688fa1` | 2 s send timeout plus an `AlarmFeedReset` marker. Tradeoff documented: an abandoned send is not followed by a reset |
| R21-D3 | Low | Dashboard | Browse search ran on every keystroke | Done | D: `db78336` | Debounce, off-dispatcher, indexed |
| R21-D4 | Medium | Dashboard | The full snapshot plus Configuration went out every second | Done | D: `6733519` | Configuration is sent once per connection; sends have a timeout |
| R21-D5 | Low | Dashboard | `AlarmsPage` re-sorted twice per render | Done | D: `2a7d6fa` | `DashboardAlarmView` memo |
| R21-D6 | Low | Dashboard | `BrowsePage` disposal was unbounded | Done | D: `db78336` | |
| R21-D7 | Low | Dashboard | Two viewers thrashed the advise LRU | Done | D: `5d4aefc` | Per-viewer share |
| R21-D8 | Low | Dashboard | Disconnected circuits kept polling | Done | D: `db78336` | 30 s retention plus a connection tracker |
| R21-D9 | Low | Dashboard | `GatewayStatus` was hardcoded (cycle-1 `SEC-27`) | Done | D: `6733519` | Derived from health checks |
| R21-D10 | Low | Docs | `docs/GatewayDashboardDesign.md` described counters that do not exist | Done | D: `2c963b7` | Doc rewritten to match the shipped pages |
| R21-D11 | Low | Dashboard | `BrowsePage` search CTS cancel/dispose race | Open | — | Only a caught `ObjectDisposedException`. Next step: swap-and-dispose under a lock when the page is next touched |
| R21-D12 | Low | Dashboard | Cached health status is stale while the publisher is idle | Open | — | Next step: refresh on first subscribe |
| R21-D13 | Low | Dashboard | Hub pumps are not awaited on broadcaster dispose | Open | — | Next step: await the pumps with a bounded timeout in `DisposeAsync` |
## Worker (stream E)
All rows are on `fix/review-2026-10-21-e` (`7618caf`, follow-up `38fbbde`), not yet merged. After merge, run `windev-worker-ci.ps1 -Mode test` and update these rows to plain `Done`.
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-K1 | Medium | Worker | The write-completion cache and value cache were never evicted | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Ghost-entry eviction on `AddItem` added in the follow-up |
| R21-K2 | Medium | Worker | `ReadBulk` was serial (N x timeout) | Done (merge 2013573) | E: `7618caf`, `38fbbde` | Shared deadline; dedupe and periodic message pumping in the follow-up |
| R21-K3 | Medium | Worker | One alarm-poll failure faulted the session (cycle-1 `WRK-05`) | Done (merge 2013573) | E: `7618caf` | Consecutive-failure threshold of 3 |
| R21-K4 | Low | Worker | The primary alarm consumer leaked when the standby build failed | Done (merge 2013573) | E: `7618caf` | |
| R21-K5 | Low | Worker | `StaRuntime` drained the whole queue before pumping messages | Done (merge 2013573) | E: `7618caf` | |
| R21-K6 | Medium | Worker | STA thread death after start was silent (cycle-1 `WRK-02`) | Done (merge 2013573) | E: `7618caf` | `LoopFault` becomes a `StaHung` WorkerFault |
| R21-K7 | Low | Worker | No unhandled-exception hook (cycle-1 `WRK-09`) | Done (merge 2013573) | E: `7618caf` | Exit code 7 |
| R21-K8 | Low | Worker | `RETRIGGER` handling was undocumented | Done (merge 2fad124) | stream L | Documented as reserved, never synthesized; the proto comment is stream L's |
| R21-K9 | Medium | Tests | Live control-command facts for the worker (cycle-1 `TST-05` residual) | Done (merge 2013573) | E: `069972a`, `4ea3672` | Facts added and passing on windev |
| R21-K10 | Low | Worker | Unregister-first shutdown cleanup | Deferred | — | Needs a parity probe against the installed MXAccess before it can change cleanup order |
| R21-K11 | Medium | Security | A per-operator `AuthenticateUser` cache | Deferred | — | Security tradeoff (caching credentials' outcome); decide with the security owner before building |
| R21-K12 | Low | Worker | Batch task-completion-source allocation rework | Deferred | — | Low value |
| R21-K13 | Low | Worker | The alarm poll builds a full-BSTR `XmlDocument` every 500 ms | Deferred | — | Steady GC cost, not a correctness issue; revisit if profiling flags it |
## Clients (streams F to J)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-CN1 | Medium | .NET client | Acknowledge retried on transient failure | Done | F: `6f0b408` | No ack retry |
| R21-CN2 | Medium | .NET client | Client always owned its channel | Done | F: `6f0b408` | Caller-owned channel constructors |
| R21-CN3 | Medium | .NET client | `DisposeAsync` could throw or block (cycle-1 `CLI-06`) | Done | F: `6f0b408` | Safe, best-effort dispose |
| R21-CN4 | Low | .NET client | No typed Ping/GetSessionState/GetWorkerInfo/DrainEvents | Done | F: `6f0b408` | |
| R21-CN5 | Medium | .NET client | A session could not be re-attached by id (cycle-1 `CLI-05`) | Done | F: `6f0b408` | `AttachSession` |
| R21-CJ1 | Medium | Java client | Event queue was fixed at 16 | Done | G: `48c9323` | Configurable, default 1024 |
| R21-CJ2 | Medium | Java client | Overflow dropped buffered events | Done | G: `48c9323` | Overflow keeps buffered events |
| R21-CJ3 | Low | Java client | Interrupt did not cancel the call | Done | G: `48c9323` | |
| R21-CJ4 | Low | Java client | No typed diagnostics | Done | G: `48c9323` | |
| R21-CG1 | Medium | Go client | Event buffer fixed at 16 | Done | H: `a3a7bd2` | `EventBufferSize` option, default 1024 |
| R21-CG2 | Medium | Go client | Deprecated `DialContext` + `WithBlock` (cycle-1 `CLI-10`) | Done | H: `a3a7bd2` | `grpc.NewClient` |
| R21-CG3 | Low | Go client | No typed diagnostics | Done | H: `a3a7bd2` | |
| R21-CG4 | Low | Go client | `go vet` copylocks error in `types_alias_test.go` | Done | see commit "fix(go-client): go vet copylocks" | The test declares pointer types; `go vet ./...` is clean |
| R21-CR1 | Medium | Rust client | Blocking file read on an async path (CA file) | Done | I: `0345726` | Async read |
| R21-CR2 | Low | Rust client | No typed diagnostics | Done | I: `0345726` | |
| R21-CP1 | Low | Python client | No typed diagnostics | Done | J: `99347d3`, `d716dec` | The client already had no ack retry, no event buffer, and cancels on exit |
## Docs and test gaps (stream L)
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-L1 | Low | Docs | `operator_full_name` is silently dropped; proto comment and docs do not say so | Done (merge 2fad124) | stream L | |
| R21-L2 | Low | Contracts | `RETRIGGER` and the `WorkerFaultCategory` values `INVALID_ARGUMENTS` and `GATEWAY_AUTHENTICATION_FAILED` are reserved but not labeled so | Done (merge 2fad124) | stream L | Proto comments |
| R21-L3 | Low | Docs | Stale notes in `stillpending.md` (sections 1.4, 3.4, 7.1) and the deferred-followups header | Done (merge 2fad124) | stream L | |
| R21-L4 | Low | Docs | The 2026-08-19 tracker header says CLI-47..50, IPC-36/37, TST-35 are `Not started` while the rows say `Done` | Done | K | Header row corrected in this stream |
| R21-L5 | Low | Tests | No tests for `GatewayConfigPathRules` | Done (merge 2fad124) | stream L | |
| R21-L6 | Low | Tests | No tests for `DashboardRedactor` | Done (merge 2fad124) | stream L | |
| R21-L7 | Low | Tests | No tests for `EventSubscriberReplayAttachment` | Done (merge 2fad124) | stream L | |
| R21-L8 | Low | Tests | No tests for the `MxStatusDetailText` table | Deferred | — | Worker-side test; needs a windev run, not done in stream L |
## External / blocked
| ID | Sev | Area | Description | Status | Commit / stream | Reason / note |
|---|---|---|---|---|---|---|
| R21-X1 | Medium | Security | TLS on the shared GLAuth (cycle-3 `SEC-37`) | Blocked (external) | — | Lives in `scadaproj/infra/glauth/` (separate repo and docker host); after it lands, move the deployed service to `Production` |
| R21-X2 | Low | CI | Gitea runner registration-token reset (cycle-3 `TST-32`) | Blocked (external) | — | Web-UI action on the Gitea admin page |
| R21-X3 | Medium | OtOpcUa | `GalaxyDriver.ShutdownAsync` leaks sessions | Blocked (external) | — | Defect is in the OtOpcUa repo; the gateway's lease expiry and detach grace eventually reap a leaked session |
## Cycle-1 rows re-triaged as still open (promoted)
These are cycle-1 findings that no later tracker re-listed. Each was checked against the integration branch, and the cycle-1 row now points here. The `Medium` rows carry correctness, security or operability weight (`R21-O1`, `R21-O8`, `R21-O9`, `R21-O16`, `R21-O21` to `R21-O23`, `R21-O30`, `R21-O31`); the `Low` rows are hygiene to batch with the next change that touches the file.
| ID | Cycle-1 ID | Sev | Area | Description | Status | Next step |
|---|---|---|---|---|---|---|
| R21-O1 | WRK-03 | Medium | Worker | `TryStartCommandTask` returns silently when `_acceptingCommands` is false, so a command racing shutdown gets no reply and no log (`Ipc/WorkerPipeSession.cs:1240`) | Open | Reply `WorkerUnavailable` (at minimum log `WorkerCommandResultDropped`) and add `CommandAfterShutdown_RepliesWorkerUnavailable`; verify on windev |
| R21-O2 | WRK-08 | Low | Worker | Residual events queued after `WorkerShutdown` are discarded, and the shutdown section of `docs/MxAccessWorkerInstanceDesign.md` does not say so | Open | Document the discard as v1 behavior (docs-only) |
| R21-O3 | WRK-10 | Low | Worker | `WorkerApplication.Run` catch blocks log the exception type but never the message | Open | Add `exception_message` through `WorkerLogRedactor` in the three catch blocks; add a `WorkerApplicationTests` assertion |
| R21-O4 | WRK-16 | Low | Worker | Seven near-identical `CreateEnvelope` overload pairs and eight `WorkerPipeClient` constructors | Open | Refactor when `WorkerPipeSession` is next touched; existing IPC tests are the guard |
| R21-O5 | WRK-17 | Low | Worker | Gateway death surfaces as `EndOfStream` and exits 6 (`ProtocolViolation`) instead of 5 | Open | Map `EndOfStream` to `PipeConnectionFailed`; add a `WorkerApplicationTests` case |
| R21-O6 | WRK-18 | Low | Worker | Event-queue overflow exits as generic `UnexpectedFailure` (1) | Open | Add a dedicated exit code (stream E already uses 7 for unhandled exceptions, so use 8) and update the overflow paragraph of `docs/MxAccessWorkerInstanceDesign.md` |
| R21-O7 | WRK-19 | Low | Worker | The documented per-command start/end log with correlation id and latency does not exist | Open | Add level-gated `WorkerCommandStarted`/`WorkerCommandCompleted` in `StaCommandDispatcher.ExecuteQueuedCommandAsync` |
| R21-O8 | GWC-05 | Medium | Gateway | Worker pipe is created with no ACL and no `CurrentUserOnly` (`SessionWorkerClientFactory.CreatePipe`); only the unguessable name and `maxNumberOfServerInstances: 1` limit access | Open | Add `PipeOptions.CurrentUserOnly` (or a `PipeSecurity` for the worker account); confirm the worker runs as the same user on windev; add a launch test |
| R21-O9 | GWC-09 | Medium | Gateway | `WorkerProcessStartedProbe` only checks `HasExited`, so `StartupProbeRetryAttempts` and `StartupProbeRetryDelayMilliseconds` can never retry anything | Open | Either make the probe wait for a real readiness signal, or remove the two options and their docs so config does not promise a retry that cannot happen |
| R21-O10 | GWC-13 | Low | Gateway | Worker-ready wait is a 25 ms poll loop (`GatewaySession`, `pollIntervalMs = 25`) | Open | Replace with a `TaskCompletionSource` signalled on state change |
| R21-O11 | GWC-17 | Low | Gateway | `SparseArrayExpander` throws `Grpc.Core.RpcException` from the Sessions layer | Open | Throw a domain exception and map it to `InvalidArgument` in the gRPC layer |
| R21-O12 | GWC-18 | Low | Gateway | `GatewaySession` exposes `DisposeAsync` without implementing `IAsyncDisposable` | Open | Declare the interface; check `await using` call sites |
| R21-O13 | GWC-19 | Low | Gateway | `GatewaySession.KillWorker(string)` has no production callers (only test comments mention it) | Open | Delete it and fix the comments, or document it as the synchronous kill primitive |
| R21-O14 | GWC-20 | Low | Gateway | `HeartbeatCheckInterval` is fed from `HeartbeatIntervalSeconds`, conflating the send and check intervals | Open | Split the option or rename it, and update `docs/GatewayConfiguration.md` |
| R21-O15 | GWC-21 | Low | Gateway | `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` are compile-time constants (`WorkerClientOptions`) | Open | Expose both under `MxGateway:Worker:*` with validation, or record them as intentionally fixed |
| R21-O16 | SEC-13 | Medium | Security | `GatewayLogRedactor.SensitiveCommandMethods` lacks `WriteSecuredBulk` and `WriteSecured2Bulk`. Latent in cycle 1, but `GatewayLogRedactorSeam` now calls `RedactCommandValue`, so a secured-bulk payload would be logged unredacted if value logging is enabled | Open | Add both bulk kinds plus a redactor test; cross-check every command kind that carries a password |
| R21-O17 | SEC-16 | Low | Security | `apikey --pepper` is accepted on the command line (`ApiKeyAdminCommandLineParser`) and lands in shell history and the process list | Open | Prefer an env var or stdin prompt; keep the flag only with a warning |
| R21-O18 | SEC-18 | Low | Security | `DashboardApiKeyManagementService` detects a missing pepper by matching `"pepper unavailable"` in an exception message | Open | Catch a typed exception from `ZB.MOM.WW.Auth.ApiKeys`, or check the pepper before the call |
| R21-O19 | SEC-23 | Low | Security | The options validator does not reject a `__Host-` `CookieName` combined with `RequireHttpsCookie=false`, a cookie browsers silently drop | Open | Add the cross-field check to `ValidateDashboard` |
| R21-O20 | SEC-24 | Low | Security | `EffectiveDashboardConfiguration` omits `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie` and `CookieName`, so the Settings page cannot show that login is disabled | Open | Project the four fields in `GatewayConfigurationProvider` |
| R21-O21 | CLI-07 | Medium | .NET client | `ExecuteSafeUnaryAsync` wraps the retry pipeline in `CancelAfter(DefaultCallTimeout)` while each attempt has the same deadline, so a `DeadlineExceeded` first attempt consumes the whole budget and retries never run | Open | Make the outer budget `MaxAttempts x DefaultCallTimeout` (or drop the outer cap); add a wire test with a slow first attempt |
| R21-O22 | CLI-09 | Medium | Go client | Go wraps every RPC failure in `GatewayError`; there is no typed `Unauthenticated` vs `PermissionDenied` mapping, though `ClientLibrariesDesign.md` requires one | Open | Add sentinel or typed errors usable with `errors.Is`; keep credential redaction |
| R21-O23 | CLI-11 | Medium | Go client | `mxgw-go` has no flag for strict TLS validation, so `--tls` without a CA is always skip-verify | Open | Add `--require-certificate-validation` wired to `Options.RequireCertificateValidation` |
| R21-O24 | CLI-19 | Low | .NET client | `InternalsVisibleTo` is declared in both `Properties/AssemblyInfo.cs` and the csproj | Open | Delete one (cosmetic) |
| R21-O25 | CLI-22 | Low | Go client | `newCorrelationID` returns an empty string if `crypto/rand` fails | Open | Fall back to a counter-based id |
| R21-O26 | CLI-27 | Low | Python client | `Session.close()` has no lock around `_closed`, and a repeat close returns a synthesized reply | Open | Guard with an `asyncio.Lock` and cache the real reply, as .NET and Go do |
| R21-O27 | CLI-31 | Low | Rust client | The Rust CLI `main.rs` is now 3,023 lines | Open | Split by command group when next touched |
| R21-O28 | CLI-33 | Low | Clients | Slow-consumer behavior is documented in the Go README but not in the Java README or the cross-language `docs/ClientBehaviorFixtures.md` | Open | Add one comparison table now that this cycle changed the Go and Java buffers |
| R21-O29 | TST-07 | Low | Tests | Real-clock sleeps with negative assertions remain (for example `WorkerClientTests.cs:975`, 150 ms) | Open | Convert to a fake time provider where one exists; otherwise leave |
| R21-O30 | TST-09 | Medium | Ops | Health checks cover the auth store and sessions only; Galaxy SQL, LDAP and worker-executable presence are unchecked (`GatewayApplication.cs:118`) | Open | Add a Ready-tag worker-exe presence check first, then a bounded Galaxy SQL ping; zero sessions must stay Healthy because the Overview dashboard polls the Active tier |
| R21-O31 | TST-10 | Medium | Ops | No deploy or upgrade script or runbook for the NSSM-wrapped service | Open | Write a deployment doc from the windev/wonder deploy notes; respect the current ban on remote execution on the wonder boxes |
| R21-O32 | TST-18 | Low | Tests | `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` timer/startup wiring has no direct tests | Open | Add timer and startup tests with a fake time provider |
| R21-O33 | TST-21 | Low | Ops | Serilog file sink rolls daily with no `retainedFileCountLimit` or size cap | Open | Set `retainedFileCountLimit` and `fileSizeLimitBytes` in `appsettings.json`; document |
| R21-O34 | TST-22 | Low | Docs | The Configuration Shape JSON block in `docs/GatewayConfiguration.md` omits `DisableLogin`, `AutoLoginUser`, `CookieName`, `Tls` and `Alarms:Fallback` | Open | Add the keys to the block (docs-only) |
## Cycle-1 triage summary
The 68 cycle-1 rows that were `Not started` with no later re-listing resolved as follows.
| Verdict | Count | Rows |
|---|:-:|---|
| Done | 21 | WRK-02, WRK-05, WRK-09, WRK-13, WRK-20, GWC-10, GWC-12, GWC-22, IPC-10, IPC-12, SEC-14, SEC-17, SEC-19, SEC-21, SEC-26, SEC-27, CLI-05, CLI-06, CLI-10, CLI-13, TST-14 |
| Won't fix | 13 | GWC-11, IPC-11, SEC-15, CLI-14, CLI-17, CLI-20, CLI-23, CLI-25, CLI-28, CLI-32, TST-19, TST-20, WRK-14 |
| Open (promoted) | 34 | `R21-O1` through `R21-O34` above |
`WRK-02`, `WRK-05` and `WRK-09` are closed by stream E and become plain `Done` when `fix/review-2026-10-21-e` merges.
The seven rows singled out for assessment: `WRK-03` Open (`R21-O1`: the worker still returns silently from `TryStartCommandTask` after shutdown starts); `WRK-05` Done (stream E); `CLI-06` Done (stream F); `CLI-07` Open (`R21-O21`: the outer `CancelAfter` still equals the per-attempt deadline); `CLI-09` Open (`R21-O22`: no typed auth errors in Go); `CLI-11` Open (`R21-O23`: no strict-TLS flag on `mxgw-go`); `CLI-13` Done (stream G).
## Change log
| Date | Change |
|---|---|
| 2026-10-22 | Initial tracker from the 2026-10-21 review outcomes. Stream E is recorded as done but unmerged; stream L items are in progress. 68 cycle-1 rows re-triaged and the 34 still-open ones promoted. |
## Final integration review (stream M)
| ID | Severity | Area | Finding | Status | Commit | Notes |
|---|---|---|---|---|---|---|
| R21-M1 | Medium | Sessions | A worker that dies right after its handshake left the session `Ready` with a dead worker until the lease expired | Done | M: `a178d68` | Attach refuses a terminal worker; the pump observes terminal state on its first read |
| R21-M2 | Low | Metrics | One worker fault was counted 1+K times (K = number of `StreamEvents` subscribers) | Done | M: `a0b5e91` | |
| R21-M3 | Low | Sessions | A cancelled open, or one drained at shutdown, was reported as `OpenFailed` | Done | M: `a178d68` | Now `Cancelled` and `GatewayStopping` respectively |
| R21-M4 | Low | Docs | Unattached-grace exemptions, subscriber-limit status code, stale event-stream description in Grpc.md | Done | M: `3ffba83` | |
| R21-M5 | Low | Docs | Client docs described the ReadBulk timeout as per-tag | Done | M: `dd62335` | Worker-side comments (`MxAccessCommandExecutor.cs:15`, `MxAccessValueCache.cs:219`, `MxAccessValueCacheTests.cs:246`) still say per-tag: Open |
+70 -69
View File
@@ -11,7 +11,7 @@ Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation
- Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` (depends-on) column within a tier.
- When a fix lands: flip Status to `Done`, and per the repo rule, update the affected docs in the same commit.
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action).
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action) · `Open` (re-triaged 2026-10-21: still present, promoted to the [2026-10-21 tracker](../2026-10-21/remediation/00-tracking.md)).
## Severity roll-up
@@ -63,24 +63,24 @@ Full design + implementation for each row lives in the linked domain doc under i
| GWC-02 | High | P0 | M | — | Done | Faulted sessions are never swept |
| GWC-03 | High | P0 | S | — | Done | Documented sparse-array max-length bound is unimplemented |
| GWC-04 | Medium | P1 | M | — | Done | Full event channel stalls the worker read loop behind command replies |
| GWC-05 | Medium | — | S | — | Not started | Worker pipe created with no ACL / no CurrentUserOnly |
| GWC-05 | Medium | — | S | — | Open | Worker pipe created with no ACL / no CurrentUserOnly — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O8 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-06 | Medium | P2 | S | GWC-07,08 | Done | Stopwatch allocated per streamed event |
| GWC-07 | Medium | P2 | S | GWC-06,08 | Done | Every mapped event is deep-cloned |
| GWC-08 | Medium | P2 | M | GWC-06,07 | Done | Pipe framing allocates per frame and writes twice |
| GWC-09 | Medium | — | M | — | Not started | Startup probe is a no-op; its retry pipeline can never retry |
| GWC-10 | Medium | — | S | — | Not started | Envelope `sequence` monotonicity specified but not enforced |
| GWC-11 | Low | — | S | — | Not started | `_workerClient` written under lock but read lock-free |
| GWC-12 | Low | — | S | — | Not started | `WorkerClient.DisposeAsync` not safe against double-dispose |
| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
| GWC-09 | Medium | — | M | — | Open | Startup probe is a no-op; its retry pipeline can never retry — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O9 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-10 | Medium | — | S | — | Done | Envelope `sequence` monotonicity specified but not enforced — **2026-10-21 triage:** resolved by spec change, not enforcement: `gateway.md:440` states `sequence` is a diagnostic aid, stamped in each side's single write path and not validated on receive |
| GWC-11 | Low | — | S | — | Won't fix | `_workerClient` written under lock but read lock-free — **2026-10-21 triage:** reference reads are atomic and every ready/state decision reads `_workerClient` under `_syncRoot` (`EvaluateReadyUnderLock`); only diagnostic properties read it lock-free |
| GWC-12 | Low | — | S | — | Done | `WorkerClient.DisposeAsync` not safe against double-dispose — **2026-10-21 triage:** closed by cycle-3 GWC-32 (`DisposeAsync` claims `_disposed` under `_syncRoot`; see the 2026-08-19 tracker) |
| GWC-13 | Low | — | M | — | Open | Worker-ready wait is a 25 ms poll loop — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O10 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-14 | Low | P2 | M | — | Done | Replay ring is a `LinkedList` with a node alloc per event |
| GWC-15 | Low | P2 | S | — | Done | Per-event gauge reads `ChannelReader.Count` every event |
| GWC-16 | Low | — | S | — | Won't fix | `Invoke` resolves the session twice per command — measured negligible, deliberately not collapsed (see domain doc) |
| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
| GWC-20 | Low | — | S | — | Not started | Heartbeat config semantics conflated (send vs check interval) |
| GWC-21 | Low | — | S | GWC-04 | Not started | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable |
| GWC-22 | Low | — | S | — | Not started | `StreamDisconnected` always labeled `"Detached"` |
| GWC-17 | Low | — | M | — | Open | Sessions layer throws `Grpc.Core.RpcException` (layering leak) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O11 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-18 | Low | — | S | — | Open | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O12 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-19 | Low | — | S | — | Open | Dead `GatewaySession.KillWorker(string)` + stale doc — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O13 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-20 | Low | — | S | — | Open | Heartbeat config semantics conflated (send vs check interval) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O14 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-21 | Low | — | S | GWC-04 | Open | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O15 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| GWC-22 | Low | — | S | — | Done | `StreamDisconnected` always labeled `"Detached"` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G7), `6ae5ed9`: `StreamDisconnected` records the real reason |
| GWC-23 | Info | — | — | — | N/A | `MaxEventSubscribersPerSession` dead in default single-subscriber mode |
### Worker — [20-worker.md](20-worker.md)
@@ -88,25 +88,25 @@ Full design + implementation for each row lives in the linked domain doc under i
| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| WRK-01 | High | P0 | M | — | Done | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
| WRK-02 | Medium | — | M | — | Not started | STA thread death after startup is silent; future work hangs forever |
| WRK-03 | Medium | — | S | WRK-02 | Not started | Commands after shutdown starts are dropped with no reply |
| WRK-02 | Medium | — | M | — | Done | STA thread death after startup is silent; future work hangs forever — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K6): an STA loop fault raises a `StaHung` WorkerFault; stream E `7618caf`, merged |
| WRK-03 | Medium | — | S | WRK-02 | Open | Commands after shutdown starts are dropped with no reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O1 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-04 | Medium | — | M | — | Done | Envelope `sequence` can appear out of order on the wire |
| WRK-05 | Medium | — | M | — | Not started | One transient alarm-poll failure kills the whole session |
| WRK-05 | Medium | — | M | — | Done | One transient alarm-poll failure kills the whole session — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K3): consecutive alarm-poll failure threshold of 3 in `MxAccessStaSession`; stream E `7618caf`, merged |
| WRK-06 | Medium | P2 | S | — | Done | `MXSTATUS_PROXY` conversion reflects per field, per event |
| WRK-07 | Medium | P1 | M | WRK-04 | Done | Documented outbound write priority not implemented |
| WRK-08 | Low | — | S | — | Not started | Residual event queue discarded at graceful shutdown, undocumented |
| WRK-09 | Low | — | S | — | Not started | No `AppDomain.UnhandledException` hook |
| WRK-10 | Low | — | S | — | Not started | Top-level catch logs exception type but never message |
| WRK-08 | Low | — | S | — | Open | Residual event queue discarded at graceful shutdown, undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O2 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-09 | Low | — | S | — | Done | No `AppDomain.UnhandledException` hook — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-K7): `AppDomain.UnhandledException` hook, exit code 7; stream E `7618caf`, merged |
| WRK-10 | Low | — | S | — | Open | Top-level catch logs exception type but never message — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O3 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-11 | Low | P2 | S | — | Done | Every accepted event is defensively cloned on enqueue |
| WRK-12 | Low | P2 | M | WRK-04 | Done | No event batching per envelope; one flush per event; 25 ms poll |
| WRK-13 | Low | — | S | — | Not started | Frame writer allocates a fresh buffer per frame; reader pools |
| WRK-14 | Low | — | M | — | Not started | Private-field naming split `_camelCase` vs `camelCase` |
| WRK-13 | Low | — | S | — | Done | Frame writer allocates a fresh buffer per frame; reader pools — **2026-10-21 triage:** worker `WorkerFrameWriter` rents the frame from `ArrayPool` (`Ipc/WorkerFrameWriter.cs:675`); the reader was already pooled |
| WRK-14 | Low | — | M | — | Won't fix | Private-field naming split `_camelCase` vs `camelCase` — **2026-10-21 triage:** cosmetic; the style guide permits both `_camelCase` and bare `camelCase` where established, and a mass rename is churn without behavior change |
| WRK-15 | Low | P2 | S | — | Done | Doc drift: STA thread name and heartbeat-counter note |
| WRK-16 | Low | — | S | — | Not started | Boilerplate duplication in IPC envelope/ctor overloads |
| WRK-17 | Low | — | S | — | Not started | Gateway death exits with wrong exit code (6 not 5) |
| WRK-18 | Low | — | S | — | Not started | Event-queue overflow exits as generic `UnexpectedFailure` |
| WRK-19 | Low | — | S | — | Not started | Command start/end logging with correlation id absent |
| WRK-20 | Low | — | M | WRK-01..04 | Not started | Test-coverage gaps for the failure modes above |
| WRK-16 | Low | — | S | — | Open | Boilerplate duplication in IPC envelope/ctor overloads — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O4 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-17 | Low | — | S | — | Open | Gateway death exits with wrong exit code (6 not 5) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O5 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-18 | Low | — | S | — | Open | Event-queue overflow exits as generic `UnexpectedFailure` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O6 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-19 | Low | — | S | — | Open | Command start/end logging with correlation id absent — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O7 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| WRK-20 | Low | — | M | WRK-01..04 | Done | Test-coverage gaps for the failure modes above — **2026-10-21 triage:** STA-death, sequence-monotonic, long-ReadBulk and late-reply tests exist; the one residual (command-after-shutdown reply) is carried by WRK-03 as R21-O1 |
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
@@ -121,9 +121,9 @@ Full design + implementation for each row lives in the linked domain doc under i
| IPC-07 | Medium | P2 | S | — | Done | `docs/Grpc.md` says six RPCs; there are seven |
| IPC-08 | Medium | — | M | — | Done | `WorkerCancel` defined and handled but never sent — resolved by `7c1ea12`/`a1a38b5`; residual (caller-cancellation leg) tracked as cycle-3 IPC-34 |
| IPC-09 | Medium | P1 | M | IPC-01 | Done | Known codegen fragilities have no in-repo guards |
| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
| IPC-12 | Low | — | S | — | Not started | Gateway frame writer has no write lock; integrity rests on undocumented invariant |
| IPC-10 | Low | — | S | — | Done | Envelope `sequence` is write-only; monotonicity never validated — **2026-10-21 triage:** same resolution as GWC-10: `gateway.md:440` documents `sequence` as a diagnostic aid |
| IPC-11 | Low | — | S | — | Won't fix | No protocol version negotiation despite `supported_protocol_version` name — **2026-10-21 triage:** the review itself recommended no change; exact-match protocol version is correct for a lockstep-deployed pair, and the next bump should introduce a min/max range |
| IPC-12 | Low | — | S | — | Done | Gateway frame writer has no write lock; integrity rests on undocumented invariant — **2026-10-21 triage:** IPC-13 collapsed the gateway write to one pooled-buffer write, and `WorkerClient.WriteLoopAsync` is its only caller |
| IPC-13 | Low | P2 | S | IPC-05 | Done | Gateway writer serializes twice and issues two stream writes |
| IPC-14 | Low | P2 | S | IPC-05 | Done | Gateway reader allocates fresh array per frame; worker rents from `ArrayPool` |
| IPC-15 | Low | P2 | M | — | Done | Worker event delivery 25 ms poll with per-event write+flush, no batching |
@@ -151,21 +151,21 @@ Full design + implementation for each row lives in the linked domain doc under i
| SEC-10 | Medium | P1 | L | — | Done | No API-key expiry |
| SEC-11 | Medium | P1 | M | SEC-08 | Done | No rate limiting or lockout on either auth surface |
| SEC-12 | Medium | P1 | M | — | Done | Dashboard Close/Kill bypass the canonical audit store |
| SEC-13 | Low | — | S | SEC-30 | Not started | Redactor's credential-command list omits secured-bulk variants |
| SEC-14 | Low | — | S | SEC-02,20 | Not started | `/metrics` + `/health` unauthenticated; a metric leaks session ids |
| SEC-15 | Low | — | S | — | Not started | GET `/logout` skips antiforgery |
| SEC-16 | Low | — | S | — | Not started | CLI accepts the pepper as a command-line argument |
| SEC-17 | Low | — | S | — | Not started | Interceptor does not override client-streaming/duplex handlers |
| SEC-18 | Low | — | S | — | Not started | Pepper-unavailable detection matches library message text |
| SEC-19 | Low | — | S | — | Not started | Canonical audit store re-issues `CREATE TABLE` per write/read |
| SEC-13 | Low | — | S | SEC-30 | Open | Redactor's credential-command list omits secured-bulk variants — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O16 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-14 | Low | — | S | SEC-02,20 | Done | `/metrics` + `/health` unauthenticated; a metric leaks session ids — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G11), `65bca80` + `82a856b`: opt-in `Metrics:RequireAuthentication`; `/health` stays open by design |
| SEC-15 | Low | — | S | — | Won't fix | GET `/logout` skips antiforgery — **2026-10-21 triage:** accepted nuisance: GET `/logout` skips antiforgery deliberately (comment in `DashboardEndpointRouteBuilderExtensions.cs`); the POST path validates antiforgery |
| SEC-16 | Low | — | S | — | Open | CLI accepts the pepper as a command-line argument — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O17 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-17 | Low | — | S | — | Done | Interceptor does not override client-streaming/duplex handlers — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-G2), `19cc74f`: client- and duplex-streaming calls authenticate and fail closed to admin |
| SEC-18 | Low | — | S | — | Open | Pepper-unavailable detection matches library message text — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O18 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-19 | Low | — | S | — | Done | Canonical audit store re-issues `CREATE TABLE` per write/read — **2026-10-21 triage:** `SqliteCanonicalAuditStore` latches schema creation, so `CREATE TABLE IF NOT EXISTS` runs once (a failure leaves the latch open) |
| SEC-20 | Low | P1 | S | — | Done | `session_id` metric tag is unbounded cardinality |
| SEC-21 | Low | — | M | — | Not started | Snapshot publisher works every second regardless of audience |
| SEC-21 | Low | — | M | — | Done | Snapshot publisher works every second regardless of audience — **2026-10-21 triage:** `DashboardSnapshotFeed` is idle-gated (no viewers, no timer), API-key summaries refresh every 15 s, and configuration is sent once per connection (R21-D4, `6733519`) |
| SEC-22 | Low | P2 | M | — | Done | `docs/Authentication.md` documents types no longer in this repo |
| SEC-23 | Low | — | S | SEC-03,04 | Not started | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` |
| SEC-24 | Low | — | S | SEC-04 | Not started | Effective-config view omits the riskiest dashboard flags |
| SEC-23 | Low | — | S | SEC-03,04 | Open | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O19 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-24 | Low | — | S | SEC-04 | Open | Effective-config view omits the riskiest dashboard flags — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O20 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| SEC-25 | Low | P2 | M | SEC-02 | Done | Per-session EventsHub ACL is an acknowledged TODO |
| SEC-26 | Low | — | M | — | Not started | Audit trail has no retention or pruning |
| SEC-27 | Low | — | S | — | Not started | Dashboard `GatewayStatus` is hardcoded Healthy |
| SEC-26 | Low | — | M | — | Done | Audit trail has no retention or pruning — **2026-10-21 triage:** `Security/Audit/AuditDrainService.cs` prunes by `AuditRetentionDays` (validated in `GatewayOptionsValidator`) on an hourly sweep |
| SEC-27 | Low | — | S | — | Done | Dashboard `GatewayStatus` is hardcoded Healthy — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-D9), `6733519`: `GatewayStatus` derives from health checks |
| SEC-28 | Info | — | S | — | N/A | Positive security observations (preserve under change) |
| SEC-29 | Info | — | S | — | N/A | UI-stack rule verified compliant |
| SEC-30 | Info | P2 | S | SEC-13 | Done | Value-logging feature is unwired end-to-end |
@@ -178,35 +178,35 @@ Full design + implementation for each row lives in the linked domain doc under i
| CLI-02 | High | P1 | M | — | Done | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) |
| CLI-03 | High | P0 | M | — | Done | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
| CLI-04 | High | P2 | L | CLI-15 | Done | Typed-command parity gap across all five clients |
| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
| CLI-05 | Medium | — | S | — | Done | .NET session cannot be re-attached to an existing session id — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN5), `6f0b408`: `MxGatewayClient.AttachSession` |
| CLI-06 | Medium | — | S | — | Done | .NET `DisposeAsync` blocks/throws on unreachable gateway — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CN3), `6f0b408`: `MxGatewaySession.DisposeAsync` is best-effort and never throws |
| CLI-07 | Medium | — | S | — | Open | .NET retry budget self-defeats on `DeadlineExceeded` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O21 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-08 | Medium | — | S | CLI-03 | Done | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) — landed via 2026-07-12 [CLI-38](../2026-07-12/remediation/50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1) |
| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
| CLI-09 | Medium | — | M | — | Open | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O22 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-10 | Medium | — | M | — | Done | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CG2), `a3a7bd2`: `grpc.NewClient` replaces `DialContext` + `WithBlock` |
| CLI-11 | Medium | — | S | — | Open | Go CLI cannot opt into strict TLS validation — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O23 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-12 | Medium | P2 | S | — | Done | Java docs still say "Java 21" after the JDK 17 retarget |
| CLI-13 | Medium | — | M | — | Not started | Java event-stream buffer hardcoded 16, cancel-on-overflow |
| CLI-14 | Medium | — | S | — | Not started | Python default TLS is blocking TOFU pin with silent `localhost` SNI |
| CLI-13 | Medium | — | M | — | Done | Java event-stream buffer hardcoded 16, cancel-on-overflow — **2026-10-21 triage:** closed by the 2026-10-21 review (R21-CJ1/CJ2), `48c9323`: event queue is configurable (default 1024) and overflow keeps buffered events |
| CLI-14 | Medium | — | S | — | Won't fix | Python default TLS is blocking TOFU pin with silent `localhost` SNI — **2026-10-21 triage:** documented and intentional: the Python README describes the trust-on-first-use default and `--require-certificate-validation`; the `localhost` SNI default matches the gateway's self-signed SAN (`options.py:160`) |
| CLI-15 | Medium | P2 | M | — | Done | No client handles `ReplayGap` or offers a reconnect helper |
| CLI-16 | Medium | P2 | S | CLI-12 | Done | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` |
| CLI-17 | Medium | — | M | CLI-14 | Not started | TLS default posture inconsistent across the five clients |
| CLI-17 | Medium | — | M | CLI-14 | Won't fix | TLS default posture inconsistent across the five clients — **2026-10-21 triage:** documented divergence (`docs/CrossLanguageSmokeMatrix.md`); a strict default would break every deployment on the gateway's self-signed certificate. Strict opt-in exists per client; the Go CLI gap is R21-O23 |
| CLI-18 | Low | P2 | S | — | Done | .NET csproj records no `<Version>` |
| CLI-19 | Low | — | S | — | Not started | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) |
| CLI-20 | Low | — | S | CLI-17 | Not started | .NET `--tls` without CA installs accept-all callback |
| CLI-19 | Low | — | S | — | Open | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O24 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-20 | Low | — | S | CLI-17 | Won't fix | .NET `--tls` without CA installs accept-all callback — **2026-10-21 triage:** same decision as CLI-17: documented, intentional default; `RequireCertificateValidation` opts in |
| CLI-21 | Low | P2 | S | — | Done | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases |
| CLI-22 | Low | — | S | — | Not started | Go `newCorrelationID` swallows `crypto/rand` error → empty id |
| CLI-23 | Low | — | S | — | Not started | Go nil-vs-empty bulk short-circuit asymmetry |
| CLI-22 | Low | — | S | — | Open | Go `newCorrelationID` swallows `crypto/rand` error → empty id — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O25 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-23 | Low | — | S | — | Won't fix | Go nil-vs-empty bulk short-circuit asymmetry — **2026-10-21 triage:** cosmetic; an empty bulk is a local no-op in some helpers and a wire call in others |
| CLI-24 | Low | — | S | — | Done | Java `MxEventStream` single-consumer constraint undocumented (closed 2026-08-07 per 2026-07-12 review old-tracker action; documented at MxEventStream.java:25 "Single consumer") |
| CLI-25 | Low | — | S | — | Not started | Java `close()` does not await channel termination |
| CLI-25 | Low | — | S | — | Won't fix | Java `close()` does not await channel termination — **2026-10-21 triage:** `closeAndAwaitTermination()` is the documented awaiting path; the leak window is bounded by JVM exit |
| CLI-26 | Low | P2 | S | — | Done | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) |
| CLI-27 | Low | — | S | — | Not started | Python `Session.close()` not concurrency-safe; synthesizes reply |
| CLI-28 | Low | — | S | — | Not started | Python circular-import workaround at end of `session.py` |
| CLI-27 | Low | — | S | — | Open | Python `Session.close()` not concurrency-safe; synthesizes reply — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O26 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-28 | Low | — | S | — | Won't fix | Python circular-import workaround at end of `session.py` — **2026-10-21 triage:** the import cycle is deliberate, marked `# noqa: E402`, and covered by the Python test suite |
| CLI-29 | Low | P2 | S | — | Done | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) |
| CLI-30 | Low | — | S | CLI-04 | Done | Rust has no `unregister` typed helper |
| CLI-31 | Low | — | M | — | Not started | Rust CLI is a single 2,699-line `main.rs` |
| CLI-32 | Low | — | S | — | Not started | Client-side bulk caps differ (.NET/Java unbounded) |
| CLI-33 | Low | — | S | CLI-01,13 | Not started | Per-language event backpressure semantics undocumented |
| CLI-31 | Low | — | M | — | Open | Rust CLI is a single 2,699-line `main.rs` — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O27 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-32 | Low | — | S | — | Won't fix | Client-side bulk caps differ (.NET/Java unbounded) — **2026-10-21 triage:** the gateway is the authority for bulk caps; the divergence affects only the error type for an oversized call |
| CLI-33 | Low | — | S | CLI-01,13 | Open | Per-language event backpressure semantics undocumented — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O28 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| CLI-34 | Low | — | S | — | Done | Python `build/`/`.pytest_cache/` present on disk (untracked) (closed 2026-08-07 per 2026-07-12 review old-tracker action; both gitignored in clients/python/.gitignore) |
### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)
@@ -219,22 +219,22 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
| TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *` → `scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
| TST-06 | Medium | — | M | — | Materially narrowed | Dashboard live-data path untested — `DashboardLiveDataServiceTests`, `EventsHubTests`, `EventsHubViewerRegistryTests`, `AlarmsHubPublisherTests`, `SessionDetailsPageEventAclTests` now exist (landed with SEC-25/TST-16); needs a re-audit-and-close pass, not re-implementation |
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
| TST-07 | Medium | — | S | — | Open | Real-clock sleeps with negative assertions are latent flakes — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O29 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
| TST-10 | Medium | — | M | — | Not started | Deployment/upgrade undocumented and hand-rolled |
| TST-09 | Medium | — | M | — | Open | Health checks cover only the auth store — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O30 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-10 | Medium | — | M | — | Open | Deployment/upgrade undocumented and hand-rolled — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O31 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-11 | Medium | P2 | M | — | Done | No version discipline on server; client versions drift |
| TST-12 | Medium | P0 | S | — | Done | CLAUDE.md misstates default retention behaviour |
| TST-13 | Medium | P2 | S | — | Done | gateway.md carries stale design-era sketches |
| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
| TST-14 | Medium | P2 | S | — | Done | Repo-root working artifacts need triage — **2026-10-21 triage:** root `oldtasks.md` and the `*-docs-*.md` working files are no longer in the tree; `stillpending.md` remains as the backlog and stream L is refreshing it |
| TST-15 | Medium | P2 | M | TST-04 | Done | Dashboard EventsHub has no per-session ACL |
| TST-16 | Medium | P1 | S | — | Done | `Dashboard:ShowTagValues` is a dead flag |
| TST-17 | Medium | — | S | — | Materially changed | Vendor-gated alarm parity residuals silently lossy — `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName`; the vendor 8-arg leg is recorded "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, `d1ae43d`, `540a118`); "silent drop, unchanged" is no longer accurate — needs a worker-domain refresh pass |
| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
| TST-18 | Low | — | S | — | Open | Hosted-service wrappers untested — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O32 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-19 | Low | — | S | — | Won't fix | Keep FakeWorkerHarness canned replies in lockstep — **2026-10-21 triage:** ongoing discipline rather than a work item; the harness shares production framing and the nightly live tier (`nightly-windev`) catches canned-reply drift |
| TST-20 | Low | — | S | — | Won't fix | E2E script papers over a real advise-without-consumer sharp edge — **2026-10-21 triage:** documented sharp edge in `docs/GatewayTesting.md`; the e2e script's interleaved drains are the intended workaround |
| TST-21 | Low | — | S | — | Open | Log rotation configured but minimal — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O33 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-22 | Low | — | S | — | Open | Config-shape JSON block omits documented keys — **2026-10-21 triage:** still present on the integration branch; promoted to R21-O34 in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) |
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
| TST-24 | Low | P2 | M | TST-03 | Done | Client wire behaviour has no automated verification — closed 2026-08-10. Go/Rust/Java already had real-server wire tests (the finding's premise was stale); the genuine gaps were .NET (transport-interface fake everywhere, no server package) and Python (stub monkeypatch everywhere but one opt-in TLS test). Added `MxGatewayClientWireTests` + `WireFakeGatewayServer` (Kestrel h2c) and `tests/test_wire_fake_gateway.py` (`grpc.aio` loopback), plus a `dotnet test` step for the .NET client in the `portable` CI job. Caught a real bug: Python `connect()` built the `grpc.aio` channel inside `asyncio.to_thread` and failed for every non-stub connection |
@@ -253,6 +253,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
| Date | Change |
|---|---|
| 2026-10-22 | **Cycle-1 `Not started` rows re-triaged against the 2026-10-21 integration branch** (stream K). All 68 rows that no later tracker had re-listed now carry a verdict: 21 `Done` (closed by earlier work or by the 2026-10-21 review), 13 `Won't fix` (reason on the row), 34 `Open` (promoted as `R21-O1`..`R21-O34` in [`../2026-10-21/remediation/00-tracking.md`](../2026-10-21/remediation/00-tracking.md) with a next step). Rows closed by stream E are marked pending merge of that stream. |
| 2026-08-17 | **TST-16 → `Done`: `Dashboard:ShowTagValues` now gates every dashboard value surface** (branch `feat/followup-closeout`). The flag was live on one seam only (the events-hub mirror, shipped with SEC-25); the two recorded residuals are closed, so it is no longer a partly-dead flag. **Alarms hub:** `AlarmsHubPublisher` took `IOptions<GatewayOptions>` and now clears `current_value`/`limit_value` from a **deep clone** of each `AlarmFeedMessage` on both value-bearing payload arms (`transition`, `active_alarm`) before the `/hubs/alarms` broadcast. Clone-only, matching `DashboardEventBroadcaster.RedactValues`: the source message fans out from one feed to gRPC `StreamAlarms` subscribers and the alarms page, none of which this dashboard-display flag governs. The redaction switches on the value-bearing arms and passes everything else through untouched and uncloned, so a payload arm added later (the `snapshot_status` arm landing alongside this change) is forwarded unbroken rather than silently dropped. **`/browse`:** the gate lives at the service boundary per TST-16's own recommendation — `DashboardLiveDataService` substitutes `DashboardTagValue.RedactedValueText` (`[redacted]`) for the formatted value text, so the value never enters the render tree and `BrowsePage.razor` needed no change; data type, quality, source timestamp, and read errors stay real, keeping the panel diagnostic. A visible marker, not a blank, so a suppressed value is distinguishable from a tag that read back empty. Tests: new `AlarmsHubPublisherTests` (4 — both arms redacted with metadata intact, source not mutated, `true` passes values through as the same instance, valueless arms forwarded as the same instance) and two `DashboardLiveDataServiceTests` cases (`false` → `[redacted]`, `true` → formatted); `DashboardEventBroadcasterTests` re-run as regression. Docs: `docs/GatewayConfiguration.md`'s `ShowTagValues` row rewritten from "gates only the events hub mirror, two still-open residuals" to the three seams it now covers, with the "no gRPC client is affected" scope statement; `docs/GatewayDashboardDesign.md` gained the redaction note on the `AlarmsHub` row and had its Browse-panel paragraph corrected (it claimed the panel "always shows values regardless of `Dashboard:ShowTagValues`"). |
| 2026-08-17 | **Alarm-snapshot truncation now has a structural degraded-status signal** (branch `feat/deferred-closeout`, commits `693a78d` + `b9fb0dd`). No review ID — this is branch work outside the 153-finding register, recorded here so the tracker is not silent on a shipped change to the alarm surface. Before it, a capped `GetXmlCurrentAlarms2` fetch suppressed absence-implies-Clear inference and said so only in a rate-limited worker stderr warning, so no client and no operator could tell a complete active set from a capped one. Two additive proto3 booleans carry the verdict out — `QueryActiveAlarmsReplyPayload.snapshot_truncated = 2` and `ActiveAlarmSnapshot.from_truncated_snapshot = 16` (per record, because `QueryActiveAlarms` is a bare `stream ActiveAlarmSnapshot` with no envelope; the reply payload states it too, since a prefix filter can leave zero records and a capped fetch with nothing to report still has to say so). Flow: `WnWrapAlarmConsumer` → `AlarmDispatcher` / `IAlarmCommandHandler` → `MxAccessCommandExecutor` reply → `GatewayAlarmMonitor` → `IGatewayAlarmService.SnapshotTruncated` → `AlarmsPage` banner. `b9fb0dd` then made the pairing structural after review: `IMxAccessAlarmConsumer` and `IAlarmCommandHandler` expose one accessor (`SnapshotActiveAlarms(out bool truncated)` / `QueryActive(..., out bool snapshotTruncated)`) satisfied from a single lock acquisition, so the snapshot and its verdict can no longer be read across a poll; the separate `LastSnapshotTruncated` property is gone from every layer. Detection is deliberately unchanged (`fetchedRecordCount >= maxAlarmsPerFetch`); switching to `ALARM_RECORDS/@COUNT` stays blocked on probe evidence (`docs/AlarmProbeFindings.md`). Not latched, and dropped with the cache generation by `ClearCache`. Additive gateway metadata about our fetch mechanics, not MXAccess behaviour — no synthesized event, so not a parity deviation. Docs: `gateway.md` alarm surface, `docs/DesignDecisions.md`. |
| 2026-08-17 | **TST-15 → `Done` (discharges the ACL half of SEC-25): per-session dashboard event ACL shipped** (branch `feat/deferred-closeout`, commits `693a78d` + `7ec0b35`). Implements `docs/plans/2026-07-10-dashboard-session-acl-tst15.md`, whose header is now `Implemented` with as-built notes in its §12. `IDashboardSessionAcl.CanViewSession` is the single decision **both** subscribe seams consult — `EventsHub.SubscribeSession` (denial is a `HubException`; the caller is neither joined to the group nor registered in `EventsHubViewerRegistry`, so the mirror stays off) and `SessionDetailsPage`'s in-process subscribe (inline denial, no subscription) — so neither path is the weaker one and the `TODO(per-session-acl)` is gone. Decision order, fail-closed on every branch: authenticated Administrator → allow (evaluated **before** the registry lookup, so Admin × unknown-session allows — pinned by a test because reordering the two checks is a plausible refactor); session not found → deny; untagged session → `Dashboard:UntaggedSessionVisibility` (`AdminOnly` default); else allow iff `session.Tags ∩ zb:dashboardtag` claims, ordinal-ignore-case. Session tags are inherited from the owning API key's `dashboard_tags` constraint (`apikey --dashboard-tags`, already in the `ApiKeyConstraints` JSON blob — no SQLite migration) and never from the client's wire request. Viewer grants come from `Dashboard:GroupToTag` over the user's LDAP groups, stamped at cookie login (`DashboardAuthenticator.CreatePrincipal`) and **re-resolved, not copied**, at hub-token mint (`HubTokenService.Issue`), so the 5-minute token lifetime bounds a stale grant. Anonymous localhost is an empty-grant Viewer; `Dashboard:DisableLogin` auto-login carries both roles and so takes the admin bypass unchanged. Tests: `DashboardSessionAclTests` (decision table, every branch asserted in its denying direction too), `EventsHubTests`, `DashboardAuthenticatorTests`, `HubTokenServiceTests`, a `GatewayOptionsTests` case proving `Dashboard:GroupToTag` keeps its ordinal-ignore-case lookup through configuration binding, and two `[LiveLdapFact]`s in `DashboardLdapLiveTests` that drive a real bind against the shared GLAuth (`gw-viewer` → `team-a` grant admits the `team-a` session and refuses the `team-b` one; `multi-role` bypasses on the sessions its own grant does not cover). The live pair needed **no GLAuth change** — the tag layer is config-side, keyed on the existing `GwAdmin`/`GwReader` groups (recorded in `glauth.md`). Docs: `docs/Sessions.md` (session-tag model), `gateway.md` + CLAUDE.md dashboard-auth paragraphs, `docs/GatewayDashboardDesign.md` (three passages that described the ACL as outstanding), `docs/GatewayConfiguration.md` (`ShowTagValues` row: redaction is now the second of two layers, not the only one), `docs/Authentication.md` (`--dashboard-tags` is the only *constraint* flag that splits on commas). |
+25 -1
View File
@@ -137,6 +137,30 @@ gateway's own sentinel observable. Normal events pass through with
`IsReplayGap == false` and `ReplayGap == null`. The gap is only ever produced by
`StreamEvents`; the diagnostic drain path never emits it. If you already consume
the raw `StreamEventsAsync` (or the client-level stream), the
### Caller-owned channels, resuming sessions, diagnostics
`MxGatewayClient.Create(GrpcChannel, MxGatewayClientOptions)` and
`GalaxyRepositoryClient.Create(GrpcChannel, MxGatewayClientOptions)` build a
client over a channel the caller created (shared channel, custom handler or
TLS). The client never disposes a caller-owned channel; `Create(options)` still
creates and owns its own channel.
`MxGatewayClient.AttachSession(sessionId)` returns an `MxGatewaySession` bound
to an existing session id without calling OpenSession, so a restarted process
can resume `StreamEventsAsync(afterWorkerSequence)`. The gateway only allows the
API key that opened the session to attach; any other key gets `PermissionDenied`.
The attached session's `OpenSessionReply` carries only the session id.
`MxGatewaySession` has typed diagnostics: `PingAsync`, `GetSessionStateAsync`,
`GetWorkerInfoAsync`, and `DrainEventsAsync` (removes queued events from the
worker; call until empty). `DisposeAsync` closes the session best-effort and does
not throw when the gateway is unreachable or the client is already disposed.
Retry applies only to idempotent calls (`CloseSessionRawAsync`, ping,
session-state and worker-info commands, Galaxy reads). `AcknowledgeAlarmAsync`
is never retried because it is a non-idempotent operator write that re-sends the
operator password and can double-acknowledge in the Historian.
`AsStreamItemsAsync()` extension projects any `IAsyncEnumerable<MxEvent>` into
the same `MxEventStreamItem` surface.
@@ -146,7 +170,7 @@ the active alarms the gateway's central monitor currently holds),
keyed by the same monitor), and `AcknowledgeAlarmAsync` (acknowledges as
the operator: the gateway authenticates `OperatorUser`/`OperatorPassword` and
writes the comment to the alarm's `AckMsg`, so the Historian records the
operator). All three accept a cancellation
operator). The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. All three accept a cancellation
token and pass through the `MxGateway:Alarms` configuration on the
server — when alarms are disabled, the gateway returns an empty list / empty
stream rather than failing.
@@ -81,24 +81,22 @@ public sealed class MxGatewayClientAlarmsTests
[Fact]
public async Task AcknowledgeAlarmAsync_HonorsCancellation()
{
// Acks are routed through the safe-unary retry pipeline (idempotent at the
// MxAccess level), so the transport-side cancellation token is a linked one
// rather than the caller's original. Verify cancellation by tripping the source
// and asserting the call observes it.
// Acks bypass the retry pipeline (non-idempotent), so the caller's token
// reaches the transport unchanged.
using CancellationTokenSource cancellation = new();
cancellation.Cancel();
FakeGatewayTransport transport = CreateTransport();
await using MxGatewayClient client = CreateClient(transport);
await Assert.ThrowsAnyAsync<OperationCanceledException>(() =>
client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest
{
AlarmFullReference = "Tank01.Level.HiHi",
Comment = string.Empty,
OperatorUser = "alice",
},
cancellation.Token));
await client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest
{
AlarmFullReference = "Tank01.Level.HiHi",
Comment = string.Empty,
OperatorUser = "alice",
},
cancellation.Token);
Assert.Equal(cancellation.Token, Assert.Single(transport.AcknowledgeAlarmCalls).CallOptions.CancellationToken);
}
/// <summary>AcknowledgeAlarmAsync maps unauthenticated RPC exception to typed exception.</summary>
@@ -0,0 +1,241 @@
using Grpc.Core;
using Grpc.Net.Client;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>Tests for ack no-retry, caller-owned channels, safe dispose, typed diagnostics, and AttachSession.</summary>
public sealed class MxGatewayClientRemediationTests
{
/// <summary>An acknowledge is a non-idempotent operator write and is never retried.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AcknowledgeAlarmAsync_DoesNotRetryTransientFailure()
{
FakeGatewayTransport transport = CreateTransport();
transport.AcknowledgeAlarmExceptions.Enqueue(new RpcException(new Status(StatusCode.Unavailable, "down")));
await using MxGatewayClient client = new(transport.Options, transport);
await Assert.ThrowsAsync<RpcException>(() => client.AcknowledgeAlarmAsync(
new AcknowledgeAlarmRequest { OperatorUser = "op", OperatorPassword = "pw" }));
Assert.Single(transport.AcknowledgeAlarmCalls);
}
/// <summary>A client created over a caller-owned channel does not dispose it.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Create_WithExistingChannel_DoesNotDisposeChannel()
{
using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost:1");
MxGatewayClient gateway = MxGatewayClient.Create(channel, Options());
GalaxyRepositoryClient galaxy = GalaxyRepositoryClient.Create(channel, Options());
await gateway.DisposeAsync();
await galaxy.DisposeAsync();
// A disposed channel throws ObjectDisposedException when a call is created.
var invoker = channel.CreateCallInvoker();
Exception? ex = Record.Exception(() => invoker.AsyncUnaryCall(
new Method<byte[], byte[]>(
MethodType.Unary,
"s",
"m",
Marshallers.Create(b => b, b => b),
Marshallers.Create(b => b, b => b)),
null,
new CallOptions(deadline: DateTime.UtcNow.AddMilliseconds(1)),
[]).Dispose());
Assert.IsNotType<ObjectDisposedException>(ex);
}
/// <summary>Create over an existing channel rejects a null channel.</summary>
[Fact]
public void Create_WithNullChannel_Throws()
{
Assert.Throws<ArgumentNullException>(() => MxGatewayClient.Create(null!, Options()));
Assert.Throws<ArgumentNullException>(() => GalaxyRepositoryClient.Create(null!, Options()));
}
/// <summary>DisposeAsync swallows a gateway failure during close and still disposes.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Session_DisposeAsync_SwallowsCloseFailure()
{
FakeGatewayTransport transport = CreateTransport();
transport.CloseSessionExceptions.Enqueue(new MxGatewayException("unreachable"));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await session.DisposeAsync();
Assert.Single(transport.CloseSessionCalls);
await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
}
/// <summary>DisposeAsync after the client is disposed does not throw.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Session_DisposeAsync_AfterClientDisposed_DoesNotThrow()
{
FakeGatewayTransport transport = CreateTransport();
MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await client.DisposeAsync();
await session.DisposeAsync();
await Assert.ThrowsAsync<ObjectDisposedException>(() => session.CloseAsync());
}
/// <summary>PingAsync sends a Ping command carrying the message.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task PingAsync_SendsPingCommand()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.Ping));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await session.PingAsync("hello");
MxCommandRequest request = Assert.Single(transport.InvokeCalls).Request;
Assert.Equal(MxCommandKind.Ping, request.Command.Kind);
Assert.Equal("hello", request.Command.Ping.Message);
Assert.Equal("session-fixture", request.SessionId);
}
/// <summary>GetSessionStateAsync returns the typed state.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetSessionStateAsync_ReturnsState()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.GetSessionState);
reply.SessionState = new SessionStateReply { State = SessionState.Ready };
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
SessionState state = await session.GetSessionStateAsync();
Assert.Equal(SessionState.Ready, state);
Assert.Equal(MxCommandKind.GetSessionState, Assert.Single(transport.InvokeCalls).Request.Command.Kind);
}
/// <summary>GetSessionStateAsync throws on a reply without payload.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetSessionStateAsync_MissingPayload_ThrowsMalformed()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.GetSessionState));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
await Assert.ThrowsAsync<MxGatewayMalformedReplyException>(() => session.GetSessionStateAsync());
}
/// <summary>GetWorkerInfoAsync returns the worker info payload.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GetWorkerInfoAsync_ReturnsWorkerInfo()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.GetWorkerInfo);
reply.WorkerInfo = new WorkerInfoReply { WorkerProcessId = 42, WorkerVersion = "1.2.3" };
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
WorkerInfoReply info = await session.GetWorkerInfoAsync();
Assert.Equal(42, info.WorkerProcessId);
Assert.Equal("1.2.3", info.WorkerVersion);
}
/// <summary>DrainEventsAsync sends max_events and returns the events.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DrainEventsAsync_ReturnsEventsAndSendsMax()
{
FakeGatewayTransport transport = CreateTransport();
MxCommandReply reply = OkReply(MxCommandKind.DrainEvents);
reply.DrainEvents = new DrainEventsReply();
reply.DrainEvents.Events.Add(new MxEvent { SessionId = "session-fixture", WorkerSequence = 5 });
transport.AddInvokeReply(reply);
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
IReadOnlyList<MxEvent> events = await session.DrainEventsAsync(10);
Assert.Equal(5UL, Assert.Single(events).WorkerSequence);
Assert.Equal(10U, Assert.Single(transport.InvokeCalls).Request.Command.DrainEvents.MaxEvents);
}
/// <summary>DrainEventsAsync treats a missing payload as an empty drain.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DrainEventsAsync_MissingPayload_ReturnsEmpty()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddInvokeReply(OkReply(MxCommandKind.DrainEvents));
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = await client.OpenSessionAsync();
Assert.Empty(await session.DrainEventsAsync());
}
/// <summary>AttachSession binds to an existing id without opening a session and resumes the stream.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AttachSession_BindsExistingIdWithoutOpening_AndResumesStream()
{
FakeGatewayTransport transport = CreateTransport();
transport.AddEvent(new MxEvent { SessionId = "existing", WorkerSequence = 8 });
await using MxGatewayClient client = new(transport.Options, transport);
MxGatewaySession session = client.AttachSession("existing");
List<MxEvent> received = [];
await foreach (MxEvent e in session.StreamEventsAsync(afterWorkerSequence: 7))
{
received.Add(e);
}
Assert.Empty(transport.OpenSessionCalls);
Assert.Equal("existing", session.SessionId);
StreamEventsRequest request = Assert.Single(transport.StreamEventsCalls).Request;
Assert.Equal("existing", request.SessionId);
Assert.Equal(7UL, request.AfterWorkerSequence);
Assert.Single(received);
}
/// <summary>AttachSession rejects blank ids and a disposed client.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AttachSession_InvalidArguments_Throw()
{
FakeGatewayTransport transport = CreateTransport();
MxGatewayClient client = new(transport.Options, transport);
Assert.Throws<ArgumentException>(() => client.AttachSession(" "));
await client.DisposeAsync();
Assert.Throws<ObjectDisposedException>(() => client.AttachSession("x"));
}
private static MxCommandReply OkReply(MxCommandKind kind) => new()
{
SessionId = "session-fixture",
Kind = kind,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
};
private static MxGatewayClientOptions Options() => new()
{
Endpoint = new Uri("http://localhost:5000"),
ApiKey = "test-api-key",
};
private static FakeGatewayTransport CreateTransport() => new(Options());
}
@@ -22,6 +22,7 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
private const int BrowseChildrenPageSize = 500;
private readonly GrpcChannel? _channel;
private readonly bool _ownsChannel;
private readonly IGalaxyRepositoryClientTransport _transport;
private readonly ResiliencePipeline _safeUnaryRetryPipeline;
private bool _disposed;
@@ -48,9 +49,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
private GalaxyRepositoryClient(
GrpcChannel channel,
IGalaxyRepositoryClientTransport transport)
IGalaxyRepositoryClientTransport transport,
bool ownsChannel)
{
_channel = channel;
_ownsChannel = ownsChannel;
_transport = transport;
Options = transport.Options;
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -95,7 +98,32 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
channel,
new GrpcGalaxyRepositoryClientTransport(
options,
new GalaxyRepository.GalaxyRepositoryClient(channel)));
new GalaxyRepository.GalaxyRepositoryClient(channel)),
ownsChannel: true);
}
/// <summary>
/// Creates a Galaxy Repository client over a caller-owned <see cref="GrpcChannel"/>.
/// The client does not dispose the channel; the caller controls its lifetime and any
/// TLS/handler configuration.
/// </summary>
/// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
/// <param name="options">Client options (API key, timeouts, retry).</param>
/// <returns>A new client instance that does not own the channel.</returns>
public static GalaxyRepositoryClient Create(
GrpcChannel channel,
MxGatewayClientOptions options)
{
ArgumentNullException.ThrowIfNull(channel);
ArgumentNullException.ThrowIfNull(options);
options.Validate();
return new GalaxyRepositoryClient(
channel,
new GrpcGalaxyRepositoryClientTransport(
options,
new GalaxyRepository.GalaxyRepositoryClient(channel)),
ownsChannel: false);
}
/// <summary>
@@ -436,7 +464,11 @@ public sealed class GalaxyRepositoryClient : IAsyncDisposable
}
_disposed = true;
_channel?.Dispose();
if (_ownsChannel)
{
_channel?.Dispose();
}
return ValueTask.CompletedTask;
}
@@ -14,7 +14,8 @@ namespace ZB.MOM.WW.MxGateway.Client;
/// </summary>
public sealed class MxGatewayClient : IAsyncDisposable
{
private readonly GrpcChannel _channel;
private readonly GrpcChannel? _channel;
private readonly bool _ownsChannel;
private readonly IMxGatewayClientTransport _transport;
private readonly ResiliencePipeline _safeUnaryRetryPipeline;
private bool _disposed;
@@ -36,14 +37,16 @@ public sealed class MxGatewayClient : IAsyncDisposable
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
options.Retry,
options.LoggerFactory?.CreateLogger<MxGatewayClient>());
_channel = null!;
_channel = null;
}
private MxGatewayClient(
GrpcChannel channel,
IMxGatewayClientTransport transport)
IMxGatewayClientTransport transport,
bool ownsChannel)
{
_channel = channel;
_ownsChannel = ownsChannel;
_transport = transport;
Options = transport.Options;
_safeUnaryRetryPipeline = MxGatewayClientRetryPolicy.Create(
@@ -88,7 +91,58 @@ public sealed class MxGatewayClient : IAsyncDisposable
channel,
new GrpcMxGatewayClientTransport(
options,
new MxAccessGateway.MxAccessGatewayClient(channel)));
new MxAccessGateway.MxAccessGatewayClient(channel)),
ownsChannel: true);
}
/// <summary>
/// Creates a gateway client over a caller-owned <see cref="GrpcChannel"/>. The client
/// does not dispose the channel; the caller controls its lifetime and any TLS/handler
/// configuration. <see cref="MxGatewayClientOptions.Endpoint"/> is still validated but
/// the channel's own address is used for transport.
/// </summary>
/// <param name="channel">An existing channel, shared or pre-configured by the caller.</param>
/// <param name="options">Client configuration options (API key, timeouts, retry).</param>
/// <returns>A new gateway client instance that does not own the channel.</returns>
public static MxGatewayClient Create(
GrpcChannel channel,
MxGatewayClientOptions options)
{
ArgumentNullException.ThrowIfNull(channel);
ArgumentNullException.ThrowIfNull(options);
options.Validate();
return new MxGatewayClient(
channel,
new GrpcMxGatewayClientTransport(
options,
new MxAccessGateway.MxAccessGatewayClient(channel)),
ownsChannel: false);
}
/// <summary>
/// Binds a session object to an existing gateway session id without calling
/// OpenSession, so a reconnecting process can resume
/// <see cref="MxGatewaySession.StreamEventsAsync"/> with <c>afterWorkerSequence</c>.
/// The gateway only allows the API key that opened the session to stream its events
/// or invoke commands on it; any other key is rejected with PermissionDenied. The
/// returned session's <see cref="MxGatewaySession.OpenSessionReply"/> carries only the
/// session id.
/// </summary>
/// <param name="sessionId">The id of a session previously opened with this client's API key.</param>
/// <returns>A session bound to <paramref name="sessionId"/>.</returns>
public MxGatewaySession AttachSession(string sessionId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(sessionId);
ThrowIfDisposed();
return new MxGatewaySession(
this,
new OpenSessionReply
{
SessionId = sessionId,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
});
}
/// <summary>
@@ -199,11 +253,13 @@ public sealed class MxGatewayClient : IAsyncDisposable
ArgumentNullException.ThrowIfNull(request);
ThrowIfDisposed();
// Deliberately not wrapped in the retry pipeline: an acknowledge is a
// non-idempotent operator write that re-sends the operator password and
// can double-ack in the Historian.
try
{
return await ExecuteSafeUnaryAsync(
token => _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(token)),
cancellationToken).ConfigureAwait(false);
return await _transport.AcknowledgeAlarmAsync(request, CreateCallOptions(cancellationToken))
.ConfigureAwait(false);
}
catch (MxGatewayException ex) when (request.OperatorPassword.Length > 0)
{
@@ -266,7 +322,11 @@ public sealed class MxGatewayClient : IAsyncDisposable
}
_disposed = true;
_channel?.Dispose();
if (_ownsChannel)
{
_channel?.Dispose();
}
return ValueTask.CompletedTask;
}
@@ -1,3 +1,4 @@
using Grpc.Core;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client;
@@ -640,7 +641,7 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// </summary>
/// <param name="serverHandle">The ServerHandle from register.</param>
/// <param name="tagAddresses">Tag addresses to read (one per result).</param>
/// <param name="timeout">Per-call timeout for the snapshot lifecycle path; <see cref="TimeSpan.Zero"/> uses the gateway default.</param>
/// <param name="timeout">One shared snapshot deadline for the whole command, not a per-tag budget: the worker advises every uncached tag, waits once for all of them, then tears them down. <see cref="TimeSpan.Zero"/> uses the worker default (1000 ms).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>One <see cref="BulkReadResult"/> per requested tag, in request order.</returns>
public async Task<IReadOnlyList<BulkReadResult>> ReadBulkAsync(
@@ -1452,6 +1453,106 @@ public sealed class MxGatewaySession : IAsyncDisposable
return _client.InvokeAsync(request, cancellationToken);
}
/// <summary>
/// Sends a PING diagnostic command to the worker and throws if the reply is a failure.
/// </summary>
/// <param name="message">Optional message echoed to the worker; defaults to empty.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A task that completes when the worker acknowledges the ping.</returns>
public async Task PingAsync(
string? message = null,
CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand { Message = message ?? string.Empty },
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
}
/// <summary>
/// Gets the gateway's lifecycle state for this session.
/// </summary>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The current session state.</returns>
public async Task<SessionState> GetSessionStateAsync(CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.GetSessionState,
GetSessionState = new GetSessionStateCommand(),
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.SessionState?.State
?? throw CreateMalformed(reply, "GetSessionState");
}
/// <summary>
/// Gets diagnostic information about the worker process serving this session.
/// </summary>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The worker process id, version, and MXAccess COM identifiers.</returns>
public async Task<WorkerInfoReply> GetWorkerInfoAsync(CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.GetWorkerInfo,
GetWorkerInfo = new GetWorkerInfoCommand(),
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.WorkerInfo
?? throw CreateMalformed(reply, "GetWorkerInfo");
}
/// <summary>
/// Drains events queued in the worker through the diagnostic command path. This removes
/// events from the worker queue; it is not a substitute for <see cref="StreamEventsAsync"/>
/// and is not retried automatically. The reply is bounded by server-side caps, so call
/// repeatedly until it returns an empty list.
/// </summary>
/// <param name="maxEvents">Maximum events to return; 0 lets the server choose.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The drained events, possibly empty.</returns>
public async Task<IReadOnlyList<MxEvent>> DrainEventsAsync(
uint maxEvents = 0,
CancellationToken cancellationToken = default)
{
MxCommandReply reply = await InvokeCommandAsync(
new MxCommand
{
Kind = MxCommandKind.DrainEvents,
DrainEvents = new DrainEventsCommand { MaxEvents = maxEvents },
},
cancellationToken)
.ConfigureAwait(false);
reply.EnsureProtocolSuccess().EnsureMxAccessSuccess();
return reply.DrainEvents?.Events.ToArray() ?? [];
}
private static MxGatewayMalformedReplyException CreateMalformed(MxCommandReply reply, string operation)
{
return new MxGatewayMalformedReplyException(
$"{operation} returned a malformed reply: OK reply carried no typed payload",
reply.SessionId,
reply.CorrelationId,
reply.ProtocolStatus,
reply.HasHresult ? reply.Hresult : null,
reply.Statuses.ToArray());
}
/// <summary>
/// Streams events from the worker for this session, optionally starting after a given sequence number.
/// </summary>
@@ -1505,8 +1606,20 @@ public sealed class MxGatewaySession : IAsyncDisposable
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
await CloseAsync().ConfigureAwait(false);
_closeLock.Dispose();
try
{
await CloseAsync().ConfigureAwait(false);
}
catch (Exception ex) when (ex is MxGatewayException or ObjectDisposedException or RpcException)
{
// Best-effort close: the gateway may be unreachable or the owning client
// already disposed. Disposal must not throw or leak the lock; the gateway
// reaps the session on its own idle/detach timers.
}
finally
{
_closeLock.Dispose();
}
}
private Task<MxCommandReply> InvokeCommandAsync(
+24 -1
View File
@@ -100,6 +100,29 @@ judges each `MXSTATUS_PROXY` entry by its category — an entry fails when
`Category` is not `MX_STATUS_CATEGORY_OK`, and the raw `Success` member is a
diagnostic that never decides the verdict. A nil entry is success.
`Dial` is connect-or-fail: it creates the connection with `grpc.NewClient`
(the endpoint resolves with the `passthrough` scheme unless it names one) and
then waits for the channel to become READY, bounded by `DialTimeout` (default
10 s) or the context deadline. A failure is returned as a `GatewayError` with
`Op == "dial"`. Passing a connection to `NewClient`/`NewGalaxyClient` stays lazy.
### Event buffering
The channel returned by `Session.Events`/`EventsAfter` buffers
`Options.EventBufferSize` data results (default `DefaultEventBufferSize`, 1024).
If the consumer falls behind and the buffer fills, the stream is cancelled; the
results already buffered are still delivered in order, followed by a final
result whose error matches `errors.Is(err, mxgateway.ErrSlowConsumer)`. Raise
`EventBufferSize` for bursty feeds, or use `SubscribeEvents`/`SubscribeEventsAfter`,
which block instead of dropping and are the non-dropping alternative.
### Diagnostics
`Session.Ping`, `GetSessionState`, `GetWorkerInfo`, and `DrainEvents` are typed
wrappers over the diagnostic commands (`PingRaw` remains for the raw reply).
`DrainEvents(ctx, maxEvents)` returns fewer events than requested when the
server caps the reply; call it until it returns an empty slice.
### Reconnect-replay gap
Each `EventResult` carries exactly one of `Event`, `ReplayGap`, or `Err`. When
@@ -144,7 +167,7 @@ snapshots, `Client.StreamAlarms` for the server-streaming feed, and
operator's Galaxy credentials (`OperatorUser` required, `OperatorPassword`,
optional `OperatorDomain`) so the gateway authenticates as the operator and the
Historian records that operator; an empty password is allowed, and the password
is scrubbed from any returned error. The streaming
is scrubbed from any returned error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The streaming
call returns a `StreamAlarmsClient`; cancel its context to terminate the
stream. All three pass straight through to the gateway's central alarm
monitor.
+2 -2
View File
@@ -571,7 +571,7 @@ func runReadBulk(ctx context.Context, args []string, stdout, stderr io.Writer) e
sessionID := flags.String("session-id", "", "gateway session id")
serverHandle := flags.Int("server-handle", 0, "MXAccess server handle")
items := flags.String("items", "", "comma-separated tag addresses")
timeoutMs := flags.Int("timeout-ms", 0, "per-tag snapshot timeout in milliseconds (0 = worker default)")
timeoutMs := flags.Int("timeout-ms", 0, "snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms)")
if err := flags.Parse(args); err != nil {
return err
@@ -754,7 +754,7 @@ func runBenchReadBulk(ctx context.Context, args []string, stdout, stderr io.Writ
tagStart := flags.Int("tag-start", 1, "first machine number")
tagPrefix := flags.String("tag-prefix", "TestMachine_", "tag prefix (machine number appended as %03d)")
tagAttribute := flags.String("tag-attribute", "TestChangingInt", "attribute appended to each tag prefix")
timeoutMs := flags.Int("timeout-ms", 1500, "per-tag snapshot timeout in milliseconds")
timeoutMs := flags.Int("timeout-ms", 1500, "snapshot deadline in milliseconds, shared by all tags in each ReadBulk command")
if err := flags.Parse(args); err != nil {
return err
+34 -2
View File
@@ -15,10 +15,12 @@ import (
"context"
"crypto/tls"
"errors"
"strings"
"time"
pb "gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/internal/generated"
"google.golang.org/grpc"
"google.golang.org/grpc/connectivity"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/types/known/durationpb"
@@ -61,11 +63,10 @@ func Dial(ctx context.Context, opts Options) (*Client, error) {
grpc.WithTransportCredentials(transportCredentials),
grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
grpc.WithBlock(),
}
dialOptions = append(dialOptions, opts.DialOptions...)
conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
if err != nil {
return nil, &GatewayError{Op: "dial", Err: err}
}
@@ -265,3 +266,34 @@ func (o OpenSessionOptions) Request() *OpenSessionRequest {
}
return req
}
// dialConnected creates a connection with grpc.NewClient and then waits for it
// to become READY, bounded by ctx. This preserves the connect-or-fail contract
// the removed grpc.DialContext + grpc.WithBlock combination gave Dial callers.
// The endpoint is resolved with the passthrough scheme unless it already names a
// scheme, matching DialContext's historical default (NewClient defaults to dns).
func dialConnected(ctx context.Context, endpoint string, dialOptions []grpc.DialOption) (*grpc.ClientConn, error) {
target := endpoint
if !strings.Contains(target, "://") {
target = "passthrough:///" + target
}
conn, err := grpc.NewClient(target, dialOptions...)
if err != nil {
return nil, err
}
conn.Connect()
for {
state := conn.GetState()
if state == connectivity.Ready {
return conn, nil
}
if state == connectivity.Shutdown {
_ = conn.Close()
return nil, errors.New("connection shut down while connecting")
}
if !conn.WaitForStateChange(ctx, state) {
_ = conn.Close()
return nil, ctx.Err()
}
}
}
+241 -4
View File
@@ -121,7 +121,7 @@ func TestEventsAfterCancelsStreamWhenCompatibilityChannelIsAbandoned(t *testing.
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -155,7 +155,7 @@ func TestEventsSlowConsumerYieldsErrSlowConsumerBeforeClose(t *testing.T) {
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -209,7 +209,7 @@ func TestEventsFullBufferTerminalErrorKeepsRootCause(t *testing.T) {
streamEventCount: eventBufferSize,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -280,7 +280,7 @@ func TestSubscribeEventsFullBufferDeliversTerminalError(t *testing.T) {
streamEventCount: eventBufferSize + eventBufferReservedSlots,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClient(t, fake)
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: eventBufferSize})
defer cleanup()
session := NewSessionForID(client, "session-1")
@@ -1047,3 +1047,240 @@ func TestWriteArrayElementsSendsWriteCommandWithSparseOneof(t *testing.T) {
t.Errorf("element float value = %v, want 1.5", elem.GetValue().GetFloatValue())
}
}
// newBufconnClientWithOptions mirrors newBufconnClient but lets a test tune Options.
func newBufconnClientWithOptions(t *testing.T, fake *fakeGatewayServer, opts Options) (*Client, func()) {
t.Helper()
listener := bufconn.Listen(bufSize)
server := grpc.NewServer()
pb.RegisterMxAccessGatewayServer(server, fake)
go func() { _ = server.Serve(listener) }()
opts.Endpoint = "bufnet"
opts.APIKey = "test-api-key"
opts.Plaintext = true
opts.DialOptions = append(opts.DialOptions, grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
return listener.DialContext(ctx)
}))
client, err := Dial(context.Background(), opts)
if err != nil {
t.Fatalf("Dial() error = %v", err)
}
return client, func() {
client.Close()
server.Stop()
listener.Close()
}
}
func TestEventsHonorsConfiguredEventBufferSize(t *testing.T) {
// 100 events exceed the old fixed 16-slot buffer; a 256-slot buffer must
// hold them all without a slow-consumer termination.
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: 100,
streamTerminalErr: status.Error(codes.Internal, "boom"),
}
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 256})
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.Events(context.Background())
if err != nil {
t.Fatalf("Events() error = %v", err)
}
<-fake.streamDone
time.Sleep(250 * time.Millisecond)
count := 0
var last EventResult
for res := range events {
if res.Event != nil {
count++
}
last = res
}
if count != 100 {
t.Fatalf("delivered events = %d, want 100", count)
}
if errors.Is(last.Err, ErrSlowConsumer) || status.Code(last.Err) != codes.Internal {
t.Fatalf("terminal err = %v, want the genuine Internal error", last.Err)
}
}
func TestEventsOverflowDeliversBufferedEventsBeforeSlowConsumerError(t *testing.T) {
fake := &fakeGatewayServer{
streamStarted: make(chan struct{}),
streamDone: make(chan struct{}),
streamEventCount: 64,
}
client, cleanup := newBufconnClientWithOptions(t, fake, Options{EventBufferSize: 4})
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.Events(context.Background())
if err != nil {
t.Fatalf("Events() error = %v", err)
}
<-fake.streamStarted
time.Sleep(250 * time.Millisecond)
buffered := 0
var last EventResult
for res := range events {
if res.Event != nil {
buffered++
if got := res.Event.GetWorkerSequence(); got != uint64(buffered) {
t.Fatalf("event %d has sequence %d, want in-order", buffered, got)
}
}
last = res
}
if buffered != 4 {
t.Fatalf("buffered events delivered = %d, want 4 (all data slots)", buffered)
}
if !errors.Is(last.Err, ErrSlowConsumer) {
t.Fatalf("final err = %v, want ErrSlowConsumer", last.Err)
}
}
func TestEventBufferSizeDefault(t *testing.T) {
if got := (Options{}).eventBufferSize(); got != DefaultEventBufferSize {
t.Fatalf("default = %d, want %d", got, DefaultEventBufferSize)
}
if got := (Options{EventBufferSize: 7}).eventBufferSize(); got != 7 {
t.Fatalf("explicit = %d, want 7", got)
}
}
func TestDialFailsWhenEndpointUnreachable(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_, err := Dial(ctx, Options{
Endpoint: "bufnet",
Plaintext: true,
DialTimeout: 300 * time.Millisecond,
DialOptions: []grpc.DialOption{grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
return nil, errors.New("refused")
})},
})
if err == nil {
t.Fatal("Dial() error = nil, want connect failure")
}
var gwErr *GatewayError
if !errors.As(err, &gwErr) || gwErr.Op != "dial" {
t.Fatalf("Dial() error = %v, want *GatewayError Op=dial", err)
}
}
func okStatus() *pb.ProtocolStatus {
return &pb.ProtocolStatus{Code: pb.ProtocolStatusCode_PROTOCOL_STATUS_CODE_OK}
}
func TestSessionPingReturnsEchoedMessage(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_PING,
ProtocolStatus: okStatus(),
DiagnosticMessage: "pong:hi",
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
got, err := session.Ping(context.Background(), "hi")
if err != nil {
t.Fatalf("Ping() error = %v", err)
}
if got != "pong:hi" {
t.Fatalf("Ping() = %q, want pong:hi", got)
}
if fake.invokeRequest.GetCommand().GetPing().GetMessage() != "hi" {
t.Fatalf("ping payload not sent: %v", fake.invokeRequest)
}
}
func TestSessionGetSessionState(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_SessionState{SessionState: &pb.SessionStateReply{
State: pb.SessionState_SESSION_STATE_CREATING,
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
got, err := session.GetSessionState(context.Background())
if err != nil {
t.Fatalf("GetSessionState() error = %v", err)
}
if got != pb.SessionState_SESSION_STATE_CREATING {
t.Fatalf("state = %v", got)
}
if fake.invokeRequest.GetCommand().GetKind() != pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE {
t.Fatalf("kind = %v", fake.invokeRequest.GetCommand().GetKind())
}
}
func TestSessionGetWorkerInfo(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_WorkerInfo{WorkerInfo: &pb.WorkerInfoReply{
WorkerProcessId: 42, WorkerVersion: "1.2.3", MxaccessProgid: "prog", MxaccessClsid: "clsid",
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
info, err := session.GetWorkerInfo(context.Background())
if err != nil {
t.Fatalf("GetWorkerInfo() error = %v", err)
}
if info.GetWorkerProcessId() != 42 || info.GetWorkerVersion() != "1.2.3" {
t.Fatalf("info = %v", info)
}
}
func TestSessionDrainEvents(t *testing.T) {
fake := &fakeGatewayServer{invokeReply: &pb.MxCommandReply{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
ProtocolStatus: okStatus(),
Payload: &pb.MxCommandReply_DrainEvents{DrainEvents: &pb.DrainEventsReply{
Events: []*pb.MxEvent{{WorkerSequence: 1}, {WorkerSequence: 2}},
}},
}}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
events, err := session.DrainEvents(context.Background(), 10)
if err != nil {
t.Fatalf("DrainEvents() error = %v", err)
}
if len(events) != 2 || events[1].GetWorkerSequence() != 2 {
t.Fatalf("events = %v", events)
}
if fake.invokeRequest.GetCommand().GetDrainEvents().GetMaxEvents() != 10 {
t.Fatalf("max_events not sent: %v", fake.invokeRequest)
}
}
func TestSessionDiagnosticsErrorOnMissingPayload(t *testing.T) {
fake := &fakeGatewayServer{}
client, cleanup := newBufconnClient(t, fake)
defer cleanup()
session := NewSessionForID(client, "session-1")
if _, err := session.GetWorkerInfo(context.Background()); err == nil {
t.Fatal("GetWorkerInfo() with empty reply: want error")
}
if _, err := session.GetSessionState(context.Background()); err == nil {
t.Fatal("GetSessionState() with empty reply: want error")
}
if _, err := session.DrainEvents(context.Background(), 1); err == nil {
t.Fatal("DrainEvents() with empty reply: want error")
}
}
+1 -2
View File
@@ -99,11 +99,10 @@ func DialGalaxy(ctx context.Context, opts Options) (*GalaxyClient, error) {
grpc.WithTransportCredentials(transportCredentials),
grpc.WithUnaryInterceptor(unaryAuthInterceptor(opts.APIKey)),
grpc.WithStreamInterceptor(streamAuthInterceptor(opts.APIKey)),
grpc.WithBlock(),
}
dialOptions = append(dialOptions, opts.DialOptions...)
conn, err := grpc.DialContext(dialCtx, opts.Endpoint, dialOptions...)
conn, err := dialConnected(dialCtx, opts.Endpoint, dialOptions)
if err != nil {
return nil, &GatewayError{Op: "dial", Err: err}
}
+15
View File
@@ -38,6 +38,21 @@ type Options struct {
// no CACertFile is pinned. Default false: the gateway's self-signed cert is
// accepted without verification (internal-tool posture).
RequireCertificateValidation bool
// EventBufferSize is the number of data results the channel returned by
// Session.Events/EventsAfter buffers before a slow consumer is cancelled with
// ErrSlowConsumer. Zero or negative applies DefaultEventBufferSize. It does not
// affect SubscribeEvents, which blocks instead of dropping.
EventBufferSize int
}
// DefaultEventBufferSize is the default Options.EventBufferSize.
const DefaultEventBufferSize = 1024
func (o Options) eventBufferSize() int {
if o.EventBufferSize <= 0 {
return DefaultEventBufferSize
}
return o.EventBufferSize
}
// BrowseChildrenOptions configures lazy Galaxy hierarchy walks performed by
+76 -8
View File
@@ -18,8 +18,9 @@ import (
const maxBulkItems = 1000
// eventBufferSize is the number of buffered event slots on the
// Events/EventsAfter (cancel-when-full) results channel.
// eventBufferSize is the data-slot count the unit tests use for the legacy
// fixed-size assertions; the runtime value comes from Options.EventBufferSize
// (default DefaultEventBufferSize).
const eventBufferSize = 16
// eventBufferReservedSlots is the extra capacity reserved beyond
@@ -548,8 +549,9 @@ func (s *Session) WriteSecured2Bulk(ctx context.Context, serverHandle int32, ent
// MXAccess COM has no synchronous Read; the worker satisfies this by returning the
// most recent cached OnDataChange value when the tag is already advised (WasCached=true),
// or by taking a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
// otherwise. timeout bounds the wait per tag in the snapshot case; pass zero to use the
// worker default. Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
// otherwise. timeout is one shared deadline for the whole command, not a per-tag budget:
// the worker advises every uncached tag, waits once for all of them, then tears them down.
// Pass zero to use the worker default (1000 ms). Per-tag failures (timeout, invalid tag) appear as BulkReadResult entries
// with WasSuccessful=false; the call never returns an error for per-tag MXAccess failures.
//
// A non-nil but empty tagAddresses slice is treated as a no-op and returns an empty
@@ -666,6 +668,68 @@ func buildSparseArrayValue(elementDataType MxDataType, totalLength uint32, eleme
}
}
// Ping sends a diagnostic PING command and returns the message the gateway
// echoed back in the reply's DiagnosticMessage field.
func (s *Session) Ping(ctx context.Context, message string) (string, error) {
reply, err := s.PingRaw(ctx, message)
if err != nil {
return "", err
}
return reply.GetDiagnosticMessage(), nil
}
// GetSessionState returns the gateway's current lifecycle state for the session.
func (s *Session) GetSessionState(ctx context.Context) (SessionState, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_SESSION_STATE,
Payload: &pb.MxCommand_GetSessionState{GetSessionState: &pb.GetSessionStateCommand{}},
})
if err != nil {
return 0, err
}
payload := reply.GetSessionState()
if payload == nil {
return 0, errors.New("mxgateway: get session state reply carried no session_state payload")
}
return payload.GetState(), nil
}
// GetWorkerInfo returns diagnostic information about the session's worker
// process (process id, version, MXAccess ProgID and CLSID).
func (s *Session) GetWorkerInfo(ctx context.Context) (*WorkerInfoReply, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_GET_WORKER_INFO,
Payload: &pb.MxCommand_GetWorkerInfo{GetWorkerInfo: &pb.GetWorkerInfoCommand{}},
})
if err != nil {
return nil, err
}
payload := reply.GetWorkerInfo()
if payload == nil {
return nil, errors.New("mxgateway: get worker info reply carried no worker_info payload")
}
return payload, nil
}
// DrainEvents removes and returns up to maxEvents queued events through the
// diagnostic DRAIN_EVENTS command. The reply is bounded server-side, so it can
// carry fewer events than requested or queued; call repeatedly until it returns
// an empty slice. It is a diagnostic tool, not a replacement for Events.
func (s *Session) DrainEvents(ctx context.Context, maxEvents uint32) ([]*MxEvent, error) {
reply, err := s.invokeCommand(ctx, &pb.MxCommand{
Kind: pb.MxCommandKind_MX_COMMAND_KIND_DRAIN_EVENTS,
Payload: &pb.MxCommand_DrainEvents{DrainEvents: &pb.DrainEventsCommand{MaxEvents: maxEvents}},
})
if err != nil {
return nil, err
}
payload := reply.GetDrainEvents()
if payload == nil {
return nil, errors.New("mxgateway: drain events reply carried no drain_events payload")
}
return payload.GetEvents(), nil
}
// PingRaw sends a diagnostic PING command and returns the raw reply.
// The message is echoed back by the gateway in the reply's DiagnosticMessage field.
func (s *Session) PingRaw(ctx context.Context, message string) (*MxCommandReply, error) {
@@ -1001,7 +1065,9 @@ func stringSecrets(values ...*MxValue) []string {
// The returned channel is buffered. If the consumer falls behind and the buffer
// overflows with data, the stream is terminated and a final EventResult carrying
// a GatewayError that wraps ErrSlowConsumer is delivered before the channel
// closes; match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
// closes. Results already buffered are delivered first, in order, ahead of that
// terminal result. The buffer holds Options.EventBufferSize data results
// (default DefaultEventBufferSize); match it with errors.Is(res.Err, ErrSlowConsumer) to distinguish a
// slow-consumer drop from a graceful server end. A genuine stream error is
// reported as itself even under overflow — it is never relabeled as
// ErrSlowConsumer, so the underlying gRPC status stays inspectable. Use
@@ -1046,7 +1112,8 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
return nil, err
}
results := make(chan EventResult, eventBufferSize+eventBufferReservedSlots)
dataSlots := s.client.opts.eventBufferSize()
results := make(chan EventResult, dataSlots+eventBufferReservedSlots)
done := make(chan struct{})
go func() {
defer close(results)
@@ -1062,7 +1129,7 @@ func (s *Session) subscribeEventsAfter(ctx context.Context, afterWorkerSequence
if gap := event.GetReplayGap(); gap != nil {
result = EventResult{ReplayGap: gap}
}
if !sendEventResult(streamCtx, results, result, cancelWhenResultBufferFull, cancel) {
if !sendEventResult(streamCtx, results, result, dataSlots, cancelWhenResultBufferFull, cancel) {
return
}
continue
@@ -1126,6 +1193,7 @@ func sendEventResult(
ctx context.Context,
results chan<- EventResult,
result EventResult,
dataSlots int,
cancelWhenBufferFull bool,
cancel context.CancelFunc,
) bool {
@@ -1136,7 +1204,7 @@ func sendEventResult(
// grows by our own sends and shrinks as the consumer reads; the reserve
// therefore always survives to carry ErrSlowConsumer. A plain buffered
// send would instead consume the reserved slot as ordinary data.
if len(results) >= eventBufferSize {
if len(results) >= dataSlots {
// The consumer fell behind and the data slots are full. Cancel the
// stream, then use the reserved terminal slot to enqueue a loud
// ErrSlowConsumer result so overflow is always observable rather
+2
View File
@@ -28,6 +28,8 @@ type (
MxCommandReply = pb.MxCommandReply
// MxCommand is the discriminated union of MXAccess command payloads.
MxCommand = pb.MxCommand
// WorkerInfoReply is the GetWorkerInfo diagnostic reply payload.
WorkerInfoReply = pb.WorkerInfoReply
// MxEvent is one ordered event delivered on a session event stream.
MxEvent = pb.MxEvent
// ReplayGap is the gateway sentinel payload signalling that a resumed event
+3 -2
View File
@@ -10,8 +10,9 @@ import (
// the two status payloads carried by AlarmFeedMessage.
func TestAlarmFeedStatusAliasesArePublic(t *testing.T) {
t.Helper()
var provider mxgateway.AlarmProviderStatus
var snapshot mxgateway.AlarmSnapshotStatus
// Pointers, not values: protobuf messages embed a mutex and must not be copied.
var provider *mxgateway.AlarmProviderStatus
var snapshot *mxgateway.AlarmSnapshotStatus
_ = provider
_ = snapshot
}
+25 -1
View File
@@ -79,6 +79,30 @@ only stops the client from waiting; it does not abort an in-flight MXAccess COM
call on the worker STA. It is a **single-consumer** surface: drive
`hasNext()`/`next()` (or `nextItem()`) from one thread only.
### Event buffer, overflow, and interrupts
The iterator buffers events in a bounded queue whose size is
`MxGatewayClientOptions.builder().eventQueueCapacity(n)` (default 1024; it was a
hard-coded 16, which a short burst could overflow). If the consumer falls more
than `n` events behind, the client cancels the call, keeps the events already
buffered, and throws `MxGatewayException` only after you have drained them. Resume
with `streamEventsAfter(lastSeenWorkerSequence)`. If the consuming thread is
interrupted, the call is cancelled, the interrupt flag is restored, and the
iterator throws a `CANCELLED` `StatusRuntimeException`.
`streamEventsAsync(request, StreamObserver<MxEvent>)` is the non-dropping
alternative: it delivers each event to your observer on the gRPC thread with no
client-side queue, so nothing is discarded by the client (a slow observer
instead applies normal gRPC flow control).
### Session diagnostics
`MxGatewaySession` exposes typed helpers for the worker diagnostic commands:
`ping(message)` (returns the raw `MxCommandReply`), `getSessionState()`
(`SessionState`), `getWorkerInfo()` (`WorkerInfoReply`: process id, version,
ProgID, CLSID) and `drainEvents(maxEvents)` (`List<MxEvent>`; replies are
bounded server-side, so loop until an empty list).
### Reconnect-replay gap signal
When you resume a stream with `streamEventsAfter(afterWorkerSequence)` and the
@@ -118,7 +142,7 @@ yields alarm-feed messages from the gateway's central monitor), and
The ack carries operator credentials: `operator_user` is required, and
`operator_password` / `operator_domain` let the gateway authenticate as that
operator so the Historian records the real operator. An empty password is
allowed. The password is scrubbed from exceptions the client throws. Close the subscription to cancel the underlying gRPC stream.
allowed. The password is scrubbed from exceptions the client throws. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. Close the subscription to cancel the underlying gRPC stream.
`ActiveAlarmSnapshot.getFromTruncatedSnapshot()` reports that the record came
from a provider fetch which hit the per-fetch cap: the snapshot set may omit
@@ -816,6 +816,11 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
*/
ALARM_TRANSITION_KIND_CLEAR(3),
/**
* <pre>
* Reserved: never produced by the gateway or worker. Retained so the numeric
* value is not reused.
* </pre>
*
* <code>ALARM_TRANSITION_KIND_RETRIGGER = 4;</code>
*/
ALARM_TRANSITION_KIND_RETRIGGER(4),
@@ -848,6 +853,11 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
*/
public static final int ALARM_TRANSITION_KIND_CLEAR_VALUE = 3;
/**
* <pre>
* Reserved: never produced by the gateway or worker. Retained so the numeric
* value is not reused.
* </pre>
*
* <code>ALARM_TRANSITION_KIND_RETRIGGER = 4;</code>
*/
public static final int ALARM_TRANSITION_KIND_RETRIGGER_VALUE = 4;
@@ -37643,11 +37653,23 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
getOperatorDomainBytes();
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The operatorFullName.
*/
java.lang.String getOperatorFullName();
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The bytes for operatorFullName.
*/
@@ -37939,6 +37961,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
@SuppressWarnings("serial")
private volatile java.lang.Object operatorFullName_ = "";
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The operatorFullName.
*/
@@ -37956,6 +37984,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The bytes for operatorFullName.
*/
@@ -38855,6 +38889,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
private java.lang.Object operatorFullName_ = "";
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The operatorFullName.
*/
@@ -38871,6 +38911,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return The bytes for operatorFullName.
*/
@@ -38888,6 +38934,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @param value The operatorFullName to set.
* @return This builder for chaining.
@@ -38901,6 +38953,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @return This builder for chaining.
*/
@@ -38911,6 +38969,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 6;</code>
* @param value The bytes for operatorFullName to set.
* @return This builder for chaining.
@@ -39717,11 +39781,23 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
getOperatorDomainBytes();
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The operatorFullName.
*/
java.lang.String getOperatorFullName();
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The bytes for operatorFullName.
*/
@@ -40109,6 +40185,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
@SuppressWarnings("serial")
private volatile java.lang.Object operatorFullName_ = "";
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The operatorFullName.
*/
@@ -40126,6 +40208,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The bytes for operatorFullName.
*/
@@ -41263,6 +41351,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
private java.lang.Object operatorFullName_ = "";
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The operatorFullName.
*/
@@ -41279,6 +41373,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return The bytes for operatorFullName.
*/
@@ -41296,6 +41396,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @param value The operatorFullName to set.
* @return This builder for chaining.
@@ -41309,6 +41415,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @return This builder for chaining.
*/
@@ -41319,6 +41431,12 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
* AlarmAckByName that carried it returns -55. operator_domain IS used (the
* AuthenticateUser identity is "&lt;operator_domain&gt;&#92;&lt;operator_user&gt;").
* </pre>
*
* <code>string operator_full_name = 8;</code>
* @param value The bytes for operatorFullName to set.
* @return This builder for chaining.
@@ -49007,11 +49125,15 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* reply returns the cached value WITHOUT modifying the existing
* subscription (was_cached = true).
* - Otherwise the worker takes the snapshot lifecycle itself: AddItem +
* Advise, wait up to `timeout_ms` for the first OnDataChange, then
* UnAdvise + RemoveItem before returning. The session is left exactly
* as it was before the call (was_cached = false).
* Advise, wait for the first OnDataChange, then UnAdvise + RemoveItem
* before returning. The session is left exactly as it was before the
* call (was_cached = false).
*
* `timeout_ms == 0` uses the gateway-configured default (1000 ms).
* `timeout_ms` is one deadline shared by the whole command, not a per-tag
* budget: the worker adds and advises every uncached tag, waits once for all
* of them, then tears them all down. A tag that has not delivered by the
* deadline gets its own was_successful = false result.
* `timeout_ms == 0` uses the worker's default (1000 ms).
* </pre>
*
* Protobuf type {@code mxaccess_gateway.v1.ReadBulkCommand}
@@ -49306,11 +49428,15 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* reply returns the cached value WITHOUT modifying the existing
* subscription (was_cached = true).
* - Otherwise the worker takes the snapshot lifecycle itself: AddItem +
* Advise, wait up to `timeout_ms` for the first OnDataChange, then
* UnAdvise + RemoveItem before returning. The session is left exactly
* as it was before the call (was_cached = false).
* Advise, wait for the first OnDataChange, then UnAdvise + RemoveItem
* before returning. The session is left exactly as it was before the
* call (was_cached = false).
*
* `timeout_ms == 0` uses the gateway-configured default (1000 ms).
* `timeout_ms` is one deadline shared by the whole command, not a per-tag
* budget: the worker adds and advises every uncached tag, waits once for all
* of them, then tears them all down. A tag that has not delivered by the
* deadline gets its own was_successful = false result.
* `timeout_ms == 0` uses the worker's default (1000 ms).
* </pre>
*
* Protobuf type {@code mxaccess_gateway.v1.ReadBulkCommand}
@@ -51949,11 +52075,19 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
getSessionIdBytes();
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The correlationId.
*/
java.lang.String getCorrelationId();
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The bytes for correlationId.
*/
@@ -52725,6 +52859,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
@SuppressWarnings("serial")
private volatile java.lang.Object correlationId_ = "";
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The correlationId.
*/
@@ -52742,6 +52880,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The bytes for correlationId.
*/
@@ -55268,6 +55410,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
private java.lang.Object correlationId_ = "";
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The correlationId.
*/
@@ -55284,6 +55430,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return The bytes for correlationId.
*/
@@ -55301,6 +55451,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
}
}
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @param value The correlationId to set.
* @return This builder for chaining.
@@ -55314,6 +55468,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @return This builder for chaining.
*/
@@ -55324,6 +55482,10 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
return this;
}
/**
* <pre>
* Echoes the request's client_correlation_id.
* </pre>
*
* <code>string correlation_id = 2;</code>
* @param value The bytes for correlationId to set.
* @return This builder for chaining.
@@ -80409,7 +80571,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -80420,7 +80582,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -80431,7 +80593,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -80895,7 +81057,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -80909,7 +81071,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -80923,7 +81085,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82407,7 +82569,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82420,7 +82582,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82437,7 +82599,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82459,7 +82621,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82479,7 +82641,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82506,7 +82668,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82525,7 +82687,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82539,7 +82701,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -82556,7 +82718,7 @@ public final class MxaccessGateway extends com.google.protobuf.GeneratedFile {
* <pre>
* When the alarm originally entered the active state. Preserved across
* acknowledge transitions so the Part 9 condition keeps the original raise
* time. Unset on retrigger from a previously-cleared condition.
* time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
* </pre>
*
* <code>.google.protobuf.Timestamp original_raise_timestamp = 6;</code>
@@ -216,6 +216,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
*/
WORKER_FAULT_CATEGORY_UNSPECIFIED(0),
/**
* <pre>
* Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
* arguments, gateway authentication) exit via WorkerExitCode before the pipe
* exists, so there is no channel to carry a fault. Names kept so the numbers
* are not reused.
* </pre>
*
* <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
*/
WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS(1),
@@ -276,6 +283,13 @@ public final class MxaccessWorker extends com.google.protobuf.GeneratedFile {
*/
public static final int WORKER_FAULT_CATEGORY_UNSPECIFIED_VALUE = 0;
/**
* <pre>
* Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
* arguments, gateway authentication) exit via WorkerExitCode before the pipe
* exists, so there is no channel to carry a fault. Names kept so the numbers
* are not reused.
* </pre>
*
* <code>WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;</code>
*/
public static final int WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS_VALUE = 1;
@@ -1262,7 +1262,7 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(
names = "--timeout-ms",
defaultValue = "0",
description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
int timeoutMs;
ReadBulkCommand(MxGatewayCliClientFactory clientFactory) {
@@ -1510,7 +1510,7 @@ public final class MxGatewayCli implements Callable<Integer> {
@Option(
names = "--timeout-ms",
defaultValue = "0",
description = "Per-tag snapshot timeout in milliseconds (0 = worker default).")
description = "Snapshot deadline in milliseconds, shared by all tags in the command (0 = worker default, 1000 ms).")
int timeoutMs;
@Option(names = "--iterations", defaultValue = "10", description = "Number of ReadBulk calls to perform.")
@@ -19,8 +19,12 @@ import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
* <p>Events arrive on a background gRPC thread and are buffered in a bounded
* blocking queue; the iterator drains them on the calling thread. Closing the
* stream cancels the underlying gRPC call. If the queue overflows the call is
* cancelled and a follow-up call to {@link #next()} throws
* {@link MxGatewayException}.
* cancelled, the events already buffered remain readable, and the
* {@link MxGatewayException} is thrown only after they are drained (the consumer
* can then resume with {@code after_worker_sequence}). The capacity comes from
* {@link MxGatewayClientOptions#eventQueueCapacity()}. Callers that must never be
* dropped should use {@link MxGatewayClient#streamEventsAsync}, which delivers on
* the gRPC thread with the caller's own flow control.
*
* <p><strong>Single consumer.</strong> This stream is not safe to drain from
* more than one thread. Interleave {@link #hasNext()}/{@link #next()} (or
@@ -46,10 +50,15 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
private final BlockingQueue<Object> queue;
private volatile ClientCallStreamObserver<StreamEventsRequest> requestStream;
private volatile boolean closed;
private final int capacity;
private boolean overflowed;
private Object next;
MxEventStream(int capacity) {
queue = new ArrayBlockingQueue<>(capacity);
this.capacity = capacity;
// Two extra slots are reserved so the overflow error and END sentinel
// can always be enqueued behind a full buffer of received events.
queue = new ArrayBlockingQueue<>(capacity + 2);
}
ClientResponseObserver<StreamEventsRequest, MxEvent> observer() {
@@ -143,12 +152,16 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
return queue.take();
} catch (InterruptedException error) {
Thread.currentThread().interrupt();
ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
if (stream != null) {
stream.cancel("interrupted while reading events", error);
}
return new StatusRuntimeException(Status.CANCELLED.withDescription("interrupted while reading events"));
}
}
}
private void offer(Object value) {
private synchronized void offer(Object value) {
Objects.requireNonNull(value, "value");
if (value == END) {
if (!queue.offer(value)) {
@@ -157,14 +170,29 @@ public final class MxEventStream implements Iterator<MxEvent>, AutoCloseable {
}
return;
}
if (!queue.offer(value)) {
if (overflowed) {
return;
}
if (value instanceof MxEvent) {
if (queue.size() < capacity) {
queue.offer(value);
return;
}
// Keep the events already received so the consumer can drain them
// and resume from the last one via after_worker_sequence; the
// overflow error is delivered behind them.
overflowed = true;
ClientCallStreamObserver<StreamEventsRequest> stream = requestStream;
if (stream != null) {
stream.cancel("client event stream queue overflowed", null);
}
queue.clear();
queue.offer(new MxGatewayException("gateway stream events queue overflowed"));
queue.offer(END);
return;
}
if (!queue.offer(value)) {
queue.clear();
queue.offer(value);
}
}
}
@@ -245,7 +245,7 @@ public final class MxGatewayClient implements AutoCloseable {
* @return an iterator-style stream of events
*/
public MxEventStream streamEvents(StreamEventsRequest request) {
MxEventStream stream = new MxEventStream(16);
MxEventStream stream = new MxEventStream(options.eventQueueCapacity());
withStreamDeadline(rawAsyncStub()).streamEvents(request, stream.observer());
return stream;
}
@@ -15,6 +15,8 @@ public final class MxGatewayClientOptions {
private static final Duration DEFAULT_CONNECT_TIMEOUT = Duration.ofSeconds(10);
private static final Duration DEFAULT_CALL_TIMEOUT = Duration.ofSeconds(30);
private static final int DEFAULT_MAX_GRPC_MESSAGE_BYTES = 16 * 1024 * 1024;
/** Default {@code StreamEvents} iterator buffer; sized to absorb bursts without dropping a slow consumer. */
static final int DEFAULT_EVENT_QUEUE_CAPACITY = 1024;
private final String endpoint;
private final String apiKey;
@@ -26,6 +28,7 @@ public final class MxGatewayClientOptions {
private final Duration callTimeout;
private final Duration streamTimeout;
private final int maxGrpcMessageBytes;
private final int eventQueueCapacity;
private MxGatewayClientOptions(Builder builder) {
endpoint = requireText(builder.endpoint, "endpoint");
@@ -40,6 +43,10 @@ public final class MxGatewayClientOptions {
maxGrpcMessageBytes = builder.maxGrpcMessageBytes <= 0
? DEFAULT_MAX_GRPC_MESSAGE_BYTES
: builder.maxGrpcMessageBytes;
if (builder.eventQueueCapacity <= 0) {
throw new IllegalArgumentException("eventQueueCapacity must be positive");
}
eventQueueCapacity = builder.eventQueueCapacity;
}
/**
@@ -149,6 +156,17 @@ public final class MxGatewayClientOptions {
return maxGrpcMessageBytes;
}
/**
* Returns the capacity of the buffer behind {@link MxGatewayClient#streamEvents}.
* A consumer that falls more than this many events behind has its stream
* cancelled with an overflow error after the buffered events are drained.
*
* @return the event buffer capacity (default 1024)
*/
public int eventQueueCapacity() {
return eventQueueCapacity;
}
@Override
public String toString() {
return "MxGatewayClientOptions{"
@@ -199,6 +217,7 @@ public final class MxGatewayClientOptions {
private Duration callTimeout;
private Duration streamTimeout;
private int maxGrpcMessageBytes;
private int eventQueueCapacity = DEFAULT_EVENT_QUEUE_CAPACITY;
private Builder() {
}
@@ -314,6 +333,17 @@ public final class MxGatewayClientOptions {
return this;
}
/**
* Sets the capacity of the iterator-style event stream buffer. Defaults to 1024.
*
* @param value the buffer capacity, must be positive
* @return this builder
*/
public Builder eventQueueCapacity(int value) {
eventQueueCapacity = value;
return this;
}
/**
* Builds an immutable {@link MxGatewayClientOptions} from the current state.
*
@@ -37,6 +37,13 @@ import mxaccess_gateway.v1.MxaccessGateway.RegisterCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.RemoveItemCommand;
import mxaccess_gateway.v1.MxaccessGateway.SetBufferedUpdateIntervalCommand;
import mxaccess_gateway.v1.MxaccessGateway.DrainEventsCommand;
import mxaccess_gateway.v1.MxaccessGateway.GetSessionStateCommand;
import mxaccess_gateway.v1.MxaccessGateway.GetWorkerInfoCommand;
import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
import mxaccess_gateway.v1.MxaccessGateway.PingCommand;
import mxaccess_gateway.v1.MxaccessGateway.SessionState;
import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeBulkCommand;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -548,15 +555,17 @@ public final class MxGatewaySession implements AutoCloseable {
* {@code OnDataChange} value for any tag that is already advised
* ({@code wasCached == true}) without modifying the existing subscription,
* and falls back to a full AddItem + Advise + wait + UnAdvise + RemoveItem
* snapshot lifecycle otherwise. The supplied {@code timeout} bounds the
* per-tag wait in the snapshot case; pass {@link Duration#ZERO} (or
* {@code null}) to use the worker default (1000 ms). Per-tag failures
* snapshot lifecycle otherwise. The supplied {@code timeout} is one shared
* deadline for the whole command, not a per-tag budget: the worker advises
* every uncached tag, waits once for all of them, then tears them down.
* Pass {@link Duration#ZERO} (or {@code null}) to use the worker default
* (1000 ms). Per-tag failures
* appear as {@link BulkReadResult} entries with {@code wasSuccessful == false};
* this method does not throw for per-tag MXAccess failures.
*
* @param serverHandle the {@code ServerHandle} owning the items
* @param tagAddresses the tag addresses to read
* @param timeout per-tag snapshot timeout (zero or null = worker default)
* @param timeout snapshot deadline shared by all tags in the command (zero or null = worker default)
* @return a per-tag {@link BulkReadResult} list
* @throws MxGatewayException on transport or protocol failure
* @throws NullPointerException if {@code tagAddresses} is {@code null}
@@ -1033,6 +1042,67 @@ public final class MxGatewaySession implements AutoCloseable {
.build());
}
/**
* Sends a {@code Ping} diagnostic command to the session's worker.
*
* @param message free-form text carried in the command
* @return the raw command reply
* @throws MxGatewayException on transport or protocol failure
*/
public MxCommandReply ping(String message) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_PING)
.setPing(PingCommand.newBuilder().setMessage(message == null ? "" : message))
.build());
}
/**
* Queries the worker-side state of this session.
*
* @return the reported {@link SessionState}
* @throws MxGatewayException on transport or protocol failure
*/
public SessionState getSessionState() {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE)
.setGetSessionState(GetSessionStateCommand.getDefaultInstance())
.build())
.getSessionState()
.getState();
}
/**
* Queries the worker process and MXAccess COM identity.
*
* @return the {@link WorkerInfoReply} (process id, version, ProgID, CLSID)
* @throws MxGatewayException on transport or protocol failure
*/
public WorkerInfoReply getWorkerInfo() {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_GET_WORKER_INFO)
.setGetWorkerInfo(GetWorkerInfoCommand.getDefaultInstance())
.build())
.getWorkerInfo();
}
/**
* Pulls queued events from the worker without opening a stream. The reply
* is bounded server-side and may carry fewer than {@code maxEvents}; drain
* iteratively until an empty list is returned.
*
* @param maxEvents the maximum number of events to return
* @return the drained events, possibly empty
* @throws MxGatewayException on transport or protocol failure
*/
public List<MxEvent> drainEvents(int maxEvents) {
return invokeCommand(MxCommand.newBuilder()
.setKind(MxCommandKind.MX_COMMAND_KIND_DRAIN_EVENTS)
.setDrainEvents(DrainEventsCommand.newBuilder().setMaxEvents(maxEvents))
.build())
.getDrainEvents()
.getEventsList();
}
/**
* Sends a pre-built {@link MxCommand} for this session and returns the raw
* reply, attaching a freshly generated client correlation id.
@@ -0,0 +1,93 @@
package com.zb.mom.ww.mxgateway.client;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import io.grpc.stub.ClientCallStreamObserver;
import io.grpc.stub.ClientResponseObserver;
import java.util.concurrent.atomic.AtomicInteger;
import mxaccess_gateway.v1.MxaccessGateway.MxEvent;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import org.junit.jupiter.api.Test;
final class MxEventStreamTests {
@Test
void overflowKeepsRetainedEventsThenSurfacesError() {
MxEventStream stream = new MxEventStream(3);
RecordingCall call = new RecordingCall();
ClientResponseObserver<StreamEventsRequest, MxEvent> observer = stream.observer();
observer.beforeStart(call);
for (long i = 1; i <= 5; i++) {
observer.onNext(MxEvent.newBuilder().setWorkerSequence(i).build());
}
assertTrue(call.cancelCount.get() >= 1, "overflow cancels the call");
assertEquals(1L, stream.next().getWorkerSequence());
assertEquals(2L, stream.next().getWorkerSequence());
assertEquals(3L, stream.next().getWorkerSequence());
MxGatewayException error = assertThrows(MxGatewayException.class, stream::hasNext);
assertTrue(error.getMessage().contains("overflowed"));
assertFalse(stream.hasNext());
}
@Test
void interruptCancelsCallAndRestoresInterruptFlag() {
MxEventStream stream = new MxEventStream(4);
RecordingCall call = new RecordingCall();
stream.observer().beforeStart(call);
Thread.currentThread().interrupt();
try {
assertThrows(io.grpc.StatusRuntimeException.class, stream::hasNext);
assertTrue(Thread.currentThread().isInterrupted(), "interrupt flag restored");
assertEquals(1, call.cancelCount.get(), "call cancelled on interrupt");
} finally {
Thread.interrupted();
}
}
private static final class RecordingCall extends ClientCallStreamObserver<StreamEventsRequest> {
final AtomicInteger cancelCount = new AtomicInteger();
@Override
public void cancel(String message, Throwable cause) {
cancelCount.incrementAndGet();
}
@Override
public boolean isReady() {
return true;
}
@Override
public void setOnReadyHandler(Runnable onReadyHandler) {
}
@Override
public void request(int count) {
}
@Override
public void setMessageCompression(boolean enable) {
}
@Override
public void disableAutoInboundFlowControl() {
}
@Override
public void onNext(StreamEventsRequest value) {
}
@Override
public void onError(Throwable t) {
}
@Override
public void onCompleted() {
}
}
}
@@ -51,7 +51,10 @@ import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
import mxaccess_gateway.v1.MxaccessGateway.QueryActiveAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.RegisterReply;
import mxaccess_gateway.v1.MxaccessGateway.ReplayGap;
import mxaccess_gateway.v1.MxaccessGateway.DrainEventsReply;
import mxaccess_gateway.v1.MxaccessGateway.SessionState;
import mxaccess_gateway.v1.MxaccessGateway.SessionStateReply;
import mxaccess_gateway.v1.MxaccessGateway.WorkerInfoReply;
import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.StreamEventsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -727,6 +730,63 @@ final class MxGatewayClientSessionTests {
}
}
@Test
void diagnosticsHelpersBuildTypedCommandsAndDecodeReplies() throws Exception {
AtomicReference<MxCommandRequest> last = new AtomicReference<>();
TestGatewayService service = new TestGatewayService() {
@Override
public void invoke(MxCommandRequest request, StreamObserver<MxCommandReply> responseObserver) {
last.set(request);
MxCommandReply.Builder reply = MxCommandReply.newBuilder()
.setSessionId(request.getSessionId())
.setKind(request.getCommand().getKind())
.setProtocolStatus(ok());
switch (request.getCommand().getKind()) {
case MX_COMMAND_KIND_GET_SESSION_STATE -> reply.setSessionState(
SessionStateReply.newBuilder().setState(SessionState.SESSION_STATE_READY));
case MX_COMMAND_KIND_GET_WORKER_INFO -> reply.setWorkerInfo(
WorkerInfoReply.newBuilder().setWorkerProcessId(99).setWorkerVersion("1.2.3"));
case MX_COMMAND_KIND_DRAIN_EVENTS -> reply.setDrainEvents(DrainEventsReply.newBuilder()
.addEvents(MxEvent.newBuilder().setWorkerSequence(5)));
default -> { }
}
responseObserver.onNext(reply.build());
responseObserver.onCompleted();
}
};
try (InProcessGateway gateway = InProcessGateway.start(service, new AtomicReference<>());
MxGatewayClient client = gateway.client("", Duration.ofSeconds(5))) {
MxGatewaySession session = MxGatewaySession.forSessionId(client, "existing-session");
MxCommandReply ping = session.ping("hello");
assertEquals(MxCommandKind.MX_COMMAND_KIND_PING, ping.getKind());
assertEquals("hello", last.get().getCommand().getPing().getMessage());
assertEquals(SessionState.SESSION_STATE_READY, session.getSessionState());
assertEquals(MxCommandKind.MX_COMMAND_KIND_GET_SESSION_STATE, last.get().getCommand().getKind());
WorkerInfoReply info = session.getWorkerInfo();
assertEquals(99, info.getWorkerProcessId());
assertEquals("1.2.3", info.getWorkerVersion());
List<MxEvent> events = session.drainEvents(10);
assertEquals(1, events.size());
assertEquals(5L, events.get(0).getWorkerSequence());
assertEquals(10, last.get().getCommand().getDrainEvents().getMaxEvents());
}
}
@Test
void eventQueueCapacityOptionDefaultsAndValidates() {
assertEquals(1024, MxGatewayClientOptions.builder().endpoint("x").build().eventQueueCapacity());
assertEquals(
7, MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(7).build().eventQueueCapacity());
assertThrows(
IllegalArgumentException.class,
() -> MxGatewayClientOptions.builder().endpoint("x").eventQueueCapacity(0).build());
}
private static TestGatewayService okInvokeService(AtomicReference<MxCommandRequest> commandRequest) {
return new TestGatewayService() {
@Override
+8 -1
View File
@@ -116,7 +116,7 @@ comment). The ack is made as the operator: set `operator_user` (required) and
`operator_password` (may be empty) and optionally `operator_domain` on
`AcknowledgeAlarmRequest`. The gateway authenticates the operator and writes the
comment to the alarm's `AckMsg`, so the alarm history records the operator. The
password is redacted from any raised error.
password is redacted from any raised error. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account.
Cancel the surrounding task or `aclose()` an alarm iterator to terminate the
stream.
@@ -523,3 +523,10 @@ a `~/.netrc` entry or `PIP_INDEX_URL=https://<user>:<token>@gitea.dohertylan.com
- [Client Packaging](../../docs/ClientPackaging.md)
- [Client Proto Generation](../../docs/ClientProtoGeneration.md)
- [Python Client Detailed Design](./PythonClientDesign.md)
## Typed diagnostics
`Session.ping(message)`, `get_session_state()`, `get_worker_info()`, and
`drain_events(max_events)` wrap the worker diagnostic commands. `drain_events`
returns a list that may be shorter than `max_events`; call it until it returns
an empty list.
@@ -448,8 +448,10 @@ class Session:
``OnDataChange`` value for any tag that is already advised (``was_cached =
True``) without modifying the existing subscription, and falls back to
a full AddItem + Advise + wait + UnAdvise + RemoveItem snapshot lifecycle
otherwise. ``timeout_ms`` bounds the per-tag wait in the snapshot case;
pass ``0`` to use the worker default (1000 ms).
otherwise. ``timeout_ms`` is one shared deadline for the whole command,
not a per-tag budget: the worker advises every uncached tag, waits once
for all of them, then tears them down. Pass ``0`` to use the worker
default (1000 ms).
"""
if tag_addresses is None:
raise TypeError("tag_addresses is required")
@@ -838,6 +840,55 @@ class Session:
)
return reply.activate.status
async def ping(self, message: str = "", *, correlation_id: str = "") -> pb.MxCommandReply:
"""Invoke the worker `Ping` diagnostic and return its reply."""
return await self.invoke(
pb.MxCommand(kind=pb.MX_COMMAND_KIND_PING, ping=pb.PingCommand(message=message)),
correlation_id=correlation_id,
)
async def get_session_state(self, *, correlation_id: str = "") -> int:
"""Return the worker-reported `SessionState` enum value."""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_GET_SESSION_STATE,
get_session_state=pb.GetSessionStateCommand(),
),
correlation_id=correlation_id,
)
return reply.session_state.state
async def get_worker_info(self, *, correlation_id: str = "") -> pb.WorkerInfoReply:
"""Return worker process id, version, and MXAccess ProgID/CLSID."""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_GET_WORKER_INFO,
get_worker_info=pb.GetWorkerInfoCommand(),
),
correlation_id=correlation_id,
)
return reply.worker_info
async def drain_events(
self,
max_events: int = 0,
*,
correlation_id: str = "",
) -> list[pb.MxEvent]:
"""Drain queued worker events (diagnostic).
A reply may carry fewer than *max_events* (server count and frame-size
caps); call repeatedly until an empty list is returned.
"""
reply = await self.invoke(
pb.MxCommand(
kind=pb.MX_COMMAND_KIND_DRAIN_EVENTS,
drain_events=pb.DrainEventsCommand(max_events=max_events),
),
correlation_id=correlation_id,
)
return list(reply.drain_events.events)
def stream_events(
self,
*,
@@ -392,7 +392,7 @@ def unsubscribe_bulk(**kwargs: Any) -> None:
@click.option("--server-handle", required=True, type=int, help="MXAccess server handle.")
@click.option("--items", required=True, help="Comma-separated MXAccess tag addresses.")
@click.option("--timeout-ms", default=0, type=int, show_default=True,
help="Per-tag snapshot timeout in milliseconds. 0 = worker default.")
help="Snapshot deadline in milliseconds, shared by all tags in the command. 0 = worker default (1000 ms).")
@click.option("--correlation-id", default="", help="Client correlation id.")
@click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
def read_bulk(**kwargs: Any) -> None:
@@ -234,3 +234,61 @@ async def test_suspend_and_activate_return_status() -> None:
)
status = await session.activate(12, 34)
assert status.category == pb.MX_STATUS_CATEGORY_OK
@pytest.mark.asyncio
async def test_ping_sends_message_and_returns_reply() -> None:
session, stub = await _session_with([_ok(pb.MX_COMMAND_KIND_PING)])
reply = await session.ping("hello")
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_PING
assert command.ping.message == "hello"
assert reply.kind == pb.MX_COMMAND_KIND_PING
@pytest.mark.asyncio
async def test_get_session_state_returns_state() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_GET_SESSION_STATE,
session_state=pb.SessionStateReply(state=pb.SESSION_STATE_CLOSED),
),
],
)
state = await session.get_session_state()
assert state == pb.SESSION_STATE_CLOSED
assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_SESSION_STATE
@pytest.mark.asyncio
async def test_get_worker_info_returns_info() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_GET_WORKER_INFO,
worker_info=pb.WorkerInfoReply(worker_process_id=42, worker_version="1.2"),
),
],
)
info = await session.get_worker_info()
assert info.worker_process_id == 42
assert info.worker_version == "1.2"
assert stub.invoke.requests[0].command.kind == pb.MX_COMMAND_KIND_GET_WORKER_INFO
@pytest.mark.asyncio
async def test_drain_events_sends_max_and_returns_events() -> None:
session, stub = await _session_with(
[
_ok(
pb.MX_COMMAND_KIND_DRAIN_EVENTS,
drain_events=pb.DrainEventsReply(events=[pb.MxEvent(), pb.MxEvent()]),
),
],
)
events = await session.drain_events(max_events=5)
assert len(events) == 2
command = stub.invoke.requests[0].command
assert command.kind == pb.MX_COMMAND_KIND_DRAIN_EVENTS
assert command.drain_events.max_events == 5
+1 -1
View File
@@ -40,7 +40,7 @@ prost-types = "0.13.5"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.145"
thiserror = "2.0.17"
tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time"] }
tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time", "fs"] }
tokio-stream = { version = "0.1.17", features = ["net"] }
tonic = { version = "0.13.1", features = ["transport", "tls-ring", "tls-native-roots"] }
tonic-build = "0.13.1"
+13 -1
View File
@@ -127,7 +127,7 @@ returns an `AlarmFeedStream` async stream of alarm-feed messages and
shares the gateway's central alarm monitor with every other client.
`acknowledge_alarm` requires `operator_user` and carries `operator_password` (may be
empty) and `operator_domain`; the gateway authenticates as that operator so the Historian
records them. The password is scrubbed from any returned error's `Display`/`Debug`.
records them. The operator's full name is not sent by the client and is not propagated to MXAccess (`operator_full_name` on the worker commands is accepted but ignored); the Historian takes it from the authenticated Galaxy account. The password is scrubbed from any returned error's `Display`/`Debug`.
The `from_truncated_snapshot` field on `ActiveAlarmSnapshot` reports that the record came from
a provider fetch which hit the per-fetch cap: the snapshot set may omit active
@@ -178,6 +178,18 @@ redaction. Per-item bulk failures are reported inside each result entry
`invoke_raw` / `client.invoke_raw` escape hatch performs neither check and
returns the unvalidated reply.
### Diagnostic commands
`Session` wraps the worker diagnostic commands as typed calls: `ping(message)`
(returns `()` on success), `get_session_state()` (returns `SessionState`),
`get_worker_info()` (returns `WorkerInfoReply`: pid, version, MXAccess
ProgID/CLSID) and `drain_events(max_events)` (returns `Vec<MxEvent>`; the reply
is bounded server-side, so drain until an empty result). A reply missing its
expected payload surfaces as `Error::MalformedReply`.
TLS note: `ClientOptions::with_ca_file` is read asynchronously (`tokio::fs`)
during `connect`, so connecting no longer blocks the executor thread.
## Event Streaming And Reconnect-Replay Gaps
`session.events()` / `session.events_after(after_worker_sequence)` (and the
+1 -1
View File
@@ -215,7 +215,7 @@ enum Command {
server_handle: i32,
#[arg(long, value_delimiter = ',')]
items: Vec<String>,
/// Per-tag snapshot timeout in milliseconds. `0` uses the worker default (1000 ms).
/// Snapshot deadline in milliseconds, shared by all tags in the command. `0` uses the worker default (1000 ms).
#[arg(long, default_value_t = 0)]
timeout_ms: u32,
#[arg(long)]
+1 -1
View File
@@ -187,7 +187,7 @@ impl GatewayClient {
})?;
endpoint = endpoint.connect_timeout(options.connect_timeout());
if let Some(tls) = build_tls_config(&options)? {
if let Some(tls) = build_tls_config(&options).await? {
endpoint = endpoint.tls_config(tls)?;
}
+1 -1
View File
@@ -182,7 +182,7 @@ impl GalaxyClient {
})?;
endpoint = endpoint.connect_timeout(options.connect_timeout());
if let Some(tls) = build_tls_config(&options)? {
if let Some(tls) = build_tls_config(&options).await? {
endpoint = endpoint.tls_config(tls)?;
}
+28 -15
View File
@@ -3,7 +3,6 @@
//! chain of `with_*` setters; the `Debug` impl redacts the API key.
use std::fmt;
use std::fs;
use std::path::PathBuf;
use std::time::Duration;
@@ -235,7 +234,9 @@ pub(crate) fn tls_trust_decision(options: &ClientOptions) -> TlsTrustDecision {
/// rejects the configuration early with an actionable error. The strict opt-in
/// instead loads the system trust roots so a certificate chaining to an
/// already-trusted root validates.
pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientTlsConfig>, Error> {
pub(crate) async fn build_tls_config(
options: &ClientOptions,
) -> Result<Option<ClientTlsConfig>, Error> {
let decision = tls_trust_decision(options);
if decision == TlsTrustDecision::None {
return Ok(None);
@@ -248,10 +249,13 @@ pub(crate) fn build_tls_config(options: &ClientOptions) -> Result<Option<ClientT
match decision {
TlsTrustDecision::PinnedCa => {
let ca_file = options.ca_file().expect("PinnedCa implies a CA file");
let certificate = fs::read(ca_file).map_err(|source| Error::InvalidEndpoint {
endpoint: options.endpoint().to_owned(),
detail: format!("failed to read CA file {}: {source}", ca_file.display()),
})?;
let certificate =
tokio::fs::read(ca_file)
.await
.map_err(|source| Error::InvalidEndpoint {
endpoint: options.endpoint().to_owned(),
detail: format!("failed to read CA file {}: {source}", ca_file.display()),
})?;
tls = tls.ca_certificate(Certificate::from_pem(certificate));
}
TlsTrustDecision::SystemRoots => {
@@ -334,11 +338,11 @@ mod tests {
assert!(!debug.contains("mxgw_secret"));
}
#[test]
fn plaintext_needs_no_tls() {
#[tokio::test]
async fn plaintext_needs_no_tls() {
let options = ClientOptions::new("http://127.0.0.1:5000").with_plaintext(true);
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::None);
assert!(build_tls_config(&options).unwrap().is_none());
assert!(build_tls_config(&options).await.unwrap().is_none());
}
#[test]
@@ -349,8 +353,8 @@ mod tests {
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::PinnedCa);
}
#[test]
fn strict_without_ca_uses_system_roots() {
#[tokio::test]
async fn strict_without_ca_uses_system_roots() {
// Regression for Client.Rust-031: strict verification with no pinned CA
// must verify against the system trust roots, not produce a config with
// zero trust anchors. The trust decision proves roots are consulted; the
@@ -365,15 +369,24 @@ mod tests {
"strict-no-CA must request the system trust roots"
);
assert!(
build_tls_config(&options).unwrap().is_some(),
build_tls_config(&options).await.unwrap().is_some(),
"strict-no-CA must build a usable TLS config"
);
}
#[test]
fn lenient_without_ca_is_rejected() {
#[tokio::test]
async fn lenient_without_ca_is_rejected() {
let options = ClientOptions::new("https://127.0.0.1:5000").with_plaintext(false);
assert_eq!(tls_trust_decision(&options), TlsTrustDecision::RejectNoCa);
assert!(build_tls_config(&options).is_err());
assert!(build_tls_config(&options).await.is_err());
}
#[tokio::test]
async fn missing_pinned_ca_file_errors_without_blocking_read() {
let options = ClientOptions::new("https://127.0.0.1:5000")
.with_plaintext(false)
.with_ca_file("/nonexistent/definitely-missing-ca.pem");
let error = build_tls_config(&options).await.unwrap_err();
assert!(error.to_string().contains("failed to read CA file"));
}
}
+96 -8
View File
@@ -17,14 +17,15 @@ use crate::generated::mxaccess_gateway::v1::mx_command_reply;
use crate::generated::mxaccess_gateway::v1::{
ActivateCommand, AddBufferedItemCommand, AddItem2Command, AddItemBulkCommand, AddItemCommand,
AdviseCommand, AdviseItemBulkCommand, AdviseSupervisoryCommand, ArchestrAUserToIdCommand,
AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest, MxCommand,
MxCommandKind, MxCommandReply, MxCommandRequest, MxDataType, MxSparseArray, MxSparseElement,
MxValue as ProtoMxValue, OpenSessionRequest, ReadBulkCommand, RegisterCommand,
RemoveItemBulkCommand, RemoveItemCommand, SetBufferedUpdateIntervalCommand,
AuthenticateUserCommand, BulkReadResult, BulkWriteResult, CloseSessionRequest,
DrainEventsCommand, GetSessionStateCommand, GetWorkerInfoCommand, MxCommand, MxCommandKind,
MxCommandReply, MxCommandRequest, MxDataType, MxEvent, MxSparseArray, MxSparseElement,
MxValue as ProtoMxValue, OpenSessionRequest, PingCommand, ReadBulkCommand, RegisterCommand,
RemoveItemBulkCommand, RemoveItemCommand, SessionState, SetBufferedUpdateIntervalCommand,
StreamEventsRequest, SubscribeBulkCommand, SubscribeResult, SuspendCommand, UnAdviseCommand,
UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, Write2BulkCommand,
Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry, WriteCommand,
WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
UnAdviseItemBulkCommand, UnregisterCommand, UnsubscribeBulkCommand, WorkerInfoReply,
Write2BulkCommand, Write2BulkEntry, Write2Command, WriteBulkCommand, WriteBulkEntry,
WriteCommand, WriteSecured2BulkCommand, WriteSecured2BulkEntry, WriteSecured2Command,
WriteSecuredBulkCommand, WriteSecuredBulkEntry, WriteSecuredCommand,
};
use crate::value::{MxStatus, MxValue, MxValueProjection};
@@ -532,7 +533,10 @@ impl Session {
/// returning the most recent cached `OnDataChange` value when the tag is
/// already advised (`was_cached = true`), or by taking a full AddItem +
/// Advise + wait + UnAdvise + RemoveItem snapshot lifecycle otherwise.
/// `timeout_ms == 0` lets the worker pick its default (1000 ms).
/// `timeout_ms` is one shared deadline for the whole command, not a
/// per-tag budget: the worker advises every uncached tag, waits once for
/// all of them, then tears them down. `timeout_ms == 0` lets the worker
/// pick its default (1000 ms).
/// Per-tag failures appear as `BulkReadResult` entries with
/// `was_successful = false`; the call never errors on per-tag failure.
///
@@ -960,6 +964,90 @@ impl Session {
.await
}
/// Send a `Ping` diagnostic command to the worker. A successful return
/// means the worker answered; `message` is forwarded for log correlation.
///
/// # Errors
///
/// Returns [`Error::Command`] for non-OK statuses plus transport errors.
pub async fn ping(&self, message: &str) -> Result<(), Error> {
self.invoke(
MxCommandKind::Ping,
Payload::Ping(PingCommand {
message: message.to_owned(),
}),
)
.await?;
Ok(())
}
/// Fetch the gateway-side [`SessionState`] of this session.
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a session-state
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn get_session_state(&self) -> Result<SessionState, Error> {
let reply = self
.invoke(
MxCommandKind::GetSessionState,
Payload::GetSessionState(GetSessionStateCommand {}),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::SessionState(state)) => {
Ok(SessionState::try_from(state.state).unwrap_or(SessionState::Unspecified))
}
_ => Err(Error::MalformedReply {
detail: "get_session_state reply did not carry a SessionState payload".to_owned(),
}),
}
}
/// Fetch worker process details (pid, version, MXAccess ProgID/CLSID).
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a worker-info
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn get_worker_info(&self) -> Result<WorkerInfoReply, Error> {
let reply = self
.invoke(
MxCommandKind::GetWorkerInfo,
Payload::GetWorkerInfo(GetWorkerInfoCommand {}),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::WorkerInfo(info)) => Ok(info),
_ => Err(Error::MalformedReply {
detail: "get_worker_info reply did not carry a WorkerInfo payload".to_owned(),
}),
}
}
/// Drain up to `max_events` queued events through the diagnostic command
/// (distinct from the streaming [`events`](Self::events) surface). The
/// reply is bounded server-side, so call repeatedly until it is empty.
///
/// # Errors
///
/// Returns [`Error::MalformedReply`] when the reply lacks a drain-events
/// payload, plus the errors of [`invoke`](Self::invoke).
pub async fn drain_events(&self, max_events: u32) -> Result<Vec<MxEvent>, Error> {
let reply = self
.invoke(
MxCommandKind::DrainEvents,
Payload::DrainEvents(DrainEventsCommand { max_events }),
)
.await?;
match reply.payload {
Some(mx_command_reply::Payload::DrainEvents(drain)) => Ok(drain.events),
_ => Err(Error::MalformedReply {
detail: "drain_events reply did not carry a DrainEvents payload".to_owned(),
}),
}
}
/// Issue a raw `Invoke` for an arbitrary command, without filtering on
/// the protocol status. Useful when callers need the full reply for
/// commands not yet wrapped by `Session`.
+130 -4
View File
@@ -26,12 +26,12 @@ use zb_mom_ww_mxgateway_client::generated::mxaccess_gateway::v1::{
AddItem2Reply, AddItemReply, AlarmConditionState, AlarmFeedMessage, AlarmTransitionKind,
AuthenticateUserCommand, AuthenticateUserReply, BulkReadReply, BulkReadResult,
BulkSubscribeReply, BulkWriteReply, BulkWriteResult, CloseSessionReply, CloseSessionRequest,
MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily, MxSparseArray,
MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
DrainEventsReply, MxCommandKind, MxCommandReply, MxDataType, MxEvent, MxEventFamily,
MxSparseArray, MxSparseElement, MxStatusCategory, MxStatusProxy, MxStatusSource, MxValue,
OnAlarmTransitionEvent, OpenSessionReply, OpenSessionRequest, ProtocolStatus,
ProtocolStatusCode, QueryActiveAlarmsRequest, RegisterReply, ReplayGap, SessionState,
StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, Write2BulkEntry, WriteBulkEntry,
WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
SessionStateReply, StreamAlarmsRequest, StreamEventsRequest, SubscribeResult, WorkerInfoReply,
Write2BulkEntry, WriteBulkEntry, WriteCommand, WriteSecured2BulkEntry, WriteSecuredBulkEntry,
};
use zb_mom_ww_mxgateway_client::{
next_correlation_id, ApiKey, ClientOptions, CommandError, Error, EventItem, GatewayClient,
@@ -550,6 +550,81 @@ async fn unary_invoke_maps_status_unavailable_to_error_unavailable() {
);
}
#[tokio::test]
async fn ping_round_trips_through_the_fake_gateway() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
session.ping("hello").await.unwrap();
assert_eq!(
*state.last_command_kind.lock().await,
Some(MxCommandKind::Ping as i32)
);
}
#[tokio::test]
async fn get_session_state_returns_typed_state() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let session_state = session.get_session_state().await.unwrap();
assert_eq!(session_state, SessionState::Ready);
}
#[tokio::test]
async fn get_worker_info_returns_typed_reply() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let info = session.get_worker_info().await.unwrap();
assert_eq!(info.worker_process_id, 4242);
assert_eq!(info.worker_version, "1.2.3");
}
#[tokio::test]
async fn drain_events_returns_events() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
let events = session.drain_events(10).await.unwrap();
assert_eq!(events.len(), 1);
}
#[tokio::test]
async fn get_session_state_malformed_without_payload() {
let state = Arc::new(FakeState::default());
let endpoint = spawn_fake_gateway(state.clone()).await;
let client = GatewayClient::connect(ClientOptions::new(endpoint))
.await
.unwrap();
let session = client.session("session-fixture");
*state.invoke_override.lock().await = Some(InvokeOverride::OkWithoutPayload);
let error = session.get_session_state().await.unwrap_err();
assert!(matches!(error, Error::MalformedReply { .. }));
}
#[tokio::test]
async fn read_bulk_round_trips_through_the_fake_gateway() {
let state = Arc::new(FakeState::default());
@@ -1337,6 +1412,57 @@ impl MxAccessGateway for FakeGateway {
}));
}
if kind == MxCommandKind::Ping as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::GetSessionState as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::SessionState(SessionStateReply {
state: SessionState::Ready as i32,
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::GetWorkerInfo as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::WorkerInfo(WorkerInfoReply {
worker_process_id: 4242,
worker_version: "1.2.3".to_owned(),
..WorkerInfoReply::default()
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::DrainEvents as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
correlation_id: "fake-correlation".to_owned(),
kind,
protocol_status: Some(ok_status("command ok")),
payload: Some(mx_command_reply::Payload::DrainEvents(DrainEventsReply {
events: vec![MxEvent::default()],
})),
..MxCommandReply::default()
}));
}
if kind == MxCommandKind::ReadBulk as i32 {
return Ok(Response::new(MxCommandReply {
session_id: request.session_id,
+12
View File
@@ -1051,6 +1051,18 @@ Every transition and snapshot record carries `degraded = true` and
`source_provider = ALARM_PROVIDER_MODE_SUBTAG`. Clients that require full
fidelity must wait for failback to the alarm manager.
### Retrigger transitions
`ALARM_TRANSITION_KIND_RETRIGGER` is reserved in the contract and the worker
never emits it, in either provider mode. The alarmmgr XML reports only the
four states `UNACK_ALM`, `ACK_ALM`, `UNACK_RTN`, and `ACK_RTN`, so a condition
that returns to alarm after clearing (`*_RTN` → `*_ALM`) looks the same as a
new raise, and `AlarmRecordTransitionMapper.MapTransition` reports it as
`ALARM_TRANSITION_KIND_RAISE`. The subtag source has no re-alarm counter to
observe either. The worker does not infer a retrigger from timing or
from a raise that follows a clear, because the gateway forwards only
transitions its source reports.
### Provider mode reflection
When `FailoverAlarmConsumer` switches between providers, it raises
+10
View File
@@ -113,6 +113,16 @@ library:
library returns carries no expiry timestamp, so the cache cannot cap an entry at the
key's expiry (capping it needs the donor library to surface expiry on the
verification identity). The default 15 s TTL bounds both windows.
Misses are **single-flight**: concurrent misses for the same token share one inner
verification, so a TTL expiry under load costs one store read instead of one per
in-flight RPC. The flight is keyed on the token hash *and* the key's generation, so
a miss that starts after an `Invalidate` never joins a verification that started
before it and cannot receive the pre-revoke identity. The shared inner call is not
bound to any one caller's cancellation token; a caller that gives up stops waiting
without cancelling the others. The per-key index `Invalidate` uses to find every
cached secret variant is pruned by a post-eviction callback when the cache drops an
entry on its own (TTL expiry, compaction), so it tracks the live cache instead of
growing with every token ever verified.
- **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
most one `MarkUsed` write per key per
`MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a
+18 -7
View File
@@ -10,7 +10,7 @@ Authorization runs as a single gRPC server interceptor registered for every call
The participating types live under `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/`:
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for unary and server-streaming calls.
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for every call shape: unary, server-streaming, client-streaming, and duplex-streaming.
- `GatewayGrpcScopeResolver` maps a request message (and, for `MxCommandRequest`, the inner `MxCommandKind`) to the scope string that must be present on the caller.
- `GatewayScopes` exposes the canonical scope constants used by the resolver and any downstream consumer.
- `GatewayRequestIdentityAccessor` and `IGatewayRequestIdentityAccessor` expose the verified identity to handlers and any service code that runs inside the call.
@@ -32,7 +32,7 @@ Centralizing the policy in `GatewayGrpcAuthorizationInterceptor` produces three
## Interceptor Flow
`GatewayGrpcAuthorizationInterceptor` overrides both `UnaryServerHandler` and `ServerStreamingServerHandler`. Both call the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then push the resolved identity onto the accessor for the duration of the call.
`GatewayGrpcAuthorizationInterceptor` overrides all four server handlers: `UnaryServerHandler`, `ServerStreamingServerHandler`, `ClientStreamingServerHandler`, and `DuplexStreamingServerHandler`. Each calls the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then pushes the resolved identity onto the accessor for the duration of the call. The base `Interceptor` passes any handler it does not override straight through, so a client- or duplex-streaming RPC added to a mapped service would otherwise run with no API key check at all. The unary and server-streaming handlers resolve the required scope from the request message; the client- and duplex-streaming handlers have no single request to inspect before the handler reads the stream, so they fail closed to `admin` — the same scope the resolver assigns an unrecognized request type. No public RPC uses either shape today.
```csharp
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
@@ -40,7 +40,10 @@ public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
scopeResolver.ResolveRequiredScope(request),
context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
@@ -96,7 +99,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is anonymous by default (`MxGateway:Metrics:RequireAuthentication` is opt-in), so neither key ids nor peer addresses may appear there.
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
@@ -262,9 +265,17 @@ the gateway fails closed.
Non-bulk constraint failures return gRPC `PermissionDenied`. Bulk read
commands preserve input order and return a failed `SubscribeResult` for each
denied item while still forwarding allowed items to the worker. Every denial
adds an `api_key_audit` entry with the key id, command kind, target, and
blocking constraint; secured values and raw credentials are never logged.
denied item while still forwarding allowed items to the worker. A non-bulk
denial adds one `constraint-denied` audit event (canonical `audit_event` table)
with the key id, command kind, target, and blocking constraint. A bulk command
adds **one aggregated** `constraint-denied` event per command, not one per
denied item: its target is `<commandKind>:bulk` and its details carry the
denied and total item counts, the distinct blocking constraint names, a sample
of at most 20 denied targets (`sampleTruncated` says whether more were
denied), and the client correlation id. The audit writer is a bounded channel
that drops on full, so per-item events let a single low-privilege key flush
every other pending audit record with one large bulk command. Secured values
and raw credentials are never logged.
## Scope Catalog
+38
View File
@@ -430,6 +430,44 @@ Suggested mapping:
- Galaxy SQL metadata endpoints if added: `metadata:read`
- worker shutdown diagnostics and key management: `admin`
## Metrics Endpoint Authentication (SEC-14)
Decision: `/metrics` stays anonymous by default and can be put behind
authentication with `MxGateway:Metrics:RequireAuthentication=true`; `/health`
stays anonymous unconditionally.
- **Opt-in rather than on by default.** Existing Prometheus scrapers on the
deployed hosts scrape anonymously; flipping the default would silently break
them on upgrade. Hosts whose network exposure makes the counters sensitive
enable it explicitly.
- **Health is never gated.** The family Overview dashboard (another repo) polls
`/health/active` without credentials, and liveness/readiness probes are
conventionally anonymous. Health responses carry status only.
- **Credentials reuse existing plumbing.** One `MxGateway.Metrics` scheme
accepts an API key with the `admin` scope (verified through the same cached
`IApiKeyVerifier` and the same per-peer failure limiter as gRPC, so `/metrics`
is not a way around the gRPC brute-force throttle) or, when no authorization
header is sent, the dashboard login cookie with a real dashboard role. No new
credential type or store is introduced.
- **`admin` scope, any dashboard role.** Metrics are an operator surface, so a
data-plane client key (`invoke:*`, `events:read`) is not enough. A dashboard
`Viewer` qualifies because every dashboard role already sees the same aggregate
counters on the dashboard; the exported metrics carry no session ids or tag
values.
- **No environment bypass.** The requirement reads no options, so anonymous
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
dashboard Viewer policy — do not satisfy it. Nor does dev-only
`Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
name, so the metrics scheme skips the cookie fallback while it is on rather than
letting a dev flag silently open an endpoint the operator asked to protect.
- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
the remote IP, while the gRPC one is the gRPC peer string (per connection), so
only the per-key-id aggregate is shared between the two paths.
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
scheme that falls back to the cookie internally keeps the challenge a `401`
with `WWW-Authenticate: Bearer`, which scrapers understand.
## Worker Process Identity
Decision: run workers as the gateway service identity for v1.
+42 -1
View File
@@ -38,6 +38,7 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
"LeaseSweepIntervalSeconds": 30,
"DetachGraceSeconds": 30,
"FaultedGraceSeconds": 0,
"UnattachedSessionGraceSeconds": 0,
"AllowMultipleEventSubscribers": false,
"MaxEventSubscribersPerSession": 8,
"WorkerReadyWaitTimeoutMs": 0
@@ -153,7 +154,8 @@ validation fails fast when either half of the headroom rule is violated.
| `MxGateway:Sessions:LeaseSweepIntervalSeconds` | `30` | Hosted monitor interval for closing expired leases. Active event-stream subscribers keep a session from expiring while the stream remains attached. |
| `MxGateway:Sessions:DetachGraceSeconds` | `30` | Detach-grace retention window. When positive, a session whose last external (gRPC) event-stream subscriber drops is retained in `Ready` for this many seconds so a client can reconnect; if no external subscriber re-attaches within the window, the lease monitor closes it with `detach-grace-expired`. The internal dashboard mirror does not count as an external subscriber, so a dashboard-only session still enters detach-grace. `0` disables retention and reverts to closing only on normal lease expiry. Must be zero or greater. Reconnect/replay itself is implemented separately (Task 12); this option controls retention and expiry only. The effective close happens within the next sweep cycle after the window elapses — up to `LeaseSweepIntervalSeconds` after expiry. Operators wanting a firm minimum retention bound should set `DetachGraceSeconds` greater than `LeaseSweepIntervalSeconds`. |
| `MxGateway:Sessions:FaultedGraceSeconds` | `0` | Grace window before the lease monitor reaps a faulted session (killing its worker and freeing the slot). A faulted session is permanently unusable yet otherwise pins a session slot and a live x86 worker until its normal lease expires. `0` (the default) reaps it on the next sweep cycle, bounding blast radius; a positive value keeps the faulted session observable via `GetSessionStatus` for that window before it is reclaimed, with the close reason `faulted-reaped`. Must be zero or greater. |
| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `AlreadyExists`. Set to `true` to enable fan-out via the `SessionEventDistributor`. |
| `MxGateway:Sessions:UnattachedSessionGraceSeconds` | `0` | Idle window after which the lease monitor reaps a `Ready` session that has **never** attached a `StreamEvents` subscriber, with the close reason `unattached-grace-expired`. Exists because a client that opens a session and vanishes before it ever streams — OtOpcUa's `GalaxyDriver` failing part-way through start-up, for example — otherwise pins a session slot and a live x86 worker for the full `DefaultLeaseSeconds` (30 minutes). The window is measured from the later of the session becoming `Ready` and its last unary command, so a slow worker start does not eat it and a unary-only client that keeps calling is never reaped by this rule; it stops applying once the session's first subscriber attaches (`DetachGraceSeconds` and the normal lease govern it from then on), and the gateway's own alarm-monitor session, whose subscriber is internal, is exempt the same way. The dashboard Browse live-data session (`DashboardLiveDataService`) is not exempt: it is unary-only, so it is reaped once idle for the window, and it opens a new session through `EnsureReadyAsync` on its next use. `0` (the default) disables it. Must be zero or greater. Set it comfortably above the time a well-behaved client needs between `OpenSession` and its first `StreamEvents`; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. |
| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `ResourceExhausted` (`EventSubscriberAlreadyActive`). Set to `true` to enable fan-out via the `SessionEventDistributor`. |
| `MxGateway:Sessions:MaxEventSubscribersPerSession` | `8` | Maximum number of concurrent `StreamEvents` subscribers per session when `AllowMultipleEventSubscribers` is `true`. Effectively 1 when `AllowMultipleEventSubscribers` is `false`. Must be greater than zero. |
| `MxGateway:Sessions:WorkerReadyWaitTimeoutMs` | `0` | Bounded time, in milliseconds, the gateway will wait for a worker to reach `Ready` when the session is already `Ready` but the worker state has transiently diverged (e.g. `Handshaking` after a heartbeat blip). Applies only to transient worker states; terminal states (`Faulted`/`Closing`/`Closed`/no worker) fail fast immediately regardless of this setting. `0` (the default) disables the wait and preserves the original fail-fast behavior. Must be greater than or equal to zero. |
@@ -419,6 +421,45 @@ model requires otherwise.
| `MxGateway:Security:AuditRetentionDays` | `90` | Days of canonical audit history kept in the `audit_event` table. The audit drain sweeps once at startup and hourly thereafter, deleting rows older than this window; without it the table grows without bound inside the same SQLite file the authentication hot path reads. Rows whose timestamp SQLite cannot parse are never swept. Must be greater than zero — retention can be widened but not switched off. |
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
## Metrics Options
Bound from `MxGateway:Metrics`. Governs the Prometheus `/metrics` scrape endpoint
(mounted by the shared `ZB.MOM.WW.Telemetry` package). The health endpoints
(`/healthz`, `/health/ready`, `/health/active`) are not affected by this section and
always stay anonymous: the family Overview dashboard polls `/health/active` from
another repo without credentials, so gating it would break that consumer.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access, `Authentication:Mode=Disabled`, and `Dashboard:DisableLogin` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
Configure a Prometheus scraper against an authenticated endpoint with a dedicated
admin-scoped key, for example:
```yaml
scrape_configs:
- job_name: mxgateway
authorization:
type: Bearer
credentials_file: /etc/prometheus/mxgateway.key # contains mxgw_<keyId>_<secret>
static_configs:
- targets: ["gateway-host:5130"]
```
The setting is read from configuration when endpoints are mapped, so changing it
requires a restart. Dev-only `Dashboard:DisableLogin=true` registers an
always-succeeding auto-login handler under the dashboard cookie scheme; the
metrics scheme deliberately skips its cookie fallback while that flag is on, so
`DisableLogin` never silently opens a `/metrics` endpoint that
`RequireAuthentication` protects — only an admin-scoped API key is accepted then.
The `/metrics` path reuses the gRPC API-key failure limiter
(`MxGateway:Security:ApiKeyFailure*`), but its per-peer partition is keyed on the
remote **IP address**, whereas the gRPC interceptor's is the gRPC peer string
(address and port — effectively per connection). The two paths therefore share
only the per-key-id aggregate layer (`ApiKeyFailureAggregateLimit`); per-peer
failure counts on one path do not throttle the other.
## Galaxy Options
| Option | Default | Description |
+151 -73
View File
@@ -273,9 +273,14 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
| Hub | Path | Producer | Payload | Routing |
|---|---|---|---|---|
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `SnapshotUpdated`: `DashboardSnapshot` **without** `configuration`; `ConfigurationUpdated`: `EffectiveGatewayConfiguration` | Ticks go to all connected clients on every snapshot interval, but only while at least one client is connected (see "Idle gating" below). The effective configuration is startup-static and the largest part of the record, so it is `[JsonIgnore]`d on `DashboardSnapshot` and sent once per connection instead: `OnConnectedAsync` pushes `ConfigurationUpdated` and then the current `SnapshotUpdated`, and the hub method `GetConfiguration()` returns it on demand. Each tick broadcast is bounded to 1 s; a send that misses the bound is abandoned, so one slow client cannot delay the next tick for everyone. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. Every subscription to the monitor — the first and each resubscribe — is preceded by an `AlarmFeedReset` frame (`{ generation }`, monotonic per process): the monitor primes each new subscription with a full `active_alarm` snapshot ending in `snapshot_complete`, so a client must discard its accumulated alarm state on the marker and rebuild from that snapshot rather than double-counting it. Each broadcast is bounded to 2 s and abandoned on timeout: awaiting it unbounded let one slow client stall the drain until the monitor dropped the publisher as a slow subscriber, and the resubscribe then replayed the whole snapshot to every client. The tradeoff: a client whose send is abandoned misses that one frame and receives no reset for it, so its alarm view can be stale until the next resubscribe (or until SignalR drops the stalled connection); there is no in-family consumer of `/hubs/alarms` today. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry`. Delivery to the group goes through one bounded per-session pump (see "Mirror gating"), so a slow viewer loses its oldest events instead of accumulating pending sends — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
No hub client ships in this repository: the server-rendered pages read the
in-process seams above, and the hubs serve external consumers. Those consumers
must honour `AlarmFeedReset` and keep the configuration from
`ConfigurationUpdated`, since ticks no longer carry it.
### Default cadences
@@ -343,12 +348,17 @@ subscribers. Two races motivate the extra state:
gate open, and cancelling on their behalf would stop a live pump that other
viewers depend on.
Two per-tick costs inside the snapshot itself are bounded independently of either gate:
Three per-tick costs inside the snapshot itself are bounded independently of either gate:
- the effective configuration (`EffectiveGatewayConfiguration`) is built once and
cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
at startup and never reloads, so rebuilding the whole option tree every tick
produced an identical object;
produced an identical object. For the same reason it is not serialized on the
`/hubs/snapshot` tick (see the hub table above);
- the gateway status is one `HealthCheckService.CheckHealthAsync` evaluation per
tick, bounded to 2 s and behind a zero-wait gate so concurrent enumerations (the
hub publisher and the page feed) do not both evaluate; `GetSnapshot()` reads the
cached result;
- the API key summaries are refreshed at most once every 15 seconds
(`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
read whose content changes only when an operator creates, rotates, or revokes a
@@ -359,7 +369,7 @@ Two per-tick costs inside the snapshot itself are bounded independently of eithe
Avoid pushing every MXAccess data-change event into a wider broadcast group.
Events are routed strictly per session (`session:{id}` groups on the hub,
per-session subscriber lists in process); the snapshot seams continue to carry
aggregate event counters and rates.
aggregate event counters.
### Mirror gating
@@ -394,10 +404,21 @@ group. With `ShowTagValues` true there is no clone at all — the original `MxEv
instance is handed to both audiences — so the "one clone per event" cost holds only
in the redacting configuration, and in the value-showing one both audiences share a
reference to the session pipeline's own event object. In-process delivery runs first
and synchronously — it cannot
throw, and it must not be skipped by the guard clause around the hub send — into
per-subscriber bounded drop-oldest channels, so a page that falls behind loses its
oldest queued events rather than blocking the session's event pipeline.
and synchronously — it cannot throw — into per-subscriber bounded drop-oldest
channels, so a page that falls behind loses its oldest queued events rather than
blocking the session's event pipeline.
Hub delivery has the same shape. `EventsHubViewerRegistry` counts remote hub
connections separately from `inproc-` subscribers (`HasHubViewers`), and while a
session has at least one, `Publish` writes the event into that session's
`HubPump`: a 256-deep drop-oldest channel drained by a single task that sends one
event at a time to `session:{id}`, each send bounded to 5 s. A session watched only
in process starts no pump. The pump starts on the first event published while a
hub viewer is present and is torn down when the registry raises
`HubViewersReleased` for the session's last hub viewer (unsubscribe or disconnect).
Before this, `Publish` fired a group `SendAsync` per event and returned; a slow
remote viewer made each send stay pending in the SignalR lifetime manager, and the
pending sends — each holding a cloned event — grew without bound.
The mirror subscriber itself is still registered on the `SessionEventDistributor`
for the session's whole lifetime; only the per-event work is gated. Starting and
@@ -411,55 +432,67 @@ clone and send this gate already removes.
### Dashboard home
Show top-level status:
The Overview page (`/`) shows, from the snapshot:
- gateway status,
- gateway version,
- uptime,
- open sessions,
- workers running,
- sessions faulted,
- command rate,
- command failure count,
- event rate,
- event queue depth,
- worker restart/kill count.
- gateway status badge — the aggregate status of the registered ASP.NET Core
health checks (`Healthy` / `Degraded` / `Unhealthy`), evaluated once per snapshot
tick with a 2 s bound and cached for the synchronous `GetSnapshot()` seed path. It
reads `Unknown` before the first evaluation, when an evaluation throws or times
out, and when no `HealthCheckService` is registered — never a hardcoded `Healthy`
(SEC-27);
- the snapshot's generation time;
- metric cards: uptime (with the gateway version as detail), open sessions, workers
running, worker event queue depth, failed commands, events received, faults, and
queue overflows — all cumulative counters or point-in-time gauges, not rates;
- a Galaxy Repository section: cache status, last deploy time, object count (with
area count), attribute, historized-attribute and alarm-attribute counts, and the
last refresh time;
- the recent faults list (`Dashboard:RecentFaultLimit`).
Use Bootstrap cards for individual metric summaries. Keep the layout compact
and operational.
There is no per-family command or event rate, no faulted-session counter, and no
worker restart/kill card on this page; `mxgateway.workers.killed` and
`mxgateway.workers.exited` are in the snapshot's metric list but not rendered.
Metric summaries use Bootstrap cards; the layout stays compact and operational.
### Sessions page
Show active and recent sessions in a table:
Shows the most recent sessions (`Dashboard:RecentSessionLimit`, newest first) in a
table with these columns:
- session id,
- client identity or API key display name,
- state,
- session id (links to the session details page),
- session state,
- client identity (redacted),
- backend,
- worker process id,
- worker process id and worker state,
- events received for the session (the `mxgateway.events.received` per-session
counter),
- open time,
- last client activity,
- last worker heartbeat,
- active event subscribers,
- pending commands,
- event queue depth,
- last fault summary.
- last fault summary (abbreviated, full text on hover),
- Close / Kill actions for Admin principals.
Rows should link to session details.
Event subscriber counts, pending command counts, and per-session queue depth are
not tracked per session and are not shown.
### Session details page
Show:
Shows:
- session metadata,
- worker metadata,
- command counters by method,
- event counters by family,
- active server handles and item counts if gateway shadow state has them,
- latest faults,
- last heartbeat payload,
- a session section: state badge, backend, client identity, client session name,
client correlation id (all redacted), open time, last activity, lease expiry,
events received, and last fault;
- a worker section: process id, worker state, and last heartbeat time;
- a "Recent events" table — the newest 50 events from the in-process mirror, with
worker sequence, family, server handle, item handle, and status (no values), plus
a live/offline pill; gated by the session ACL, so a caller who may not observe the
session sees "Not authorized" instead;
- admin Close session / Kill worker controls (Admin role only).
Per-method command counters, per-family event counters for the session, server
handle / item counts, and the heartbeat payload are not shown — the gateway keeps
no per-session shadow state for them.
The Sessions list, the Workers list, and this details page all render the same
admin controls when the signed-in principal carries the `Admin` role; viewers
and the localhost-anonymous bypass see no action affordances and the server
@@ -484,33 +517,31 @@ management audit path.
### Workers page
Show:
Shows one row per worker that is not closed, with:
- worker process id,
- session id,
- executable path/version,
- state,
- startup duration,
- memory and CPU if available,
- last heartbeat,
- current command correlation id,
- pending command count,
- event queue depth,
- restart/kill reason if terminal.
- worker state,
- owning session id (links to the session details page),
- last heartbeat time,
- last fault summary (the owning session's final fault),
- a Kill worker action for Admin principals.
Executable path and version, startup duration, memory/CPU, in-flight command
correlation, pending command count, and per-worker queue depth are not collected
and are not shown; the worker executable path is on the Settings page.
### Events page
Show aggregate event diagnostics:
Shows aggregate event diagnostics from the snapshot:
- event rate by session,
- event rate by event family,
- total events since start,
- events received since start,
- worker event queue depth and gRPC stream queue depth (gauges),
- queue overflow count,
- stream disconnect count,
- recent terminal faults.
- gRPC stream disconnect count,
- an "Event Families" table: cumulative events received per event family.
Do not display full tag values by default. If value display is later added, make
it opt-in and redacted.
The figures are counters since start, not rates; per-session event counts are on
the Sessions page. No tag values are shown here.
### Browse page
@@ -537,6 +568,33 @@ at the service, not in the page: one decision point, and the value never enters
the render tree. Set the flag true to make the panel the explicit opt-in
tag-value surface it was originally designed as.
The filter box is debounced: keystrokes within 150 ms coalesce into one search,
which runs on the thread pool with a cancellation token that the next keystroke
cancels, and only the latest query may publish its results. The search reads
`DashboardBrowseSearchIndex`, a singleton holding every attribute's full reference
and name lowercased once, keyed on the Galaxy cache `Sequence` and rebuilt only when
a deploy moves it. The previous filter rescanned the whole cached Galaxy with
case-insensitive comparisons, synchronously on the render dispatcher, once per
keystroke per circuit.
The page's two background loops — the 2 s live-read poll and the Galaxy
deploy-notification subscription — handle faults inside the loop and retry; only
cancellation ends them. A poll fault surfaces in the "Live read failed" banner; a
notifier fault surfaces as a "redeploy notifications failed (retrying)" warning and
resubscribes after 5 s; the tree rebuild after a redeploy runs on the renderer
dispatcher. `DisposeAsync` cancels the loops and any in-flight search, drains them
concurrently with the same 5 s bound `DashboardPageBase` uses, and disposes its
cancellation sources in a `finally`.
A closed or backgrounded tab leaves its circuit alive, disconnected, for
`DisconnectedCircuitRetentionPeriod`, which the dashboard sets to **30 seconds**
(`DashboardServiceCollectionExtensions.DisconnectedCircuitRetention`) instead of
the framework's three minutes. Within that window the poll does not read: a scoped
`DashboardCircuitConnectionTracker`, registered as the circuit's `CircuitHandler`,
reports the browser connection down and the poll skips its ticks until it comes
back. Every tick it does run is a `ReadBulk` against the shared dashboard worker,
so a closed tab previously kept the worker busy for three minutes.
### Alarms page
`/dashboard/alarms` lists the alarms the gateway's central alarm monitor
@@ -552,7 +610,10 @@ client, no loopback socket, and no hub token — and the same subscription carri
`snapshot_status` frame behind the truncated-snapshot banner ("Alarm snapshot may be
incomplete"), so a capped provider fetch is caveated the moment the monitor decides
it. The alarm rows themselves still come from the three-second poll, which also
re-asserts the truncation verdict as the reconcile baseline. If `MxGateway:Alarms:Enabled` is
re-asserts the truncation verdict as the reconcile baseline. The filtered rows and
the area list are memoized in `DashboardAlarmView` on the poll's alarm generation and
the filter inputs, so the "Showing" card and the table share one filter-and-sort per
change rather than recomputing it several times per render. If `MxGateway:Alarms:Enabled` is
false the central monitor never starts, and the page says so instead of showing
an empty list with no explanation.
@@ -599,6 +660,19 @@ overflow against the oversized set and evicts the whole excess in one pass (a
subscribes nothing new evicts nothing, but neither can it grow the set. A browse
page requests far fewer tags than the cap, so in practice the set settles at 256.
The 256 is shared between viewers rather than one global LRU. Each Browse page
reads under its own viewer id, every advised tag records the viewer that read it
last, and each active viewer (one that read within the last 30 s) owns a share of
`256 / active viewers`, floored at 32. When a read needs room, eviction first takes
the least-recently-read tags of *other* viewers that are over their share — an
inactive viewer's share is zero, so a closed page's tags go first — and then the
reader's own tags that this read did not touch. A viewer within its share is never
evicted by someone else's read, so with several viewers the cap is soft: the set
can sit above 256 rather than strip a neighbour. Under the global LRU, two circuits
browsing different areas evicted each other's whole panels on every poll and
re-advised them two seconds later. With a single viewer the behaviour is exactly
the global LRU described above.
The Alarms page does **not** use the dashboard session: alarm data comes from
the gateway's always-on central monitor. `QueryAlarmsAsync` reads
`IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the
@@ -665,20 +739,24 @@ Secrets and pepper values are never logged.
### Settings page
Show read-only effective configuration:
Shows the read-only effective configuration:
- worker executable path,
- configured timeouts,
- queue capacities,
- auth mode,
- SQLite auth database path with sensitive parts redacted if needed,
- dashboard enabled state,
- authentication: mode, SQLite auth database path, pepper secret name, and whether
migrations run on startup;
- LDAP: enabled, server and port, fallback servers, transport, search base,
service account DN, service password (redacted), and the username and group
attributes;
- the LDAP group mappings that decide what a signed-in user may see —
`Dashboard:GroupToRole` (group → `Administrator` / `Viewer`) and
`Dashboard:GroupToTag` (group → the session tags it grants),
- `Dashboard:UntaggedSessionVisibility`, which decides who sees a session whose
owning API key carries no dashboard tags,
- protocol version.
`Dashboard:GroupToTag` (group → the session tags it grants);
- worker: executable path, required architecture, and the startup, shutdown, and
heartbeat-grace timeouts;
- sessions: default command timeout and max sessions;
- events: queue capacity and backpressure policy;
- dashboard: enabled, anonymous localhost, snapshot interval, recent fault and
session limits, `ShowTagValues`, and `Dashboard:UntaggedSessionVisibility`
(who sees a session whose owning API key carries no dashboard tags);
- the worker protocol version.
Do not show API key secrets or pepper values.
+11 -3
View File
@@ -177,9 +177,17 @@ return a successful close result with the final known state.
`WorkerClient.ShutdownAsync` sends `WorkerShutdown`, waits for the worker read,
write, and heartbeat loops to stop, and waits for the launched worker process to
exit within the same shutdown timeout. If the pipe loops or process exit exceed
the timeout, the close operation fails with `ShutdownTimeout`; `GatewaySession`
then kills the worker process tree before surfacing the close failure.
exit within the same shutdown timeout. The timeout starts before the
`WorkerShutdown` envelope is queued, not after: the outbound channel is bounded,
and a worker that has stopped reading the pipe parks the write loop in a pipe
write while the channel behind it fills with commands and their cancels. Because
the heartbeat watchdog only watches a `Ready` client, nothing else would end that
stall once the client is `Closing`, so an enqueue bound to the caller's token
alone waited for that token — for the lease sweep, until host stop — with the
session's close lock held. If queuing the envelope, the pipe loops, or process
exit exceed the timeout, the close operation fails with `ShutdownTimeout` (the
fault cancels the loops and kills the worker); `GatewaySession` then runs its
kill fallback before surfacing the close failure.
### Invoke
+23 -45
View File
@@ -82,6 +82,8 @@ return mapper.MapCommandReply(workerReply);
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies. To keep the two ids joinable, `Invoke` logs the pair (command kind, session id, client correlation id, worker envelope correlation id — no values) once per call at `Debug`, so a client error report quoting its correlation id can be matched to gateway and worker log lines.
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
Selected control commands have additional public behavior beyond their protobuf shape:
@@ -94,6 +96,8 @@ Selected control commands have additional public behavior beyond their protobuf
`StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
Each response write receives the call's cancellation token, so a write parked on a slow or vanished client is released when the call is cancelled rather than waiting on the transport. A write that then fails with an `IOException` or `OperationCanceledException` while the call token is cancelled is a routine client disconnect: it maps to `Cancelled` and logs nothing. The same `IOException` on a live call keeps the generic `Unavailable` + warning mapping, because that is a real transport failure. `StreamAlarms` applies the same rule.
### `AcknowledgeAlarm`
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
@@ -103,6 +107,7 @@ Request fields that identify the acknowledging operator:
- `operator_user` (required): the Galaxy account name. An empty value is rejected with `InvalidArgument` so the real operator is always recorded.
- `operator_password` (field 6): the operator's Galaxy password. May be empty because Galaxy accounts can have empty passwords. It is a credential and is never logged.
- `operator_domain` (field 7): optional. When set, the worker authenticates `domain\user`; when empty it authenticates `user`.
- `operator_full_name`: not a field on `AcknowledgeAlarmRequest`. The worker commands (`AcknowledgeAlarmCommand` field 6, `AcknowledgeAlarmByNameCommand` field 8) carry one, but it is accepted and **not propagated** to MXAccess: the 8-argument `AlarmAckByName` that carried it returns -55 (see [Alarm Client Discovery](./AlarmClientDiscovery.md)). `operator_domain` is used.
The worker calls `AuthenticateUser` as the operator, then writes `operator_comment` to `<TagName>.AckMsg` (subtag mode: the configured ack-comment subtag) on a dedicated MXAccess server registered as `MxGateway:Alarms:AckClientName`, and waits a bounded time for `OnWriteComplete`. The AVEVA runtime treats that write as the acknowledgement, and the Historian (`Runtime.dbo.v_AlarmEventHistory2`) records the operator name, domain and full name from the authenticated account, with the ack client name as `OperatorNode`. The alarm-client ack calls (`AlarmAckByName`, `AlarmAckByGUID`) are not used because a headless consumer's acks are accepted and dropped; see [Alarm Probe Findings](./AlarmProbeFindings.md).
@@ -125,6 +130,8 @@ The worker calls `AuthenticateUser` as the operator, then writes `operator_comme
`StreamAlarms` is a server-streaming, **session-less** RPC that attaches to the gateway's central alarm feed. The handler delegates to `IGatewayAlarmService.StreamAsync`. The stream opens with a `provider_status` and a `snapshot_status` `AlarmFeedMessage` (the current provider mode and snapshot-completeness verdict), then one `AlarmFeedMessage` carrying an `active_alarm` per currently-active alarm (the ConditionRefresh snapshot), then a single `snapshot_complete`, then a `transition` for every subsequent raise / acknowledge / clear — interleaved with a further `provider_status` on each failover/failback and a further `snapshot_status` on each change of the truncation verdict. It is served by the always-on `GatewayAlarmMonitor`, which owns a single gateway-managed worker session and fans out to every attached client — clients no longer open a session of their own. `alarm_filter_prefix`, when set, scopes the stream to a sub-tree.
A subscriber's opening snapshot is taken from the monitor's memoized alarm-set projection (the same one `QueryActiveAlarms` and the dashboard read), captured under the broadcast lock in the same critical section that registers the subscriber, so no transition can fall between the snapshot and the live stream; the prefix filter runs afterwards, outside the lock. Each subscriber has a bounded queue (2048 messages). A subscriber that falls that far behind is dropped and its stream ends with `ResourceExhausted` ("Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."), the same status a `StreamEvents` overflow carries; the remedy is to reconnect, which re-sends the snapshot. The drop is not logged as a gateway failure.
### `QueryActiveAlarms`
`QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `events:read` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
@@ -294,6 +301,8 @@ Every handler wraps its body in `try { ... } catch (Exception exception) when (e
- `SessionManagerException` is mapped by its `ErrorCode`.
- `WorkerClientException` is mapped by its `ErrorCode`.
Two stream-specific cases are handled before `MapException`: on `StreamEvents` and `StreamAlarms` a write failure after the client cancelled becomes `Cancelled` without a log entry, and on `StreamAlarms` an `AlarmFeedSubscriberOverflowException` (slow-subscriber drop) becomes `ResourceExhausted` without a log entry.
Anything else is logged at warning and surfaced as `Unavailable` with a generic message — clients see a retryable status, and the unexpected exception is captured in gateway logs rather than leaked over the wire.
```csharp
@@ -302,74 +311,43 @@ StatusCode statusCode = exception.ErrorCode switch
SessionManagerErrorCode.SessionNotFound => StatusCode.NotFound,
SessionManagerErrorCode.SessionNotReady => StatusCode.FailedPrecondition,
SessionManagerErrorCode.EventSubscriberAlreadyActive => StatusCode.ResourceExhausted,
SessionManagerErrorCode.EventSubscriberLimitReached => StatusCode.ResourceExhausted,
SessionManagerErrorCode.EventQueueOverflow => StatusCode.ResourceExhausted,
SessionManagerErrorCode.SessionLimitExceeded => StatusCode.ResourceExhausted,
SessionManagerErrorCode.OpenFailed => StatusCode.Unavailable,
SessionManagerErrorCode.CloseFailed => StatusCode.Unavailable,
SessionManagerErrorCode.PermissionDenied => StatusCode.PermissionDenied,
SessionManagerErrorCode.GatewayStopping => StatusCode.Unavailable,
_ => StatusCode.Unavailable,
};
```
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same.
While the gateway is stopping, clients see three different outcomes. An in-flight `Invoke` gets `Cancelled` (`GatewayShutdown`). An active `StreamEvents` ends cleanly with `OK`, because the drain closes its session and completes the subscriber channel. A new `OpenSession` gets `Unavailable` (`GatewayStopping`), including an open whose session the drain closed while its worker was still starting. An `OpenSession` the caller cancels while its worker is starting gets `Cancelled`, not `Unavailable`, and is not counted as an open fault.
## Event Streaming Model
`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the channel, backpressure policy, and metric bookkeeping can be unit-tested without spinning up a Kestrel host.
`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the attach, replay, filtering and metric bookkeeping can be unit-tested without spinning up a Kestrel host. It does not read the worker itself: each session owns one `SessionEventDistributor` whose single pump drains the worker event channel, maps each `WorkerEvent` to the public `MxEvent`, appends it to the session's replay ring, and fans it to every registered subscriber. A `StreamEvents` call is one of those subscribers. [Sessions](./Sessions.md) describes the distributor, the replay ring and detach-grace in full; this section covers the per-RPC boundary.
The pipeline has three stages:
The pipeline has four stages:
1. The session is resolved via `sessionManager.TryGetSession`. A miss raises `SessionManagerException(SessionNotFound)` so the handler reports `StatusCode.NotFound`.
2. `session.AttachEventSubscriber(...)` enforces the single-subscriber-per-session rule (or allows multiple subscribers if `Sessions:AllowMultipleEventSubscribers` is enabled). The returned `IDisposable` is released in the `finally` block, ensuring the subscriber slot is freed even when the client cancels mid-stream.
3. A `Channel<MxEvent>` decouples the worker-side producer from the gRPC writer. The channel is bounded by `Events:QueueCapacity` and configured for a single reader and writer:
2. The attach is owner-scoped: the caller's API key id must match the key that opened the session, or the call fails with `PermissionDenied` before anything is attached or delivered.
3. The call registers a subscriber on the distributor. A fresh stream (`after_worker_sequence == 0`) uses `session.AttachEventSubscriber(...)`. A resume uses `session.AttachEventSubscriberWithReplay(...)`, which snapshots the replay ring for events newer than the cursor and registers the live subscriber under one lock, so the replay-to-live handoff has no gap and no duplicate. In single-subscriber mode (the default) a second attach fails with `EventSubscriberAlreadyActive`; with `Sessions:AllowMultipleEventSubscribers` enabled, an attach beyond `MaxEventSubscribersPerSession` fails with `EventSubscriberLimitReached`. Both map to `ResourceExhausted`. The lease is released in the `finally` block, so the subscriber slot is freed even when the client cancels mid-stream.
4. The call yields events from the lease's bounded channel (capacity `Events:QueueCapacity`). On a resume it first yields a `ReplayGap` sentinel when the cursor predates the oldest retained event, then the replayed batch, then live events filtered to sequences above the last replayed one. Per-session ordering is preserved end-to-end, because the pump is the single writer to each subscriber channel and writes in `WorkerSequence` order.
```csharp
Channel<MxEvent> eventQueue = Channel.CreateBounded<MxEvent>(
new BoundedChannelOptions(options.Value.Events.QueueCapacity)
{
SingleReader = true,
SingleWriter = true,
FullMode = BoundedChannelFullMode.Wait,
AllowSynchronousContinuations = false,
});
```
The producer reads `WorkerEvent`s from the session, maps them, and writes to the channel. Per-session ordering is preserved end-to-end: events arrive from the worker in `WorkerSequence` order, the producer is the single writer, and the consumer drains FIFO. The producer also honours the request's `AfterWorkerSequence` cursor by skipping any event whose `WorkerSequence` is at or before the requested cutoff, which lets clients resume after a disconnect without server-side replay state.
The replay ring is server-side state that outlives a single RPC: with the default `DetachGraceSeconds` and `ReplayRetentionSeconds`, a client that drops can reconnect with its last `worker_sequence` and receive what it missed, as long as the events are still retained.
### Backpressure
When `TryWrite` fails the queue is full. The handling depends on `Events:BackpressurePolicy`:
```csharp
if (!writer.TryWrite(publicEvent))
{
string message = $"Session {session.SessionId} event stream queue overflowed.";
metrics.QueueOverflow("grpc-event-stream");
if (options.Value.Events.BackpressurePolicy == EventBackpressurePolicy.FailFast)
{
session.MarkFaulted(message);
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
}
else
{
logger.LogDebug(
"Disconnecting event stream for session {SessionId} after queue overflow.",
session.SessionId);
}
writer.TryComplete(new SessionManagerException(
SessionManagerErrorCode.EventQueueOverflow,
message));
return;
}
```
Under `FailFast` the session is faulted so subsequent commands return `FailedPrecondition`; the client must reopen. Under the default policy only the stream is dropped and the session continues to accept commands, leaving recovery to the client (typically a fresh `StreamEvents` call with an updated `AfterWorkerSequence`). Either way, the consumer side observes `StatusCode.ResourceExhausted` via the `EventQueueOverflow` mapping above.
The pump never blocks on a slow subscriber: it writes to each subscriber channel without waiting. When one subscriber's channel is full, the distributor's overflow handler records `QueueOverflow("grpc-event-stream")` and completes that subscriber's channel with `SessionManagerException(EventQueueOverflow)`, which the client observes as `ResourceExhausted`. Under `Events:BackpressurePolicy = FailFast` (the default) in single-subscriber mode the session is also faulted (and `Fault(EventQueueOverflow)` recorded), so subsequent commands return `FailedPrecondition` and the client must reopen. In multi-subscriber mode, or under `DisconnectSubscriber`, only that stream is dropped and the session keeps serving commands and other subscribers; the client recovers with a fresh `StreamEvents` that resumes from its last `worker_sequence`. The internal dashboard mirror is registered as an internal subscriber, so a slow dashboard only disconnects itself and never faults the session.
### Cancellation and cleanup
The handler creates a linked cancellation token (`streamCts`) so that completing the consumer (client disconnect, error, or graceful end-of-stream) also cancels the producer. The `finally` block cancels the source, disposes the subscriber slot, awaits the producer (swallowing the expected cancellation), and emits `StreamDisconnected("Detached")` so dashboards see the disconnection regardless of cause.
The `finally` block removes the subscriber's backlog-gauge contribution, disposes the lease, and emits `StreamDisconnected(reason)` so dashboards see every disconnection. The reason is the bounded constant (`EventStreamDisconnectReasons`) for how the stream ended — `SessionClosed`, `Overflow`, `WorkerFault`, `ClientCancelled`, `Error`, or `Detached` — rather than one constant for every cause, so an operator can tell slow consumers from worker faults from routine client disconnects.
`WorkerClientException` thrown by the producer marks the session as faulted before completing the channel — the worker is presumed gone, and any subsequent command on that session must observe the fault rather than silently retry.
When the worker event stream ends with a `WorkerClientException`, the pump completes every subscriber channel with it. Each subscriber marks the session faulted, so any subsequent command observes the fault rather than silently retrying, and records the `WorkerFault` disconnect reason. The fault itself is counted once in `mxgateway.faults`, by `WorkerClient` when it faults, however many subscribers observe it.
## Authorization Interceptor Integration
+5 -5
View File
@@ -45,7 +45,7 @@ All counters are `Counter<long>`. Tag values come from the call sites listed und
| `mxgateway.workers.killed` | `reason` | Forced terminations of worker processes. |
| `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
| `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
| `mxgateway.grpc.streams.disconnected` | `reason` | Detachments of the dashboard or client gRPC event stream. |
| `mxgateway.grpc.streams.disconnected` | `reason` | Ends of a client `StreamEvents` stream. `reason` is one of a fixed set (`EventStreamDisconnectReasons`): `SessionClosed` (the session event stream completed), `Overflow` (subscriber fell behind its bounded queue), `WorkerFault` (the worker event stream faulted), `ClientCancelled` (the client cancelled or disconnected), `Error` (unclassified), `Detached` (the consumer stopped reading without cancelling). |
| `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |
### Histograms
@@ -124,7 +124,7 @@ The recording call sites describe the code paths that write into each instrument
```csharp
_metrics.SessionOpened();
...
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString()); // not for a cancelled open or GatewayStopping
...
_metrics.SessionClosed();
...
@@ -181,14 +181,14 @@ IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
...
backlogRegistration.Dispose();
metrics.StreamDisconnected("Detached");
metrics.StreamDisconnected(disconnectReason); // an EventStreamDisconnectReasons constant
...
metrics.QueueOverflow("grpc-event-stream");
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
...
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
```
A worker fault seen by a subscriber records only the `WorkerFault` disconnect reason, not a fault. `WorkerClient` records the fault once when it faults, and every subscriber of the session observes that same fault, so counting it per subscriber made one worker fault read as 1+K faults for K subscribers.
Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.
`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):
+85 -10
View File
@@ -133,6 +133,9 @@ starts the STA, or touches MXAccess:
| `2` | `InvalidArguments` | Required arguments are missing or unknown arguments are present. |
| `3` | `InvalidProtocolVersion` | `--protocol-version` is not numeric or does not match the supported worker protocol. |
| `4` | `MissingNonce` | `MXGATEWAY_WORKER_NONCE` is absent or empty. |
| `5` | `PipeConnectionFailed` | An `IOException` or `TimeoutException` escapes the pipe client. |
| `6` | `ProtocolViolation` | A `WorkerFrameProtocolException` escapes the pipe client. |
| `7` | `UnhandledException` | An exception reaches the `AppDomain.UnhandledException` hook (see `docs/WorkerBootstrap.md`). |
Bootstrap logs use `WorkerConsoleLogger` key/value output. `WorkerLogRedactor`
redacts fields whose names indicate nonce, secret, password, token,
@@ -270,12 +273,33 @@ is still responsive. Shutdown marks the runtime as closing, wakes the pump,
rejects new commands, cancels queued work, uninitializes COM on the STA, and
waits for the thread to exit.
`ProcessQueuedCommands` pumps pending Windows messages between queued work
items whenever more work remains, not only after the whole queue drains. A
burst of queued commands would otherwise hold every inbound COM callback
(`OnDataChange`, `OnWriteComplete`) behind the entire backlog.
If the STA loop itself throws after `Start` has returned (a failed
`MsgWaitForMultipleObjectsEx`, or an exception escaping a dispatched message),
`StaRuntime` records the exception as `LoopFault`, fails every later
`InvokeAsync` with an `InvalidOperationException` wrapping it (never the
graceful `StaRuntimeShutdownException`), cancels queued work, and invokes the
handler registered through `RegisterLoopFaultHandler` — unless a requested
shutdown is already in progress, in which case the death is part of a
deliberate teardown and is not reported. `MxAccessStaSession` registers a
handler that records a `StaHung` `WorkerFault` on the event queue (diagnostic
text carries the exception type and HRESULT only, never the message),
so the IPC drain loop sends the fault to the gateway and ends the pipe session
and the worker process. Before this, a post-start loop death was stored in the
startup-exception field that only `Start` reads, leaving a live worker whose
commands queued forever. Startup failures still surface from `Start` itself.
### Inner Completion Waits
Two commands hold the STA while waiting for a COM event they just provoked: the
unary write path waits for its `OnWriteComplete`
(`MxAccessWriteCompletionCache.TryWaitForCompletion`), and `ReadBulk` waits per
tag for the first `OnDataChange` (`MxAccessValueCache.TryWaitForUpdate`). Both
(`MxAccessWriteCompletionCache.TryWaitForCompletion`), and `ReadBulk` waits for
the first `OnDataChange` of every snapshot tag
(`MxAccessValueCache.TryWaitForUpdates`). Both
run the same loop shape as the outer pump, for the same reason — the event they
are waiting for *is* a Windows message, so the thread must keep dispatching to
receive it:
@@ -297,7 +321,7 @@ The idle slice is a Win32 wait (`StaWaitHelper.WaitForSignalOrMessages`), never
`Thread.Sleep`. A sleeping STA pumps no messages, so a sleep-polled loop could
only dispatch the awaited COM event at poll-tick granularity while stalling
*every other* event for the same tick — up to 1.5 s for a write completion and
up to `timeout_ms` per tag for `ReadBulk`. The Win32 wait returns the instant a
up to `timeout_ms` for `ReadBulk`. The Win32 wait returns the instant a
message needs pumping, so the apartment dispatches continuously for the whole
wait. Each cache also sets an `AutoResetEvent` from its update path (outside the
cache lock) so a cross-thread producer wakes the waiter immediately; in the live
@@ -317,8 +341,29 @@ nothing wakes the wait — a process with no STA message queue (unit tests drive
these caches from ordinary threads, standing in for the STA by updating the
cache from a fake `pumpStep`) must not block for a full poll interval. Timeouts,
deadline math, and return values are unchanged by the wait mechanism: an expired
write wait still yields the empty-`statuses` unconfirmed reply, and an expired
per-tag `ReadBulk` wait still reports its own timeout.
write wait still yields the empty-`statuses` unconfirmed reply, and every
`ReadBulk` tag that did not deliver before the deadline still reports its own
timeout.
`ReadBulk` runs its snapshot lifecycle in phases so the STA is held for at most
one `timeout_ms` per command, not one per tag: (1) every tag is validated and
probed against the cached-and-advised fast path; (2) every remaining tag is
added and advised; (3) one wait with a shared deadline collects the first
`OnDataChange` per snapshot handle and returns as soon as all have delivered;
(4) every snapshot handle is unadvised and removed, best-effort. Per-tag
results are unchanged: request order, the blank-address failure, the cached
result, an `AddItem` failure (item handle 0) or `Advise` failure (its own item
handle) failing only that tag, and the `ReadBulk timed out after <n> ms waiting
for first OnDataChange.` message for each tag that never delivered. The cache
probes all run before any snapshot `AddItem`, so a later tag in the same
request can never be served from an earlier tag's snapshot subscription.
When MXAccess returns the same item handle for two tags in one batch (a
duplicate address), the handle is advised, unadvised, and removed once, and
every duplicate receives the shared outcome. Phases 2 and 4 make one or two
COM calls per tag without waiting, so they call the pump step every
`ReadBulkPumpInterval` (16) tags; a large batch therefore keeps dispatching
inbound callbacks and refreshing `LastStaActivityUtc` instead of drifting
toward `HeartbeatStuckCeiling`.
The write wait's budget is `MxGateway:Worker:WriteCompletionWaitMilliseconds`
(default 1500). It applies to `Write`, `Write2`, `WriteSecured`, and
@@ -671,7 +716,20 @@ The worker records plain and supervisory advice separately only after the COM
call returns normally. Successful `UnAdvise` removes all tracked advice for the
server and item pair because the public MXAccess cleanup method has no plain
versus supervisory selector. Successful `RemoveItem` and `Unregister` also clear
related advice state from the worker registry. Failed advice and cleanup calls
related advice state from the worker registry, and evict the per-handle
caches the event sink fills: `RemoveItem` drops the handle pair's
`MxAccessValueCache` and `MxAccessWriteCompletionCache` entries, and
`Unregister` drops every entry recorded under the server handle (MXAccess
removes the server's items with it, so the per-item eviction never runs for
them). Without this, an `AddItem` → `Write` → `RemoveItem` churn grew the
completion cache by one entry per cycle for the life of the worker.
`AddItem`, `AddItem2`, and `AddBufferedItem` also drop both cache entries for
the handle they receive when the registry does not already track it. An
`OnDataChange` or `OnWriteComplete` that was queued when `RemoveItem` or
`Unregister` ran is dispatched after the eviction and recreates the entry;
when MXAccess later reuses that item id, `ReadBulk`'s cached path would
otherwise serve the stale value as `was_cached = true`. A handle the
registry still tracks belongs to a live item, so its entries are kept. Failed advice and cleanup calls
leave registry state unchanged so diagnostics continue to reflect the last
successful MXAccess-owned state transition.
@@ -905,6 +963,23 @@ the STA more than 75 s without pumping *should* fault — that is the ceiling's
contract). The flag is named generically so any future non-dispatcher STA work
reuses it.
A `PollOnce` that throws does not fault the session on the first failure. The
poll loop counts consecutive failures and records the alarm-poll fault
(`MxaccessEventConversionFailed`) only when the count reaches
`MxAccessStaSession.DefaultAlarmPollFailureThreshold` (3, the same default as
`AlarmFailoverConfig.consecutive_failure_threshold`); a successful poll resets
the count. Each failure below the threshold writes one
`level=Warning event=AlarmPollFailed` line to stderr carrying the exception
type, HRESULT, and counts only — the exception message is left out because it
can echo alarm or tag names. A single transient COM failure in
`GetXmlCurrentAlarms2` used to fault the whole session, taking every data
subscription with it; a persistent failure, including an STA-affinity
regression that fails every tick, still faults within three poll intervals.
A poll failure seen after the STA loop itself has died (`StaRuntime.LoopFault`
is set) ends the poll loop without recording anything: the loop-fault handler
has already recorded the `StaHung` fault, and the failure is only the
fast-fail `InvokeAsync` returns afterwards.
The in-flight-command suppression itself is bounded by
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
`HeartbeatGrace`). The motivating case for the suppression is a legitimately
@@ -921,7 +996,7 @@ has been stale for longer than `HeartbeatStuckCeiling`, the watchdog fires
healthy long-running commands: `StaRuntime.PumpPendingMessages()` refreshes
`LastStaActivityUtc` (via `MarkActivity()`) every time it runs, and long-hold
STA commands invoke it on every wait iteration (`ReadBulk` routes its
per-tag wait through the `pumpStep` wired from `StaRuntime.PumpPendingMessages`).
shared snapshot wait through the `pumpStep` wired from `StaRuntime.PumpPendingMessages`).
A command that keeps pumping therefore keeps its activity timestamp fresh and
never reaches the ceiling, while a genuinely stuck STA — one that has stopped
pumping — accrues staleness and faults correctly. The ceiling is thus the
@@ -1019,10 +1094,10 @@ and relies on the gateway to kill the process.
## Fault Handling
Worker fault categories:
Worker fault categories (`InvalidArguments` and `GatewayAuthenticationFailed`
are reserved and never emitted: bootstrap failures exit via `WorkerExitCode`
before the pipe exists):
- `InvalidArguments`
- `GatewayAuthenticationFailed`
- `ProtocolMismatch`
- `ProtocolViolation`
- `PipeDisconnected`
+97 -47
View File
@@ -80,7 +80,7 @@ private void EnsureSessionCapacity()
}
```
`SessionManager` also defines five close-reason constants — `DefaultCloseReason` (`"client-close"`), `GatewayShutdownReason` (`"gateway-shutdown"`), `LeaseExpiredReason` (`"lease-expired"`), `FaultedReason` (`"faulted-reaped"`), and `DetachGraceExpiredReason` (`"detach-grace-expired"`) — so that the metrics and worker shutdown paths agree on a fixed vocabulary.
`SessionManager` also defines six close-reason constants — `DefaultCloseReason` (`"client-close"`), `GatewayShutdownReason` (`"gateway-shutdown"`), `LeaseExpiredReason` (`"lease-expired"`), `FaultedReason` (`"faulted-reaped"`), `DetachGraceExpiredReason` (`"detach-grace-expired"`), and `UnattachedGraceExpiredReason` (`"unattached-grace-expired"`) — so that the metrics and worker shutdown paths agree on a fixed vocabulary.
### SessionRegistry (ISessionRegistry)
@@ -137,18 +137,19 @@ public sealed record SessionOpenRequest(
### SessionCloseResult
`SessionCloseResult` is the record returned from a successful close. `AlreadyClosed` distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting.
`SessionCloseResult` is the record returned from a successful close. `AlreadyClosed` distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting. `WorkerShutdownFailure` is set when the worker refused a graceful shutdown and was killed instead — still a successful close (see [Close](#close)); the manager logs it and the gRPC reply does not expose it.
```csharp
public sealed record SessionCloseResult(
string SessionId,
SessionState FinalState,
bool AlreadyClosed);
bool AlreadyClosed,
Exception? WorkerShutdownFailure = null);
```
### SessionCloseStartedException
`SessionCloseStartedException` is `internal` and is only thrown from inside `GatewaySession.CloseAsync` when the close path has already begun mutating worker state and a subsequent step fails. `SessionManager.CloseSessionCoreAsync` catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as `SessionManagerException` with `CloseFailed`. The intermediate type exists so the public API surface only ever exposes `SessionManagerException`.
`SessionCloseStartedException` is `internal` and is only thrown from inside `GatewaySession.CloseAsync` when the close path has already begun mutating worker state and a subsequent step fails — in practice, when the worker's graceful shutdown failed *and* the kill fall-back failed too, so a worker may still be running. `SessionManager.CloseSessionCoreAsync` catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as `SessionManagerException` with `CloseFailed`. The intermediate type exists so the public API surface only ever exposes `SessionManagerException`.
### SessionManagerException and SessionManagerErrorCode
@@ -162,26 +163,26 @@ public sealed record SessionCloseResult(
| `EventSubscriberLimitReached` | In multi-subscriber mode, an attach exceeded `MaxEventSubscribersPerSession` concurrent external subscribers. |
| `EventQueueOverflow` | Reserved for the worker event channel overflow path. |
| `SessionLimitExceeded` | `MaxSessions` is in use. |
| `OpenFailed` | `OpenSessionAsync` failed; the inner exception carries the cause. |
| `CloseFailed` | A close started but did not complete cleanly; the session is removed and faulted. |
| `OpenFailed` | `OpenSessionAsync` failed; the inner exception carries the cause. A caller-cancelled open and an open refused because the gateway is stopping do not use this code. |
| `CloseFailed` | A close started but did not complete cleanly — the worker's graceful shutdown and the kill fall-back both failed (or a kill threw); the session is removed and faulted. A shutdown failure whose kill succeeded is a successful close, not this. |
| `PermissionDenied` | The caller may not perform the operation on this session (for example, attaching `StreamEvents` to a session opened by a different API key). |
| `GatewayStopping` | The gateway is stopping and refuses new sessions (see [Shutdown Coordination](#shutdown-coordination)). Maps to gRPC `Unavailable`; not recorded as a fault. |
## Lifecycle
### Open
`SessionManager.OpenSessionAsync` allocates a session slot, builds the `GatewaySession`, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step:
`SessionManager.OpenSessionAsync` allocates a session slot, builds the `GatewaySession`, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step. Two outcomes are not open failures and skip the fault metric and the warning: `GatewayStopping` (see [Stopping gate](#stopping-gate)) and the caller's own cancellation, which is rethrown as the `OperationCanceledException` so the gRPC layer maps it to `Cancelled` rather than `Unavailable`:
```csharp
catch (OperationCanceledException exception) when (cancellationToken.IsCancellationRequested)
{
await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
throw;
}
catch (Exception exception)
{
session?.MarkFaulted(exception.Message);
if (session is not null)
{
_registry.TryRemove(session.SessionId, out _);
await session.DisposeAsync().ConfigureAwait(false);
}
ReleaseSessionSlot();
await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
_logger.LogWarning(
exception,
@@ -195,7 +196,19 @@ catch (Exception exception)
}
```
The order — fault, deregister, dispose, release slot, record metric, log, rethrow — matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine.
`RollBackOpenAsync` faults the session and, when it had been registered, routes it through `RemoveSessionAsync` — the single removal path every teardown uses, which deregisters, drops the session's per-session event metrics, releases the slot and disposes, in that order. The order matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine. A session that was never registered (id collision) releases the slot and disposes directly. The open-session gauge is not touched by the rollback: `SessionOpened` is recorded inside the attach gate described next, after which nothing in the open sequence can fail.
#### Close or kill during worker startup
The session is registered — and therefore visible to the dashboard's Kill button, the lease sweep and the host drain — for the whole worker startup, which can last up to `MxGateway:Worker:StartupTimeoutSeconds`. A kill or drain that lands in that window finds no worker client to stop, flips the session to `Closed`, removes it and disposes it. The worker client that `SessionWorkerClientFactory.CreateAsync` then returns must not be stored on that dead session and handed back to the caller as a successful open: nothing would ever shut the worker down (the close already ran) and the client would hold a `Closed` session.
`GatewaySession.AttachWorkerClient` therefore returns `false`, under `_syncRoot`, when the session is already `Closing`, `Closed` or `Faulted` or has been disposed. The manager runs the end of the open sequence — attach, `MarkReady`, `SessionOpened` — through `TryAttachWorkerClientAndMarkReadyAsync`, which holds the per-session close gate (`_closeLock`) so a concurrent `CloseAsync` / `KillWorkerWithCloseGateAsync` runs either entirely before it (the attach is refused) or entirely after it (an ordinary close of a Ready session). `TryAttachWorkerClientAndMarkReadyAsync` returns a `SessionAttachOutcome` rather than a bool, because the open reports each refusal differently. On a refused attach the manager kills and disposes the late worker client; the registry is already empty and the slot was released by whichever teardown closed the session. A refusal caused by the host drain fails the open with `GatewayStopping`, the same code as an open refused at the door, so the client retries against the restarted gateway. A refusal caused by an admin kill fails it with `OpenFailed`.
The attach also refuses a worker client that is already `Faulted` or `Closed` (`SessionAttachOutcome.WorkerTerminal`). A worker can die right after its handshake, for example from an STA loop fault or exit code 7. Attaching it would hand the caller a `Ready` session over a dead worker, and nothing would notice until the lease expired. The client is discarded and the open fails with `OpenFailed`. A worker that dies just after this check is caught by the event pump instead (see [Worker exit outside a gateway close](#worker-exit-outside-a-gateway-close)).
Once `CreateAsync` has returned, a live worker exists that only the open path knows about, so the gate wait is **not cancellable** (it waits on `CancellationToken.None`; the gate is only ever held for the duration of one close or kill) and the manager discards the client on *any* exception out of the attach step — the caller's own cancellation (checked before the attach, so an abandoned open never yields a session), a disposed gate, anything thrown while becoming Ready. A cancellation surfaces as the `OperationCanceledException` itself; any other failure surfaces as `OpenFailed` with the cause as inner exception. An exception that escaped before the discard would be an orphaned x86 worker, which a restarted gateway terminates rather than adopts.
Because such a session was never counted as opened, it must not be counted as closed either. `GatewaySession.OpenRecorded` is set inside the same gate right after `SessionOpened` is recorded, and every teardown path (`KillWorkerAsync`, `CloseSessionCoreAsync`, the close-failed branch) records `SessionClosed` / `SessionRemoved` only when it is set — so `sessions.opened`, `sessions.closed` and the `sessions.open` gauge stay consistent across the startup window.
### Run
@@ -215,11 +228,13 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
The same one-distributor-per-session rule is enforced at the other end of the lifecycle. Session disposal disposes the distributor and latches the fact, so an attach that passed its readiness gate and then lost the CPU until the session was torn down throws `SessionNotReady` rather than constructing a replacement distributor. A replacement can never be pumped — the pump-start flag is a one-way latch — so its subscriber would block on an empty channel that is never completed until the client's own deadline expired, with no error, log or metric to explain it. Every attach entry point (`AttachEventSubscriber`, `AttachEventSubscriberWithReplay`, `AttachInternalEventSubscriber`) fails this way, and each rolls back the subscriber-count slot it took, so a caller that loses the race to teardown sees the same failure it would have seen a moment later from the readiness gate.
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration (`TouchClientActivity` is the only path that moves the lease). `IsLeaseExpired` cooperates with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
#### Teardown parallelism
A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses`, a compile-time constant of `4` in `SessionManager`. Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace, then unattached-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses`, a compile-time constant of `4` in `SessionManager`. Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
`TryBeginCloseIfExpired` is a *reservation*, not the close: it writes `Closing` but leaves the session's close-started flag alone, so the `CloseAsync` the sweep issues next is the first close to run and reports `AlreadyClosed: false`. That is what lets `CloseSessionCoreAsync` account `SessionClosed` for lease-expired, faulted-reaped and detach-grace-expired closes exactly once; a reservation that also set the flag made every sweep-driven close look like a second concurrent close, and swept sessions never left the `sessions.open` gauge. A client close that races the reservation still counts once: whichever `CloseAsync` takes the close lock first does the accounting, and the other finds the session `Closed`.
Splitting the phases moves the TOCTOU re-check earlier, and that is an accepted trade rather than an unchanged behavior: selection now flips **every** chosen session to `Closing` up front, before any teardown runs, whereas the sequential sweep re-checked session *N* only after sessions *1..N-1* had finished closing. A client that re-attaches a subscriber while the close phase is running therefore loses a race it could previously win — the eligibility snapshot is taken at one instant for the whole pass. Expiry evaluation itself is unaffected, because `now` is a parameter and is not re-read per session.
@@ -239,9 +254,21 @@ Mechanically: when the last external subscriber detaches and `DetachGraceSeconds
`DetachGraceSeconds` controls retention and expiry only; the reconnect/replay path that re-attaches a dropped client to a retained session is described in [Reconnect and replay](#reconnect-and-replay).
#### Worker exit outside a gateway close
`IWorkerClient` has no state-change callback, so a worker that reached `Closed` or `Faulted` *without the gateway closing it* — the client sent the `ShutdownWorker` control command and the worker exited cleanly, the worker crashed, its pipe broke, a heartbeat timed out — used to leave the session reporting `Ready` until its lease expired (a live finding: `WorkerClient` was `Closed` on the shutdown ack, the session still `Ready`, even with an event stream attached). The session now observes it from its own worker-event pump source (`MapWorkerEventsAsync`): the single distributor pump is the session's one always-on view of the worker (it runs from `MarkReady` via the dashboard mirror), and `WorkerClient` enters its terminal state *before* it completes the event channel, so when the enumeration ends the terminal state is visible. `ObserveWorkerTerminalState` then faults a `Ready` session whose worker is `Closed`/`Faulted`. The pump also calls it when its first readiness check (`GetReadyWorkerClientAsync`) fails. A worker that dies between the open's attach and the pump's first read never reaches the event enumeration, so its `finally` never runs. Without that second call the session would stay `Ready` over the dead worker until its lease expired.
The choice is **`Faulted`, not `Closed`**: `Closed` means the gateway closed the session, and a `Closed` session left in the registry would linger until its lease expired with nothing reaping it. A faulted session is permanently unusable (every command fails the readiness check, as the worker is gone), stays observable via `GetSessionStatus` with `FinalFault` naming the worker state, and is reaped by the faulted-session sweep below after `FaultedGraceSeconds` — which is where `SessionClosed` is accounted, exactly once, through the ordinary `CloseSessionCoreAsync` path (`WorkerClient.ShutdownAsync` on a `Closed` client returns immediately, so no kill fallback or warning). Only a `Ready` session follows its worker: during a gateway-initiated close the worker closing is the expected outcome, not a fault.
#### Faulted-session reaping
A session that faults (for example, a slow single-subscriber client that overflows its event queue under the `FailFast` backpressure policy) is left permanently unusable — every command fails `EvaluateReadyUnderLock` — but its worker keeps running and it still pins one of `MxGateway:Sessions:MaxSessions` slots. Rather than waiting up to `DefaultLeaseSeconds` for the lease to expire, `SessionManager.CloseExpiredLeasesAsync` also checks `IsFaultedReapable(now)` and reaps a faulted session through the same `TryBeginCloseIfExpired` → `CloseSessionCoreAsync` teardown, with the distinct `FaultedReason` (`"faulted-reaped"`). `MarkFaulted` stamps a fault timestamp from the session's `TimeProvider`; `MxGateway:Sessions:FaultedGraceSeconds` (default `0`) bounds how long the faulted session is retained before the next sweep reaps it — `0` reaps it on the next sweep, a positive value keeps it observable via `GetSessionStatus` for that window first. Sweep precedence when several conditions fire at once is lease-expiry, then faulted, then detach-grace.
A session that faults (for example, a slow single-subscriber client that overflows its event queue under the `FailFast` backpressure policy) is left permanently unusable — every command fails `EvaluateReadyUnderLock` — but its worker keeps running and it still pins one of `MxGateway:Sessions:MaxSessions` slots. Rather than waiting up to `DefaultLeaseSeconds` for the lease to expire, `SessionManager.CloseExpiredLeasesAsync` also checks `IsFaultedReapable(now)` and reaps a faulted session through the same `TryBeginCloseIfExpired` → `CloseSessionCoreAsync` teardown, with the distinct `FaultedReason` (`"faulted-reaped"`). `MarkFaulted` stamps a fault timestamp from the session's `TimeProvider`; `MxGateway:Sessions:FaultedGraceSeconds` (default `0`) bounds how long the faulted session is retained before the next sweep reaps it — `0` reaps it on the next sweep, a positive value keeps it observable via `GetSessionStatus` for that window first. Sweep precedence when several conditions fire at once is lease-expiry, then faulted, then detach-grace, then unattached-grace.
#### Never-subscribed session reaping
Detach-grace only helps a session that *had* a subscriber. A client that opens a session and then disappears before it ever calls `StreamEvents` — OtOpcUa's `GalaxyDriver` failing part-way through its start-up is the motivating case — leaves a `Ready` session with no subscriber, no detach timestamp and a lease 30 minutes out, pinning one of `MaxSessions` and a live x86 worker for all of `DefaultLeaseSeconds`. `MxGateway:Sessions:UnattachedSessionGraceSeconds` (default `0`, off) closes that gap: when positive, `CloseExpiredLeasesAsync` also checks `IsUnattachedGraceExpired(now)` and reaps, with the distinct `UnattachedGraceExpiredReason` (`"unattached-grace-expired"`), any `Ready` session that has never had a subscriber of either kind and has been idle for the window.
Four deliberate limits keep the rule from catching legitimate sessions. It applies only to `Ready` sessions, and its idle clock starts at `Ready` (`MarkReady` stamps `_readyAtUtc`), so a slow worker start neither reaps the session mid-startup nor eats the window before the client even has the session id. Idle is measured from the later of that Ready stamp and the session's *last client activity* (its most recent unary command), so a unary-only client (reads and writes, never streams) is not reaped while it keeps calling; the rule is an idle timeout for sessions that never streamed, not a deadline to stream. It stops applying the moment the first subscriber attaches, after which detach-grace and the normal lease govern the session. And an *internal* in-process subscriber counts as attached for this rule (`AttachInternalEventSubscriber` sets a separate `_everHadInternalEventSubscriber` latch): the gateway's own `GatewayAlarmMonitor` session only ever attaches that way, and it must not be reaped by its own gateway every cycle. The latch is separate from `_everHadEventSubscriber` because that one drives detach-grace, which is keyed on external subscribers only. The dashboard mirror, which every session registers at `MarkReady`, is deliberately *not* a subscriber for this rule. The alarm monitor is the only gateway-owned session exempted. The dashboard Browse page's live-data session (`DashboardLiveDataService`) is unary-only: it registers and reads but never attaches a subscriber, so once it has been idle for the window the rule reaps it. That is expected. The service checks its cached session on every call (`EnsureReadyAsync`) and opens and registers a new one when the old one is gone. Like the other sweep conditions it is re-verified inside `TryBeginCloseIfExpired`, so a subscriber that attaches between selection and the close wins the race. Operators should set the window comfortably above a well-behaved client's open-to-first-`StreamEvents` time; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later.
#### Reconnect and replay
@@ -267,9 +294,10 @@ The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream a
### Close
`GatewaySession.CloseAsync` is serialized by a per-session `SemaphoreSlim` (`_closeLock`) so only one close runs at a time, but every read/write of `_state` still passes through `_syncRoot` (via `TryBeginClose` and `MarkClosed`). The close path therefore obeys the same lock discipline as `TransitionTo` / `MarkFaulted`: it transitions to `Closing`, asks the worker client to shut down within `ShutdownTimeout`, and on success transitions to `Closed`. `DisposeAsync` waits on `_closeLock` once before disposing the semaphore so an in-flight close's `Release()` cannot race against the dispose. If `WorkerClient.ShutdownAsync` throws, the session falls back to `IWorkerClient.Kill` (forced close):
`GatewaySession.CloseAsync` is serialized by a per-session `SemaphoreSlim` (`_closeLock`) so only one close runs at a time, but every read/write of `_state` still passes through `_syncRoot` (via `TryBeginClose` and `MarkClosed`). The close path therefore obeys the same lock discipline as `TransitionTo` / `MarkFaulted`: it transitions to `Closing`, asks the worker client to shut down within `ShutdownTimeout`, and on success transitions to `Closed`. `DisposeAsync` waits on `_closeLock` once before disposing the semaphore so an in-flight close's `Release()` cannot race against the dispose. `WorkerClient.ShutdownAsync` is bounded by `ShutdownTimeout` end to end — queuing the `WorkerShutdown` envelope included, since a worker that stopped reading the pipe can leave the bounded outbound channel full with no watchdog left to clear it once the client is `Closing` — so a close never holds `_closeLock` longer than the timeout before reaching the fallback. If `WorkerClient.ShutdownAsync` throws, the session falls back to `IWorkerClient.Kill` (forced close):
```csharp
Exception? workerShutdownFailure = null;
if (_workerClient is not null)
{
try
@@ -289,54 +317,76 @@ if (_workerClient is not null)
new AggregateException(exception, killException));
}
throw;
workerShutdownFailure = exception;
}
}
MarkClosed();
return new SessionCloseResult(SessionId, SessionState.Closed, alreadyClosing, workerShutdownFailure);
```
If both graceful shutdown and the kill fall-back fail, the original and kill exceptions are bundled into an `AggregateException` and surfaced as `SessionCloseStartedException`. `SessionManager.CloseSessionCoreAsync` then translates that into a `SessionManagerException` with `CloseFailed` and removes the session.
A kill that succeeds completes the close **successfully**: the worker is gone and the session is `Closed`, exactly as if the shutdown had worked, and the caller's `CloseSession` reply says so. The graceful failure is not lost — it travels back as `SessionCloseResult.WorkerShutdownFailure`, and `SessionManager.CloseSessionCoreAsync` logs it at warning level ("did not shut down gracefully and was killed"), which is the operator signal that workers are hanging on shutdown. The kill itself is counted by `WorkerClient.Kill` (`mxgateway.workers.killed`). Nothing is faulted and no `CloseFailed` fault is recorded, because nothing failed from the session's point of view; reporting the common hung-worker case that way had made every such close look like a teardown failure.
`GatewaySession.KillWorker` is the unconditional forced-close path used by shutdown when graceful close itself throws, and also by `SessionManager.KillWorkerAsync` — the explicit kill path that the dashboard's admin Kill button invokes. `KillWorkerAsync` skips `WorkerClient.ShutdownAsync` entirely, so `KillCount` increments while `ShutdownCount` does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful `CloseSessionAsync`.
Only if both the graceful shutdown and the kill fall-back fail are the original and kill exceptions bundled into an `AggregateException` and surfaced as `SessionCloseStartedException`. `SessionManager.CloseSessionCoreAsync` then translates that into a `SessionManagerException` with `CloseFailed`, faults the session, records the fault metric and removes the session — a worker may still be running, which is what `CloseFailed` is reserved for.
**Losing a race to a concurrent close.** A caller that looked a session up (a kill, a client close, a sweep's snapshot) and then found its close gate already disposed — because a concurrent close finished, removed the session, accounted `SessionClosed` and disposed it in between — sees `ObjectDisposedException` from the gate. Both `KillWorkerAsync` and `CloseSessionCoreAsync` catch that case *when the session is no longer registered* and return `AlreadyClosed: true` with no accounting, no fault and no log: the close that won already did all of it. (The old kill catch recorded a fault, faulted the session and decremented the open gauge a second time; the sweep rethrew it as a sweep failure.) A sweep does not count such a session among the sessions it closed. A disposed gate on a session that *is* still registered is a real failure and takes the normal `CloseFailed` path.
`SessionManager.KillWorkerAsync` is the explicit kill path that the dashboard's admin Kill button invokes and that `ShutdownAsync` falls back to when a graceful close throws. It goes through `GatewaySession.KillWorkerWithCloseGateAsync`, which skips `WorkerClient.ShutdownAsync` entirely, so `KillCount` increments while `ShutdownCount` does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful `CloseSessionAsync`.
## Shutdown Coordination
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. Sessions are drained with the same bounded fan-out the lease sweep uses (`MaxParallelSessionCloses`), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions — *every* exception, including from the fallback — calls `KillWorkerAsync` on an uncancellable token, and removes the session, so that neither one stuck worker nor one failing teardown can block or abort the rest of the host's drain:
```csharp
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
async (session, _) =>
MarkStopping();
for (int pass = 0; pass < MaxShutdownDrainPasses; pass++)
{
if (_registry.Count == 0)
{
try
return;
}
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
async (session, _) =>
{
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
_logger.LogWarning(
exception,
"Graceful shutdown failed for session {SessionId}; killing worker.",
session.SessionId);
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
&& registeredSession is not null)
try
{
try
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
_logger.LogWarning(
exception,
"Graceful shutdown failed for session {SessionId}; killing worker.",
session.SessionId);
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
&& registeredSession is not null)
{
// Not the caller's token: the kill is the last-resort orphan preventer.
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
}
catch (Exception killException)
{
_logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
try
{
// Not the caller's token: the kill is the last-resort orphan preventer.
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
}
catch (Exception killException)
{
_logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
}
}
}
}
}).ConfigureAwait(false);
}).ConfigureAwait(false);
}
```
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.
### Stopping gate
A drain that only ever looked at one snapshot could be outrun by `OpenSession`: the host raises `ApplicationStopping` *before* hosted services stop, and Kestrel keeps accepting gRPC calls until its own `StopAsync` runs, so a session opened behind the snapshot would have nobody left to close it and its x86 worker would survive the gateway as an orphan. `SessionManager` therefore keeps a stopping flag that is set by the host's `IHostApplicationLifetime.ApplicationStopping` (the manager registers for it in its constructor when the lifetime is injected) and, as a fallback for hosts without one, at the start of `ShutdownAsync`. Once set it is never cleared. `OpenSessionAsync` checks it twice: on entry, before a slot is taken, and again right after the session is registered. The second check is what closes the race — the flag is written before the drain takes its registry snapshot, so an open that does not observe it there registered its session before that snapshot and is drained, while one that does observe it rolls its own session back. Either way the caller receives `SessionManagerException` with `GatewayStopping`, which the gRPC layer maps to `Unavailable` so the client retries against the restarted gateway; it is not an open failure and records no fault metric. An open that passed both checks but whose session the drain closed while its worker was still starting gets the same code: the attach is refused, the late worker is discarded, and the manager rechecks the flag before it reports the refusal.
The drain loop above is the backstop to that gate: it re-snapshots until the registry is empty, bounded at `MaxShutdownDrainPasses` (`3`) so a session that somehow cannot be removed cannot hold the host stop forever. Anything still registered after the last pass is logged and left to the orphan terminator on the next start.
## Dependency Injection
`SessionServiceCollectionExtensions.AddGatewaySessions` registers the four singletons and the hosted service:
+6 -1
View File
@@ -117,8 +117,13 @@ public static WorkerBootstrapResult Failure(WorkerExitCode exitCode, IEnumerable
| `MissingNonce` | 4 | The `MXGATEWAY_WORKER_NONCE` environment variable is null, empty, or whitespace. |
| `PipeConnectionFailed` | 5 | An `IOException` or `TimeoutException` escapes the pipe client. |
| `ProtocolViolation` | 6 | A `WorkerFrameProtocolException` escapes the pipe client. |
| `UnhandledException` | 7 | An exception reaches `AppDomain.CurrentDomain.UnhandledException` — thrown on a thread outside `WorkerApplication.Run`'s `try/catch` (a background task continuation, a timer, a COM callback thread). |
`InvalidArguments`, `InvalidProtocolVersion`, and `MissingNonce` originate in the parser; the others originate in `WorkerApplication.Run`'s `try/catch` around the pipe client.
`InvalidArguments`, `InvalidProtocolVersion`, and `MissingNonce` originate in the parser; `UnhandledException` originates in the process-wide hook described below; the others originate in `WorkerApplication.Run`'s `try/catch` around the pipe client.
### Unhandled-exception hook
The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because they can carry tag values or credentials passed through a command, the same rule every other worker failure line follows. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
## Environment Abstraction
+13 -3
View File
@@ -86,10 +86,16 @@ StaWorkItem<T> workItem = new(command, cancellationToken);
lock (gate)
{
if (loopFault is not null)
{
return Task.FromException<T>(new InvalidOperationException(
"The worker STA loop terminated unexpectedly.",
loopFault));
}
if (shutdownRequested)
{
return Task.FromException<T>(
new InvalidOperationException("The worker STA runtime is shutting down."));
return Task.FromException<T>(new StaRuntimeShutdownException());
}
commandQueue.Enqueue(workItem);
@@ -101,7 +107,11 @@ return workItem.Task;
`StaWorkItem<T>` uses an `Interlocked.CompareExchange` on `started` so that exactly one of three outcomes happens: the STA thread executes the delegate, an external `CancellationToken` cancellation fires first, or `CancelBeforeExecution` runs during shutdown. `Execute` runs on the STA thread, sets `Completion` from `command()`, and propagates exceptions through `TrySetException` so the awaiting caller observes them.
`ProcessQueuedCommands` is the only consumer of the queue and runs on the STA thread, so each work item is guaranteed to execute on the apartment that owns the COM object.
`ProcessQueuedCommands` is the only consumer of the queue and runs on the STA thread, so each work item is guaranteed to execute on the apartment that owns the COM object. It calls `PumpPendingMessages` between work items whenever more work is queued, so a backlog of commands cannot hold inbound COM callbacks until the whole queue drains.
### Loop failure after start
An exception thrown by the loop after `Start()` returned — `WaitForWorkOrMessages` reporting `WAIT_FAILED`, or an exception escaping a dispatched message — is not a startup failure, and nothing reads `startupException` after `Start()`. `ThreadMain` instead stores it as `LoopFault` under `gate`, so every later `InvokeAsync` fails immediately with an `InvalidOperationException` whose inner exception is the loop fault, and work already queued is cancelled by the `finally` block. It then invokes the handler registered with `RegisterLoopFaultHandler`; a handler registered after the loop died runs immediately on the registering thread. Neither happens when `Shutdown` was already requested, because a loop that throws during a deliberate teardown is not a worker fault. `MxAccessStaSession` registers a handler that records a `StaHung` `WorkerFault` on the event queue, which the IPC drain loop sends to the gateway before ending the pipe session and the worker process.
## Command Dispatch
+1 -1
View File
@@ -1,7 +1,7 @@
# Deferred Follow-ups Implementation Plan
**Date:** 2026-06-14
**Status:** D1 executed (commit 4af24b9 — `mxgateway.alarms.provider_switches` emitted in `DashboardSnapshotService.cs:198`). D2 resolved as no-op (see resolution section below). D3–D5 remain pending (ops/validation, no code).
**Status:** D1 executed (commit 4af24b9 — `mxgateway.alarms.provider_switches` emitted in `DashboardSnapshotService.cs:198`). D2 resolved as no-op (see resolution section below). D3 (ForceSubtag validation) was exercised through the forced-subtag work (`2026-06-15-forced-subtag-mode-fix.md`); D4 (windev backup prune) has no completion record in the repo and is treated as pending; D5 (wonder redeploy) was superseded by later wonder deployments. D3–D5 are ops/validation with no code.
**Context:** After the alarm-subtag-fallback cleanup (merged `5976770`) and its redeploy to
windev (10.100.0.48), five items remain deferred. This plan handles all five. They are
independent — execute in any order, or cherry-pick. Items D1–D2 are code (branch off `main`);
+12 -3
View File
@@ -446,7 +446,9 @@ Rules:
enforcement is ever added, this is the property it will rely on.
- `correlation_id` links a command to its reply; it is authoritative on the
envelope, and the inner `MxCommandReply.correlation_id` echoes it for
MXAccess parity.
MXAccess parity. The envelope id never reaches public clients: the gateway
overwrites the public `MxCommandReply.correlation_id` with the request's
`client_correlation_id` (empty when the client sent none).
- Events carry their own correlation id or leave it empty.
- Replies must preserve MXAccess HRESULT/status information even when the
command is also represented as a protocol-level failure.
@@ -569,8 +571,15 @@ returns one result per requested entry — per-entry failures populate
advised (`was_cached = true`, no subscription side-effects), or by
taking a full `AddItem` + `Advise` + wait-for-first-OnDataChange +
`UnAdvise` + `RemoveItem` snapshot lifecycle when no live subscription
exists (`was_cached = false`). Per-tag timeouts surface as
`was_successful = false` rather than throwing. The cache lives on the
exists (`was_cached = false`). `timeout_ms` is one shared deadline for
the whole command, not a per-tag budget: the worker adds and advises
every uncached tag first, waits once for all of them, then tears them
all down, so the STA is held for at most one `timeout_ms` (plus the
add/advise and teardown COM calls, during which it pumps every 16 tags).
Each tag that has not delivered by the deadline surfaces its own
`was_successful = false` timeout rather than throwing. Duplicate
addresses that MXAccess resolves to the same item handle share one
subscription and receive the same result. The cache lives on the
worker's `MxAccessValueCache`, populated by `MxAccessBaseEventSink` on
every `OnDataChange` after the event clears the outbound queue.
Required scope: `invoke:read`. `timeout_ms == 0` uses the worker's
@@ -727,6 +727,10 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
[pbr::OriginalName("ALARM_TRANSITION_KIND_RAISE")] Raise = 1,
[pbr::OriginalName("ALARM_TRANSITION_KIND_ACKNOWLEDGE")] Acknowledge = 2,
[pbr::OriginalName("ALARM_TRANSITION_KIND_CLEAR")] Clear = 3,
/// <summary>
/// Reserved: never produced by the gateway or worker. Retained so the numeric
/// value is not reused.
/// </summary>
[pbr::OriginalName("ALARM_TRANSITION_KIND_RETRIGGER")] Retrigger = 4,
}
@@ -12714,6 +12718,11 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
/// <summary>Field number for the "operator_full_name" field.</summary>
public const int OperatorFullNameFieldNumber = 6;
private string operatorFullName_ = "";
/// <summary>
/// Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
/// AlarmAckByName that carried it returns -55. operator_domain IS used (the
/// AuthenticateUser identity is "&lt;operator_domain>\&lt;operator_user>").
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public string OperatorFullName {
@@ -13384,6 +13393,11 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
/// <summary>Field number for the "operator_full_name" field.</summary>
public const int OperatorFullNameFieldNumber = 8;
private string operatorFullName_ = "";
/// <summary>
/// Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
/// AlarmAckByName that carried it returns -55. operator_domain IS used (the
/// AuthenticateUser identity is "&lt;operator_domain>\&lt;operator_user>").
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public string OperatorFullName {
@@ -16191,11 +16205,15 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
/// reply returns the cached value WITHOUT modifying the existing
/// subscription (was_cached = true).
/// - Otherwise the worker takes the snapshot lifecycle itself: AddItem +
/// Advise, wait up to `timeout_ms` for the first OnDataChange, then
/// UnAdvise + RemoveItem before returning. The session is left exactly
/// as it was before the call (was_cached = false).
/// Advise, wait for the first OnDataChange, then UnAdvise + RemoveItem
/// before returning. The session is left exactly as it was before the
/// call (was_cached = false).
///
/// `timeout_ms == 0` uses the gateway-configured default (1000 ms).
/// `timeout_ms` is one deadline shared by the whole command, not a per-tag
/// budget: the worker adds and advises every uncached tag, waits once for all
/// of them, then tears them all down. A tag that has not delivered by the
/// deadline gets its own was_successful = false result.
/// `timeout_ms == 0` uses the worker's default (1000 ms).
/// </summary>
[global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")]
public sealed partial class ReadBulkCommand : pb::IMessage<ReadBulkCommand>
@@ -17527,6 +17545,9 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
/// <summary>Field number for the "correlation_id" field.</summary>
public const int CorrelationIdFieldNumber = 2;
private string correlationId_ = "";
/// <summary>
/// Echoes the request's client_correlation_id.
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public string CorrelationId {
@@ -26125,7 +26146,7 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
/// <summary>
/// When the alarm originally entered the active state. Preserved across
/// acknowledge transitions so the Part 9 condition keeps the original raise
/// time. Unset on retrigger from a previously-cleared condition.
/// time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
@@ -132,6 +132,12 @@ namespace ZB.MOM.WW.MxGateway.Contracts.Proto {
public enum WorkerFaultCategory {
[pbr::OriginalName("WORKER_FAULT_CATEGORY_UNSPECIFIED")] Unspecified = 0,
/// <summary>
/// Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
/// arguments, gateway authentication) exit via WorkerExitCode before the pipe
/// exists, so there is no channel to carry a fault. Names kept so the numbers
/// are not reused.
/// </summary>
[pbr::OriginalName("WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS")] InvalidArguments = 1,
[pbr::OriginalName("WORKER_FAULT_CATEGORY_GATEWAY_AUTHENTICATION_FAILED")] GatewayAuthenticationFailed = 2,
[pbr::OriginalName("WORKER_FAULT_CATEGORY_PROTOCOL_MISMATCH")] ProtocolMismatch = 3,
@@ -405,6 +405,9 @@ message AcknowledgeAlarmCommand {
string operator_user = 3;
string operator_node = 4;
string operator_domain = 5;
// Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
// AlarmAckByName that carried it returns -55. operator_domain IS used (the
// AuthenticateUser identity is "<operator_domain>\<operator_user>").
string operator_full_name = 6;
// Operator credential used for AuthenticateUser before the AckMsg write.
// Implementations must keep this field out of logs, metrics labels, command
@@ -435,6 +438,9 @@ message AcknowledgeAlarmByNameCommand {
string operator_user = 5;
string operator_node = 6;
string operator_domain = 7;
// Accepted for wire compatibility but NOT propagated to MXAccess: the 8-arg
// AlarmAckByName that carried it returns -55. operator_domain IS used (the
// AuthenticateUser identity is "<operator_domain>\<operator_user>").
string operator_full_name = 8;
// Operator credential used for AuthenticateUser before the AckMsg write.
// Implementations must keep this field out of logs, metrics labels, command
@@ -518,11 +524,15 @@ message WriteSecured2BulkEntry {
// reply returns the cached value WITHOUT modifying the existing
// subscription (was_cached = true).
// - Otherwise the worker takes the snapshot lifecycle itself: AddItem +
// Advise, wait up to `timeout_ms` for the first OnDataChange, then
// UnAdvise + RemoveItem before returning. The session is left exactly
// as it was before the call (was_cached = false).
// Advise, wait for the first OnDataChange, then UnAdvise + RemoveItem
// before returning. The session is left exactly as it was before the
// call (was_cached = false).
//
// `timeout_ms == 0` uses the gateway-configured default (1000 ms).
// `timeout_ms` is one deadline shared by the whole command, not a per-tag
// budget: the worker adds and advises every uncached tag, waits once for all
// of them, then tears them all down. A tag that has not delivered by the
// deadline gets its own was_successful = false result.
// `timeout_ms == 0` uses the worker's default (1000 ms).
message ReadBulkCommand {
int32 server_handle = 1;
repeated string tag_addresses = 2;
@@ -549,6 +559,7 @@ message ShutdownWorkerCommand {
message MxCommandReply {
string session_id = 1;
// Echoes the request's client_correlation_id.
string correlation_id = 2;
MxCommandKind kind = 3;
ProtocolStatus protocol_status = 4;
@@ -880,7 +891,7 @@ message OnAlarmTransitionEvent {
// When the alarm originally entered the active state. Preserved across
// acknowledge transitions so the Part 9 condition keeps the original raise
// time. Unset on retrigger from a previously-cleared condition.
// time. Unset when the raise time is unknown (RETRIGGER is reserved and never produced).
google.protobuf.Timestamp original_raise_timestamp = 6;
// When this specific transition occurred (raise time on Raise, ack time on
@@ -930,6 +941,8 @@ enum AlarmTransitionKind {
ALARM_TRANSITION_KIND_RAISE = 1;
ALARM_TRANSITION_KIND_ACKNOWLEDGE = 2;
ALARM_TRANSITION_KIND_CLEAR = 3;
// Reserved: never produced by the gateway or worker. Retained so the numeric
// value is not reused.
ALARM_TRANSITION_KIND_RETRIGGER = 4;
}
@@ -127,6 +127,10 @@ enum WorkerState {
enum WorkerFaultCategory {
WORKER_FAULT_CATEGORY_UNSPECIFIED = 0;
// Values 1 and 2 are reserved: never emitted. Bootstrap failures (bad
// arguments, gateway authentication) exit via WorkerExitCode before the pipe
// exists, so there is no channel to carry a fault. Names kept so the numbers
// are not reused.
WORKER_FAULT_CATEGORY_INVALID_ARGUMENTS = 1;
WORKER_FAULT_CATEGORY_GATEWAY_AUTHENTICATION_FAILED = 2;
WORKER_FAULT_CATEGORY_PROTOCOL_MISMATCH = 3;
@@ -1052,6 +1052,231 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
}
}
/// <summary>
/// TST-05: the real worker answers the four non-terminating control commands off-STA in
/// <c>WorkerPipeSession.HandleControlCommandAsync</c>. The default suite only ever sees the
/// FakeWorkerHarness canned replies, so this drives <c>Ping</c>, <c>GetSessionState</c>,
/// <c>GetWorkerInfo</c>, and <c>DrainEvents</c> through the gateway against a live worker and
/// asserts each reply carries real worker state rather than an <c>INVALID_REQUEST</c>.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[LiveMxAccessFact]
public async Task GatewaySession_WithLiveWorker_ControlCommands_ReturnRealWorkerState()
{
string workerExecutablePath = IntegrationTestEnvironment.ResolveLiveMxAccessWorkerExecutablePath();
Assert.True(
File.Exists(workerExecutablePath),
$"Live MXAccess worker executable was not found at {workerExecutablePath}. Build the worker or set {IntegrationTestEnvironment.LiveMxAccessWorkerExecutableVariableName}.");
TestWorkerProcessFactory processFactory = new(output);
await using GatewayServiceFixture fixture = new(workerExecutablePath, processFactory, output);
string? sessionId = null;
try
{
OpenSessionReply openReply = await fixture.Service.OpenSession(
new OpenSessionRequest
{
ClientSessionName = "live-mxaccess-control",
ClientCorrelationId = "live-open-control",
CommandTimeout = Duration.FromTimeSpan(CommandTimeout),
},
new TestServerCallContext()).ConfigureAwait(false);
sessionId = openReply.SessionId;
Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
Assert.True(openReply.WorkerProcessId > 0);
MxCommandReply pingReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-ping", new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand { Message = "live-ping-echo" },
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("Ping", pingReply);
Assert.Equal(ProtocolStatusCode.Ok, pingReply.ProtocolStatus.Code);
Assert.Equal(MxCommandKind.Ping, pingReply.Kind);
MxCommandReply stateReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-get-session-state", new MxCommand
{
Kind = MxCommandKind.GetSessionState,
GetSessionState = new GetSessionStateCommand(),
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("GetSessionState", stateReply);
Assert.Equal(ProtocolStatusCode.Ok, stateReply.ProtocolStatus.Code);
Assert.Equal(MxCommandKind.GetSessionState, stateReply.Kind);
Assert.NotNull(stateReply.SessionState);
Assert.Equal(SessionState.Ready, stateReply.SessionState.State);
MxCommandReply infoReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-get-worker-info", new MxCommand
{
Kind = MxCommandKind.GetWorkerInfo,
GetWorkerInfo = new GetWorkerInfoCommand(),
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("GetWorkerInfo", infoReply);
Assert.Equal(ProtocolStatusCode.Ok, infoReply.ProtocolStatus.Code);
Assert.Equal(MxCommandKind.GetWorkerInfo, infoReply.Kind);
Assert.NotNull(infoReply.WorkerInfo);
output.WriteLine(
$"GetWorkerInfo pid={infoReply.WorkerInfo.WorkerProcessId} version={infoReply.WorkerInfo.WorkerVersion} "
+ $"progid={infoReply.WorkerInfo.MxaccessProgid} clsid={infoReply.WorkerInfo.MxaccessClsid}");
Assert.Equal(openReply.WorkerProcessId, infoReply.WorkerInfo.WorkerProcessId);
Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.WorkerVersion));
Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.MxaccessProgid));
Assert.False(string.IsNullOrWhiteSpace(infoReply.WorkerInfo.MxaccessClsid));
// No StreamEvents subscriber is attached, so whatever the worker queued since start
// is still drainable; the reply must carry the DrainEvents payload (possibly empty).
MxCommandReply drainReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-drain-events", new MxCommand
{
Kind = MxCommandKind.DrainEvents,
DrainEvents = new DrainEventsCommand { MaxEvents = 16 },
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("DrainEvents", drainReply);
Assert.Equal(ProtocolStatusCode.Ok, drainReply.ProtocolStatus.Code);
Assert.Equal(MxCommandKind.DrainEvents, drainReply.Kind);
Assert.Equal(MxCommandReply.PayloadOneofCase.DrainEvents, drainReply.PayloadCase);
Assert.True(drainReply.DrainEvents.Events.Count <= 16);
output.WriteLine($"DrainEvents drained={drainReply.DrainEvents.Events.Count}");
// A control command must not disturb the session: it is still Ready afterwards.
Assert.True(fixture.TryGetSession(sessionId, out GatewaySession? session));
Assert.Equal(SessionState.Ready, session!.State);
}
finally
{
await ShutDownAsync(fixture, processFactory, sessionId, streamTask: null).ConfigureAwait(false);
}
}
/// <summary>
/// TST-05: <c>ShutdownWorker</c> writes its OK reply before the worker shuts down, so the
/// reply reaches the caller (a reply written after the pipe closed would surface as a
/// pipe-disconnected failure instead), the worker process then exits on its own, and a later
/// command on the session does not succeed. Uses its own fixture and worker because it
/// terminates the worker.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[LiveMxAccessFact]
public async Task GatewaySession_WithLiveWorker_ShutdownWorker_RepliesOkThenWorkerExits()
{
string workerExecutablePath = IntegrationTestEnvironment.ResolveLiveMxAccessWorkerExecutablePath();
Assert.True(
File.Exists(workerExecutablePath),
$"Live MXAccess worker executable was not found at {workerExecutablePath}. Build the worker or set {IntegrationTestEnvironment.LiveMxAccessWorkerExecutableVariableName}.");
TestWorkerProcessFactory processFactory = new(output);
await using GatewayServiceFixture fixture = new(workerExecutablePath, processFactory, output);
using RecordingServerStreamWriter<MxEvent> eventWriter = new();
string? sessionId = null;
Task? streamTask = null;
using CancellationTokenSource streamCancellation = new();
try
{
OpenSessionReply openReply = await fixture.Service.OpenSession(
new OpenSessionRequest
{
ClientSessionName = "live-mxaccess-shutdown-worker",
ClientCorrelationId = "live-open-shutdown-worker",
CommandTimeout = Duration.FromTimeSpan(CommandTimeout),
},
new TestServerCallContext()).ConfigureAwait(false);
sessionId = openReply.SessionId;
Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
// The gateway observes a worker's pipe closing through the event-stream read loop
// (the same path the abnormal-exit fact relies on); without a subscriber the session
// would sit in Ready until the heartbeat grace expired.
streamTask = fixture.Service.StreamEvents(
new StreamEventsRequest { SessionId = sessionId },
eventWriter,
new TestServerCallContext(streamCancellation.Token));
MxCommandReply shutdownReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-shutdown-worker", new MxCommand
{
Kind = MxCommandKind.ShutdownWorker,
ShutdownWorker = new ShutdownWorkerCommand
{
GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(5)),
},
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("ShutdownWorker", shutdownReply);
Assert.Equal(ProtocolStatusCode.Ok, shutdownReply.ProtocolStatus.Code);
Assert.Equal(MxCommandKind.ShutdownWorker, shutdownReply.Kind);
// The worker exits on its own — no CloseSession, no kill. WaitForProcessesAsync
// throws on timeout, which fails the test.
await processFactory.WaitForProcessesAsync(StreamShutdownTimeout).ConfigureAwait(false);
// The session can no longer serve commands once its worker is gone: either the
// gateway rejects the call outright or the reply is not Ok.
MxCommandReply? afterReply = null;
Exception? afterException = null;
try
{
afterReply = await fixture.Service.Invoke(
CreateControlRequest(sessionId, "live-ping-after-shutdown", new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand(),
}),
new TestServerCallContext()).ConfigureAwait(false);
LogReply("PingAfterShutdown", afterReply);
}
catch (Exception exception)
{
afterException = exception;
output.WriteLine($"PingAfterShutdown rejected: {exception.GetType().Name}: {exception.Message}");
}
Assert.True(
afterException is not null || afterReply!.ProtocolStatus.Code != ProtocolStatusCode.Ok,
"A command sent after ShutdownWorker must not succeed against the exited worker.");
// Logged, not asserted: the gateway's WorkerClient moves to Closed on the shutdown
// ack, but GatewaySession.State is not driven from that transition today, so the
// public session state still reads Ready here (observed live on windev). That is a
// gateway-side gap, not a worker one — this fact pins the worker half of the
// contract: OK reply first, then a clean exit.
string observedState = fixture.TryGetSession(sessionId, out GatewaySession? session)
? session.State.ToString()
: "<removed>";
output.WriteLine($"ShutdownWorker observed_session_state={observedState}");
}
finally
{
// The worker is gone; a CloseSession round-trip would only log a cleanup failure.
streamCancellation.Cancel();
await ShutDownAsync(fixture, processFactory, sessionId: null, streamTask).ConfigureAwait(false);
}
}
private static MxCommandRequest CreateControlRequest(
string sessionId,
string correlationId,
MxCommand command)
{
return new MxCommandRequest
{
SessionId = sessionId,
ClientCorrelationId = correlationId,
Command = command,
};
}
/// <summary>
/// Closes the session and drains the event stream / worker processes without letting a
/// cleanup timeout mask the original failure from the test body.
@@ -1704,6 +1929,20 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
/// </summary>
public WriteOptions? WriteOptions { get; set; }
/// <summary>
/// Records the message unless the token is already cancelled.
/// </summary>
/// <param name="message">The message to write.</param>
/// <param name="cancellationToken">Token that cancels the write.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public Task WriteAsync(T message, CancellationToken cancellationToken)
{
// The interface default throws NotSupportedException for a cancelable token; the
// gateway passes the call's token on every streamed write.
cancellationToken.ThrowIfCancellationRequested();
return WriteAsync(message);
}
/// <summary>
/// Records the message and signals any pending waiter.
/// </summary>
@@ -2112,26 +2351,26 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
private sealed class AllowAllConstraintEnforcer : IConstraintEnforcer
{
/// <inheritdoc />
public Task<ConstraintFailure?> CheckReadTagAsync(
public ValueTask<ConstraintFailure?> CheckReadTagAsync(
ApiKeyIdentity? identity,
string tagAddress,
CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
/// <inheritdoc />
public Task<ConstraintFailure?> CheckReadHandleAsync(
public ValueTask<ConstraintFailure?> CheckReadHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
int itemHandle,
CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
/// <inheritdoc />
public Task<ConstraintFailure?> CheckWriteHandleAsync(
public ValueTask<ConstraintFailure?> CheckWriteHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
int itemHandle,
CancellationToken cancellationToken) => Task.FromResult<ConstraintFailure?>(null);
CancellationToken cancellationToken) => ValueTask.FromResult<ConstraintFailure?>(null);
/// <inheritdoc />
public Task RecordDenialAsync(
@@ -2141,6 +2380,14 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
ConstraintFailure failure,
string? correlationId,
CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public Task RecordBulkDenialAsync(
ApiKeyIdentity? identity,
string commandKind,
ConstraintDenialSummary summary,
string? correlationId,
CancellationToken cancellationToken) => Task.CompletedTask;
}
}
@@ -0,0 +1,39 @@
namespace ZB.MOM.WW.MxGateway.Server.Alarms;
/// <summary>
/// Completes an alarm-feed subscriber that fell behind its bounded queue and was dropped.
/// </summary>
/// <remarks>
/// A typed exception rather than a bare <see cref="InvalidOperationException"/> so the gRPC
/// layer can map it to <c>ResourceExhausted</c> — the status a <c>StreamEvents</c> overflow
/// carries — without treating a slow client as a gateway failure (an unrecognized exception is
/// logged as a warning with its stack and surfaced as <c>Unavailable</c>). The remedy is the
/// same as for an event-stream overflow: reconnect, which re-snapshots the alarm set.
/// </remarks>
public sealed class AlarmFeedSubscriberOverflowException : Exception
{
/// <summary>The message every dropped subscriber is completed with.</summary>
public const string DefaultMessage =
"Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot.";
/// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
public AlarmFeedSubscriberOverflowException()
: base(DefaultMessage)
{
}
/// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
/// <param name="message">The error message.</param>
public AlarmFeedSubscriberOverflowException(string message)
: base(message)
{
}
/// <summary>Initializes a new instance of the <see cref="AlarmFeedSubscriberOverflowException"/> class.</summary>
/// <param name="message">The error message.</param>
/// <param name="innerException">The exception that caused this one.</param>
public AlarmFeedSubscriberOverflowException(string message, Exception innerException)
: base(message, innerException)
{
}
}
@@ -967,7 +967,9 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
}
// Caller holds _sync. Pushes a feed message to every matching subscriber;
// a subscriber that has fallen behind is completed with an error and dropped.
// a subscriber that has fallen behind is completed with an AlarmFeedSubscriberOverflowException
// and dropped. The typed fault is what lets StreamAlarms answer ResourceExhausted instead of
// treating a slow client as a gateway failure.
private void Broadcast(AlarmFeedMessage message, string reference)
{
for (int index = _subscribers.Count - 1; index >= 0; index--)
@@ -980,8 +982,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
if (!subscriber.Channel.Writer.TryWrite(message))
{
subscriber.Channel.Writer.TryComplete(new InvalidOperationException(
"Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."));
subscriber.Channel.Writer.TryComplete(new AlarmFeedSubscriberOverflowException());
_subscribers.RemoveAt(index);
}
}
@@ -997,8 +998,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
Subscriber subscriber = _subscribers[index];
if (!subscriber.Channel.Writer.TryWrite(message))
{
subscriber.Channel.Writer.TryComplete(new InvalidOperationException(
"Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."));
subscriber.Channel.Writer.TryComplete(new AlarmFeedSubscriberOverflowException());
_subscribers.RemoveAt(index);
}
}
@@ -1046,9 +1046,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
_subscribers.Add(subscriber);
providerStatus = BuildProviderStatus();
snapshotTruncated = _snapshotTruncated;
snapshot = _alarms.Values
.Where(alarm => prefix.Length == 0
|| alarm.AlarmFullReference.StartsWith(prefix, StringComparison.Ordinal))
// Take the memoized projection rather than cloning the whole set per attach: the
// clone runs at most once per alarm-set generation (shared with CurrentAlarms), so a
// burst of reconnecting subscribers no longer holds the broadcast lock for one full
// copy each. The array is never mutated once built — a write to _alarms replaces it
// with null — so capturing the reference here pins exactly the generation this
// subscriber registered against, which is what keeps snapshot-then-live ordering.
snapshot = _currentAlarmsProjection ??= _alarms.Values
.Select(alarm => alarm.Clone())
.ToArray();
}
@@ -1068,8 +1073,15 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
SnapshotStatus = new AlarmSnapshotStatus { Truncated = snapshotTruncated },
};
// The prefix filter runs here, outside _sync, over the immutable projection.
foreach (ActiveAlarmSnapshot alarm in snapshot)
{
if (prefix.Length != 0
&& !alarm.AlarmFullReference.StartsWith(prefix, StringComparison.Ordinal))
{
continue;
}
yield return new AlarmFeedMessage { ActiveAlarm = alarm };
}
@@ -52,4 +52,7 @@ public sealed class GatewayOptions
/// login / API-key rate limiting).
/// </summary>
public SecurityOptions Security { get; init; } = new();
/// <summary>Gets Prometheus <c>/metrics</c> endpoint options.</summary>
public MetricsOptions Metrics { get; init; } = new();
}
@@ -325,6 +325,10 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
options.FaultedGraceSeconds,
"MxGateway:Sessions:FaultedGraceSeconds must be zero or greater (0 reaps a faulted session on the next sweep).",
builder);
AddIfNegative(
options.UnattachedSessionGraceSeconds,
"MxGateway:Sessions:UnattachedSessionGraceSeconds must be zero or greater (0 disables reaping of never-subscribed sessions).",
builder);
AddIfNegative(
options.WorkerReadyWaitTimeoutMs,
"MxGateway:Sessions:WorkerReadyWaitTimeoutMs must be greater than or equal to zero.",
@@ -0,0 +1,15 @@
namespace ZB.MOM.WW.MxGateway.Server.Configuration;
/// <summary>Options for the Prometheus <c>/metrics</c> scrape endpoint (<c>MxGateway:Metrics</c>).</summary>
public sealed class MetricsOptions
{
/// <summary>
/// Gets a value indicating whether <c>/metrics</c> requires an authenticated caller. When
/// <see langword="true"/>, a scrape must present either an API key carrying the <c>admin</c>
/// scope (<c>Authorization: Bearer mxgw_…</c>) or a dashboard login cookie holding a real
/// dashboard role; anonymous localhost access does not qualify. Default <see langword="false"/>
/// keeps the endpoint anonymous for existing scrapers. The <c>/health</c> endpoints stay
/// anonymous either way.
/// </summary>
public bool RequireAuthentication { get; init; }
}
@@ -57,6 +57,28 @@ public sealed class SessionOptions
/// </summary>
public int FaultedGraceSeconds { get; init; }
/// <summary>
/// Gets the idle window, in seconds, after which the lease monitor reaps a session that
/// has <em>never</em> attached an event-stream subscriber. A client that opens a session
/// and vanishes before it ever streams (for example OtOpcUa's <c>GalaxyDriver</c> failing
/// part-way through its start-up) otherwise pins a session slot and a live x86 worker for
/// the full <see cref="DefaultLeaseSeconds"/> (30 minutes by default). The window is
/// measured from the later of the session becoming Ready and its most recent unary
/// command, so a slow worker start does not eat it and a client that only ever reads and
/// writes is not reaped while it keeps calling; it stops applying the moment the session's
/// first subscriber — external, or the gateway's own internal alarm-monitor subscriber —
/// attaches (from then on <see cref="DetachGraceSeconds"/> and the normal lease govern
/// it). A value of <c>0</c> (the default) disables the sweep. Must be greater than or
/// equal to zero.
/// </summary>
/// <remarks>
/// Set it comfortably above the time a well-behaved client needs between
/// <c>OpenSession</c> and its first <c>StreamEvents</c>, and remember the close lands on
/// the next sweep after the window elapses (up to <see cref="LeaseSweepIntervalSeconds"/>
/// later).
/// </remarks>
public int UnattachedSessionGraceSeconds { get; init; }
/// <summary>
/// Gets a value indicating whether multiple event subscribers are allowed per session.
/// </summary>
@@ -58,6 +58,8 @@
<MetricCard Label="Active (unacked)" Value="@_unackedCount.ToString("N0")" />
<MetricCard Label="Acknowledged" Value="@_ackedCount.ToString("N0")" />
<MetricCard Label="Total Active" Value="@_alarms.Count.ToString("N0")" />
@* FilteredAlarms() is memoized on (alarm generation, filter inputs), so this card and the
table below share one computation per change instead of filtering twice per render. *@
<MetricCard Label="Showing" Value="@FilteredAlarms().Count.ToString("N0")" />
</section>
@@ -189,6 +191,10 @@
private int _maxSeverity = 1000;
private string _search = string.Empty;
// Bumped whenever _alarms is replaced, so the memoized view recomputes exactly then.
private readonly DashboardAlarmView _view = new();
private long _alarmGeneration;
// Upper bound on waiting for either background loop while disposing, mirroring
// DashboardPageBase's snapshot-watch drain: both loops marshal renders through the
// renderer's dispatcher and disposal can run on that same dispatcher, so the wait is
@@ -295,42 +301,13 @@
return _workerPid is int pid ? $"{refreshed} · worker pid {pid}" : refreshed;
}
private IReadOnlyList<string> Areas()
{
return _alarms
.Select(alarm => alarm.Area)
.Where(area => !string.IsNullOrWhiteSpace(area))
.Distinct(StringComparer.OrdinalIgnoreCase)
.OrderBy(area => area, StringComparer.OrdinalIgnoreCase)
.ToArray();
}
private IReadOnlyList<string> Areas() => _view.Areas(_alarmGeneration, _alarms);
private IReadOnlyList<DashboardActiveAlarm> FilteredAlarms()
{
string query = _search.Trim();
return _alarms
.Where(MatchesState)
.Where(alarm => _areaFilter.Length == 0
|| string.Equals(alarm.Area, _areaFilter, StringComparison.OrdinalIgnoreCase))
.Where(alarm => alarm.Severity >= _minSeverity && alarm.Severity <= _maxSeverity)
.Where(alarm => query.Length == 0
|| alarm.Reference.Contains(query, StringComparison.OrdinalIgnoreCase)
|| alarm.Source.Contains(query, StringComparison.OrdinalIgnoreCase)
|| alarm.Description.Contains(query, StringComparison.OrdinalIgnoreCase))
.OrderByDescending(alarm => alarm.Severity)
.ThenByDescending(alarm => alarm.LastTransition ?? DateTimeOffset.MinValue)
.ToArray();
}
private bool MatchesState(DashboardActiveAlarm alarm)
{
return alarm.State switch
{
AlarmConditionState.Active => _showActive,
AlarmConditionState.ActiveAcked => _showAcked,
_ => true,
};
}
private IReadOnlyList<DashboardActiveAlarm> FilteredAlarms() =>
_view.Filtered(
_alarmGeneration,
new DashboardAlarmFilterCriteria(_showActive, _showAcked, _areaFilter, _minSeverity, _maxSeverity, _search),
_alarms);
private static string StateText(AlarmConditionState state)
{
@@ -439,6 +416,7 @@
_lastRefresh = DateTimeOffset.UtcNow;
_alarms.Clear();
_alarms.AddRange(result.Alarms);
_alarmGeneration++;
_unackedCount = _alarms.Count(alarm => alarm.State == AlarmConditionState.Active);
_ackedCount = _alarms.Count(alarm => alarm.State == AlarmConditionState.ActiveAcked);
StateHasChanged();
@@ -4,6 +4,8 @@
@inject IDashboardLiveDataService LiveData
@inject IDashboardBrowseService BrowseService
@inject IGalaxyDeployNotifier DeployNotifier
@inject DashboardBrowseSearchIndex SearchIndex
@inject DashboardCircuitConnectionTracker Circuit
@using ZB.MOM.WW.GalaxyRepository
@using ZB.MOM.WW.GalaxyRepository.Grpc
@@ -36,7 +38,15 @@
}
else if (!string.IsNullOrWhiteSpace(Search))
{
@if (_searchMatches.Count == 0)
@if (!string.IsNullOrWhiteSpace(_searchError))
{
<div class="alert alert-danger">Search failed: @_searchError</div>
}
else if (_searchPending && _searchMatches.Count == 0)
{
<div class="empty-state">Searching&hellip;</div>
}
else if (_searchMatches.Count == 0)
{
<div class="empty-state">No attributes match &ldquo;@Search&rdquo;.</div>
}
@@ -115,6 +125,13 @@
<div class="alert alert-danger">Live read failed: @_readError</div>
}
@if (!string.IsNullOrWhiteSpace(_deployError))
{
<div class="alert alert-warning">
Galaxy redeploy notifications failed (retrying): @_deployError
</div>
}
@if (_subscribed.Count == 0)
{
<div class="empty-state">
@@ -194,13 +211,35 @@
@code {
private const int SearchResultLimit = 300;
// Keystrokes inside this window coalesce into one search: typing a tag name no
// longer rescans the Galaxy once per character.
private static readonly TimeSpan SearchDebounce = TimeSpan.FromMilliseconds(150);
// Delay before resubscribing to deploy notifications after the stream ends or faults.
private static readonly TimeSpan DeployResubscribeDelay = TimeSpan.FromSeconds(5);
// Upper bound on waiting for the page's background loops while disposing, the same
// bound DashboardPageBase and AlarmsPage use: the loops marshal renders through the
// renderer's dispatcher and disposal can run on that same dispatcher, so the wait is
// bounded rather than unconditional.
private static readonly TimeSpan LoopDrainTimeout = TimeSpan.FromSeconds(5);
// Identifies this page to the shared live-data service, which budgets the advise set
// per viewer so another circuit's reads cannot evict this page's whole panel.
private readonly string _viewerId = "browse-" + Guid.NewGuid().ToString("N");
private List<DashboardBrowseNode> _roots = [];
private ulong _cacheSequence;
private string? _staleBanner;
private CancellationTokenSource _deployCts = new();
private readonly CancellationTokenSource _deployCts = new();
private Task? _deployTask;
private string? _deployError;
private string _search = string.Empty;
private IReadOnlyList<GalaxyAttribute> _searchMatches = [];
private bool _searchPending;
private string? _searchError;
private CancellationTokenSource? _searchCts;
private Task? _searchTask;
private readonly List<string> _subscribed = [];
private readonly Dictionary<string, DashboardTagValue> _values = new(StringComparer.Ordinal);
// Per-tag bookkeeping for the Updated column: the signature of the value
@@ -225,8 +264,8 @@
BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
_roots = [.. roots.Nodes];
_cacheSequence = roots.CacheSequence;
_pollTask = PollLoopAsync();
_deployTask = SubscribeToDeployEventsAsync();
_pollTask = PollLoopAsync(_cts.Token);
_deployTask = SubscribeToDeployEventsAsync(_deployCts.Token);
}
private async Task LoadChildrenAsync(DashboardBrowseNode node)
@@ -249,30 +288,55 @@
await InvokeAsync(StateHasChanged);
}
private async Task SubscribeToDeployEventsAsync()
// Faults are handled inside the loop: a notifier stream that faults or ends is
// surfaced on the page and resubscribed after a delay, so the tree does not silently
// stop tracking redeploys. Only cancellation ends the loop, so DrainAsync never
// observes a faulted task.
private async Task SubscribeToDeployEventsAsync(CancellationToken cancellationToken)
{
try
while (!cancellationToken.IsCancellationRequested)
{
await foreach (GalaxyDeployEventInfo info in DeployNotifier
.SubscribeAsync(_deployCts.Token)
.ConfigureAwait(false))
try
{
// First Latest replay echoes the sequence we already projected
// from — skip those to avoid a spurious "redeployed" banner.
if (info.Sequence == 0 || (ulong)info.Sequence == _cacheSequence)
await foreach (GalaxyDeployEventInfo info in DeployNotifier
.SubscribeAsync(cancellationToken)
.ConfigureAwait(false))
{
continue;
}
// First Latest replay echoes the sequence we already projected
// from — skip those to avoid a spurious "redeployed" banner.
if (info.Sequence == 0 || (ulong)info.Sequence == _cacheSequence)
{
continue;
}
BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
_roots = [.. roots.Nodes];
_cacheSequence = roots.CacheSequence;
_staleBanner = "Galaxy redeployed — tree refreshed.";
await InvokeAsync(StateHasChanged);
await InvokeAsync(() =>
{
BrowseLevelResult roots = BrowseService.GetRoots(new BrowseFilterArgs());
_roots = [.. roots.Nodes];
_cacheSequence = roots.CacheSequence;
_staleBanner = "Galaxy redeployed — tree refreshed.";
_deployError = null;
StateHasChanged();
}).ConfigureAwait(false);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
await ReportAsync(() => _deployError = ex.Message).ConfigureAwait(false);
}
try
{
await Task.Delay(DeployResubscribeDelay, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return;
}
}
catch (OperationCanceledException)
{
}
}
@@ -288,42 +352,101 @@
+ $"{entry.AlarmAttributeCount:N0} alarm attributes";
}
// Bound to the filter box with @bind:event="oninput", so the setter runs once per
// keystroke on the render dispatcher. It only schedules the search; the scan itself
// runs after the debounce, off the dispatcher, and a newer keystroke cancels it.
private string Search
{
get => _search;
set
{
_search = value ?? string.Empty;
_searchMatches = ComputeSearch(_search);
ScheduleSearch(_search);
}
}
private IReadOnlyList<GalaxyAttribute> ComputeSearch(string rawQuery)
private void ScheduleSearch(string query)
{
string query = rawQuery.Trim();
if (query.Length == 0)
CancelQuietly(_searchCts);
_searchCts = null;
_searchError = null;
if (query.Trim().Length == 0)
{
return [];
_searchMatches = [];
_searchPending = false;
return;
}
List<GalaxyAttribute> matches = [];
foreach (GalaxyObject galaxyObject in GalaxyCache.Current.Objects)
// Linked to the page token, so disposal cancels an in-flight search too. The
// search task owns the source and disposes it when it finishes.
CancellationTokenSource searchCts = CancellationTokenSource.CreateLinkedTokenSource(_cts.Token);
_searchCts = searchCts;
_searchPending = true;
_searchTask = RunSearchAsync(query, searchCts);
}
private async Task RunSearchAsync(string query, CancellationTokenSource searchCts)
{
CancellationToken cancellationToken = searchCts.Token;
try
{
foreach (GalaxyAttribute attr in galaxyObject.Attributes)
await Task.Delay(SearchDebounce, cancellationToken).ConfigureAwait(false);
GalaxyHierarchyCacheEntry entry = GalaxyCache.Current;
IReadOnlyList<GalaxyAttribute> matches = await Task.Run(
() => SearchIndex.Search(entry, query, SearchResultLimit, cancellationToken),
cancellationToken)
.ConfigureAwait(false);
await InvokeAsync(() =>
{
if (attr.FullTagReference.Contains(query, StringComparison.OrdinalIgnoreCase)
|| attr.AttributeName.Contains(query, StringComparison.OrdinalIgnoreCase))
// Only the latest query may publish: a slower, superseded scan that
// finishes late must not overwrite newer results.
if (ReferenceEquals(_searchCts, searchCts) && !cancellationToken.IsCancellationRequested)
{
matches.Add(attr);
if (matches.Count >= SearchResultLimit)
{
return matches;
}
_searchMatches = matches;
_searchPending = false;
StateHasChanged();
}
}
}).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Superseded by a newer keystroke, or the page is going away.
}
catch (Exception ex)
{
await ReportAsync(() =>
{
if (ReferenceEquals(_searchCts, searchCts))
{
_searchPending = false;
_searchError = ex.Message;
}
}).ConfigureAwait(false);
}
finally
{
searchCts.Dispose();
}
}
// The search task disposes its own source, so a cancel can race that disposal.
private static void CancelQuietly(CancellationTokenSource? source)
{
if (source is null)
{
return;
}
return matches;
try
{
source.Cancel();
}
catch (ObjectDisposedException)
{
// The search already finished; there is nothing left to cancel.
}
}
private static string FormatType(GalaxyAttribute attr)
@@ -422,17 +545,66 @@
: "When the dashboard first observed this value — MXAccess did not supply a source timestamp for this tag.";
}
private async Task PollLoopAsync()
// Fault handling sits inside the loop, matching AlarmsPage: a fault on one tick is
// surfaced in the "Live read failed" banner and retried on the next tick rather than
// ending polling for the life of the page. Cancellation is the only exit, so the loop
// task never faults — which is what DrainAsync in DisposeAsync relies on.
//
// A tick is skipped while the circuit's browser is disconnected: a closed tab keeps
// its circuit for the retention period, and polling ReadBulk on the shared dashboard
// worker for a page nobody can see is pure cost. The first tick after a reconnect
// refreshes as normal.
private async Task PollLoopAsync(CancellationToken cancellationToken)
{
try
using PeriodicTimer timer = new(TimeSpan.FromSeconds(2));
while (true)
{
using PeriodicTimer timer = new(TimeSpan.FromSeconds(2));
while (await timer.WaitForNextTickAsync(_cts.Token).ConfigureAwait(false))
try
{
if (!await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false))
{
return;
}
}
catch (OperationCanceledException)
{
return;
}
if (!Circuit.IsConnected)
{
continue;
}
try
{
await InvokeAsync(RefreshValuesAsync).ConfigureAwait(false);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
await ReportAsync(() => _readError = ex.Message).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
}
// Best-effort: the fault being reported may itself be an InvokeAsync against a
// disposed renderer, in which case the dispatch fails the same way and there is no
// page left to show it on.
private async Task ReportAsync(Action update)
{
try
{
await InvokeAsync(() =>
{
update();
StateHasChanged();
}).ConfigureAwait(false);
}
catch
{
}
}
@@ -445,7 +617,7 @@
}
string[] tags = [.. _subscribed];
DashboardLiveReadResult result = await LiveData.ReadAsync(tags, _cts.Token);
DashboardLiveReadResult result = await LiveData.ReadAsync(tags, _viewerId, _cts.Token);
_readError = result.Error;
_workerPid = result.WorkerProcessId;
DateTimeOffset now = DateTimeOffset.UtcNow;
@@ -454,7 +626,7 @@
// Stamp the observed-change time only when the value/quality
// signature actually changes, so the Updated column does not
// tick on every poll for a static tag.
string signature = $"{value.ValueText}{value.Quality}{value.Ok}";
string signature = $"{value.ValueText}{value.Quality}{value.Ok}";
if (!_valueSignature.TryGetValue(value.TagAddress, out string? previous)
|| previous != signature)
{
@@ -471,31 +643,49 @@
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await _cts.CancelAsync();
await _deployCts.CancelAsync();
if (_pollTask is not null)
try
{
try
{
await _pollTask;
}
catch (OperationCanceledException)
{
}
await _cts.CancelAsync().ConfigureAwait(false);
await _deployCts.CancelAsync().ConfigureAwait(false);
// Drained together, not in sequence: a wedged dispatcher blocks every loop at
// once, so sequential drains would multiply the bound. DrainAsync tolerates null.
await Task.WhenAll(DrainAsync(_pollTask), DrainAsync(_deployTask), DrainAsync(_searchTask))
.ConfigureAwait(false);
}
if (_deployTask is not null)
finally
{
try
{
await _deployTask;
}
catch (OperationCanceledException)
{
}
// Always released, even if cancellation or a drain throws. A search still in
// flight disposes its own linked source.
_cts.Dispose();
_deployCts.Dispose();
GC.SuppressFinalize(this);
}
}
// The accepted cost of the bound is an abandoned loop that keeps its timer or
// notifier subscription until it does unwind; waiting forever on a dispatcher that
// is not draining would wedge the circuit teardown itself.
private static async Task DrainAsync(Task? loop)
{
if (loop is null)
{
return;
}
_cts.Dispose();
_deployCts.Dispose();
GC.SuppressFinalize(this);
try
{
await loop.WaitAsync(LoopDrainTimeout).ConfigureAwait(false);
}
catch (TimeoutException)
{
}
catch (OperationCanceledException)
{
}
catch
{
// Other disposal-time errors are best-effort.
}
}
}
@@ -10,7 +10,7 @@
"Ready" or "Healthy" or "Active" => StatusState.Ok,
"Creating" or "StartingWorker" or "WaitingForPipe" or "InitializingWorker" or "Closing"
or "Stale" or "Degraded" or "Expiring" => StatusState.Warn,
"Faulted" or "Unavailable" or "Expired" => StatusState.Bad,
"Faulted" or "Unavailable" or "Expired" or "Unhealthy" => StatusState.Bad,
_ => StatusState.Idle,
};
}
@@ -0,0 +1,125 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>The Alarms page's filter inputs, compared by value to decide whether a filtered view is stale.</summary>
/// <param name="ShowActive">Include unacknowledged <c>Active</c> alarms.</param>
/// <param name="ShowAcked">Include <c>ActiveAcked</c> alarms.</param>
/// <param name="Area">Area to match (case-insensitive), or empty for all.</param>
/// <param name="MinSeverity">Lowest severity shown.</param>
/// <param name="MaxSeverity">Highest severity shown.</param>
/// <param name="Search">Reference/source/description text search; trimmed before matching.</param>
public sealed record DashboardAlarmFilterCriteria(
bool ShowActive,
bool ShowAcked,
string Area,
int MinSeverity,
int MaxSeverity,
string Search);
/// <summary>
/// Memoized projections of the Alarms page's alarm list: the filtered, sorted rows and
/// the distinct area list. A render used to run the full filter-and-sort twice (the
/// "Showing" card and the table) and the area projection once more, on every render —
/// including the renders that typing in the search box triggers per keystroke. Each
/// projection is now recomputed only when the alarm list (its generation) or, for the
/// rows, the filter criteria change.
/// </summary>
/// <remarks>
/// Owned by one page instance and touched only from its render dispatcher, so it is not
/// synchronized.
/// </remarks>
public sealed class DashboardAlarmView
{
private long _filteredGeneration = -1;
private DashboardAlarmFilterCriteria? _filteredCriteria;
private IReadOnlyList<DashboardActiveAlarm> _filtered = [];
private long _areasGeneration = -1;
private IReadOnlyList<string> _areas = [];
/// <summary>Gets how many times the filtered rows were actually computed (diagnostic, for tests).</summary>
internal int FilterComputations { get; private set; }
/// <summary>Gets how many times the area list was actually computed (diagnostic, for tests).</summary>
internal int AreaComputations { get; private set; }
/// <summary>Returns the alarms matching <paramref name="criteria"/>, highest severity then newest first.</summary>
/// <param name="generation">
/// Version of <paramref name="alarms"/>; the caller bumps it whenever it replaces the list's contents.
/// </param>
/// <param name="criteria">The current filter inputs.</param>
/// <param name="alarms">The full active-alarm list.</param>
/// <returns>The filtered rows; the same instance while generation and criteria are unchanged.</returns>
public IReadOnlyList<DashboardActiveAlarm> Filtered(
long generation,
DashboardAlarmFilterCriteria criteria,
IReadOnlyList<DashboardActiveAlarm> alarms)
{
ArgumentNullException.ThrowIfNull(criteria);
ArgumentNullException.ThrowIfNull(alarms);
if (generation == _filteredGeneration && criteria == _filteredCriteria)
{
return _filtered;
}
_filtered = Apply(criteria, alarms);
_filteredGeneration = generation;
_filteredCriteria = criteria;
FilterComputations++;
return _filtered;
}
/// <summary>Returns the distinct non-blank areas of <paramref name="alarms"/>, sorted case-insensitively.</summary>
/// <param name="generation">Version of <paramref name="alarms"/>.</param>
/// <param name="alarms">The full active-alarm list.</param>
/// <returns>The area list; the same instance while the generation is unchanged.</returns>
public IReadOnlyList<string> Areas(long generation, IReadOnlyList<DashboardActiveAlarm> alarms)
{
ArgumentNullException.ThrowIfNull(alarms);
if (generation == _areasGeneration)
{
return _areas;
}
_areas = alarms
.Select(alarm => alarm.Area)
.Where(area => !string.IsNullOrWhiteSpace(area))
.Distinct(StringComparer.OrdinalIgnoreCase)
.OrderBy(area => area, StringComparer.OrdinalIgnoreCase)
.ToArray();
_areasGeneration = generation;
AreaComputations++;
return _areas;
}
private static DashboardActiveAlarm[] Apply(
DashboardAlarmFilterCriteria criteria,
IReadOnlyList<DashboardActiveAlarm> alarms)
{
string query = criteria.Search.Trim();
return alarms
.Where(alarm => MatchesState(criteria, alarm))
.Where(alarm => criteria.Area.Length == 0
|| string.Equals(alarm.Area, criteria.Area, StringComparison.OrdinalIgnoreCase))
.Where(alarm => alarm.Severity >= criteria.MinSeverity && alarm.Severity <= criteria.MaxSeverity)
.Where(alarm => query.Length == 0
|| alarm.Reference.Contains(query, StringComparison.OrdinalIgnoreCase)
|| alarm.Source.Contains(query, StringComparison.OrdinalIgnoreCase)
|| alarm.Description.Contains(query, StringComparison.OrdinalIgnoreCase))
.OrderByDescending(alarm => alarm.Severity)
.ThenByDescending(alarm => alarm.LastTransition ?? DateTimeOffset.MinValue)
.ToArray();
}
private static bool MatchesState(DashboardAlarmFilterCriteria criteria, DashboardActiveAlarm alarm)
{
return alarm.State switch
{
AlarmConditionState.Active => criteria.ShowActive,
AlarmConditionState.ActiveAcked => criteria.ShowAcked,
_ => true,
};
}
}
@@ -0,0 +1,117 @@
using ZB.MOM.WW.GalaxyRepository;
using ZB.MOM.WW.GalaxyRepository.Grpc;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>
/// Flat, pre-lowercased attribute index behind the Browse page's filter box, shared by
/// every circuit and rebuilt only when the Galaxy cache sequence changes.
/// </summary>
/// <remarks>
/// The filter used to walk every object and attribute of the cached Galaxy, doing two
/// case-insensitive <c>Contains</c> per attribute, synchronously in the property setter
/// bound to <c>oninput</c> — so a large Galaxy was rescanned on the render dispatcher
/// once per keystroke, per circuit. The page now debounces and runs the search off the
/// dispatcher; this index makes each search a single pass of ordinal substring checks
/// over strings lowercased once per Galaxy deploy rather than once per comparison.
/// Lowercasing is invariant-culture, which matches the previous
/// <see cref="StringComparison.OrdinalIgnoreCase"/> behaviour for the ASCII tag names
/// Galaxy uses.
/// </remarks>
public sealed class DashboardBrowseSearchIndex
{
// How often a scan checks its cancellation token. A superseded keystroke stops
// within this many attributes rather than finishing the whole Galaxy.
private const int CancellationCheckInterval = 1024;
private IndexSnapshot? _snapshot;
private int _buildCount;
/// <summary>Gets how many times the index has been (re)built (diagnostic, for tests).</summary>
internal int BuildCount => Volatile.Read(ref _buildCount);
/// <summary>
/// Returns up to <paramref name="limit"/> attributes whose full tag reference or
/// attribute name contains <paramref name="rawQuery"/>, case-insensitively, in
/// Galaxy object order.
/// </summary>
/// <param name="entry">The current Galaxy cache entry; its sequence keys the index.</param>
/// <param name="rawQuery">The filter text as typed; it is trimmed.</param>
/// <param name="limit">Maximum number of matches.</param>
/// <param name="cancellationToken">Cancelled when a newer query supersedes this one.</param>
/// <returns>The matches; empty for a blank query.</returns>
/// <exception cref="OperationCanceledException"><paramref name="cancellationToken"/> was cancelled.</exception>
public IReadOnlyList<GalaxyAttribute> Search(
GalaxyHierarchyCacheEntry entry,
string rawQuery,
int limit,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(entry);
string query = (rawQuery ?? string.Empty).Trim().ToLowerInvariant();
if (query.Length == 0 || limit <= 0)
{
return [];
}
IndexedAttribute[] attributes = GetOrBuild(entry, cancellationToken);
List<GalaxyAttribute> matches = [];
for (int i = 0; i < attributes.Length; i++)
{
if (i % CancellationCheckInterval == 0)
{
cancellationToken.ThrowIfCancellationRequested();
}
IndexedAttribute candidate = attributes[i];
if (candidate.FullTagReference.Contains(query, StringComparison.Ordinal)
|| candidate.AttributeName.Contains(query, StringComparison.Ordinal))
{
matches.Add(candidate.Attribute);
if (matches.Count >= limit)
{
break;
}
}
}
return matches;
}
private IndexedAttribute[] GetOrBuild(GalaxyHierarchyCacheEntry entry, CancellationToken cancellationToken)
{
IndexSnapshot? current = Volatile.Read(ref _snapshot);
if (current is not null && current.Sequence == entry.Sequence)
{
return current.Attributes;
}
// A racing build for the same sequence is harmless: the projection is pure, so
// either winner stores equivalent content.
List<IndexedAttribute> attributes = [];
foreach (GalaxyObject galaxyObject in entry.Objects)
{
cancellationToken.ThrowIfCancellationRequested();
foreach (GalaxyAttribute attribute in galaxyObject.Attributes)
{
attributes.Add(new IndexedAttribute(
attribute.FullTagReference.ToLowerInvariant(),
attribute.AttributeName.ToLowerInvariant(),
attribute));
}
}
IndexSnapshot built = new(entry.Sequence, [.. attributes]);
Volatile.Write(ref _snapshot, built);
Interlocked.Increment(ref _buildCount);
return built.Attributes;
}
private sealed record IndexSnapshot(long Sequence, IndexedAttribute[] Attributes);
private readonly record struct IndexedAttribute(
string FullTagReference,
string AttributeName,
GalaxyAttribute Attribute);
}
@@ -0,0 +1,42 @@
using Microsoft.AspNetCore.Components.Server.Circuits;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>
/// Per-circuit view of whether the browser is still attached. Registered scoped, so a
/// page injecting it sees its own circuit, and also registered as that circuit's
/// <see cref="CircuitHandler"/> so Blazor drives it.
/// </summary>
/// <remarks>
/// A closed or backgrounded browser tab leaves its circuit alive, disconnected, for
/// <see cref="DashboardServiceCollectionExtensions.DisconnectedCircuitRetention"/>
/// before Blazor disposes it, and every timer on the page keeps running in that
/// window. The Browse page's two-second <c>ReadBulk</c> poll is the costly one — it
/// drives the shared dashboard worker — so it skips its tick while
/// <see cref="IsConnected"/> is false and resumes on reconnect.
/// </remarks>
public sealed class DashboardCircuitConnectionTracker : CircuitHandler
{
private volatile bool _connected = true;
/// <summary>
/// Gets a value indicating whether the circuit's browser connection is up. True
/// until Blazor reports the connection down, and also outside a circuit (prerender),
/// where no handler callbacks ever arrive.
/// </summary>
public bool IsConnected => _connected;
/// <inheritdoc />
public override Task OnConnectionUpAsync(Circuit circuit, CancellationToken cancellationToken)
{
_connected = true;
return Task.CompletedTask;
}
/// <inheritdoc />
public override Task OnConnectionDownAsync(Circuit circuit, CancellationToken cancellationToken)
{
_connected = false;
return Task.CompletedTask;
}
}
@@ -34,20 +34,38 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
//
// The bound is per-read, not absolute: a read may never evict a tag it is itself
// about to return, so a single read of more distinct tags than the cap leaves the
// set that large. The invariant EvictForAsync actually maintains is
// set that large. With a single viewer the invariant EvictForAsync maintains is
//
// |advise set| after a read <= max(MaxSubscribedTags, distinct tags in that read)
//
// and any overshoot is squeezed back out by the next read that subscribes a tag
// (with several active viewers the per-viewer share below can hold it higher, by
// at most what the other viewers hold within their shares) and any overshoot is squeezed back out by the next read that subscribes a tag
// (see EvictForAsync). A browse page requests far fewer tags than the cap, so in
// practice the set settles at MaxSubscribedTags.
private const int MaxSubscribedTags = 256;
// Per-viewer soft cap: MaxSubscribedTags is shared by every active viewer (Browse
// circuit), so each owns at most MaxSubscribedTags / activeViewers tags before its
// own least-recently-read tags become the first eviction candidates — never fewer
// than this floor. Without the split, two circuits browsing different areas
// thrashed the single global LRU: each read evicted the other circuit's whole set,
// which then re-advised it two seconds later.
private const int MinTagsPerViewer = 32;
// A viewer that has not read within this window no longer counts as active: its
// share is released and its tags become the first eviction candidates. Generous
// against the Browse page's 2 s poll, so a briefly paused circuit keeps its share.
private static readonly TimeSpan ActiveViewerWindow = TimeSpan.FromSeconds(30);
// Viewer id for callers that do not identify themselves.
private const string SharedViewerId = "";
private static readonly TimeSpan ReadTimeout = TimeSpan.FromSeconds(5);
private readonly ISessionManager _sessionManager;
private readonly IGatewayAlarmService _alarmService;
private readonly ILogger<DashboardLiveDataService> _logger;
private readonly TimeProvider _timeProvider;
private readonly SemaphoreSlim _gate = new(1, 1);
/// <summary>
@@ -70,6 +88,10 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
private readonly LinkedList<SubscribedTag> _recency = new();
// Per-viewer ownership and activity. A tag is owned by the viewer that read it
// last; only touched under _gate.
private readonly Dictionary<string, ViewerState> _viewers = new(StringComparer.Ordinal);
private GatewaySession? _session;
private int _serverHandle;
private bool _disposed;
@@ -79,22 +101,31 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
/// <param name="alarmService">Gateway central alarm service.</param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Diagnostic logger.</param>
/// <param name="timeProvider">Clock for viewer activity; defaults to system time.</param>
public DashboardLiveDataService(
ISessionManager sessionManager,
IGatewayAlarmService alarmService,
IOptions<GatewayOptions> options,
ILogger<DashboardLiveDataService> logger)
ILogger<DashboardLiveDataService> logger,
TimeProvider? timeProvider = null)
{
ArgumentNullException.ThrowIfNull(options);
_sessionManager = sessionManager ?? throw new ArgumentNullException(nameof(sessionManager));
_alarmService = alarmService ?? throw new ArgumentNullException(nameof(alarmService));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_showTagValues = options.Value.Dashboard.ShowTagValues;
_timeProvider = timeProvider ?? TimeProvider.System;
}
/// <inheritdoc />
public Task<DashboardLiveReadResult> ReadAsync(
IReadOnlyCollection<string> tagAddresses,
CancellationToken cancellationToken) => ReadAsync(tagAddresses, viewerId: null, cancellationToken);
/// <inheritdoc />
public async Task<DashboardLiveReadResult> ReadAsync(
IReadOnlyCollection<string> tagAddresses,
string? viewerId,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(tagAddresses);
@@ -109,15 +140,19 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
(GatewaySession session, int serverHandle) = await EnsureReadyAsync(cancellationToken)
.ConfigureAwait(false);
string[] toSubscribe = TouchAndCollectNewTags(tagAddresses, out int justReadCount);
string viewer = viewerId ?? SharedViewerId;
DateTimeOffset now = _timeProvider.GetUtcNow();
TouchViewer(viewer, now);
string[] toSubscribe = TouchAndCollectNewTags(tagAddresses, viewer, out HashSet<string> justRead);
if (toSubscribe.Length > 0)
{
await EvictForAsync(session, serverHandle, toSubscribe.Length, justReadCount, cancellationToken)
await EvictForAsync(session, serverHandle, toSubscribe.Length, justRead, viewer, now, cancellationToken)
.ConfigureAwait(false);
IReadOnlyList<SubscribeResult> subscribeResults = await session
.SubscribeBulkAsync(serverHandle, toSubscribe, cancellationToken)
.ConfigureAwait(false);
TrackSubscribed(toSubscribe, subscribeResults);
TrackSubscribed(toSubscribe, subscribeResults, viewer);
}
IReadOnlyList<BulkReadResult> results = await session
@@ -175,18 +210,20 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
}
// Promotes every already-advised tag in this read to the front of the recency
// list and returns the tags that still need subscribing (distinct, in request
// order). `justReadCount` is how many distinct tags of this read were already
// advised — they now occupy the front of the list and must never be evicted to
// make room for the same read's new tags. Callers must hold _gate.
// list, hands its ownership to the reading viewer, and returns the tags that
// still need subscribing (distinct, in request order). `justRead` is every
// distinct tag of this read — none of them may be evicted to make room for the
// same read's new tags. Callers must hold _gate.
//
// Every tag of one read is equally recently read; the recency list needs a total
// order anyway, so the whole service uses one tie-break: later in the request wins.
// Promoting in request order gives that here, and TrackSubscribed inserts new tags
// the same way.
private string[] TouchAndCollectNewTags(IReadOnlyCollection<string> tagAddresses, out int justReadCount)
private string[] TouchAndCollectNewTags(
IReadOnlyCollection<string> tagAddresses,
string viewer,
out HashSet<string> justRead)
{
int touched = 0;
List<string> toSubscribe = [];
HashSet<string> seen = new(StringComparer.OrdinalIgnoreCase);
@@ -200,10 +237,14 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
_recency.AddFirst(node);
}
if (seen.Add(tag))
if (!string.Equals(node.Value.Owner, viewer, StringComparison.Ordinal))
{
touched++;
AdjustOwned(node.Value.Owner, -1);
AdjustOwned(viewer, +1);
node.Value = node.Value with { Owner = viewer };
}
seen.Add(tag);
}
else if (seen.Add(tag))
{
@@ -211,7 +252,7 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
}
}
justReadCount = touched;
justRead = seen;
return [.. toSubscribe];
}
@@ -221,7 +262,17 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
// from tracking regardless, and the session-invalidation path already handles
// gateway/worker drift. Callers must hold _gate.
//
// Eviction stops at the tags this read just touched (`justReadCount`), so a read
// Victims are chosen in two passes, each least-recently-read first:
// 1. tags of OTHER viewers that are over their share (PerViewerCap; an
// inactive viewer's share is zero);
// 2. the reading viewer's own tags that this read did not touch.
// A viewer within its share is never evicted by someone else's read, so the
// cap is soft: when every other viewer sits within its share and the reader's
// own older tags run out, the set ends over MaxSubscribedTags rather than
// stripping a neighbour. With one viewer, pass 1 is empty and pass 2 is the
// plain global LRU.
//
// Eviction never touches the tags this read just touched (`justRead`), so a read
// whose own distinct tags outnumber the cap ends over it — see MaxSubscribedTags
// for the exact invariant. That overshoot is not sticky: the next read that
// subscribes anything computes `overflow` against the oversized set and evicts the
@@ -236,26 +287,56 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
GatewaySession session,
int serverHandle,
int incomingCount,
int justReadCount,
HashSet<string> justRead,
string reader,
DateTimeOffset now,
CancellationToken cancellationToken)
{
int overflow = _subscribed.Count + incomingCount - MaxSubscribedTags;
int evictable = _subscribed.Count - justReadCount;
int evictCount = Math.Min(overflow, evictable);
if (evictCount <= 0)
if (overflow <= 0)
{
return;
}
List<int> evictedHandles = new(evictCount);
for (int i = 0; i < evictCount && _recency.Last is { } oldest; i++)
int share = PerViewerCap(now);
List<int> evictedHandles = [];
int evictCount = 0;
// Pass 1: other viewers over their share.
for (LinkedListNode<SubscribedTag>? node = _recency.Last; node is not null && overflow > 0;)
{
_recency.RemoveLast();
_subscribed.Remove(oldest.Value.TagAddress);
if (oldest.Value.ItemHandle != 0)
LinkedListNode<SubscribedTag>? previous = node.Previous;
string owner = node.Value.Owner;
if (!string.Equals(owner, reader, StringComparison.Ordinal)
&& !justRead.Contains(node.Value.TagAddress)
&& OwnedCount(owner) > (IsActive(owner, now) ? share : 0))
{
evictedHandles.Add(oldest.Value.ItemHandle);
Evict(node, evictedHandles);
overflow--;
evictCount++;
}
node = previous;
}
// Pass 2: the reader's own tags this read did not touch.
for (LinkedListNode<SubscribedTag>? node = _recency.Last; node is not null && overflow > 0;)
{
LinkedListNode<SubscribedTag>? previous = node.Previous;
if (string.Equals(node.Value.Owner, reader, StringComparison.Ordinal)
&& !justRead.Contains(node.Value.TagAddress))
{
Evict(node, evictedHandles);
overflow--;
evictCount++;
}
node = previous;
}
if (evictCount == 0)
{
return;
}
_logger.LogDebug(
@@ -282,12 +363,81 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
}
}
// Removes one tag from tracking, releasing its owner's count and collecting its
// item handle for the batch unadvise. Callers must hold _gate.
private void Evict(LinkedListNode<SubscribedTag> node, List<int> evictedHandles)
{
_recency.Remove(node);
_subscribed.Remove(node.Value.TagAddress);
AdjustOwned(node.Value.Owner, -1);
if (node.Value.ItemHandle != 0)
{
evictedHandles.Add(node.Value.ItemHandle);
}
}
// Each active viewer's share of MaxSubscribedTags, floored at MinTagsPerViewer.
// Callers must hold _gate.
private int PerViewerCap(DateTimeOffset now)
{
int active = _viewers.Values.Count(state => now - state.LastReadAt <= ActiveViewerWindow);
return Math.Max(MinTagsPerViewer, MaxSubscribedTags / Math.Max(1, active));
}
private bool IsActive(string viewer, DateTimeOffset now) =>
_viewers.TryGetValue(viewer, out ViewerState? state) && now - state.LastReadAt <= ActiveViewerWindow;
private int OwnedCount(string viewer) =>
_viewers.TryGetValue(viewer, out ViewerState? state) ? state.OwnedCount : 0;
// Records a read by `viewer` and forgets viewers that are both inactive and own
// nothing, so the map does not grow one entry per Browse page ever opened.
// Callers must hold _gate.
private void TouchViewer(string viewer, DateTimeOffset now)
{
if (!_viewers.TryGetValue(viewer, out ViewerState? state))
{
state = new ViewerState();
_viewers[viewer] = state;
}
state.LastReadAt = now;
foreach (KeyValuePair<string, ViewerState> entry in _viewers.ToArray())
{
if (entry.Value.OwnedCount <= 0 && now - entry.Value.LastReadAt > ActiveViewerWindow)
{
_viewers.Remove(entry.Key);
}
}
}
// Callers must hold _gate.
private void AdjustOwned(string viewer, int delta)
{
if (!_viewers.TryGetValue(viewer, out ViewerState? state))
{
if (delta <= 0)
{
return;
}
state = new ViewerState { LastReadAt = DateTimeOffset.MinValue };
_viewers[viewer] = state;
}
state.OwnedCount = Math.Max(0, state.OwnedCount + delta);
}
// Records the freshly advised tags as the most recently read, keeping each
// tag's item handle so eviction can unadvise it. Tags the worker failed to
// advise are still tracked (matching the pre-cap behaviour of not retrying
// them on every read) but carry no handle, so eviction just forgets them.
// Callers must hold _gate.
private void TrackSubscribed(IReadOnlyList<string> tagAddresses, IReadOnlyList<SubscribeResult> results)
private void TrackSubscribed(
IReadOnlyList<string> tagAddresses,
IReadOnlyList<SubscribeResult> results,
string viewer)
{
Dictionary<string, int> handles = new(results.Count, StringComparer.OrdinalIgnoreCase);
foreach (SubscribeResult result in results)
@@ -303,7 +453,8 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
foreach (string tag in tagAddresses)
{
handles.TryGetValue(tag, out int itemHandle);
_subscribed[tag] = _recency.AddFirst(new SubscribedTag(tag, itemHandle));
_subscribed[tag] = _recency.AddFirst(new SubscribedTag(tag, itemHandle, viewer));
AdjustOwned(viewer, +1);
}
}
@@ -314,6 +465,10 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
{
_subscribed.Clear();
_recency.Clear();
foreach (ViewerState state in _viewers.Values)
{
state.OwnedCount = 0;
}
}
// Returns a Ready session + its Register server handle, opening a fresh
@@ -424,5 +579,14 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
// One entry of the advise set. ItemHandle is the handle the worker bound for
// the tag, or 0 when the subscribe failed and there is nothing to unadvise.
private readonly record struct SubscribedTag(string TagAddress, int ItemHandle);
// Owner is the viewer that read the tag most recently.
private readonly record struct SubscribedTag(string TagAddress, int ItemHandle, string Owner);
// One viewer's slice of the advise set.
private sealed class ViewerState
{
public int OwnedCount { get; set; }
public DateTimeOffset LastReadAt { get; set; }
}
}
@@ -1,6 +1,7 @@
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.Cookies;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components.Server.Circuits;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
@@ -16,6 +17,16 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// </summary>
public static class DashboardServiceCollectionExtensions
{
/// <summary>
/// How long a disconnected dashboard circuit is kept for reconnect before Blazor
/// disposes it. The framework default is three minutes, during which a closed Browse
/// tab's page — and its two-second live-read poll against the shared dashboard
/// worker — keeps running. Thirty seconds still rides out a network blip or a
/// laptop lid; a longer outage reloads the page, which costs nothing here because
/// every page re-seeds from in-process state.
/// </summary>
public static readonly TimeSpan DisconnectedCircuitRetention = TimeSpan.FromSeconds(30);
/// <summary>
/// Registers all dashboard services, authentication, and Razor components.
/// </summary>
@@ -54,6 +65,14 @@ public static class DashboardServiceCollectionExtensions
// nothing in this process builds a hub connection or needs a token for one.
services.AddSingleton<HubTokenService>();
services.AddScoped<IDashboardBrowseService, DashboardBrowseService>();
// Singleton: the lowercased attribute index is keyed on the Galaxy cache sequence
// and shared by every circuit's Browse filter, so it is built once per deploy.
services.AddSingleton<DashboardBrowseSearchIndex>();
// Scoped (one per circuit) and registered as that circuit's CircuitHandler, so the
// Browse page can pause its live-read poll while its browser is disconnected.
services.AddScoped<DashboardCircuitConnectionTracker>();
services.AddScoped<CircuitHandler>(
static provider => provider.GetRequiredService<DashboardCircuitConnectionTracker>());
// Singleton: EventsHub instances are transient (one per hub invocation), so the
// subscriber bookkeeping they share with the broadcaster must outlive them.
services.AddSingleton<Hubs.EventsHubViewerRegistry>();
@@ -76,7 +95,8 @@ public static class DashboardServiceCollectionExtensions
services.AddAntiforgery();
services.AddCascadingAuthenticationState();
services.AddRazorComponents()
.AddInteractiveServerComponents();
.AddInteractiveServerComponents(circuit =>
circuit.DisconnectedCircuitRetentionPeriod = DisconnectedCircuitRetention);
services.AddSignalR();
// DEV/TEST ONLY. Read directly from configuration here because authentication scheme
@@ -1,7 +1,33 @@
using System.Text.Json.Serialization;
using ZB.MOM.WW.MxGateway.Server.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>
/// One dashboard refresh: gateway status, sessions, workers, metrics, faults, API
/// keys, the effective configuration, and the Galaxy summary.
/// </summary>
/// <remarks>
/// <see cref="Configuration"/> is excluded from JSON serialization. It is
/// startup-static, yet it is the largest part of the record, and the only wire this
/// record crosses is the <c>/hubs/snapshot</c> tick sent every snapshot interval.
/// Remote clients receive it once per connection instead, as
/// <see cref="Hubs.DashboardSnapshotHub.ConfigurationMessage"/> (or on demand via
/// <see cref="Hubs.DashboardSnapshotHub.GetConfiguration"/>). In-process pages read the
/// property directly and are unaffected.
/// </remarks>
/// <param name="GeneratedAt">When the snapshot was built.</param>
/// <param name="GatewayStartedAt">When the gateway process started.</param>
/// <param name="GatewayUptime">Elapsed time since start.</param>
/// <param name="GatewayStatus">Aggregate health-check status, or <c>Unknown</c>.</param>
/// <param name="GatewayVersion">Server assembly version.</param>
/// <param name="Sessions">Most recent sessions, newest first.</param>
/// <param name="Workers">Workers that are not closed.</param>
/// <param name="Metrics">Gateway counters and gauges.</param>
/// <param name="Faults">Recent session and worker faults.</param>
/// <param name="ApiKeys">API key summaries (no secrets).</param>
/// <param name="Configuration">Effective configuration; not serialized on the hub tick.</param>
/// <param name="Galaxy">Galaxy hierarchy cache summary.</param>
public sealed record DashboardSnapshot(
DateTimeOffset GeneratedAt,
DateTimeOffset GatewayStartedAt,
@@ -13,5 +39,5 @@ public sealed record DashboardSnapshot(
IReadOnlyList<DashboardMetricSummary> Metrics,
IReadOnlyList<DashboardFaultSummary> Faults,
IReadOnlyList<DashboardApiKeySummary> ApiKeys,
EffectiveGatewayConfiguration Configuration,
[property: JsonIgnore] EffectiveGatewayConfiguration Configuration,
DashboardGalaxySummary Galaxy);
@@ -1,4 +1,5 @@
using System.Runtime.CompilerServices;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
@@ -14,7 +15,16 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
public sealed class DashboardSnapshotService : IDashboardSnapshotService
{
private const string HealthyStatus = "Healthy";
/// <summary>
/// Status shown before the first health evaluation completes, when the health
/// evaluation fails, and when no <see cref="HealthCheckService"/> is registered.
/// Deliberately not "Healthy": claiming health that was never checked is how the
/// dashboard used to show a green gateway status throughout an outage.
/// </summary>
internal const string UnknownStatus = "Unknown";
/// <summary>Upper bound on one health evaluation, so a hung check cannot stall the snapshot tick.</summary>
private static readonly TimeSpan GatewayStatusRefreshTimeout = TimeSpan.FromSeconds(2);
/// <summary>
/// Minimum spacing between API key list reads. The list is a SQLite query whose
@@ -39,7 +49,10 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
private readonly int _recentFaultLimit;
private readonly int _recentSessionLimit;
private readonly ILogger<DashboardSnapshotService> _logger;
private readonly HealthCheckService? _healthCheckService;
private readonly SemaphoreSlim _apiKeySummaryRefreshGate = new(1, 1);
private readonly SemaphoreSlim _gatewayStatusRefreshGate = new(1, 1);
private string _gatewayStatus = UnknownStatus;
private IReadOnlyList<DashboardApiKeySummary> _apiKeySummaries = Array.Empty<DashboardApiKeySummary>();
private long _apiKeySummariesRefreshedAtTicks = NeverRefreshedTicks;
// The effective configuration is built from IOptions<GatewayOptions> and is startup-static:
@@ -65,6 +78,11 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
/// <param name="options">Gateway configuration options.</param>
/// <param name="timeProvider">Provider for current time; defaults to system time.</param>
/// <param name="logger">Optional logger for the dashboard snapshot service.</param>
/// <param name="healthCheckService">
/// The gateway's registered health checks. <see cref="DashboardSnapshot.GatewayStatus"/>
/// is the aggregate status of one evaluation per snapshot tick; without it the
/// status stays <see cref="UnknownStatus"/>.
/// </param>
public DashboardSnapshotService(
ISessionRegistry sessionRegistry,
GatewayMetrics metrics,
@@ -73,7 +91,8 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
IApiKeyAdminStore apiKeyAdminStore,
IOptions<GatewayOptions> options,
TimeProvider? timeProvider = null,
ILogger<DashboardSnapshotService>? logger = null)
ILogger<DashboardSnapshotService>? logger = null,
HealthCheckService? healthCheckService = null)
{
_sessionRegistry = sessionRegistry ?? throw new ArgumentNullException(nameof(sessionRegistry));
_metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
@@ -88,6 +107,7 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
_recentFaultLimit = options.Value.Dashboard.RecentFaultLimit;
_recentSessionLimit = options.Value.Dashboard.RecentSessionLimit;
_logger = logger ?? NullLogger<DashboardSnapshotService>.Instance;
_healthCheckService = healthCheckService;
}
/// <inheritdoc />
@@ -111,7 +131,7 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
GeneratedAt: generatedAt,
GatewayStartedAt: _gatewayStartedAt,
GatewayUptime: generatedAt - _gatewayStartedAt,
GatewayStatus: HealthyStatus,
GatewayStatus: Volatile.Read(ref _gatewayStatus),
GatewayVersion: typeof(DashboardSnapshotService).Assembly.GetName().Version?.ToString() ?? "unknown",
Sessions: sessionSummaries,
Workers: workerSummaries,
@@ -171,6 +191,7 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
}
await RefreshApiKeySummariesAsync(cancellationToken).ConfigureAwait(false);
await RefreshGatewayStatusAsync(cancellationToken).ConfigureAwait(false);
yield return GetSnapshot();
using PeriodicTimer timer = new(_snapshotInterval, _timeProvider);
@@ -192,10 +213,57 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
}
await RefreshApiKeySummariesAsync(cancellationToken).ConfigureAwait(false);
await RefreshGatewayStatusAsync(cancellationToken).ConfigureAwait(false);
yield return GetSnapshot();
}
}
/// <summary>
/// Evaluates the registered health checks once and caches the aggregate status
/// for <see cref="GetSnapshot"/>, which is synchronous and also serves page seeds
/// and hub connects between ticks. One evaluation per tick, not per
/// <see cref="GetSnapshot"/> call: the checks include an auth-store query, and the
/// tick is already the cadence at which the dashboard refreshes.
/// </summary>
/// <param name="cancellationToken">Token to cancel the evaluation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
private async Task RefreshGatewayStatusAsync(CancellationToken cancellationToken)
{
if (_healthCheckService is null)
{
return;
}
// Zero-timeout gate: a concurrent enumeration (the hub publisher and the page
// feed each run one) skips rather than queueing a second evaluation.
if (!await _gatewayStatusRefreshGate.WaitAsync(0, cancellationToken).ConfigureAwait(false))
{
return;
}
try
{
using CancellationTokenSource timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeout.CancelAfter(GatewayStatusRefreshTimeout);
HealthReport report = await _healthCheckService.CheckHealthAsync(timeout.Token).ConfigureAwait(false);
Volatile.Write(ref _gatewayStatus, report.Status.ToString());
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception exception)
{
// A health evaluation that cannot complete is not evidence of health.
Volatile.Write(ref _gatewayStatus, UnknownStatus);
_logger.LogDebug(exception, "Dashboard gateway health evaluation failed.");
}
finally
{
_gatewayStatusRefreshGate.Release();
}
}
private static DashboardSessionSummary CreateSessionSummary(
GatewaySession session,
GatewayMetricsSnapshot metricsSnapshot)
@@ -0,0 +1,14 @@
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Payload of the <see cref="AlarmsHub.ResetMessage"/> frame: the publisher has
/// (re)subscribed to the central alarm monitor, which primes the subscription with
/// a full active-alarm snapshot. A client discards its accumulated alarm state on
/// receipt and rebuilds it from the <c>active_alarm</c> rows up to the next
/// <c>snapshot_complete</c>.
/// </summary>
/// <param name="Generation">
/// Monotonic subscription counter for this gateway process, starting at 1. Lets a
/// client tell a resubscribe from a duplicate delivery.
/// </param>
public sealed record AlarmFeedReset(long Generation);
@@ -23,11 +23,19 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
[Authorize(Policy = DashboardAuthenticationDefaults.AlarmsPolicy)]
public sealed class AlarmsHub : Hub
{
/// <summary>Group every admitted connection joins on connect.</summary>
public const string AllAlarmsGroup = "__alarms__";
/// <summary>Method name used to push <c>AlarmFeedMessage</c> values to clients.</summary>
public const string AlarmMessage = "AlarmFeed";
/// <summary>
/// Method name of the <see cref="AlarmFeedReset"/> marker sent ahead of every
/// (re)subscription to the central monitor. A client discards its accumulated
/// alarm state on receipt and rebuilds it from the snapshot that follows.
/// </summary>
public const string ResetMessage = "AlarmFeedReset";
/// <inheritdoc />
public override async Task OnConnectedAsync()
{
@@ -31,18 +31,69 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// narrowing this loop could apply even in principle. Adding a filter here would imply a
/// per-caller scope the feed does not carry.
/// </para>
/// <para>
/// Each broadcast is time-bounded (<see cref="DefaultSendTimeout"/>). A group send
/// completes only once every member accepted the frame, so awaiting it unbounded
/// let one slow browser stall this loop; the stalled loop stopped draining its
/// monitor subscription, the monitor dropped it as a slow subscriber, and the
/// resubscribe replayed the whole active-alarm snapshot to every client. A send that
/// misses the bound is abandoned and the loop moves on.
/// </para>
/// <para>
/// Every subscription — the first and each resubscribe — is preceded by an
/// <see cref="AlarmsHub.ResetMessage"/> frame. The monitor primes every new
/// subscription with a full snapshot, so a client that keeps accumulating
/// <c>active_alarm</c> rows across a resubscribe would double-count or keep alarms
/// that cleared while the feed was down; the marker tells it to discard its state
/// and rebuild from the snapshot that follows.
/// </para>
/// </remarks>
/// <param name="alarmService">The gateway's central alarm feed.</param>
/// <param name="hubContext">Hub context used to broadcast to the alarms group.</param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort broadcast failures.</param>
public sealed class AlarmsHubPublisher(
IGatewayAlarmService alarmService,
IHubContext<AlarmsHub> hubContext,
IOptions<GatewayOptions> options,
ILogger<AlarmsHubPublisher> logger) : BackgroundService
public sealed class AlarmsHubPublisher : BackgroundService
{
private readonly bool _showTagValues = options.Value.Dashboard.ShowTagValues;
/// <summary>Upper bound on one alarm broadcast before it is abandoned.</summary>
internal static readonly TimeSpan DefaultSendTimeout = TimeSpan.FromSeconds(2);
private readonly IGatewayAlarmService _alarmService;
private readonly IHubContext<AlarmsHub> _hubContext;
private readonly ILogger<AlarmsHubPublisher> _logger;
private readonly TimeSpan _sendTimeout;
private readonly bool _showTagValues;
private long _subscriptionGeneration;
/// <summary>Initializes a new instance of the <see cref="AlarmsHubPublisher"/> class.</summary>
/// <param name="alarmService">The gateway's central alarm feed.</param>
/// <param name="hubContext">Hub context used to broadcast to the alarms group.</param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort broadcast failures.</param>
public AlarmsHubPublisher(
IGatewayAlarmService alarmService,
IHubContext<AlarmsHub> hubContext,
IOptions<GatewayOptions> options,
ILogger<AlarmsHubPublisher> logger)
: this(alarmService, hubContext, options, logger, DefaultSendTimeout)
{
}
/// <summary>Initializes a new instance with a custom send timeout (test hook).</summary>
/// <param name="alarmService">The gateway's central alarm feed.</param>
/// <param name="hubContext">Hub context used to broadcast to the alarms group.</param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort broadcast failures.</param>
/// <param name="sendTimeout">Upper bound on one broadcast.</param>
internal AlarmsHubPublisher(
IGatewayAlarmService alarmService,
IHubContext<AlarmsHub> hubContext,
IOptions<GatewayOptions> options,
ILogger<AlarmsHubPublisher> logger,
TimeSpan sendTimeout)
{
ArgumentNullException.ThrowIfNull(options);
_alarmService = alarmService ?? throw new ArgumentNullException(nameof(alarmService));
_hubContext = hubContext ?? throw new ArgumentNullException(nameof(hubContext));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_sendTimeout = sendTimeout;
_showTagValues = options.Value.Dashboard.ShowTagValues;
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
@@ -54,7 +105,16 @@ public sealed class AlarmsHubPublisher(
{
try
{
await foreach (AlarmFeedMessage message in alarmService
// Sent before the subscription opens, so it reaches clients ahead of
// the snapshot the monitor primes the new subscription with.
long generation = Interlocked.Increment(ref _subscriptionGeneration);
await BroadcastAsync(
AlarmsHub.ResetMessage,
new AlarmFeedReset(generation),
stoppingToken)
.ConfigureAwait(false);
await foreach (AlarmFeedMessage message in _alarmService
.StreamAsync(alarmFilterPrefix: null, stoppingToken)
.ConfigureAwait(false))
{
@@ -63,18 +123,8 @@ public sealed class AlarmsHubPublisher(
break;
}
try
{
AlarmFeedMessage outbound = _showTagValues ? message : RedactValues(message);
await hubContext.Clients
.Group(AlarmsHub.AllAlarmsGroup)
.SendAsync(AlarmsHub.AlarmMessage, outbound, stoppingToken)
.ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "Alarm broadcast failed; continuing.");
}
AlarmFeedMessage outbound = _showTagValues ? message : RedactValues(message);
await BroadcastAsync(AlarmsHub.AlarmMessage, outbound, stoppingToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
@@ -83,7 +133,7 @@ public sealed class AlarmsHubPublisher(
}
catch (Exception ex)
{
logger.LogWarning(ex, "Alarm subscription faulted; reconnecting in 5s.");
_logger.LogWarning(ex, "Alarm subscription faulted; reconnecting in 5s.");
try
{
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken).ConfigureAwait(false);
@@ -96,6 +146,40 @@ public sealed class AlarmsHubPublisher(
}
}
/// <summary>
/// Sends one frame to the alarms group within the send timeout. A slow or failed
/// send is logged and dropped so the feed keeps draining; only host shutdown
/// propagates.
/// </summary>
/// <param name="method">Client method name.</param>
/// <param name="argument">The frame payload.</param>
/// <param name="stoppingToken">Host shutdown token.</param>
/// <returns>A task that completes when the send finished or was abandoned.</returns>
private async Task BroadcastAsync(string method, object argument, CancellationToken stoppingToken)
{
try
{
bool sent = await BoundedHubSend.SendAsync(
_hubContext.Clients.Group(AlarmsHub.AllAlarmsGroup),
method,
argument,
_sendTimeout,
stoppingToken)
.ConfigureAwait(false);
if (!sent)
{
_logger.LogDebug(
"Alarm broadcast {Method} exceeded {Timeout} and was abandoned; continuing.",
method,
_sendTimeout);
}
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogWarning(ex, "Alarm broadcast failed; continuing.");
}
}
/// <summary>
/// Produces a copy of <paramref name="source"/> with the alarm value fields
/// cleared, leaving every other field — reference, severity, state, category,
@@ -0,0 +1,52 @@
using Microsoft.AspNetCore.SignalR;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Time-bounded SignalR send shared by the dashboard hub publishers. A group or
/// broadcast send completes only once every target connection has accepted the
/// frame, so one remote viewer with a full transport buffer would otherwise hold
/// the publisher's single drain loop for as long as that viewer stays slow.
/// </summary>
internal static class BoundedHubSend
{
/// <summary>
/// Sends <paramref name="argument"/> as <paramref name="method"/> and waits at most
/// <paramref name="timeout"/> for it. The timeout cancels the send's own token, so
/// the lifetime manager abandons the pending write to the slow connection, and the
/// wait itself is bounded too, so a proxy that ignores the token cannot stall the
/// caller either.
/// </summary>
/// <param name="proxy">Target client proxy (a group or all connections).</param>
/// <param name="method">Client method name.</param>
/// <param name="argument">The single message argument.</param>
/// <param name="timeout">Upper bound on the send.</param>
/// <param name="stoppingToken">Host shutdown token; its cancellation propagates.</param>
/// <returns>
/// <see langword="true"/> when the send completed in time; <see langword="false"/>
/// when it was abandoned at the timeout.
/// </returns>
/// <exception cref="OperationCanceledException"><paramref name="stoppingToken"/> was cancelled.</exception>
public static async Task<bool> SendAsync(
IClientProxy proxy,
string method,
object argument,
TimeSpan timeout,
CancellationToken stoppingToken)
{
using CancellationTokenSource bounded = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
bounded.CancelAfter(timeout);
try
{
await proxy.SendAsync(method, argument, bounded.Token)
.WaitAsync(bounded.Token)
.ConfigureAwait(false);
return true;
}
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
{
return false;
}
}
}
@@ -10,10 +10,11 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// Broadcasts MxEvents to the two dashboard audiences for a session: remote
/// <see cref="EventsHub"/> clients subscribed to the session's group, and
/// in-process subscribers opened through
/// <see cref="IDashboardSessionEventSubscriber.Subscribe"/>. Fire-and-forget: we
/// hand the send to the hub context and return immediately so the source gRPC
/// stream is never blocked. Errors are logged once and dropped — keeping the
/// SignalR mirror best-effort preserves the gRPC contract that exists today.
/// <see cref="IDashboardSessionEventSubscriber.Subscribe"/>. <see cref="Publish"/>
/// never blocks: both audiences are fed through bounded drop-oldest queues, so the
/// source gRPC stream is never held up by a slow dashboard viewer. Errors are
/// logged and dropped — keeping the SignalR mirror best-effort preserves the gRPC
/// contract.
/// </summary>
/// <remarks>
/// When <c>MxGateway:Dashboard:ShowTagValues</c> is false (the default), tag
@@ -26,31 +27,45 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <see cref="EventsHub"/>), while the redaction decides what a permitted
/// subscriber sees — so the value-leak seam stays closed whatever the ACL
/// admits.
/// <para>
/// Hub delivery goes through one <see cref="HubPump"/> per watched session: a
/// <see cref="HubQueueCapacity"/>-deep drop-oldest channel drained by a single task
/// that sends one event at a time, each send time-bounded. An earlier version fired
/// a group send per event and returned, so a slow remote viewer accumulated an
/// unbounded number of pending sends — each holding a cloned event — inside the
/// SignalR lifetime manager. The pump is started by the first event published while
/// a hub connection watches the session and torn down when
/// <see cref="EventsHubViewerRegistry.HubViewersReleased"/> reports the last one gone.
/// </para>
/// </remarks>
/// <param name="hubContext">Hub context used to send to the session's group.</param>
/// <param name="viewerRegistry">
/// Live-subscriber registry consulted before any per-event work is done. Both
/// audiences register here — hub connections by their SignalR connection id,
/// in-process subscriptions by a synthetic one — so the gate stays a single
/// source of truth.
/// </param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort mirror failures.</param>
public sealed class DashboardEventBroadcaster(
IHubContext<EventsHub> hubContext,
EventsHubViewerRegistry viewerRegistry,
IOptions<GatewayOptions> options,
ILogger<DashboardEventBroadcaster> logger) : IDashboardEventBroadcaster, IDashboardSessionEventSubscriber
public sealed class DashboardEventBroadcaster : IDashboardEventBroadcaster, IDashboardSessionEventSubscriber, IDisposable
{
/// <summary>
/// Queue depth per in-process subscriber. The consumer is a Blazor page
/// rendering the newest handful of events, so a burst it cannot keep up with
/// is dropped oldest-first rather than allowed to grow — same best-effort
/// contract the SignalR mirror already has.
/// contract the SignalR mirror has.
/// </summary>
private const int InProcessQueueCapacity = 256;
private readonly bool _showTagValues = options.Value.Dashboard.ShowTagValues;
/// <summary>
/// Queue depth of a session's hub-delivery pump. Same depth and drop policy as
/// the in-process feed: the remote viewer is a live view, so a burst it cannot
/// keep up with loses its oldest events rather than buffering without bound.
/// </summary>
internal const int HubQueueCapacity = 256;
private static readonly TimeSpan DefaultHubSendTimeout = TimeSpan.FromSeconds(5);
private readonly IHubContext<EventsHub> _hubContext;
private readonly EventsHubViewerRegistry _viewerRegistry;
private readonly ILogger<DashboardEventBroadcaster> _logger;
private readonly TimeSpan _hubSendTimeout;
private readonly bool _showTagValues;
private readonly object _syncRoot = new();
@@ -63,6 +78,9 @@ public sealed class DashboardEventBroadcaster(
private readonly Dictionary<string, InProcessSubscription[]> _inProcessSubscribers =
new(StringComparer.Ordinal);
/// <summary>Live hub-delivery pumps per session. Guarded by <see cref="_syncRoot"/>.</summary>
private readonly Dictionary<string, HubPump> _hubPumps = new(StringComparer.Ordinal);
/// <summary>
/// Total live in-process subscribers, read without the lock so the common
/// case — nobody has a session-details page open — never contends on it.
@@ -70,6 +88,61 @@ public sealed class DashboardEventBroadcaster(
/// </summary>
private int _inProcessSubscriberCount;
private bool _disposed;
/// <summary>Initializes a new instance of the <see cref="DashboardEventBroadcaster"/> class.</summary>
/// <param name="hubContext">Hub context used to send to the session's group.</param>
/// <param name="viewerRegistry">
/// Live-subscriber registry consulted before any per-event work is done. Both
/// audiences register here — hub connections by their SignalR connection id,
/// in-process subscriptions by a synthetic one — so the gate stays a single
/// source of truth.
/// </param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort mirror failures.</param>
public DashboardEventBroadcaster(
IHubContext<EventsHub> hubContext,
EventsHubViewerRegistry viewerRegistry,
IOptions<GatewayOptions> options,
ILogger<DashboardEventBroadcaster> logger)
: this(hubContext, viewerRegistry, options, logger, DefaultHubSendTimeout)
{
}
/// <summary>Initializes a new instance with a custom hub send timeout (test hook).</summary>
/// <param name="hubContext">Hub context used to send to the session's group.</param>
/// <param name="viewerRegistry">Live-subscriber registry.</param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort mirror failures.</param>
/// <param name="hubSendTimeout">Upper bound on one group send before it is abandoned.</param>
internal DashboardEventBroadcaster(
IHubContext<EventsHub> hubContext,
EventsHubViewerRegistry viewerRegistry,
IOptions<GatewayOptions> options,
ILogger<DashboardEventBroadcaster> logger,
TimeSpan hubSendTimeout)
{
ArgumentNullException.ThrowIfNull(options);
_hubContext = hubContext ?? throw new ArgumentNullException(nameof(hubContext));
_viewerRegistry = viewerRegistry ?? throw new ArgumentNullException(nameof(viewerRegistry));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_hubSendTimeout = hubSendTimeout;
_showTagValues = options.Value.Dashboard.ShowTagValues;
_viewerRegistry.HubViewersReleased += OnHubViewersReleased;
}
/// <summary>Gets the number of live hub-delivery pumps (one per hub-watched session).</summary>
internal int ActiveHubPumpCount
{
get
{
lock (_syncRoot)
{
return _hubPumps.Count;
}
}
}
/// <inheritdoc />
public void Publish(string sessionId, MxEvent mxEvent)
{
@@ -81,46 +154,129 @@ public sealed class DashboardEventBroadcaster(
// Every session's dashboard-mirror subscriber calls Publish for every event,
// whether or not a browser is on that session's page. Without this gate the
// steady state — no dashboard viewer at all — still paid a deep protobuf
// clone (redaction is on by default) plus a send to an empty SignalR group
// per event. Bail before both.
if (!viewerRegistry.HasViewers(sessionId))
// clone (redaction is on by default) per event. Bail before it.
if (!_viewerRegistry.HasViewers(sessionId))
{
return;
}
MxEvent outbound = _showTagValues ? mxEvent : RedactValues(mxEvent);
// In-process delivery first: it is synchronous, cannot throw, and must not be
// skipped by the early return the hub send's guard clause takes.
DeliverInProcess(sessionId, outbound);
// Wrap the Task acquisition in a try/catch so a hypothetical synchronous throw
// from SendAsync (e.g. an implementation that throws before returning the Task)
// cannot escape Publish. The interface contract is never-throw; fire-and-forget.
Task send;
// In-process-only viewers need no hub send at all; the pump exists only
// while a remote connection is in the session's group.
if (_viewerRegistry.HasHubViewers(sessionId))
{
EnqueueForHub(sessionId, outbound);
}
}
/// <summary>
/// Stops every hub pump and detaches from the viewer registry. The singleton is
/// disposed with the host; a pump's in-flight send is bounded by its timeout.
/// </summary>
public void Dispose()
{
HubPump[] pumps;
lock (_syncRoot)
{
if (_disposed)
{
return;
}
_disposed = true;
pumps = [.. _hubPumps.Values];
_hubPumps.Clear();
}
_viewerRegistry.HubViewersReleased -= OnHubViewersReleased;
foreach (HubPump pump in pumps)
{
pump.Complete();
}
}
/// <summary>
/// Queues an event on the session's hub pump, starting the pump on first use.
/// The write happens under the lock so it cannot land on a pump that a
/// concurrent teardown has already completed.
/// </summary>
/// <param name="sessionId">Session the event belongs to.</param>
/// <param name="outbound">The event as the dashboard should see it.</param>
private void EnqueueForHub(string sessionId, MxEvent outbound)
{
lock (_syncRoot)
{
// Re-check under the lock: the last viewer may have left between the
// caller's unlocked check and here, and its release handler (which takes
// this lock) would then never see a pump created after it ran.
if (_disposed || !_viewerRegistry.HasHubViewers(sessionId))
{
return;
}
if (!_hubPumps.TryGetValue(sessionId, out HubPump? pump))
{
pump = new HubPump(this, sessionId, HubQueueCapacity);
_hubPumps[sessionId] = pump;
}
pump.TryWrite(outbound);
}
}
/// <summary>
/// Tears down a session's hub pump once its last hub viewer has gone. Re-checks
/// the registry under the lock: a viewer that re-subscribed in between keeps the
/// pump, and one that arrives after the teardown gets a fresh pump on the next
/// event.
/// </summary>
/// <param name="sessionId">Session whose last hub viewer left.</param>
private void OnHubViewersReleased(string sessionId)
{
HubPump? pump;
lock (_syncRoot)
{
if (_viewerRegistry.HasHubViewers(sessionId) || !_hubPumps.Remove(sessionId, out pump))
{
return;
}
}
pump.Complete();
}
/// <summary>
/// Sends one event to the session's group, bounded by the hub send timeout.
/// Never throws: a failed or abandoned send is logged and the pump moves on.
/// </summary>
/// <param name="sessionId">Session whose group receives the event.</param>
/// <param name="outbound">The event to send.</param>
/// <returns>A task that completes when the send finished or was abandoned.</returns>
private async Task SendToHubAsync(string sessionId, MxEvent outbound)
{
try
{
send = hubContext.Clients
.Group(EventsHub.GroupName(sessionId))
.SendAsync(EventsHub.EventMessage, outbound);
bool sent = await BoundedHubSend.SendAsync(
_hubContext.Clients.Group(EventsHub.GroupName(sessionId)),
EventsHub.EventMessage,
outbound,
_hubSendTimeout,
CancellationToken.None)
.ConfigureAwait(false);
if (!sent)
{
_logger.LogDebug(
"Dashboard event mirror to session {SessionId} timed out after {Timeout}; continuing.",
sessionId,
_hubSendTimeout);
}
}
catch (Exception ex)
{
logger.LogDebug(ex, "Dashboard event mirror to session {SessionId} threw synchronously.", sessionId);
return;
}
if (!send.IsCompletedSuccessfully)
{
_ = send.ContinueWith(
t =>
{
if (t.Exception is { } ex)
{
logger.LogDebug(ex, "Dashboard event mirror to session {SessionId} failed.", sessionId);
}
},
TaskScheduler.Default);
_logger.LogDebug(ex, "Dashboard event mirror to session {SessionId} failed.", sessionId);
}
}
@@ -133,7 +289,7 @@ public sealed class DashboardEventBroadcaster(
// usable for a subscriber that has no SignalR connection behind it. The
// "inproc-" prefix cannot collide with a SignalR connection id and makes the
// origin obvious in a debugger.
string connectionId = "inproc-" + Guid.NewGuid().ToString("N");
string connectionId = EventsHubViewerRegistry.InProcessConnectionIdPrefix + Guid.NewGuid().ToString("N");
InProcessSubscription subscription = new(this, sessionId, connectionId, InProcessQueueCapacity);
// Register before the subscriber becomes a delivery target, exactly as
@@ -142,7 +298,7 @@ public sealed class DashboardEventBroadcaster(
// Publish's gate still reports the session unwatched, silently dropping events
// it should receive. The cost of this order is at worst a redaction clone that
// reaches nobody for the width of the window.
viewerRegistry.AddViewer(connectionId, sessionId);
_viewerRegistry.AddViewer(connectionId, sessionId);
lock (_syncRoot)
{
@@ -227,12 +383,12 @@ public sealed class DashboardEventBroadcaster(
}
}
viewerRegistry.RemoveViewer(subscription.ConnectionId, subscription.SessionId);
_viewerRegistry.RemoveViewer(subscription.ConnectionId, subscription.SessionId);
// The synthetic connection id is used once and never reconnects, so nothing
// else will ever call ReleaseConnection for it; without this the registry
// would retain an empty per-connection entry per subscription ever opened.
viewerRegistry.ReleaseConnection(subscription.ConnectionId);
_viewerRegistry.ReleaseConnection(subscription.ConnectionId);
}
/// <summary>
@@ -328,4 +484,56 @@ public sealed class DashboardEventBroadcaster(
_channel.Writer.TryComplete();
}
}
/// <summary>
/// One session's hub-delivery pump: a bounded drop-oldest channel and the single
/// task that drains it into the session's SignalR group, one bounded send at a
/// time. The drain is the only reader, so at most one send per session is ever
/// pending in the lifetime manager.
/// </summary>
private sealed class HubPump
{
private readonly Channel<MxEvent> _channel;
/// <summary>Initializes a new instance of the <see cref="HubPump"/> class and starts its drain.</summary>
/// <param name="owner">Broadcaster whose send path the drain uses.</param>
/// <param name="sessionId">Session whose group the pump feeds.</param>
/// <param name="capacity">Queue depth before the oldest queued event is dropped.</param>
internal HubPump(DashboardEventBroadcaster owner, string sessionId, int capacity)
{
_channel = Channel.CreateBounded<MxEvent>(new BoundedChannelOptions(capacity)
{
// DropOldest, not Wait: Publish runs on the gRPC event pipeline and
// must never block, and the newest events are the ones a live view wants.
FullMode = BoundedChannelFullMode.DropOldest,
SingleReader = true,
SingleWriter = false,
});
// Task.Run so the drain never starts inline on the publishing thread.
Completion = Task.Run(() => DrainAsync(owner, sessionId));
}
/// <summary>Gets the drain task; it completes once the channel is completed and emptied.</summary>
internal Task Completion { get; }
/// <summary>Queues an event, dropping the oldest queued one when full. Never blocks.</summary>
/// <param name="mxEvent">The event to queue.</param>
internal void TryWrite(MxEvent mxEvent) => _channel.Writer.TryWrite(mxEvent);
/// <summary>Completes the channel so the drain exits once it is empty.</summary>
internal void Complete() => _channel.Writer.TryComplete();
private async Task DrainAsync(DashboardEventBroadcaster owner, string sessionId)
{
ChannelReader<MxEvent> reader = _channel.Reader;
while (await reader.WaitToReadAsync().ConfigureAwait(false))
{
while (reader.TryRead(out MxEvent? next))
{
await owner.SendToHubAsync(sessionId, next).ConfigureAwait(false);
}
}
}
}
}
@@ -1,13 +1,15 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.SignalR;
using ZB.MOM.WW.MxGateway.Server.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// SignalR hub that pushes a fresh <see cref="DashboardSnapshot"/> on every
/// snapshot refresh. New connections receive the current snapshot
/// immediately via <see cref="OnConnectedAsync"/>; subsequent refreshes are
/// broadcast by <see cref="DashboardSnapshotPublisher"/>.
/// snapshot refresh. New connections receive the effective configuration and the
/// current snapshot immediately via <see cref="OnConnectedAsync"/>; subsequent
/// refreshes are broadcast by <see cref="DashboardSnapshotPublisher"/> without the
/// configuration, which is startup-static and serialized only once per connection.
/// </summary>
/// <remarks>
/// Connections are counted into <see cref="DashboardSnapshotHubConnectionCounter"/>
@@ -22,13 +24,28 @@ public sealed class DashboardSnapshotHub(
/// <summary>Method name used to push snapshot updates to clients.</summary>
public const string SnapshotMessage = "SnapshotUpdated";
/// <summary>
/// Method name of the one-per-connection effective-configuration push. Snapshot
/// ticks omit the configuration, so a client keeps the copy it got here.
/// </summary>
public const string ConfigurationMessage = "ConfigurationUpdated";
/// <summary>
/// Returns the effective gateway configuration — the same object
/// <see cref="ConfigurationMessage"/> carries — for a client that needs it again.
/// </summary>
/// <returns>The effective configuration.</returns>
public EffectiveGatewayConfiguration GetConfiguration() => snapshotService.GetSnapshot().Configuration;
/// <inheritdoc />
public override async Task OnConnectedAsync()
{
// Count the viewer before seeding it, so the publisher resumes its tick
// no later than the first snapshot this connection renders.
connectionCounter.Increment();
await Clients.Caller.SendAsync(SnapshotMessage, snapshotService.GetSnapshot()).ConfigureAwait(false);
DashboardSnapshot snapshot = snapshotService.GetSnapshot();
await Clients.Caller.SendAsync(ConfigurationMessage, snapshot.Configuration).ConfigureAwait(false);
await Clients.Caller.SendAsync(SnapshotMessage, snapshot).ConfigureAwait(false);
await base.OnConnectedAsync().ConfigureAwait(false);
}
@@ -26,11 +26,20 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// connected. While the count is zero the publisher does not advance the snapshot
/// enumerator at all, so the producing iterator stays suspended and builds nothing.
/// </para>
/// <para>
/// Each broadcast is bounded by a send timeout (one second by default, the default
/// snapshot interval). A broadcast completes only once every connection accepted the
/// frame, so awaiting it unbounded let one slow remote client delay every tick for
/// everyone; a send that misses the bound is abandoned and the next tick goes out on
/// time. The tick omits the effective configuration (see
/// <see cref="DashboardSnapshot"/>), so each broadcast is the volatile state only.
/// </para>
/// </remarks>
public sealed class DashboardSnapshotPublisher : BackgroundService
{
private static readonly TimeSpan DefaultReconnectDelay = TimeSpan.FromSeconds(5);
private static readonly TimeSpan DefaultIdlePollInterval = TimeSpan.FromSeconds(1);
private static readonly TimeSpan DefaultSendTimeout = TimeSpan.FromSeconds(1);
private readonly IDashboardSnapshotService _snapshotService;
private readonly IHubContext<DashboardSnapshotHub> _hubContext;
@@ -38,6 +47,7 @@ public sealed class DashboardSnapshotPublisher : BackgroundService
private readonly ILogger<DashboardSnapshotPublisher> _logger;
private readonly TimeSpan _reconnectDelay;
private readonly TimeSpan _idlePollInterval;
private readonly TimeSpan _sendTimeout;
/// <summary>Initializes a new instance of the DashboardSnapshotPublisher class.</summary>
/// <param name="snapshotService">The snapshot service to subscribe to.</param>
@@ -64,13 +74,15 @@ public sealed class DashboardSnapshotPublisher : BackgroundService
/// <param name="logger">The logger instance.</param>
/// <param name="reconnectDelay">The delay before reconnecting after a subscription failure.</param>
/// <param name="idlePollInterval">How often the idle publisher re-checks for a connected viewer.</param>
/// <param name="sendTimeout">Upper bound on one broadcast; defaults to one second.</param>
internal DashboardSnapshotPublisher(
IDashboardSnapshotService snapshotService,
IHubContext<DashboardSnapshotHub> hubContext,
DashboardSnapshotHubConnectionCounter connectionCounter,
ILogger<DashboardSnapshotPublisher> logger,
TimeSpan reconnectDelay,
TimeSpan idlePollInterval)
TimeSpan idlePollInterval,
TimeSpan? sendTimeout = null)
{
_snapshotService = snapshotService;
_hubContext = hubContext;
@@ -78,6 +90,7 @@ public sealed class DashboardSnapshotPublisher : BackgroundService
_logger = logger;
_reconnectDelay = reconnectDelay;
_idlePollInterval = idlePollInterval;
_sendTimeout = sendTimeout ?? DefaultSendTimeout;
}
/// <inheritdoc />
@@ -117,10 +130,19 @@ public sealed class DashboardSnapshotPublisher : BackgroundService
try
{
await _hubContext.Clients
.All
.SendAsync(DashboardSnapshotHub.SnapshotMessage, snapshot, stoppingToken)
bool sent = await BoundedHubSend.SendAsync(
_hubContext.Clients.All,
DashboardSnapshotHub.SnapshotMessage,
snapshot,
_sendTimeout,
stoppingToken)
.ConfigureAwait(false);
if (!sent)
{
_logger.LogDebug(
"Snapshot broadcast exceeded {Timeout} and was abandoned; the next tick proceeds.",
_sendTimeout);
}
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
@@ -19,11 +19,27 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// simply goes away never calls <c>UnsubscribeSession</c>; the hub's
/// <c>OnDisconnectedAsync</c> releases everything the connection held.
/// </para>
/// <para>
/// Hub viewers are also counted on their own, separately from the in-process
/// subscribers that register under an <see cref="InProcessConnectionIdPrefix"/>
/// id: <see cref="DashboardEventBroadcaster"/> runs one bounded hub-delivery pump
/// per session only while a remote connection is watching it, and tears that pump
/// down on <see cref="HubViewersReleased"/>.
/// </para>
/// </remarks>
public sealed class EventsHubViewerRegistry
{
/// <summary>
/// Prefix of the synthetic connection ids in-process subscribers register under.
/// A SignalR connection id never carries it, so it is what tells the two
/// audiences apart for the hub-only count.
/// </summary>
public const string InProcessConnectionIdPrefix = "inproc-";
private readonly ConcurrentDictionary<string, int> _viewersBySession = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, int> _hubViewersBySession = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, ConcurrentDictionary<string, byte>> _sessionsByConnection =
new(StringComparer.Ordinal);
@@ -53,8 +69,19 @@ public sealed class EventsHubViewerRegistry
}
_viewersBySession.AddOrUpdate(sessionId, 1, static (_, count) => count + 1);
if (IsHubConnection(connectionId))
{
_hubViewersBySession.AddOrUpdate(sessionId, 1, static (_, count) => count + 1);
}
}
/// <summary>
/// Raised with a session id when its last remote hub viewer leaves (unsubscribe
/// or disconnect). In-process subscribers do not raise it. Handlers run on the
/// releasing thread and must not throw or block.
/// </summary>
public event Action<string>? HubViewersReleased;
/// <summary>
/// Records that <paramref name="connectionId"/> stopped watching
/// <paramref name="sessionId"/>. A removal with no matching
@@ -75,7 +102,7 @@ public sealed class EventsHubViewerRegistry
return;
}
ReleaseSession(sessionId);
Release(connectionId, sessionId);
}
/// <summary>
@@ -107,7 +134,7 @@ public sealed class EventsHubViewerRegistry
// same detached set must not let the session be decremented twice.
if (sessions.TryRemove(sessionId, out _))
{
ReleaseSession(sessionId);
Release(connectionId, sessionId);
}
}
}
@@ -120,32 +147,63 @@ public sealed class EventsHubViewerRegistry
&& _viewersBySession.TryGetValue(sessionId, out int count)
&& count > 0;
/// <summary>
/// Gets a value indicating whether any remote <see cref="EventsHub"/> connection —
/// as opposed to an in-process subscriber — is watching the session.
/// </summary>
/// <param name="sessionId">Session id to test.</param>
/// <returns><see langword="true"/> when at least one hub connection is subscribed.</returns>
public bool HasHubViewers(string sessionId) =>
!string.IsNullOrEmpty(sessionId)
&& _hubViewersBySession.TryGetValue(sessionId, out int count)
&& count > 0;
private static bool IsHubConnection(string connectionId) =>
!connectionId.StartsWith(InProcessConnectionIdPrefix, StringComparison.Ordinal);
/// <summary>
/// Releases one subscription from both counts it was recorded in, and raises
/// <see cref="HubViewersReleased"/> when it was the session's last hub viewer.
/// </summary>
/// <param name="connectionId">Connection that held the subscription.</param>
/// <param name="sessionId">Session id whose counts are released.</param>
private void Release(string connectionId, string sessionId)
{
ReleaseSession(_viewersBySession, sessionId);
if (IsHubConnection(connectionId) && ReleaseSession(_hubViewersBySession, sessionId))
{
HubViewersReleased?.Invoke(sessionId);
}
}
/// <summary>
/// Decrements the session's viewer count, dropping the entry entirely at
/// zero so the dictionary does not grow one key per session ever viewed.
/// The compare-and-swap loop keeps the decrement correct against a
/// concurrent <see cref="AddViewer"/> on the same session.
/// </summary>
/// <param name="counts">The count map to decrement.</param>
/// <param name="sessionId">Session id whose count is released.</param>
private void ReleaseSession(string sessionId)
/// <returns><see langword="true"/> when this release dropped the count to zero.</returns>
private static bool ReleaseSession(ConcurrentDictionary<string, int> counts, string sessionId)
{
while (true)
{
if (!_viewersBySession.TryGetValue(sessionId, out int count))
if (!counts.TryGetValue(sessionId, out int count))
{
return;
return false;
}
if (count <= 1)
{
if (_viewersBySession.TryRemove(new KeyValuePair<string, int>(sessionId, count)))
if (counts.TryRemove(new KeyValuePair<string, int>(sessionId, count)))
{
return;
return true;
}
}
else if (_viewersBySession.TryUpdate(sessionId, count - 1, count))
else if (counts.TryUpdate(sessionId, count - 1, count))
{
return;
return false;
}
}
}
@@ -20,6 +20,24 @@ public interface IDashboardLiveDataService
IReadOnlyCollection<string> tagAddresses,
CancellationToken cancellationToken);
/// <summary>
/// Reads the supplied tags on behalf of one viewer (a Browse page circuit).
/// The shared advise set is budgeted per active viewer, so the viewer id is
/// what stops one circuit's reads from evicting another circuit's whole set.
/// Never throws, as <see cref="ReadAsync(IReadOnlyCollection{string}, CancellationToken)"/>.
/// </summary>
/// <param name="tagAddresses">Fully-qualified tag references to read.</param>
/// <param name="viewerId">
/// Stable id of the reading viewer for the life of its page; <see langword="null"/>
/// reads under one shared anonymous viewer.
/// </param>
/// <param name="cancellationToken">Token to cancel the read.</param>
/// <returns>The read result, or an error-bearing result on failure.</returns>
Task<DashboardLiveReadResult> ReadAsync(
IReadOnlyCollection<string> tagAddresses,
string? viewerId,
CancellationToken cancellationToken) => ReadAsync(tagAddresses, cancellationToken);
/// <summary>
/// Queries the currently-active alarm set for the dashboard session.
/// Never throws — failures are surfaced in
@@ -113,6 +113,7 @@ public static class GatewayApplication
builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
builder.Services.AddSqliteAuthStore(builder.Configuration);
builder.Services.AddGatewayGrpcAuthorization();
builder.Services.AddGatewayMetricsAuthorization();
AddLoginRateLimiter(builder);
builder.Services.AddHealthChecks()
.AddTypeActivatedCheck<AuthStoreHealthCheck>(
@@ -350,8 +351,19 @@ public static class GatewayApplication
{
endpoints.MapStaticAssets(ResolveStaticAssetsManifestPath());
// /health stays anonymous unconditionally: an off-repo Overview dashboard polls
// /health/active without credentials.
endpoints.MapZbHealth();
endpoints.MapZbMetrics();
IEndpointConventionBuilder metricsEndpoint = endpoints.MapZbMetrics();
// Read from configuration rather than IOptions<GatewayOptions>: resolving the options here
// would run whole-options validation during Build instead of at StartAsync.
if (endpoints.ServiceProvider
.GetRequiredService<IConfiguration>()
.GetValue<bool>("MxGateway:Metrics:RequireAuthentication"))
{
// Opt-in (SEC-14): an admin-scoped API key or a dashboard login with a real role.
metricsEndpoint.RequireAuthorization(MetricsAuthorizationDefaults.Policy);
}
endpoints.MapGrpcService<MxAccessGatewayService>();
endpoints.MapZbGalaxyRepository();
@@ -0,0 +1,30 @@
namespace ZB.MOM.WW.MxGateway.Server.Grpc;
/// <summary>
/// The bounded set of <c>reason</c> tag values recorded on
/// <c>mxgateway.grpc.streams.disconnected</c> when a <c>StreamEvents</c> stream ends.
/// </summary>
/// <remarks>
/// Constants rather than exception messages or free text so the metric's tag cardinality stays
/// fixed: an exporter mints one time series per distinct tag value.
/// </remarks>
public static class EventStreamDisconnectReasons
{
/// <summary>The session's event stream completed cleanly (session closed or the worker stream ended).</summary>
public const string SessionClosed = "SessionClosed";
/// <summary>The subscriber fell behind its bounded queue and was disconnected.</summary>
public const string Overflow = "Overflow";
/// <summary>The worker event stream terminated with a fault.</summary>
public const string WorkerFault = "WorkerFault";
/// <summary>The client cancelled the call or disconnected.</summary>
public const string ClientCancelled = "ClientCancelled";
/// <summary>The stream ended on an unclassified error.</summary>
public const string Error = "Error";
/// <summary>The consumer stopped reading without cancelling the call.</summary>
public const string Detached = "Detached";
}
@@ -129,6 +129,10 @@ public sealed class EventStreamService(
IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
// Why this stream ended, recorded as a bounded constant on the disconnect metric. Set at
// the point the end is observed; a consumer that simply stops iterating keeps Detached.
string disconnectReason = EventStreamDisconnectReasons.Detached;
try
{
// Emit order for a resume: the ReplayGap sentinel FIRST (only when events were
@@ -170,16 +174,35 @@ public sealed class EventStreamService(
{
// The distributor pump completes every subscriber channel with the source
// fault when the worker event stream terminates abnormally; that surfaces
// here. Mirror the original ProduceEventsAsync behavior: fault the
// session and record the metric, then propagate the terminal fault to the
// gRPC client.
// here. Fault the session and propagate the terminal fault to the gRPC
// client. The fault itself is NOT counted here: WorkerClient.SetFaulted
// already recorded it once, and every subscriber of the session sees the
// same fault, so counting it per subscriber inflated mxgateway.faults to
// 1+K. This subscriber's share is the WorkerFault disconnect reason.
disconnectReason = EventStreamDisconnectReasons.WorkerFault;
session.MarkFaulted(workerException.Message);
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
throw;
}
catch (SessionManagerException sessionException)
when (sessionException.ErrorCode == SessionManagerErrorCode.EventQueueOverflow)
{
disconnectReason = EventStreamDisconnectReasons.Overflow;
throw;
}
catch (OperationCanceledException)
{
disconnectReason = EventStreamDisconnectReasons.ClientCancelled;
throw;
}
catch (Exception)
{
disconnectReason = EventStreamDisconnectReasons.Error;
throw;
}
if (!hasMore)
{
disconnectReason = EventStreamDisconnectReasons.SessionClosed;
break;
}
@@ -214,7 +237,15 @@ public sealed class EventStreamService(
backlogRegistration.Dispose();
subscriber.Dispose();
metrics.StreamDisconnected("Detached");
// A consumer that stopped iterating mid-stream lands here with the Detached default;
// when the call token has fired, that early stop was the client going away.
if (disconnectReason == EventStreamDisconnectReasons.Detached
&& cancellationToken.IsCancellationRequested)
{
disconnectReason = EventStreamDisconnectReasons.ClientCancelled;
}
metrics.StreamDisconnected(disconnectReason);
}
}
@@ -144,6 +144,25 @@ public sealed class MxAccessGatewayService(
publicReply = bulkConstraintPlan.MergeDeniedInto(publicReply);
}
// MxCommandReply.correlation_id always echoes the caller's client_correlation_id —
// the same value the constraint-denied replies carry — so a client can match every
// reply to its request. The worker reply carries the gateway's internal envelope id,
// which is a gateway↔worker routing detail and is overwritten here, empty included.
// Record the pairing once, before the envelope id is overwritten, so a client error
// report quoting its correlation id can be matched to gateway and worker log lines
// (which carry the envelope id). Ids and the command kind only — never values.
if (logger.IsEnabled(LogLevel.Debug))
{
logger.LogDebug(
"Invoke {CommandKind} for session {SessionId}: client correlation id {ClientCorrelationId} maps to worker envelope correlation id {WorkerCorrelationId}.",
command.Kind,
request.SessionId,
request.ClientCorrelationId,
publicReply.CorrelationId);
}
publicReply.CorrelationId = request.ClientCorrelationId;
session.TrackCommandReply(commandToInvoke, publicReply);
return publicReply;
}
@@ -172,14 +191,22 @@ public sealed class MxAccessGatewayService(
// volume gateway path. Stopwatch.GetTimestamp/GetElapsedTime measure the exact
// same wall-clock span the former Stopwatch.StartNew()/.Elapsed did.
long sendStartTimestamp = Stopwatch.GetTimestamp();
await responseStream.WriteAsync(publicEvent).ConfigureAwait(false);
// The call token releases a write parked on a slow or vanished client when the
// call is cancelled, instead of leaving it to the transport's own timeout.
await responseStream.WriteAsync(publicEvent, context.CancellationToken).ConfigureAwait(false);
metrics.RecordEventStreamSend(
publicEvent.Family.ToString(),
MxEventFamilyNames.Get(publicEvent.Family),
Stopwatch.GetElapsedTime(sendStartTimestamp));
}
}
catch (Exception exception) when (exception is not RpcException)
{
if (IsClientAbort(exception, context))
{
throw ClientCancelled();
}
throw MapException(exception);
}
}
@@ -225,11 +252,24 @@ public sealed class MxAccessGatewayService(
.WithCancellation(context.CancellationToken)
.ConfigureAwait(false))
{
await responseStream.WriteAsync(message).ConfigureAwait(false);
await responseStream.WriteAsync(message, context.CancellationToken).ConfigureAwait(false);
}
}
catch (AlarmFeedSubscriberOverflowException overflow)
{
// The monitor dropped this subscriber for falling behind its bounded queue. That is a
// client-side slowness condition with a client-side remedy (reconnect and
// re-snapshot), so it maps to ResourceExhausted like a StreamEvents overflow rather
// than reaching MapException's warning-with-stack + Unavailable fallback.
throw new RpcException(new Status(StatusCode.ResourceExhausted, overflow.Message));
}
catch (Exception exception) when (exception is not RpcException)
{
if (IsClientAbort(exception, context))
{
throw ClientCancelled();
}
throw MapException(exception);
}
}
@@ -264,6 +304,17 @@ public sealed class MxAccessGatewayService(
}
}
// A response write fails with an IOException (or an operation-cancelled exception) when the
// client disconnects mid-stream. With the call token already cancelled that is a routine
// client abort, not a gateway fault: it maps to Cancelled with no warning. The same exception
// on a live call token is a genuine transport failure and keeps the MapException path.
private static bool IsClientAbort(Exception exception, ServerCallContext context) =>
context.CancellationToken.IsCancellationRequested
&& exception is IOException or OperationCanceledException;
private static RpcException ClientCancelled() =>
new(new Status(StatusCode.Cancelled, "gRPC request was canceled."));
private string? ResolveClientIdentity()
{
return identityAccessor.Current?.DisplayName ?? identityAccessor.Current?.KeyId;
@@ -488,6 +539,7 @@ public sealed class MxAccessGatewayService(
Dictionary<int, SubscribeResult> denied = [];
List<string> allowed = [];
ConstraintDenialSummary? denialSummary = null;
for (int index = 0; index < tagAddresses.Count; index++)
{
string tagAddress = tagAddresses[index];
@@ -500,8 +552,7 @@ public sealed class MxAccessGatewayService(
continue;
}
await constraintEnforcer.RecordDenialAsync(identity, command.Kind.ToString(), tagAddress, failure, correlationId, cancellationToken)
.ConfigureAwait(false);
(denialSummary ??= new ConstraintDenialSummary(tagAddresses.Count)).Add(tagAddress, failure);
denied[index] = new SubscribeResult
{
ServerHandle = serverHandle,
@@ -511,11 +562,17 @@ public sealed class MxAccessGatewayService(
};
}
if (denied.Count == 0)
if (denialSummary is null)
{
return null;
}
// One aggregated audit event per bulk command, not one per denied item (see
// ConstraintDenialSummary for why).
await constraintEnforcer
.RecordBulkDenialAsync(identity, command.Kind.ToString(), denialSummary, correlationId, cancellationToken)
.ConfigureAwait(false);
// Build the filtered command directly instead of cloning the original and clearing it:
// the clone deep-copied every denied address only to drop it. The payload's other fields
// (server_handle) are copied across explicitly. Nothing aliases the request here — these
@@ -556,6 +613,7 @@ public sealed class MxAccessGatewayService(
Dictionary<int, BulkReadResult> denied = [];
List<string> allowed = [];
ConstraintDenialSummary? denialSummary = null;
for (int index = 0; index < tagAddresses.Count; index++)
{
string tagAddress = tagAddresses[index];
@@ -568,8 +626,7 @@ public sealed class MxAccessGatewayService(
continue;
}
await constraintEnforcer.RecordDenialAsync(identity, command.Kind.ToString(), tagAddress, failure, correlationId, cancellationToken)
.ConfigureAwait(false);
(denialSummary ??= new ConstraintDenialSummary(tagAddresses.Count)).Add(tagAddress, failure);
denied[index] = new BulkReadResult
{
ServerHandle = serverHandle,
@@ -580,11 +637,17 @@ public sealed class MxAccessGatewayService(
};
}
if (denied.Count == 0)
if (denialSummary is null)
{
return null;
}
// One aggregated audit event per bulk command, not one per denied item (see
// ConstraintDenialSummary for why).
await constraintEnforcer
.RecordBulkDenialAsync(identity, command.Kind.ToString(), denialSummary, correlationId, cancellationToken)
.ConfigureAwait(false);
MxCommand filtered = new() { Kind = command.Kind };
ReadBulkCommand payload = new()
{
@@ -621,6 +684,7 @@ public sealed class MxAccessGatewayService(
Dictionary<int, BulkWriteResult> denied = [];
List<TEntry> allowed = [];
ConstraintDenialSummary? denialSummary = null;
for (int index = 0; index < entries.Count; index++)
{
TEntry entry = entries[index];
@@ -634,14 +698,7 @@ public sealed class MxAccessGatewayService(
continue;
}
await constraintEnforcer.RecordDenialAsync(
identity,
command.Kind.ToString(),
itemHandle.ToString(System.Globalization.CultureInfo.InvariantCulture),
failure,
correlationId,
cancellationToken)
.ConfigureAwait(false);
(denialSummary ??= new ConstraintDenialSummary(entries.Count)).Add(itemHandle, failure);
denied[index] = new BulkWriteResult
{
ServerHandle = serverHandle,
@@ -651,11 +708,17 @@ public sealed class MxAccessGatewayService(
};
}
if (denied.Count == 0)
if (denialSummary is null)
{
return null;
}
// One aggregated audit event per bulk command, not one per denied item (see
// ConstraintDenialSummary for why).
await constraintEnforcer
.RecordBulkDenialAsync(identity, command.Kind.ToString(), denialSummary, correlationId, cancellationToken)
.ConfigureAwait(false);
return new WriteBulkConstraintPlan(
BuildFilteredWriteBulkCommand(command, allowed),
entries.Count,
@@ -742,6 +805,7 @@ public sealed class MxAccessGatewayService(
Dictionary<int, SubscribeResult> denied = [];
List<int> allowed = [];
ConstraintDenialSummary? denialSummary = null;
for (int index = 0; index < itemHandles.Count; index++)
{
int itemHandle = itemHandles[index];
@@ -754,8 +818,7 @@ public sealed class MxAccessGatewayService(
continue;
}
await constraintEnforcer.RecordDenialAsync(identity, command.Kind.ToString(), itemHandle.ToString(System.Globalization.CultureInfo.InvariantCulture), failure, correlationId, cancellationToken)
.ConfigureAwait(false);
(denialSummary ??= new ConstraintDenialSummary(itemHandles.Count)).Add(itemHandle, failure);
denied[index] = new SubscribeResult
{
ServerHandle = serverHandle,
@@ -765,11 +828,17 @@ public sealed class MxAccessGatewayService(
};
}
if (denied.Count == 0)
if (denialSummary is null)
{
return null;
}
// One aggregated audit event per bulk command, not one per denied item (see
// ConstraintDenialSummary for why).
await constraintEnforcer
.RecordBulkDenialAsync(identity, command.Kind.ToString(), denialSummary, correlationId, cancellationToken)
.ConfigureAwait(false);
MxCommand filtered = new() { Kind = command.Kind };
AdviseItemBulkCommand payload = new() { ServerHandle = command.AdviseItemBulk.ServerHandle };
payload.ItemHandles.Add(allowed);
@@ -1035,6 +1104,7 @@ public sealed class MxAccessGatewayService(
SessionManagerErrorCode.OpenFailed => StatusCode.Unavailable,
SessionManagerErrorCode.CloseFailed => StatusCode.Unavailable,
SessionManagerErrorCode.PermissionDenied => StatusCode.PermissionDenied,
SessionManagerErrorCode.GatewayStopping => StatusCode.Unavailable,
_ => StatusCode.Unavailable,
};
@@ -0,0 +1,28 @@
using System.Collections.Frozen;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Server.Metrics;
/// <summary>
/// Cached <see cref="MxEventFamily"/> names for the per-event metric tags.
/// </summary>
/// <remarks>
/// <see cref="Enum.ToString()"/> allocates a new string on every call, and the event-received
/// and stream-send metrics tag every event with its family — once per event at the worker
/// boundary and once per event per subscriber on the stream. The names are fixed by the
/// contract, so they are computed once here. A value the contract does not define falls back
/// to <see cref="Enum.ToString()"/>, which is the same string the uncached call produced.
/// </remarks>
public static class MxEventFamilyNames
{
private static readonly FrozenDictionary<MxEventFamily, string> Names =
Enum.GetValues<MxEventFamily>()
.Distinct()
.ToFrozenDictionary(family => family, family => family.ToString());
/// <summary>Gets the name of an event family.</summary>
/// <param name="family">The event family.</param>
/// <returns>The family's name, identical to <see cref="Enum.ToString()"/>.</returns>
public static string Get(MxEventFamily family) =>
Names.TryGetValue(family, out string? name) ? name : family.ToString();
}
@@ -58,6 +58,19 @@ public interface IApiKeyCacheInvalidator
/// window and it is intentionally kept short (<see cref="SecurityOptions.ApiKeyVerificationCacheSeconds"/>,
/// default 15 s).
/// </para>
/// <para>
/// Misses are single-flight: concurrent misses for the same token and the same key generation share
/// one inner verification, so a TTL expiry under load costs one SQLite read rather than one per
/// in-flight RPC. A miss that starts after an <see cref="Invalidate"/> never joins a flight that
/// started before it (the flight is keyed on the generation too), so the revoke guarantee holds for
/// joiners as well. The shared inner call is not bound to any one caller's cancellation token; each
/// caller stops waiting on its own token without cancelling the others.
/// </para>
/// <para>
/// The key-id index is pruned by a post-eviction callback whenever the cache drops an entry on its
/// own (expiry, compaction), not only by <see cref="Invalidate"/>, so it stays proportional to the
/// live cache rather than to every token ever verified.
/// </para>
/// </remarks>
public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalidator
{
@@ -69,9 +82,20 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
private readonly TimeSpan _ttl;
// keyId -> set of cache keys currently held for that id, so a revoke/rotate can evict every
// secret variant (e.g. an old secret still within TTL after a rotate).
private readonly ConcurrentDictionary<string, ConcurrentDictionary<string, byte>> _keyIdIndex =
new(StringComparer.Ordinal);
// secret variant (e.g. an old secret still within TTL after a rotate). Guarded by _indexSync,
// which also covers the cache Set that an index entry describes: the post-eviction callback
// runs asynchronously, so Set + index and check-cache + unindex must each be atomic with
// respect to the other, or a late callback could unindex a freshly re-cached entry and leave
// it out of reach of a later Invalidate. Every operation under the lock is O(1) and the lock is
// only taken on misses, evictions and invalidations — never on a cache hit.
private readonly object _indexSync = new();
private readonly Dictionary<string, HashSet<string>> _keyIdIndex = new(StringComparer.Ordinal);
// (cache key, key generation) -> the in-flight inner verification for that pair (single-flight).
private readonly ConcurrentDictionary<(string CacheKey, long Generation), Lazy<Task<ApiKeyVerification>>> _inflight =
new();
private readonly PostEvictionDelegate _onEvicted;
// keyId -> monotonic generation, bumped by Invalidate before it evicts. A VerifyAsync snapshots
// the generation before calling the inner verifier and only caches if it is unchanged after the
@@ -103,6 +127,37 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
_inner = inner;
_cache = cache;
_ttl = ttl;
_onEvicted = OnCacheEntryEvicted;
}
/// <summary>Gets the number of cache keys currently held in the key-id index (test seam).</summary>
internal int IndexedCacheKeyCount
{
get
{
lock (_indexSync)
{
int count = 0;
foreach (HashSet<string> set in _keyIdIndex.Values)
{
count += set.Count;
}
return count;
}
}
}
/// <summary>Gets the number of key ids currently present in the key-id index (test seam).</summary>
internal int IndexedKeyIdCount
{
get
{
lock (_indexSync)
{
return _keyIdIndex.Count;
}
}
}
/// <summary>
@@ -126,11 +181,50 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
}
// Snapshot the key's generation BEFORE the inner verify so a concurrent Invalidate that runs
// while we are in the library is detected below and its repopulation discarded.
// while we are in the library is detected and its repopulation discarded. The snapshot is
// also half of the single-flight key, so a miss that starts after an Invalidate cannot join
// a flight that started before it.
string? tokenKeyId = TryParseKeyId(authorizationHeader);
long generationAtStart = tokenKeyId is null ? 0 : ReadGeneration(tokenKeyId);
(string CacheKey, long Generation) flightKey = (cacheKey, generationAtStart);
ApiKeyVerification result = await _inner.VerifyAsync(authorizationHeader, ct).ConfigureAwait(false);
Lazy<Task<ApiKeyVerification>> created = new(
() => VerifyMissAsync(authorizationHeader, cacheKey, tokenKeyId, generationAtStart));
Lazy<Task<ApiKeyVerification>> flight = _inflight.GetOrAdd(flightKey, created);
Task<ApiKeyVerification> verification = flight.Value;
if (ReferenceEquals(flight, created))
{
// The flight's creator retires it once it completes (success, failure or fault), so the
// next miss for this token starts fresh. Removing by (key, value) pair never retires a
// newer flight that replaced this one.
_ = verification.ContinueWith(
static (_, state) =>
{
var (owner, key, completed) =
((CachingApiKeyVerifier, (string, long), Lazy<Task<ApiKeyVerification>>))state!;
owner._inflight.TryRemove(
new KeyValuePair<(string CacheKey, long Generation), Lazy<Task<ApiKeyVerification>>>(key, completed));
},
(this, flightKey, flight),
CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
}
return await verification.WaitAsync(ct).ConfigureAwait(false);
}
// The shared body of one single-flight miss. Not bound to any caller's token: a caller that
// gives up must not cancel the verification other callers are waiting on.
private async Task<ApiKeyVerification> VerifyMissAsync(
string authorizationHeader,
string cacheKey,
string? tokenKeyId,
long generationAtStart)
{
ApiKeyVerification result = await _inner
.VerifyAsync(authorizationHeader, CancellationToken.None)
.ConfigureAwait(false);
if (result.Succeeded && result.Identity is not null)
{
@@ -158,11 +252,21 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
}
string keyId = result.Identity!.KeyId;
_cache.Set(cacheKey, result, new MemoryCacheEntryOptions
MemoryCacheEntryOptions entryOptions = new()
{
AbsoluteExpirationRelativeToNow = _ttl,
};
entryOptions.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration
{
EvictionCallback = _onEvicted,
State = keyId,
});
IndexCacheKey(keyId, cacheKey);
lock (_indexSync)
{
_cache.Set(cacheKey, result, entryOptions);
IndexCacheKey(keyId, cacheKey);
}
// Set-then-recheck: an Invalidate that landed between the pre-Set check and IndexCacheKey
// would have missed this entry (it was not yet indexed). If the generation has moved, evict
@@ -170,13 +274,38 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
if (tokenKeyId is not null && ReadGeneration(tokenKeyId) != generationAtStart)
{
_cache.Remove(cacheKey);
if (_keyIdIndex.TryGetValue(keyId, out ConcurrentDictionary<string, byte>? set))
lock (_indexSync)
{
set.TryRemove(cacheKey, out _);
UnindexCacheKey(keyId, cacheKey);
}
}
}
// Post-eviction callback: drops the evicted entry's key-id index slot so the index tracks the
// live cache. MemoryCache raises it asynchronously, so by the time it runs the same cache key
// may already hold a newer entry (re-verified after the expiry); that entry owns the slot, and
// the check below runs under the lock that also covers Set + index, so it cannot interleave
// with a re-cache.
private void OnCacheEntryEvicted(object key, object? value, EvictionReason reason, object? state)
{
if (reason == EvictionReason.Replaced || key is not string cacheKey || state is not string keyId)
{
// Replaced: a newer entry under the same key was Set in this entry's place and is
// indexed under the same slot already.
return;
}
lock (_indexSync)
{
if (_cache.TryGetValue(cacheKey, out _))
{
return;
}
UnindexCacheKey(keyId, cacheKey);
}
}
/// <inheritdoc />
public void Invalidate(string keyId)
{
@@ -189,12 +318,22 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
// generation refuses to cache (or self-evicts) its now-stale repopulation.
_generations.AddOrUpdate(keyId, 1, static (_, current) => current + 1);
if (_keyIdIndex.TryRemove(keyId, out ConcurrentDictionary<string, byte>? cacheKeys))
string[] cacheKeys;
lock (_indexSync)
{
foreach (string cacheKey in cacheKeys.Keys)
if (!_keyIdIndex.Remove(keyId, out HashSet<string>? indexed))
{
_cache.Remove(cacheKey);
return;
}
cacheKeys = [.. indexed];
}
// Evicted outside the lock: the eviction callbacks these raise take it themselves (and find
// the slot already gone).
foreach (string cacheKey in cacheKeys)
{
_cache.Remove(cacheKey);
}
}
@@ -257,12 +396,28 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
return new string(afterPrefix[..separator]);
}
// Caller holds _indexSync.
private void IndexCacheKey(string keyId, string cacheKey)
{
ConcurrentDictionary<string, byte> set = _keyIdIndex.GetOrAdd(
keyId,
static _ => new ConcurrentDictionary<string, byte>(StringComparer.Ordinal));
set.TryAdd(cacheKey, 0);
if (!_keyIdIndex.TryGetValue(keyId, out HashSet<string>? set))
{
set = new HashSet<string>(StringComparer.Ordinal);
_keyIdIndex[keyId] = set;
}
set.Add(cacheKey);
}
// Caller holds _indexSync. Drops the key id's slot entirely once its last cache key goes, so
// the index does not accumulate empty sets for keys that are no longer cached.
private void UnindexCacheKey(string keyId, string cacheKey)
{
if (_keyIdIndex.TryGetValue(keyId, out HashSet<string>? set)
&& set.Remove(cacheKey)
&& set.Count == 0)
{
_keyIdIndex.Remove(keyId);
}
}
private static bool TryComputeCacheKey(string? authorizationHeader, out string cacheKey)
@@ -0,0 +1,90 @@
using System.Globalization;
namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
/// <summary>
/// Accumulates the constraint denials of one bulk command so they are audited as a single
/// aggregated event rather than one event per denied item.
/// </summary>
/// <remarks>
/// A bulk command can carry thousands of items, and the audit writer is a bounded, drop-on-full
/// channel. One audit event per denied item let a single low-privilege key flood that channel and
/// push every other audit record out of it. The summary keeps the forensic signal — how many items
/// were denied, which constraints denied them, and a bounded sample of the targets — at a fixed
/// cost of one audit event per command.
/// </remarks>
public sealed class ConstraintDenialSummary
{
/// <summary>The maximum number of denied targets retained as a sample.</summary>
public const int MaxSampleTargets = 20;
private readonly List<string> _constraintNames = [];
private readonly List<string> _sampleTargets = [];
/// <summary>Initializes a new instance of the <see cref="ConstraintDenialSummary"/> class.</summary>
/// <param name="totalCount">The number of items the bulk command carried.</param>
public ConstraintDenialSummary(int totalCount)
{
TotalCount = totalCount;
}
/// <summary>Gets the number of items the bulk command carried.</summary>
public int TotalCount { get; }
/// <summary>Gets the number of items denied.</summary>
public int DeniedCount { get; private set; }
/// <summary>Gets the distinct constraint names that denied at least one item, in first-seen order.</summary>
public IReadOnlyList<string> ConstraintNames => _constraintNames;
/// <summary>Gets up to <see cref="MaxSampleTargets"/> denied targets, in item order.</summary>
public IReadOnlyList<string> SampleTargets => _sampleTargets;
/// <summary>Gets a value indicating whether more items were denied than the sample retains.</summary>
public bool SampleTruncated => DeniedCount > _sampleTargets.Count;
/// <summary>Records one denied tag-address item.</summary>
/// <param name="target">The denied tag address.</param>
/// <param name="failure">The constraint failure that denied it.</param>
public void Add(string target, ConstraintFailure failure)
{
ArgumentNullException.ThrowIfNull(failure);
AddConstraint(failure.ConstraintName);
if (_sampleTargets.Count < MaxSampleTargets)
{
_sampleTargets.Add(target);
}
DeniedCount++;
}
/// <summary>Records one denied item-handle item.</summary>
/// <param name="itemHandle">The denied MXAccess item handle.</param>
/// <param name="failure">The constraint failure that denied it.</param>
/// <remarks>The handle is formatted only while the sample still has room.</remarks>
public void Add(int itemHandle, ConstraintFailure failure)
{
ArgumentNullException.ThrowIfNull(failure);
AddConstraint(failure.ConstraintName);
if (_sampleTargets.Count < MaxSampleTargets)
{
_sampleTargets.Add(itemHandle.ToString(CultureInfo.InvariantCulture));
}
DeniedCount++;
}
private void AddConstraint(string constraintName)
{
// The constraint vocabulary is a handful of fixed names, so a linear scan beats a set.
for (int index = 0; index < _constraintNames.Count; index++)
{
if (string.Equals(_constraintNames[index], constraintName, StringComparison.Ordinal))
{
return;
}
}
_constraintNames.Add(constraintName);
}
}
@@ -25,7 +25,7 @@ public sealed class ConstraintEnforcer(
identity?.EffectiveConstraints.HasWriteConstraints ?? false;
/// <inheritdoc />
public Task<ConstraintFailure?> CheckReadTagAsync(
public ValueTask<ConstraintFailure?> CheckReadTagAsync(
ApiKeyIdentity? identity,
string tagAddress,
CancellationToken cancellationToken)
@@ -33,14 +33,14 @@ public sealed class ConstraintEnforcer(
ApiKeyConstraints constraints = identity?.EffectiveConstraints ?? ApiKeyConstraints.Empty;
if (!constraints.HasReadConstraints)
{
return Task.FromResult<ConstraintFailure?>(null);
return ValueTask.FromResult<ConstraintFailure?>(null);
}
return Task.FromResult(CheckReadTarget(constraints, tagAddress));
return ValueTask.FromResult(CheckReadTarget(constraints, tagAddress));
}
/// <inheritdoc />
public Task<ConstraintFailure?> CheckReadHandleAsync(
public ValueTask<ConstraintFailure?> CheckReadHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
@@ -50,19 +50,19 @@ public sealed class ConstraintEnforcer(
ApiKeyConstraints constraints = identity?.EffectiveConstraints ?? ApiKeyConstraints.Empty;
if (!constraints.HasReadConstraints)
{
return Task.FromResult<ConstraintFailure?>(null);
return ValueTask.FromResult<ConstraintFailure?>(null);
}
if (!session.TryGetItemRegistration(serverHandle, itemHandle, out SessionItemRegistration registration))
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure("item_handle", "Item handle is not registered in the constrained session."));
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure("item_handle", "Item handle is not registered in the constrained session."));
}
return Task.FromResult(CheckReadTarget(constraints, registration.TagAddress));
return ValueTask.FromResult(CheckReadTarget(constraints, registration.TagAddress));
}
/// <inheritdoc />
public Task<ConstraintFailure?> CheckWriteHandleAsync(
public ValueTask<ConstraintFailure?> CheckWriteHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
@@ -72,23 +72,23 @@ public sealed class ConstraintEnforcer(
ApiKeyConstraints constraints = identity?.EffectiveConstraints ?? ApiKeyConstraints.Empty;
if (!constraints.HasWriteConstraints)
{
return Task.FromResult<ConstraintFailure?>(null);
return ValueTask.FromResult<ConstraintFailure?>(null);
}
if (!session.TryGetItemRegistration(serverHandle, itemHandle, out SessionItemRegistration registration))
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure("item_handle", "Item handle is not registered in the constrained session."));
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure("item_handle", "Item handle is not registered in the constrained session."));
}
GalaxyTagLookup? target = ResolveTarget(registration.TagAddress);
if (target is null)
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure("tag_metadata", "Tag metadata is not available in the Galaxy hierarchy cache."));
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure("tag_metadata", "Tag metadata is not available in the Galaxy hierarchy cache."));
}
if (!MatchesPathOrTag(target.ContainedPath, registration.TagAddress, constraints.WriteSubtrees, constraints.WriteTagGlobs))
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure("write_scope", "Tag is outside the API key write scope."));
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure("write_scope", "Tag is outside the API key write scope."));
}
if (constraints.MaxWriteClassification is { } maxClassification)
@@ -96,18 +96,18 @@ public sealed class ConstraintEnforcer(
GalaxyAttribute? attribute = target.Attribute;
if (attribute is null)
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure("max_write_classification", "Attribute security classification is not available."));
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure("max_write_classification", "Attribute security classification is not available."));
}
if (attribute.SecurityClassification > maxClassification)
{
return Task.FromResult<ConstraintFailure?>(new ConstraintFailure(
return ValueTask.FromResult<ConstraintFailure?>(new ConstraintFailure(
"max_write_classification",
$"Attribute security classification {attribute.SecurityClassification} exceeds allowed maximum {maxClassification}."));
}
}
return Task.FromResult<ConstraintFailure?>(null);
return ValueTask.FromResult<ConstraintFailure?>(null);
}
/// <inheritdoc />
@@ -150,6 +150,46 @@ public sealed class ConstraintEnforcer(
await auditWriter.WriteAsync(auditEvent, cancellationToken).ConfigureAwait(false);
}
/// <inheritdoc />
public async Task RecordBulkDenialAsync(
ApiKeyIdentity? identity,
string commandKind,
ConstraintDenialSummary summary,
string? correlationId,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(summary);
if (summary.DeniedCount == 0)
{
return;
}
// One event for the whole command. Target stays "<commandKind>:<target>"-shaped so audit
// queries that split on ':' keep working; the "bulk" target marks the aggregated form.
AuditEvent auditEvent = new()
{
EventId = Guid.NewGuid(),
OccurredAtUtc = DateTimeOffset.UtcNow,
Actor = identity?.KeyId ?? "anonymous",
Action = "constraint-denied",
Outcome = AuditOutcome.Denied,
Category = CanonicalForwardingApiKeyAuditStore.ApiKeyCategory,
Target = $"{commandKind}:bulk",
SourceNode = null,
CorrelationId = Guid.TryParse(correlationId, out var cid) ? cid : (Guid?)null,
DetailsJson = JsonSerializer.Serialize(new BulkDenialDetails(
summary.ConstraintNames,
commandKind,
summary.DeniedCount,
summary.TotalCount,
summary.SampleTargets,
summary.SampleTruncated,
correlationId ?? "")),
};
await auditWriter.WriteAsync(auditEvent, cancellationToken).ConfigureAwait(false);
}
private ConstraintFailure? CheckReadTarget(
ApiKeyConstraints constraints,
string tagAddress)
@@ -240,4 +280,15 @@ public sealed class ConstraintEnforcer(
return false;
}
// DetailsJson shape of an aggregated bulk denial. Serialized with camelCase names to match the
// single-item envelope's keys ("commandKind", "clientCorrelationId").
private sealed record BulkDenialDetails(
[property: System.Text.Json.Serialization.JsonPropertyName("constraints")] IReadOnlyList<string> Constraints,
[property: System.Text.Json.Serialization.JsonPropertyName("commandKind")] string CommandKind,
[property: System.Text.Json.Serialization.JsonPropertyName("deniedCount")] int DeniedCount,
[property: System.Text.Json.Serialization.JsonPropertyName("totalCount")] int TotalCount,
[property: System.Text.Json.Serialization.JsonPropertyName("sampleTargets")] IReadOnlyList<string> SampleTargets,
[property: System.Text.Json.Serialization.JsonPropertyName("sampleTruncated")] bool SampleTruncated,
[property: System.Text.Json.Serialization.JsonPropertyName("clientCorrelationId")] string ClientCorrelationId);
}
@@ -30,7 +30,10 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
scopeResolver.ResolveRequiredScope(request),
context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
@@ -45,7 +48,10 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
scopeResolver.ResolveRequiredScope(request),
context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
@@ -53,15 +59,51 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
}
}
// The base Interceptor passes client- and duplex-streaming calls straight to the handler, so
// without these overrides any such RPC (none exist today, but a .proto edit could add one, and
// the shared Galaxy Repository service is mapped through the same interceptor) would be
// reachable with no API key at all. Both run the identical authentication path. There is no
// single request message to derive a scope from before the handler reads the stream, so they
// fail closed to the scope the resolver assigns an unrecognized request: admin.
/// <inheritdoc />
public override async Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(GatewayScopes.Admin, context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
return await continuation(requestStream, context).ConfigureAwait(false);
}
}
/// <inheritdoc />
public override async Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(GatewayScopes.Admin, context)
.ConfigureAwait(false);
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
using (identityScope)
{
await continuation(requestStream, responseStream, context).ConfigureAwait(false);
}
}
/// <summary>Authenticates the API key and authorizes the RPC call by required scope.</summary>
/// <typeparam name="TRequest">Request message type.</typeparam>
/// <param name="request">Request payload.</param>
/// <param name="requiredScope">The scope the call requires, resolved by the caller.</param>
/// <param name="context">RPC server call context.</param>
/// <returns>Authenticated API key identity, or null if authentication is disabled.</returns>
private async Task<ApiKeyIdentity?> AuthenticateAndAuthorizeAsync<TRequest>(
TRequest request,
private async Task<ApiKeyIdentity?> AuthenticateAndAuthorizeAsync(
string requiredScope,
ServerCallContext context)
where TRequest : class
{
if (options.Value.Authentication.Mode == AuthenticationMode.Disabled)
{
@@ -116,7 +158,6 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
ApiKeyIdentity identity = GatewayApiKeyIdentityMapper.ToGatewayIdentity(verification.Identity);
string requiredScope = scopeResolver.ResolveRequiredScope(request);
if (!identity.Scopes.Contains(requiredScope))
{
throw new RpcException(new Status(
@@ -3,6 +3,13 @@ using ZB.MOM.WW.MxGateway.Server.Sessions;
namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
/// <summary>Enforces API-key read/write constraints and records constraint denials for audit.</summary>
/// <remarks>
/// The per-item checks return <see cref="ValueTask{TResult}"/> because every production check is
/// synchronous (an in-memory Galaxy hierarchy lookup) and runs once per item of a bulk command:
/// a <see cref="Task{TResult}"/> return allocated a task per item for a value that was already
/// known. An implementation that genuinely needs to await still can.
/// </remarks>
public interface IConstraintEnforcer
{
/// <summary>
@@ -34,7 +41,7 @@ public interface IConstraintEnforcer
/// <param name="tagAddress">Tag address to check.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The constraint failure details if denied; otherwise null.</returns>
Task<ConstraintFailure?> CheckReadTagAsync(
ValueTask<ConstraintFailure?> CheckReadTagAsync(
ApiKeyIdentity? identity,
string tagAddress,
CancellationToken cancellationToken);
@@ -46,7 +53,7 @@ public interface IConstraintEnforcer
/// <param name="itemHandle">The MXAccess item handle.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The constraint failure details if denied; otherwise null.</returns>
Task<ConstraintFailure?> CheckReadHandleAsync(
ValueTask<ConstraintFailure?> CheckReadHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
@@ -60,7 +67,7 @@ public interface IConstraintEnforcer
/// <param name="itemHandle">The MXAccess item handle.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The constraint failure details if denied; otherwise null.</returns>
Task<ConstraintFailure?> CheckWriteHandleAsync(
ValueTask<ConstraintFailure?> CheckWriteHandleAsync(
ApiKeyIdentity? identity,
GatewaySession session,
int serverHandle,
@@ -85,4 +92,25 @@ public interface IConstraintEnforcer
ConstraintFailure failure,
string? correlationId,
CancellationToken cancellationToken);
/// <summary>
/// Records every constraint denial of one bulk command as a single aggregated audit event.
/// </summary>
/// <param name="identity">The API key identity.</param>
/// <param name="commandKind">The kind of bulk command whose items were denied.</param>
/// <param name="summary">The accumulated denials: count, constraint names and a bounded target sample.</param>
/// <param name="correlationId">The per-request client correlation id, if any (same handling as <see cref="RecordDenialAsync"/>).</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
/// <remarks>
/// Bulk callers use this instead of one <see cref="RecordDenialAsync"/> per item: the audit
/// channel is bounded and drops on full, so per-item events let one bulk command evict every
/// other pending audit record.
/// </remarks>
Task RecordBulkDenialAsync(
ApiKeyIdentity? identity,
string commandKind,
ConstraintDenialSummary summary,
string? correlationId,
CancellationToken cancellationToken);
}
@@ -0,0 +1,222 @@
using System.Security.Claims;
using System.Text.Encodings.Web;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authorization;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Metrics;
namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
/// <summary>Names shared by the optional <c>/metrics</c> authentication (<c>MxGateway:Metrics:RequireAuthentication</c>).</summary>
public static class MetricsAuthorizationDefaults
{
/// <summary>Authentication scheme that authenticates a metrics scrape.</summary>
public const string AuthenticationScheme = "MxGateway.Metrics";
/// <summary>Authorization policy attached to <c>/metrics</c> when authentication is required.</summary>
public const string Policy = "MxGateway.Metrics";
/// <summary>Claim type carrying one API-key scope on a principal the metrics scheme authenticated.</summary>
public const string ScopeClaimType = "mxgateway:scope";
}
/// <summary>
/// Authenticates a <c>/metrics</c> scrape from either credential the gateway already issues: an API
/// key presented as <c>Authorization: Bearer mxgw_…</c> (verified through the same cached
/// <see cref="IApiKeyVerifier"/> and per-peer failure limiter as gRPC calls), or, when no
/// authorization header is sent, the dashboard login cookie.
/// </summary>
/// <remarks>
/// One scheme fronting both credentials keeps the challenge a plain <c>401</c> with
/// <c>WWW-Authenticate: Bearer</c>, which is what a Prometheus scraper understands; listing the
/// cookie scheme on the policy directly would turn an anonymous scrape into a 302 redirect to the
/// login page. A throttled key partition answers <c>429</c>, mirroring the gRPC
/// <c>ResourceExhausted</c>. Failure reasons are never echoed to the caller and the presented
/// secret is never logged. The limiter's per-peer partition here is the remote IP, whereas the gRPC
/// interceptor's is the gRPC peer string (address and port, i.e. per connection), so the two paths
/// share only the per-key-id aggregate layer.
/// </remarks>
public sealed class MetricsAuthenticationHandler(
IOptionsMonitor<AuthenticationSchemeOptions> options,
ILoggerFactory logger,
UrlEncoder encoder,
IApiKeyVerifier apiKeyVerifier,
ApiKeyFailureLimiter failureLimiter,
IAuthenticationSchemeProvider schemeProvider,
IOptions<GatewayOptions> gatewayOptions,
GatewayMetrics metrics)
: AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
{
private bool _throttled;
/// <inheritdoc />
protected override async Task<AuthenticateResult> HandleAuthenticateAsync()
{
string? authorizationHeader = Request.Headers.Authorization;
if (string.IsNullOrWhiteSpace(authorizationHeader))
{
return await AuthenticateDashboardCookieAsync().ConfigureAwait(false);
}
// Same throttle as the gRPC interceptor: checked before the store read so online guessing
// against /metrics cannot spend a verification per attempt or bypass the gRPC limiter.
ApiKeyThrottlePartition partition = new(
Context.Connection.RemoteIpAddress?.ToString() ?? "unknown",
GatewayGrpcAuthorizationInterceptor.TryResolveKeyId(authorizationHeader));
ApiKeyThrottleDecision decision = failureLimiter.Check(
partition,
out ApiKeyFailureLimiter.PartitionResolution resolution);
if (decision is ApiKeyThrottleDecision.ThrottledByPeer or ApiKeyThrottleDecision.ThrottledByAggregate)
{
_throttled = true;
metrics.RecordAuthThrottled(decision == ApiKeyThrottleDecision.ThrottledByPeer ? "peer" : "aggregate");
return AuthenticateResult.Fail("Too many authentication attempts.");
}
ApiKeyVerification verification = await apiKeyVerifier
.VerifyAsync(authorizationHeader, Context.RequestAborted)
.ConfigureAwait(false);
if (!verification.Succeeded || verification.Identity is null)
{
failureLimiter.RecordFailure(partition);
return AuthenticateResult.Fail("Missing or invalid API key.");
}
failureLimiter.Reset(partition, resolution);
List<Claim> claims = [new Claim(ClaimTypes.Name, verification.Identity.KeyId)];
foreach (string scope in verification.Identity.Scopes)
{
claims.Add(new Claim(MetricsAuthorizationDefaults.ScopeClaimType, scope));
}
ClaimsPrincipal principal = new(new ClaimsIdentity(claims, Scheme.Name));
return AuthenticateResult.Success(new AuthenticationTicket(principal, Scheme.Name));
}
/// <inheritdoc />
protected override Task HandleChallengeAsync(AuthenticationProperties properties)
{
if (_throttled)
{
Response.StatusCode = StatusCodes.Status429TooManyRequests;
return Task.CompletedTask;
}
Response.StatusCode = StatusCodes.Status401Unauthorized;
Response.Headers.WWWAuthenticate = "Bearer";
return Task.CompletedTask;
}
// No authorization header: fall back to the dashboard login cookie, so an operator already
// signed in to the dashboard can open /metrics in the same browser.
private async Task<AuthenticateResult> AuthenticateDashboardCookieAsync()
{
// Dashboard:DisableLogin registers an always-succeeding auto-login handler under the cookie
// scheme name. Falling back to it would let every anonymous request satisfy /metrics, so
// with DisableLogin on, RequireAuthentication=true accepts API keys only.
if (gatewayOptions.Value.Dashboard.DisableLogin)
{
return AuthenticateResult.NoResult();
}
if (await schemeProvider.GetSchemeAsync(DashboardAuthenticationDefaults.AuthenticationScheme).ConfigureAwait(false)
is null)
{
return AuthenticateResult.NoResult();
}
AuthenticateResult cookie = await Context
.AuthenticateAsync(DashboardAuthenticationDefaults.AuthenticationScheme)
.ConfigureAwait(false);
return cookie.Succeeded && cookie.Principal is not null
? AuthenticateResult.Success(new AuthenticationTicket(cookie.Principal, Scheme.Name))
: AuthenticateResult.NoResult();
}
}
/// <summary>Requirement for <c>/metrics</c> access; see <see cref="MetricsAccessHandler"/>.</summary>
public sealed class MetricsAccessRequirement : IAuthorizationRequirement
{
}
/// <summary>
/// Grants <c>/metrics</c> to an API key carrying the <c>admin</c> scope or to a dashboard login
/// holding a real dashboard role (Administrator or Viewer). It reads no options, so neither
/// anonymous localhost nor <c>Authentication:Mode=Disabled</c> can satisfy it.
/// </summary>
/// <remarks>
/// A Viewer login qualifies because every dashboard role already sees the same aggregate counters
/// on the dashboard itself; the metrics carry no per-session or per-tag data. An API key needs
/// <c>admin</c> because metrics are an operator surface, not one of the data-plane scopes a client
/// key is normally issued with.
/// </remarks>
public sealed class MetricsAccessHandler : AuthorizationHandler<MetricsAccessRequirement>
{
/// <inheritdoc />
protected override Task HandleRequirementAsync(
AuthorizationHandlerContext context,
MetricsAccessRequirement requirement)
{
foreach (ClaimsIdentity identity in context.User.Identities)
{
if (!identity.IsAuthenticated)
{
continue;
}
// Scope claims count only on an identity the metrics scheme minted from a verified key.
bool isApiKeyIdentity = string.Equals(
identity.AuthenticationType,
MetricsAuthorizationDefaults.AuthenticationScheme,
StringComparison.Ordinal);
if (isApiKeyIdentity
&& identity.HasClaim(MetricsAuthorizationDefaults.ScopeClaimType, GatewayScopes.Admin))
{
context.Succeed(requirement);
return Task.CompletedTask;
}
if (!isApiKeyIdentity
&& (identity.HasClaim(identity.RoleClaimType, DashboardRoles.Admin)
|| identity.HasClaim(identity.RoleClaimType, DashboardRoles.Viewer)))
{
context.Succeed(requirement);
return Task.CompletedTask;
}
}
return Task.CompletedTask;
}
}
/// <summary>Registration for the optional <c>/metrics</c> authentication.</summary>
public static class MetricsAuthorizationServiceCollectionExtensions
{
/// <summary>
/// Registers the metrics authentication scheme, the <see cref="MetricsAuthorizationDefaults.Policy"/>
/// policy and its handler. Registration is unconditional and inert: the policy only takes effect
/// when <c>MxGateway:Metrics:RequireAuthentication</c> attaches it to the endpoint.
/// </summary>
/// <param name="services">Service collection to register into.</param>
/// <returns>The same service collection, for chaining.</returns>
public static IServiceCollection AddGatewayMetricsAuthorization(this IServiceCollection services)
{
services.AddAuthentication()
.AddScheme<AuthenticationSchemeOptions, MetricsAuthenticationHandler>(
MetricsAuthorizationDefaults.AuthenticationScheme,
_ => { });
services.AddAuthorizationBuilder()
.AddPolicy(
MetricsAuthorizationDefaults.Policy,
policy => policy
.AddAuthenticationSchemes(MetricsAuthorizationDefaults.AuthenticationScheme)
.AddRequirements(new MetricsAccessRequirement()));
services.AddSingleton<IAuthorizationHandler, MetricsAccessHandler>();
return services;
}
}
@@ -26,9 +26,11 @@ public sealed class GatewaySession
private DateTimeOffset _lastClientActivityAt;
private DateTimeOffset? _leaseExpiresAt;
private bool _closeStarted;
private bool _openRecorded;
private int _activeEventSubscriberCount;
private readonly TimeSpan _detachGrace;
private readonly TimeSpan _faultedGrace;
private readonly TimeSpan _unattachedGrace;
private readonly TimeSpan _workerReadyWaitTimeout;
private DateTimeOffset? _detachedAtUtc;
private DateTimeOffset? _faultedAtUtc;
@@ -37,6 +39,12 @@ public sealed class GatewaySession
// FAILED first attach — which still runs the rollback DetachEventSubscriber from the
// attach catch path — does not push a never-subscribed session into the grace window.
private bool _everHadEventSubscriber;
// True once an in-process (internal, non-counted) subscriber attached via
// AttachInternalEventSubscriber. Consulted only by the unattached-grace sweep rule.
private bool _everHadInternalEventSubscriber;
// Stamped once, when the session first becomes Ready; the unattached-grace idle clock cannot
// start before it.
private DateTimeOffset? _readyAtUtc;
private SessionEventDistributor? _eventDistributor;
private bool _eventDistributorStarted;
// Set under _syncRoot when DisposeAsync nulls _eventDistributor, making the
@@ -168,6 +176,12 @@ public sealed class GatewaySession
/// The tags come from the owner key, never from the client's wire request, so a client
/// cannot label its own session with another tenant's tag.
/// </param>
/// <param name="unattachedGrace">
/// Idle window after which the lease monitor reaps a Ready session that has never attached
/// an event subscriber (see <see cref="IsUnattachedGraceExpired"/>), so a client that
/// opens and vanishes does not pin a worker for the full lease. <see cref="TimeSpan.Zero"/>
/// (the default) disables it.
/// </param>
public GatewaySession(
string sessionId,
string backendName,
@@ -187,7 +201,8 @@ public sealed class GatewaySession
TimeSpan workerReadyWaitTimeout = default,
ArrayAddressNormalizer? addressNormalizer = null,
TimeSpan faultedGrace = default,
IReadOnlyList<string>? ownerDashboardTags = null)
IReadOnlyList<string>? ownerDashboardTags = null,
TimeSpan unattachedGrace = default)
{
if (string.IsNullOrWhiteSpace(sessionId))
{
@@ -230,6 +245,7 @@ public sealed class GatewaySession
_eventStreaming = eventStreaming ?? SessionEventStreaming.Default;
_detachGrace = detachGrace > TimeSpan.Zero ? detachGrace : TimeSpan.Zero;
_faultedGrace = faultedGrace > TimeSpan.Zero ? faultedGrace : TimeSpan.Zero;
_unattachedGrace = unattachedGrace > TimeSpan.Zero ? unattachedGrace : TimeSpan.Zero;
_workerReadyWaitTimeout = workerReadyWaitTimeout > TimeSpan.Zero ? workerReadyWaitTimeout : TimeSpan.Zero;
_addressNormalizer = addressNormalizer;
}
@@ -413,13 +429,151 @@ public sealed class GatewaySession
/// Attaches the worker client for this session.
/// </summary>
/// <param name="workerClient">Worker client to attach.</param>
public void AttachWorkerClient(IWorkerClient workerClient)
/// <returns>
/// <see langword="true"/> when the client was attached; <see langword="false"/> when the
/// session is already <see cref="SessionState.Closing"/>, <see cref="SessionState.Closed"/>
/// or <see cref="SessionState.Faulted"/>, or has been disposed — the caller owns the client
/// and must discard it.
/// </returns>
/// <remarks>
/// The session is registered before its worker is created, so an admin kill or the host
/// drain can close it while the worker is still starting (up to the startup timeout). A
/// worker client arriving after that must not be stored on the dead session: nothing would
/// ever shut it down (the close already ran with no client to stop) and the open would hand
/// a closed session back to the caller as if it had succeeded. The decision is taken under
/// <c>_syncRoot</c> so it cannot interleave with the close's state write.
/// </remarks>
public bool AttachWorkerClient(IWorkerClient workerClient)
{
ArgumentNullException.ThrowIfNull(workerClient);
lock (_syncRoot)
{
if (_state is SessionState.Closing or SessionState.Closed or SessionState.Faulted
|| _eventDistributorDisposed)
{
return false;
}
_workerClient = workerClient;
return true;
}
}
/// <summary>
/// Runs the end of the open sequence — attach the worker client, mark the session
/// <see cref="SessionState.Ready"/>, record it as opened — under the per-session close
/// gate, so a concurrent <see cref="CloseAsync"/> or
/// <see cref="KillWorkerWithCloseGateAsync"/> runs either entirely before it (the attach
/// is refused) or entirely after it (an ordinary close of a Ready session). Without the
/// gate a kill could land between the attach and the open accounting and the two teardown
/// bookkeepings would disagree about whether the session ever counted as open.
/// </summary>
/// <param name="workerClient">Worker client to attach.</param>
/// <param name="onReady">
/// Invoked inside the gate once the session is Ready; the caller records its open-session
/// accounting here, and <see cref="OpenRecorded"/> is set right after it returns.
/// </param>
/// <returns>
/// <see cref="SessionAttachOutcome.Attached"/> when the session became Ready;
/// <see cref="SessionAttachOutcome.SessionClosed"/> when the session refused the attach
/// (see <see cref="AttachWorkerClient"/>), including when it was disposed before the gate
/// could be taken; <see cref="SessionAttachOutcome.WorkerTerminal"/> when the worker client
/// had already reached <see cref="WorkerClientState.Faulted"/> or
/// <see cref="WorkerClientState.Closed"/>.
/// </returns>
/// <remarks>
/// Not cancellable: the worker client already exists, so the only acceptable outcomes are
/// "attached" or "refused, caller discards it". The gate is held only for the duration of
/// a close or kill, so the wait is bounded by the worker shutdown timeout. The caller
/// still decides what to do about a cancelled open before calling this.
/// <para>
/// A worker that dies right after its handshake (an STA loop fault, exit code 7) can
/// be terminal before it is attached. Attaching it would hand the caller a Ready
/// session over a dead worker that nothing notices until the lease expires, so it is
/// refused here. A worker that dies just after this check is caught by the event
/// pump's readiness failure path instead (see <c>MapWorkerEventsAsync</c>).
/// </para>
/// </remarks>
internal async ValueTask<SessionAttachOutcome> TryAttachWorkerClientAndMarkReadyAsync(
IWorkerClient workerClient,
Action onReady)
{
ArgumentNullException.ThrowIfNull(workerClient);
ArgumentNullException.ThrowIfNull(onReady);
try
{
await _closeLock.WaitAsync(CancellationToken.None).ConfigureAwait(false);
}
catch (ObjectDisposedException)
{
// Disposed while the worker was starting: the session has been torn down.
return SessionAttachOutcome.SessionClosed;
}
try
{
if (workerClient.State is WorkerClientState.Faulted or WorkerClientState.Closed)
{
// A session closed meanwhile is the stronger reason: it decides whether the open
// reports a stopping gateway rather than a failed worker.
lock (_syncRoot)
{
return _state is SessionState.Closing or SessionState.Closed or SessionState.Faulted
|| _eventDistributorDisposed
? SessionAttachOutcome.SessionClosed
: SessionAttachOutcome.WorkerTerminal;
}
}
if (!AttachWorkerClient(workerClient))
{
return SessionAttachOutcome.SessionClosed;
}
MarkReady();
onReady();
lock (_syncRoot)
{
_openRecorded = true;
}
return SessionAttachOutcome.Attached;
}
finally
{
try
{
_closeLock.Release();
}
catch (ObjectDisposedException)
{
// A double-dispose that slipped past DisposeAsync's own gate; the session is torn
// down either way and there is no one left to hand the slot to.
}
}
}
/// <summary>
/// Gets a value indicating whether the open-session accounting
/// (<c>GatewayMetrics.SessionOpened</c>) has been recorded for this session. Every
/// teardown path consults it so a session closed or killed while its worker was still
/// starting — never counted as opened — is not counted as closed either.
/// </summary>
/// <remarks>
/// Written only by <see cref="TryAttachWorkerClientAndMarkReadyAsync"/>, inside the close
/// gate; read by teardown paths that hold (or just released) the same gate, so the two
/// accountings never observe a half-finished open.
/// </remarks>
internal bool OpenRecorded
{
get
{
lock (_syncRoot)
{
return _openRecorded;
}
}
}
@@ -481,6 +635,14 @@ public sealed class GatewaySession
public void MarkReady()
{
TransitionTo(SessionState.Ready);
lock (_syncRoot)
{
// The unattached-grace idle clock starts here, not at open: the client cannot
// subscribe before it has the session, and a slow worker start must not hand it a
// session the next sweep reaps.
_readyAtUtc ??= _eventStreaming.TimeProvider.GetUtcNow();
}
StartDashboardMirror();
}
@@ -634,6 +796,15 @@ public sealed class GatewaySession
SessionEventDistributor distributor = EnsureDistributorCreated(out bool startNow);
IEventSubscriberLease lease = distributor.Register(isInternal: true);
StartPumpIfRequested(distributor, startNow);
lock (_syncRoot)
{
// An in-process consumer (the alarm monitor) is a real subscriber for the
// unattached-grace rule: its session is attached, not abandoned. It stays out of
// _everHadEventSubscriber on purpose — that latch drives detach-grace, which is keyed
// on EXTERNAL subscribers only.
_everHadInternalEventSubscriber = true;
}
return lease;
}
@@ -853,15 +1024,72 @@ public sealed class GatewaySession
[EnumeratorCancellation] CancellationToken cancellationToken)
{
MxAccessGrpcMapper mapper = _eventStreaming.Mapper;
IWorkerClient workerClient = await GetReadyWorkerClientAsync(cancellationToken).ConfigureAwait(false);
IWorkerClient workerClient;
try
{
workerClient = await GetReadyWorkerClientAsync(cancellationToken).ConfigureAwait(false);
}
catch (SessionManagerException)
{
// The worker can die between the open's attach and this first read (an STA loop
// fault or exit code 7 right after the handshake). The readiness check then fails
// here and the worker event stream below is never reached, so its finally never
// runs; without this the session would stay Ready over a dead worker until its
// lease expired.
ObserveWorkerTerminalState();
throw;
}
TouchClientActivity(_eventStreaming.TimeProvider.GetUtcNow());
await foreach (WorkerEvent workerEvent in workerClient
.ReadEventsAsync(cancellationToken)
.WithCancellation(cancellationToken)
.ConfigureAwait(false))
try
{
yield return mapper.MapEvent(workerEvent);
await foreach (WorkerEvent workerEvent in workerClient
.ReadEventsAsync(cancellationToken)
.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
yield return mapper.MapEvent(workerEvent);
}
}
finally
{
// The worker event stream only ends when the worker client has reached a terminal
// state (it enters Closed/Faulted BEFORE completing the channel) or the pump is being
// torn down. This is the session's one always-on view of the worker, so it is where
// a worker that went away on its own is noticed.
ObserveWorkerTerminalState();
}
}
// IWorkerClient has no state-change callback, so a worker that reached Closed/Faulted
// without the gateway closing it — the client sent ShutdownWorker and the worker exited,
// the worker crashed, its pipe broke, a heartbeat timed out — used to leave the session
// reporting Ready until its lease expired (live finding). The session follows the worker
// into Faulted: it is permanently unusable (every command fails the readiness check), stays
// observable via GetSessionStatus with the reason, and the faulted-session sweep reaps it
// after FaultedGraceSeconds, accounting SessionClosed exactly once there. Not Closed: Closed
// means the gateway closed it, and a Closed session still in the registry would linger until
// its lease expired with nothing reaping it. Only a Ready session follows — during a
// gateway-initiated close the worker closing is the expected outcome, not a fault.
private void ObserveWorkerTerminalState()
{
lock (_syncRoot)
{
if (_state is not SessionState.Ready || _workerClient is null)
{
return;
}
WorkerClientState workerState = _workerClient.State;
if (workerState is not (WorkerClientState.Closed or WorkerClientState.Faulted))
{
return;
}
MarkFaulted(
$"Worker for session {SessionId} reached {workerState} outside a gateway-initiated close "
+ "(worker exited or faulted).");
}
}
@@ -901,18 +1129,6 @@ public sealed class GatewaySession
}
}
/// <summary>
/// Extends the session lease to the specified expiration time.
/// </summary>
/// <param name="leaseExpiresAt">Timestamp when the lease expires.</param>
public void ExtendLease(DateTimeOffset leaseExpiresAt)
{
lock (_syncRoot)
{
_leaseExpiresAt = leaseExpiresAt;
}
}
/// <summary>
/// Determines whether the session lease has expired.
/// </summary>
@@ -969,6 +1185,25 @@ public sealed class GatewaySession
}
}
/// <summary>
/// Determines whether a <see cref="SessionState.Ready"/> session that has <em>never</em>
/// attached an event-stream subscriber has been idle — no unary command since open or
/// since its last one — for at least the configured unattached grace. The lease monitor
/// reaps such a session exactly as it reaps an expired lease, so a client that opened and
/// vanished does not pin a worker for the full lease. Always <see langword="false"/> when
/// the grace is disabled, when the session ever had a subscriber (detach-grace and the
/// normal lease govern it from then on) or when one is attached now.
/// </summary>
/// <param name="now">Current timestamp for comparison.</param>
/// <returns><see langword="true"/> if the session is reapable as never-subscribed and idle; otherwise <see langword="false"/>.</returns>
public bool IsUnattachedGraceExpired(DateTimeOffset now)
{
lock (_syncRoot)
{
return IsUnattachedGraceExpiredCore(now);
}
}
/// <summary>
/// Attaches an event subscriber and returns a lease whose
/// <see cref="IEventSubscriberLease.Reader"/> reads the fanned public
@@ -1627,6 +1862,7 @@ public sealed class GatewaySession
return new SessionCloseResult(SessionId, SessionState.Closed, AlreadyClosed: true);
}
Exception? workerShutdownFailure = null;
if (_workerClient is not null)
{
try
@@ -1635,6 +1871,12 @@ public sealed class GatewaySession
}
catch (Exception exception)
{
// The kill is the documented fallback for a worker that will not shut down
// (typically a ShutdownTimeout on a hung worker). When it succeeds the
// worker is gone and this close has done its job, so it completes as a
// normal close carrying the shutdown failure for the caller to log; only a
// kill that ALSO fails is a close failure (SessionCloseStartedException →
// CloseFailed), because then a worker may still be running.
try
{
_workerClient.Kill(reason);
@@ -1646,12 +1888,12 @@ public sealed class GatewaySession
new AggregateException(exception, killException));
}
throw;
workerShutdownFailure = exception;
}
}
MarkClosed();
return new SessionCloseResult(SessionId, SessionState.Closed, alreadyClosing);
return new SessionCloseResult(SessionId, SessionState.Closed, alreadyClosing, workerShutdownFailure);
}
catch (Exception exception) when (exception is not SessionCloseStartedException)
{
@@ -1718,6 +1960,15 @@ public sealed class GatewaySession
/// committing to <c>Closing</c>, so a reattach that wins the race leaves the session
/// in <c>Ready</c> and usable.
/// </para>
/// <para>
/// This is a <em>reservation</em>, not the close itself: it flips the state so no
/// reattach can revive the session and no other sweep pass re-selects it, but it
/// deliberately leaves <c>_closeStarted</c> untouched. The sweep's own
/// <see cref="CloseAsync"/> that follows is the first close to run, and must report
/// <c>AlreadyClosed: false</c> so the manager accounts <c>SessionClosed</c> for it;
/// setting <c>_closeStarted</c> here made that close look like a second, concurrent one
/// and every sweep-driven close went unaccounted.
/// </para>
/// </remarks>
internal bool TryBeginCloseIfExpired(DateTimeOffset now, out bool alreadyClosing)
{
@@ -1731,7 +1982,10 @@ public sealed class GatewaySession
// Re-verify eligibility atomically. If a subscriber reattached between the sweep's
// eligibility check and this point, neither condition holds and we decline.
bool eligible = IsLeaseExpiredCore(now) || IsFaultedReapableCore(now) || IsDetachGraceExpiredCore(now);
bool eligible = IsLeaseExpiredCore(now)
|| IsFaultedReapableCore(now)
|| IsDetachGraceExpiredCore(now)
|| IsUnattachedGraceExpiredCore(now);
if (!eligible)
{
alreadyClosing = false;
@@ -1739,7 +1993,6 @@ public sealed class GatewaySession
}
alreadyClosing = _closeStarted;
_closeStarted = true;
_state = SessionState.Closing;
return true;
}
@@ -1763,6 +2016,29 @@ public sealed class GatewaySession
|| _faultedAtUtc is null
|| now - _faultedAtUtc.Value >= _faultedGrace);
// Ready only: a session still starting its worker is not idle, it is waiting, and reaping it
// mid-startup would turn a slow worker start into an OpenFailed for a client that did nothing
// wrong. Idle is measured from the later of Ready and the last client activity (last unary
// command) rather than from open, so a slow start does not eat the window and a unary-only
// client that keeps calling is never reaped by this rule. An internal (in-process) subscriber
// — the gateway's own alarm monitor — counts as attached, or the gateway would reap its own
// monitor session every cycle.
private bool IsUnattachedGraceExpiredCore(DateTimeOffset now)
{
if (_unattachedGrace <= TimeSpan.Zero
|| _state is not SessionState.Ready
|| _readyAtUtc is null
|| _everHadEventSubscriber
|| _everHadInternalEventSubscriber
|| _activeEventSubscriberCount != 0)
{
return false;
}
DateTimeOffset idleSince = _lastClientActivityAt > _readyAtUtc.Value ? _lastClientActivityAt : _readyAtUtc.Value;
return now - idleSince >= _unattachedGrace;
}
// Final terminal transition; under _syncRoot to keep _state writes single-lock.
// Closed is unconditionally terminal — TransitionTo refuses to overwrite it —
// so we don't need to re-check the precondition here.
@@ -0,0 +1,26 @@
namespace ZB.MOM.WW.MxGateway.Server.Sessions;
/// <summary>
/// Result of <see cref="GatewaySession.TryAttachWorkerClientAndMarkReadyAsync"/>: whether the
/// open's worker client was attached, and if not, which side refused it. The open maps each
/// refusal to a different failure, so the reason has to survive the call.
/// </summary>
internal enum SessionAttachOutcome
{
/// <summary>The worker client was attached and the session is <see cref="SessionState.Ready"/>.</summary>
Attached,
/// <summary>
/// The session was closed, faulted or disposed while its worker was starting (an admin
/// kill or the host drain); the caller discards the client.
/// </summary>
SessionClosed,
/// <summary>
/// The worker client was already <see cref="Workers.WorkerClientState.Faulted"/> or
/// <see cref="Workers.WorkerClientState.Closed"/> — the worker died right after its
/// handshake. Attaching it would leave a Ready session over a dead worker; the caller
/// discards the client and fails the open.
/// </summary>
WorkerTerminal,
}
@@ -2,7 +2,21 @@ using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Server.Sessions;
/// <summary>Outcome of closing a session.</summary>
/// <param name="SessionId">Identifier of the closed session.</param>
/// <param name="FinalState">State the session ended in.</param>
/// <param name="AlreadyClosed">
/// <see langword="true"/> when the session was already closed (or being closed by another
/// caller) when this close acquired the close lock, so this call did no teardown of its own.
/// </param>
/// <param name="WorkerShutdownFailure">
/// The exception the worker's graceful shutdown threw when the close had to fall back to
/// killing the worker; <see langword="null"/> when the worker shut down cleanly. The close is
/// still a success — the worker is gone and the session is closed — so this is diagnostic
/// detail for the caller's log, not an error.
/// </param>
public sealed record SessionCloseResult(
string SessionId,
SessionState FinalState,
bool AlreadyClosed);
bool AlreadyClosed,
Exception? WorkerShutdownFailure = null);

Some files were not shown because too many files have changed in this diff Show More