Compare commits

...

77 Commits

Author SHA1 Message Date
Joseph Doherty bb7aa6209f chore(plans): tasks 1-12 complete through review chains
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 56s
ci / java (push) Successful in 2m12s
ci / portable (push) Successful in 8m19s
2026-08-15 21:25:15 -04:00
Joseph Doherty 3ef56be2dd fix(worker): unconditional fault observation for abandoned pipe I/O; exception-total transport dispose 2026-08-15 21:21:25 -04:00
Joseph Doherty e913dab5db fix(worker): session-owned stream disposal unblocks and observes the net48 pipe read at teardown 2026-08-15 21:06:03 -04:00
Joseph Doherty aac79579ab perf(worker): control-frame completions resolve at the class-transition flush, not after the event batch
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.

The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.

A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.

The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
2026-08-15 21:05:57 -04:00
Joseph Doherty 9871d4772d perf(worker): value cache borrows the write-once event's instances — three clones per OnDataChange removed
MxAccessValueCache.Set deep-copied the Value (recursive for an MxArray), the
SourceTimestamp, and the Statuses RepeatedField (container plus every
MxStatusProxy) on every OnDataChange. The aliasing audit found all three
removable: the sink enqueues the event first — which stamps
WorkerSequence/WorkerTimestamp inside the queue lock — and only then runs the
postPublish hook that reaches Set, so the event is write-once by then and the
queue's ownership invariant forbids later mutation. The producer never reuses
instances (fresh MxEvent per mapper call, fresh MxValue per convert), and the
alias already existed on the read side: MxAccessSession.SucceededRead puts the
cache's own Value/SourceTimestamp/status references on every BulkReadResult,
which the worker only serializes onto the IPC pipe.

Set and CachedValue now carry the ownership contract: the cache holds borrowed
references into an enqueued, write-once MxEvent; consumers may read and
serialize, never mutate. Mutation would corrupt the still-queued event AND
invalidate QueuedEvent.Size — the enqueue-time memoized serialized size the
byte-budgeted Drain charges — so a grown message could overshoot the negotiated
frame max and fault the session with MessageTooLarge. MxAccessEventQueue's
class remark, which claimed the cache keeps an independent snapshot, is
corrected to point at the borrow.

MxAccessWriteCompletionCache.Record keeps its parallel statuses.Clone()
deliberately, with a cross-reference explaining why: it takes a bare
RepeatedField whose provenance its signature cannot constrain, and it is on the
command-rate write path, not the streaming hot path.

Tests: Set_StoresIndependentSnapshot_UnaffectedByLaterEventMutation codified
the invariant being reversed, so it is replaced by
Set_BorrowsTheEventsOwnInstances_ByOwnershipContract (Assert.Same on Value,
SourceTimestamp, and the status row). Adds the missing cached-read-path test to
MxAccessCommandExecutorTests — nothing in the worker exercised was_cached ==
true end to end — asserting the cache hit, reference identity out to the
BulkReadResult, and that no COM call is made for the read.

Not built or tested here: these are net48/x86 worker files that cannot compile
on the macOS tree. Verification is deferred to the windev gate.
2026-08-15 21:03:35 -04:00
Joseph Doherty 53881c6220 docs(dashboard): in-process page feeds, two-tier idle gating, hubs as the external surface 2026-08-15 20:51:36 -04:00
Joseph Doherty 756296886b fix(dashboard): generation-scoped idle gate in UnsubscribeAsync — no zero-subscriber pump survives 2026-08-15 20:43:17 -04:00
Joseph Doherty b0f5941e46 fix(dashboard): generation-tagged feed subscribers survive pump teardown races; drain-timeout observability 2026-08-15 20:32:07 -04:00
Joseph Doherty 38dd7678f2 fix(dashboard): guard stale-session batches inside the renderer dispatch; register IDashboardSessionEventSubscriber 2026-08-15 20:26:42 -04:00
Joseph Doherty 10406a3541 fix(dashboard): mutate provider-status state inside the renderer dispatch 2026-08-15 20:17:07 -04:00
Joseph Doherty e245237c2b feat(dashboard): in-process snapshot feed replaces the pages' loopback /hubs/snapshot hop 2026-08-15 20:16:29 -04:00
Joseph Doherty e23f816bfb feat(dashboard): in-process session event subscription feeds the viewer registry — SessionDetailsPage drops its /hubs/events hop 2026-08-15 20:15:52 -04:00
Joseph Doherty d44fe1d6b5 feat(dashboard): AlarmsPage reads provider status from IGatewayAlarmService in-process 2026-08-15 20:10:51 -04:00
Joseph Doherty 59420d8568 refactor(sessions): _subscribers to plain Dictionary — every access is under _lifecycleLock 2026-08-15 20:07:15 -04:00
Joseph Doherty 94dbe9f1af perf(sessions): fold the ReadEventsAsync pass-through into MapWorkerEventsAsync — one fewer iterator per event 2026-08-15 20:07:11 -04:00
Joseph Doherty 935f002dbf perf(grpc): consume the subscriber channel directly in StreamEventsAsync — drops the ReadAllAsync iterator hop 2026-08-15 20:06:48 -04:00
Joseph Doherty 25f07f89dd fix(tests): clear the SQLite pool before deleting the secrets path-guard temp dir — Windows sharing violation 2026-08-15 20:04:59 -04:00
Joseph Doherty a756e47682 docs(plans): deferred-findings remediation plan — 14 tasks over the six deferred findings + the Windows secrets-test bug 2026-08-15 20:03:52 -04:00
Joseph Doherty 15f188e4f9 Merge perf/review-remediation: full 2026-08-15 perf-review remediation — pipe buffers, signal-driven drains, STA message-driven waits, viewer-gated dashboard mirror, pull-model metrics, async audit pipeline, parallel teardown, truncation-safe alarms 2026-08-15 19:25:09 -04:00
Joseph Doherty 2faf243189 chore(plans): all 25 tasks complete — final integration review: ready to merge
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m6s
ci / portable (push) Successful in 8m57s
2026-08-15 18:07:33 -04:00
Joseph Doherty f4b065b9f6 docs(worker): reviewer follow-up comments and tests from the remediation reviews
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m11s
ci / portable (push) Successful in 11m5s
The worker-side half of the review tail. Tests and comments only — nothing here
changes worker behavior, and none of it compiles on the macOS tree (net48/x86),
so it was reviewed line by line against the already-windev-validated files.

- MxAccessHandleRegistryTests gains the multi-candidate case behind
  MxAccessSession.TryGetCachedReadFor's fall-through: one tag under two item
  handles, the lower registered-but-unadvised and the higher advised. Asserted
  at the registry rather than the session because the session's read path needs
  a live MXAccess COM instance; what the registry owes the scan is the stable
  ascending candidate order and a per-item-handle (not per-tag) advice index,
  and both are pinned here along with the fall-through contract in prose.
- A single adversarial lifecycle test — register, advise, re-register the same
  item handle under a new tag, unadvise, unregister the server — asserting every
  index agrees after each step. The individual transitions were already covered;
  what was not was that they compose, and a stale entry in any one index
  resurrects a handle MXAccess has already retired.
- StaWaitHelperTests.WaitForSignalOrMessages_PreSignalledHandle_ReturnsImmediately
  drains pending messages first, like the other two wait tests. Without it a
  stale message can end the wait instead of the handle, failing the
  signal-consumed post-condition for an unrelated reason.
- GatewayTesting.md records the two findings from the Task 24 windev gate:
  SecretsStorePathGuardTests.CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt
  fails deterministically on Windows on main too (SQLite pooling holds secrets.db
  open across the cleanup's recursive delete; pre-existing, tracked separately),
  and the StaWaitHelper timing tests' flake signature on a loaded box is a
  message wake — the helper working as designed — not a broken wait.
2026-08-15 17:56:16 -04:00
Joseph Doherty dc2df628e3 chore(followups): reviewer-recommended tests, comments, and hardening from the remediation reviews
The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.

Hardening (behavior changes, all narrow):

- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
  fifth bulk-write kind added upstream without a filter case here would have
  shipped the DENIED entries to the worker while reporting them denied to the
  caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
  cannot date. The retention sweep deliberately preserves such rows (SQLite's
  datetime() yields NULL, so the DELETE never matches), which guaranteed the
  dashboard's recent-audit view would meet one eventually and lose the whole
  page to it. The row is now reported at DateTimeOffset.MinValue with every
  other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
  detaching the drain, so a producer that raced past the attached check takes
  the write-through branch instead of stranding its event in a buffer nobody
  reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.

Tests:

- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
  existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
  which way it fails. Nothing validates key-id length at creation, so
  docs/Diagnostics.md's "which no issued key id does" is now stated as the
  heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
  evicted between the Check and the Reset: inert, and clears nobody else's
  block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
  thread, so overshoot must be transient and proportional to the in-flight
  inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.

Comment/doc accuracy:

- EventsHubViewerRegistry.ReleaseConnection records that it relies on
  SignalR's default sequential per-connection dispatch
  (MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
  needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
  lock-free reader, but the pump's single capture point is inside _replayLock;
  the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
  "cannot be observed here" now excepts the DisposeAsync abandon path. The
  churn test names its ConcurrentDictionary bucket-order assumption and that a
  violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
  docs/Sessions.md) were wrong: the sequential drain leaked too, because
  KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
  on the first session for zero kills. Reworded to "fixes a leak the
  sequential drain also had", with the sweep-bound/shutdown-unbound
  ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
  sweep rather than cancelling it, with the bounded overrun stated.
  SessionShutdownHostedService.StopAsync records that its cancellation-logging
  branch is now unreachable.
2026-08-15 17:54:31 -04:00
Joseph Doherty 7755745f2f chore(plans): Task 24 windev gate green — slnx+x86 builds clean, worker 499/499, gateway 1039/1040 (1 pre-existing main failure, 1 isolated-pass load flake) 2026-08-15 17:44:27 -04:00
Joseph Doherty b2d8dd70ed chore(plans): Phase B implementation complete; as-built note for Task 17
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m12s
ci / java (push) Successful in 2m26s
ci / portable (push) Successful in 7m56s
2026-08-15 17:35:05 -04:00
Joseph Doherty 58d97ad4e8 perf(worker): memoize event frame size at enqueue; drain stops sizing under the STA's lock 2026-08-15 17:27:47 -04:00
Joseph Doherty b5ea6bb461 fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance 2026-08-15 17:19:41 -04:00
Joseph Doherty 7c9add3d73 fix(worker): StaWaitHelper regression tests + full-drain contract notes; consolidate wait P/Invoke 2026-08-15 17:17:10 -04:00
Joseph Doherty 896d81e286 docs(worker): record the SemaphoreSlim/STA dispatch invariant + single-waiter contract; harden fake 2026-08-15 17:13:17 -04:00
Joseph Doherty 25cbe5cd3e fix(worker): guard timestamp-format derivation against pathological culture patterns 2026-08-15 17:10:58 -04:00
Joseph Doherty 13583322b5 perf(worker): launcher-configurable event queue capacity 2026-08-15 17:10:54 -04:00
Joseph Doherty f3e1de5f37 fix(alarms): truncation-safe transitions; perf: single-pass alarm parse; configurable poll cadence 2026-08-15 17:04:06 -04:00
Joseph Doherty 94fdc18c3c perf(worker): exact-format timestamp parse and compiled status-field accessors on the event path 2026-08-15 16:59:52 -04:00
Joseph Doherty f4a6cb1db2 perf(worker): signal-driven event drain — removes the 25 ms latency floor and idle wakeups 2026-08-15 16:59:13 -04:00
Joseph Doherty dc9424d3bd perf(worker): message-driven completion waits — the STA pumps continuously while waiting 2026-08-15 16:58:54 -04:00
Joseph Doherty afec56d03b perf(worker): reverse tag index + memoized views + indexed removals in the handle registry 2026-08-15 16:58:02 -04:00
Joseph Doherty f56798aeb9 perf(worker): pooled frame buffers — brings the net48 codec up to the gateway side's GWC-30 pattern 2026-08-15 13:28:29 -04:00
Joseph Doherty 13df92fbd8 chore(plans): Phase A complete — gate green at 1015/1015 2026-08-15 13:25:05 -04:00
Joseph Doherty 95f8ba918d fix(sessions): exception-total shutdown body with non-cancellable kill fallback 2026-08-15 13:19:08 -04:00
Joseph Doherty 0bc13b5292 perf(sessions): bounded-parallel teardown in lease sweep and shutdown 2026-08-15 12:43:30 -04:00
Joseph Doherty a1a38b5538 fix(ipc): cancellation-priority timeout classification; structurally-enforced no-throw cancel send 2026-08-15 12:41:38 -04:00
Joseph Doherty 1742e38c10 fix(events): graceful unregister no longer masquerades as overflow under FailFast 2026-08-15 12:38:20 -04:00
Joseph Doherty 7b2d04605e fix(audit): write-through on completed channel, poison-batch isolation, drain-fault fallback 2026-08-15 12:37:09 -04:00
Joseph Doherty 07b83561d1 docs(events): correct the capture-under-replayLock rationale 2026-08-15 12:34:36 -04:00
Joseph Doherty f920b4cbf5 fix(dashboard): clamped CAS retry loop in snapshot hub connection counter + direct counter tests
Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.

Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.

The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.

Also documents that the API-key refresh's pre-gate time check races benignly.
2026-08-15 12:32:58 -04:00
Joseph Doherty 44ca7c8623 fix(dashboard): enforce/document the advised-set cap honestly; cover handle-0 and oversize-read paths 2026-08-15 12:31:36 -04:00
Joseph Doherty 7c1ea12331 perf(ipc): WaitAsync command timeouts + forward WorkerCancel so a timed-out COM call frees the STA 2026-08-15 12:28:09 -04:00
Joseph Doherty 7171892984 perf(grpc): O(1) unconstrained bulk fast path, direct filtered-command build, cache eviction, capacity hints 2026-08-15 12:24:17 -04:00
Joseph Doherty 88d38bb900 perf(auth): allocation-free token parsing; single partition-key build per RPC 2026-08-15 12:23:25 -04:00
Joseph Doherty 3ff073d1ea perf(grpc): transfer reply ownership instead of deep-cloning every worker reply 2026-08-15 12:22:59 -04:00
Joseph Doherty 8e2066b4bd docs(dashboard): restore the advised-set LRU paragraph dropped by the snapshot commit
77c5731 committed this file from a working copy that predated 75e3dc2's
advised-set section, silently deleting it. Puts the paragraph back verbatim;
no other content changes.
2026-08-15 12:22:33 -04:00
Joseph Doherty 9735ac3b7c perf(metrics): pull-model worker queue gauge (fixes last-writer-wins), Interlocked command counters 2026-08-15 12:21:51 -04:00
Joseph Doherty 77c5731b7b perf(dashboard): idle-gate the snapshot tick; cache static config; bound key-list refresh
The snapshot publisher broadcast to Clients.All on every ~1s tick forever, with
zero viewers. Each tick cost a session-registry snapshot and sort, a metrics
snapshot that copies dictionaries under the global metrics lock, a full rebuild
of EffectiveGatewayConfiguration, and a SQLite read of the API key table.

DashboardSnapshotHub now counts live connections into the singleton
DashboardSnapshotHubConnectionCounter (clamped at zero, since SignalR can call
OnDisconnectedAsync for a connection whose OnConnectedAsync faulted). The
publisher drives the snapshot enumerator by hand instead of await foreach: with
no connections it does not call MoveNextAsync at all, so the producing iterator
stays suspended at its yield and no snapshot is built — the gate removes the
build, not just the broadcast. It re-checks once a second, so the first viewer
resumes the tick within about one interval; that viewer is seeded immediately by
DashboardPageBase's synchronous GetSnapshot() and by the hub's OnConnectedAsync.

Two per-tick costs are bounded independently of the gate: the effective
configuration is startup-static (options are bound once at boot and never
reloaded), so it is built once and cached; and the API key summaries refresh at
most every 15s, since the list only changes when an operator creates, rotates,
or revokes a key. Only a successful refresh restarts the interval, so a failed
or timed-out read is still retried on the next tick with the previous summaries
left on screen.
2026-08-15 12:21:49 -04:00
Joseph Doherty e04b1c9199 perf(events): copy-on-write subscriber snapshot in fan-out pump 2026-08-15 12:21:42 -04:00
Joseph Doherty 75e3dc2794 perf(dashboard): LRU cap on the shared live-read session's advised set 2026-08-15 12:21:13 -04:00
Joseph Doherty f1e26fed4f perf(alarms): memoize CurrentAlarms projection, invalidate on mutation 2026-08-15 12:20:51 -04:00
Joseph Doherty ca34a2d65d fix(logging): fail-closed bearer redaction; hoist per-request logger creation 2026-08-15 12:17:27 -04:00
Joseph Doherty e2ac5d117a perf(audit): bounded async audit writer with batched inserts, one-time bootstrap, retention sweep 2026-08-15 12:17:22 -04:00
Joseph Doherty 6c5218913b perf(dashboard): gate event mirror on live viewers — no clone, no send for unwatched sessions
DashboardEventBroadcaster.Publish ran a deep protobuf Clone (redaction is on by
default) and a group SendAsync for every event of every session, before anything
checked whether a dashboard client was actually watching. In the steady state the
session:{id} group is empty, so that work was thrown away per event.

SignalR does not expose group membership, so EventsHub now mirrors its own
add/remove into a singleton EventsHubViewerRegistry, and OnDisconnectedAsync
releases everything a dropped connection held (SessionDetailsPage disposes the
connection rather than unsubscribing). Publish returns early when the session has
no viewers, before the redaction clone. Watched sessions behave exactly as before.

Lazy mirror-lease start/stop was deliberately not attempted — it entangles the
dashboard with SessionEventDistributor subscribe lifetime for no saving beyond
this gate; recorded in docs/GatewayDashboardDesign.md.
2026-08-15 12:07:33 -04:00
Joseph Doherty da8463534b perf(ipc): give worker pipes real OS buffers instead of zero-quota rendezvous 2026-08-15 12:02:43 -04:00
Joseph Doherty 9958f80026 docs(plans): perf review remediation plan 2026-08-15 12:01:18 -04:00
Joseph Doherty 5744aad028 test(dashboard): prove the Secrets nav gate exists, by its absence
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m15s
ci / java (push) Successful in 2m16s
ci / portable (push) Successful in 9m19s
The policy tests shipped at 1c30611 could not detect a deleted gate. They
assert that secrets:manage admits an Administrator and refuses a Viewer —
true, and library behaviour this repo did not author. The wiring is the
only thing that change introduced, and nothing covered it.

The point is sharper for repos where the link already existed before
gating, which includes this one: "an Administrator still sees it" is
identical to the pre-change behaviour, so it cannot distinguish a working
gate from an inert AuthorizeView. Only the negative observation proves a
gate is there at all.

SecretsNavRenderTests renders MainLayout through the framework's static
HtmlRenderer — no component-testing package, because the assertion is
about emitted markup rather than interactivity — and asserts:

- absent for a Viewer, and for an anonymous caller (the load-bearing pair)
- present for an Administrator (the control: without it, a rail that
  rendered nothing at all would satisfy both absence assertions and the
  suite would report a working gate over a blank page)
- the ungated API Keys sibling still present for a Viewer, so a later
  "consistency fix" that hides it fails loudly rather than silently
  removing read access

Confirmed non-vacuous by mutation rather than by argument: with the
AuthorizeView removed from the layout, both absence tests go red and all
three original policy tests stay green.

Build 0 warnings / 0 errors; suite 899/899 (895 + 4).
2026-08-13 10:02:48 -04:00
Joseph Doherty 87d575dce4 Merge feat/secrets-nav-role-gate: side rail's Secrets link gated on secrets:manage, the policy the page itself enforces
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m15s
ci / java (push) Successful in 2m22s
ci / portable (push) Successful in 7m50s
2026-08-13 09:46:14 -04:00
Joseph Doherty 1c30611b1e feat(dashboard): gate the side rail's Secrets link on secrets:manage
Family-wide nav sweep: the Secrets management page should be linked from
each app's UI, visible to Administrator-role users only.

The link already existed in MainLayout's Admin section. The gate did not:
the rail rendered every item for every visitor, including a Viewer and the
anonymous-localhost read-only identity. Not an access hole — the mounted
page carries [Authorize(Policy = "secrets:manage")], so a Viewer clicking
through was denied — but a dead link presented as a live one. There was
also no existing role-gated nav pattern to follow; the rail's only
AuthorizeView was the footer's signed-in/signed-out split.

Gated on the POLICY rather than a role literal, so nav visibility cannot
drift from what the page enforces. In this host the two are equivalent:
GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting
roles (secrets-manager, secrets-reveal) are unreachable. The policy form
stays correct if that ever relaxes, where a role literal would then hide
the link from users who can use the page.

API Keys is deliberately left ungated. It looks like the same case and is
not: ApiKeysPage renders for a Viewer with write affordances hidden, so
hiding its link would remove legitimate read access. The secrets page has
no read-only mode. The rule is "gate the link when the page denies the
role outright", not "gate everything under Admin".

Coverage: three tests pin the policy's verdict per principal
(Administrator admitted, Viewer refused, unauthenticated refused), and
/admin/secrets joins the canonical route list — it is the one nav
destination mounted from an RCL rather than declared here, so a routing
regression could remove it without touching this repo's pages. The
principal helper sets an authentication type deliberately: without one
the role assertions would pass vacuously for the wrong reason.

Not a rendering test — the suite has no component-testing harness, and
adding one to assert a single AuthorizeView would be a large dependency
for a small claim.

Build 0 warnings / 0 errors; suite 895/895.
2026-08-13 09:33:04 -04:00
Joseph Doherty 1cb14d22bf chore(deps): bump ZB.MOM.WW.Auth to 0.2.1 — AD continuation-referral fix
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m12s
ci / java (push) Successful in 3m18s
ci / portable (push) Successful in 9m59s
2026-08-13 08:30:53 -04:00
Joseph Doherty 55bca95ad2 Merge docs/deploy-provenance-rows: 08-11/08-12 wonder deploys recorded; backup dirs read as a provenance chain
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m28s
ci / java (push) Successful in 2m46s
ci / portable (push) Successful in 8m27s
2026-08-12 04:20:42 -04:00
Joseph Doherty 5fe96b6677 docs(runbook): record the 08-11/08-12 wonder deploys and the backup-chain technique
Adds the two wonder rows that were deliberately withheld on 2026-08-11
while the pre-55f2889 SHA was unsettled. It is settled: b948e69 (08-09)
and 0a9715d (08-11) were never competing claims about one binary, they
are two deploys two days apart.

What settled it is worth recording as a technique in its own right, so
it goes in as a fourth way to identify a build: each Server.bak.<ts>
holds the exe that deploy REPLACED, so a VersionInfo sweep across the
backups reconstructs a host's deploy history from the host alone — no
repo access, no deploy record. The subtlety that makes it readable is
that a backup's timestamp dates the NEXT deploy, not the build inside
it. Reading a file version is non-destructive, unlike opening a SQLite
store in a backup directory.

Also records the full garbage version stamp recovered from the 08-09
binary, because the failure mode is a false positive rather than a
blank: "0.1.2+fatal:..." reads like a version that succeeded and then
picked up noise, when the leading 0.1.2 is just the static base <Version>
every build carries. For a binary in that window the commit is not
recoverable from the binary at all, so finding nothing is the expected
result rather than evidence against a SHA established another way.

Provenance is stated per cell rather than uniformly: the worker SHAs on
the new rows are carried forward and marked unconfirmed, b948e69 rests
on PDB hash plus the contemporaneous record and never on a stamp, and
55f2889 was read from the live stamp, which is trustworthy only because
it postdates 0152180.
2026-08-12 04:20:38 -04:00
Joseph Doherty 2c0daee481 Merge chore/shared-lib-latest-pins: every ZB.MOM.WW pin to newest published; Auth 0.2.0 obliged the LdapOptions shadow mirror
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m24s
ci / portable (push) Successful in 9m24s
2026-08-12 04:17:34 -04:00
Joseph Doherty 62394f5b85 chore(deps): move every ZB.MOM.WW pin to the newest published version
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m11s
ci / java (push) Successful in 2m5s
ci / portable (push) Successful in 9m15s
Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui
0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry
and Telemetry.Serilog were already at the newest version on the feed.

Checked against the shared-lib source rather than the version numbers,
because these packages are versioned as a family and a bump is not by
itself evidence that the package changed:

- Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC
  failover (FallbackServers, endpoint walk with sticky preference,
  boot-time entry validation). Purely additive; the default is empty,
  which leaves single-endpoint behaviour unchanged.
- Health 0.3.0 carries a breaking change, but every line of it is in
  ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched
  the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0.
- Secrets 0.6.2 is a message-only change: one validator string literal
  gains mounted-volume guidance. SecretsStorePathRules is untouched.

The four non-csproj files are not a separate feature. Configuration/
LdapOptions is a deliberate shadow of the shared type and carries an
explicit warning to mirror any new upstream field, because AddZbLdapAuth
binds the whole MxGateway:Ldap section onto the shared options. So
FallbackServers is live on our config surface the moment the package
lands, and without the mirror an operator could configure a backup DC
that works but is invisible on the dashboard's Settings page. The
Settings row renders "none" when empty, since that is the answer someone
who believes a backup DC is configured actually needs.

Entry syntax is deliberately NOT re-validated here: the shared validator
already fails the boot on a malformed entry and owns the (internal)
parser, so a second copy would drift. Note both validators skip entirely
when Ldap:Enabled is false.

Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused
anywhere in the tree), so the upgrade could not break startup on a
newly-recognised key either way.

Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live
LDAP tests are opt-in and were not run, so the failover path itself is
covered only by the shared library's own tests.
2026-08-12 04:16:23 -04:00
Joseph Doherty 55f2889c24 Merge fix/secrets-prehost-content-root: run the store-path guard where the store is actually created
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 3m12s
ci / portable (push) Successful in 10m36s
0.6.0 landed the rules but not the enforcement on this path: the pre-host secrets
container has no IHostEnvironment, so the library skipped the content-root check
and the migrator created the database before the real host could refuse it. The
pin to 0.6.1 alone does not close that — the call site has to pass the content
root explicitly, which is why this is a code change and not a version bump.
2026-08-12 02:13:40 -04:00
Joseph Doherty 5fdd8a570a fix(secrets): run the store-path guard in the pre-host container (0.6.1)
ci / windows-x86 (push) Successful in 1m19s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m59s
ci / portable (push) Successful in 10m48s
0.6.0 put the store-path rules in the shared library, but the guard was not
running at the moment that matters here.

CreateBuilder resolves ${secret:} references before the host exists, using a
throwaway ServiceCollection that contains no IHostEnvironment — and it runs the
store migrator, which creates the database. The library resolved the content
root from IHostEnvironment alone, so it could not distinguish "no content root"
from "no host registered" and skipped the under-content-root rule entirely. The
store was created at the rejected path; the boot then failed a moment later when
the real host validated. The leftover empty database with its -wal/-shm siblings
is exactly the artifact that made the 2026-08-09 credential loss read as "the
database is there, it's just empty".

The pin alone does not close this. An app with a correctly configured path shows
no symptom and is still unprotected, because the guard simply is not running when
the store is created. 0.6.1 adds a 4-argument AddZbSecrets overload taking the
content root explicitly, and the call site has to use it. The in-host
registration below needs nothing.

Verified by removing the fix rather than by observing a clean boot — which is how
this survived its first release. With the 3-argument overload the new test fails
by finding a created database at
src/ZB.MOM.WW.MxGateway.Server/probe-secrets-*.db: inside the source tree, since
that is what the content root resolves to under test.

Two things about the test itself, both of which it would have been easy to get
subtly wrong:

It asserts no-file-created before asserting that startup threw. "It threw" is the
weaker claim, and asserting it first masks the stronger one — the run that proved
this defect would have reported "no exception was thrown" and said nothing about
the database sitting in the source tree.

The accepting case asserts the database *is* created, not merely that nothing
threw. A not-null builder is close to a tautology once no exception escaped, and
it would still pass if the pre-host container stopped opening the store at all —
which would also quietly void the rejecting case, since that one can only observe
a file the migration would otherwise have written. The two assertions hold each
other up.

Found by HistorianGateway's adoption, which probed the rejected paths instead of
observing a successful boot.
2026-08-11 08:54:06 -04:00
Joseph Doherty 9bc70d1af3 Merge feat/session-health-check-and-store-path-guard: session health probe + store-path guard + Secrets 0.6.0
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 17m53s
Three separable changes, each building on its own: the mxaccess-sessions health
check on the active tier, the content-root rule that closes the gap the 2026-08-09
credential-store loss went through, and the Secrets re-pin (0.2.3 -> 0.6.0) that
moves the same rules into the shared library.
2026-08-11 08:43:43 -04:00
Joseph Doherty 6ba52a68f0 build(secrets): re-pin ZB.MOM.WW.Secrets to 0.6.0
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m18s
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 8m14s
0.6.0 moves the store-path rules into the shared library — both the rooted check
and the content-root check — with a SecretsOptionsValidator wired into
AddZbSecrets and validated on start, so all four consuming apps get the guard
from one implementation rather than four copies. mxgw therefore adds no local
validator over the Secrets section.

This is a four-minor jump, not a re-pin: mxgw was on 0.2.3 and skips 0.3.0,
0.4.0, 0.4.1, 0.5.0 and 0.5.1 in one step. Verified rather than assumed —
diffing the 0.2.3 and 0.6.0 assemblies shows the added surface is the new path
rules and their plumbing (AddIfNotRooted, AddIfUnderContentRoot,
ComputeDefaultSqlitePath, DefaultSqlitePath, IValidateOptions, IsPathRooted,
GetFullPath) and nothing touching store or delete behaviour. Secrets.Ui is
byte-identical across the range: both assemblies are 39424 bytes and differ only
in the version stamp, because this package family shares one version across
every package even when a release touches only one of them. So the browser gate
run against the /admin/secrets delete modal on 0.2.3 still covers what ships
here.

The library default is LocalApplicationData-derived so the family's
cross-platform apps still boot locally. The gateway keeps its own
CommonApplicationData value, which always wins — see the note on
ApplyDefaultSecretsStorePath for why the difference is deliberate and why
deleting that method as a redundancy would silently move the store.
2026-08-11 08:42:28 -04:00
Joseph Doherty 882c7ca3cd fix(config): reject credential and cache paths inside the application directory
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.

MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.

GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.

The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.

Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.

Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
2026-08-11 08:42:16 -04:00
Joseph Doherty c69a1c441b feat(diagnostics): report MXAccess session health on the active probe
Adds a `mxaccess-sessions` health check reporting how many MXAccess sessions are
healthy. Each session is one worker process holding one MXAccess COM instance —
a live connection into a Galaxy — so this answers "how many Galaxy connections
are healthy" in the vocabulary the code actually uses.

Zero sessions is Healthy, deliberately, and the rest of the design follows from
that. The gateway opens a session when a client asks and holds none otherwise,
so an idle gateway is working normally. A count threshold ("unhealthy below N")
would sit red forever on a host nothing dials yet, and a permanently red probe
is one operators stop reading — which leaves them worse off than no probe. The
check therefore grades on whether the sessions that exist are usable: nothing
faulted is Healthy, some faulted beside a ready or starting one is Degraded, and
every session faulted is Unhealthy. Counts ride along as entry data for the
family Overview dashboard.

Tagged `active` rather than `ready` for the same reason. Readiness decides
whether the process should be sent traffic, and a gateway with no sessions is
ready to serve — unlike the auth store, which every call depends on. Failing
readiness here would pull a working gateway out of rotation over a condition its
own clients create.

Reads ISessionRegistry, which already exposes Snapshot(); ISessionManager stays
the command surface and grows no enumerator.
2026-08-11 08:41:54 -04:00
Joseph Doherty 22a34f7f31 Merge docs/runbook-evidence-precision: what the deployed-build identification evidence rests on
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m51s
ci / portable (push) Successful in 10m51s
2026-08-11 06:06:48 -04:00
Joseph Doherty f6b6184e70 docs(runbook): state what the identification evidence actually rests on
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m55s
ci / portable (push) Successful in 10m39s
Two precision fixes to the runbook landed in fd0e88e, both making it weaker in
the sense that matters.

The PDB source-hash entry read as though hashes settled the 2026-08-09
provenance question by themselves. They did not: a hash match and a
contemporaneous deploy record written that evening independently named the same
two commits, and the agreement is what makes the result trustworthy. A hash
match alone tells you which sources a binary was built from — not that the build
was intentional, nor which host it reached. A future reader holding only one of
the two derivations should know to look for a second.

The two-swap timeline was asserted from the investigation's timestamps, which a
later reader cannot re-derive. It is also confirmable from artifacts still on
disk — windev keeps two worker backup directories from that day and wonder one,
because there were two worker operations. Records that, so the story can be
checked against the boxes rather than believed.
2026-08-11 06:06:48 -04:00
Joseph Doherty e5dbcee17c Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 3m26s
ci / portable (push) Successful in 11m7s
2026-08-11 06:05:23 -04:00
135 changed files with 14221 additions and 907 deletions
+20
View File
@@ -140,6 +140,26 @@ Two viable A.2 designs given the probe data:
poll period; modest CPU floor because the call is cheap. Matches
the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a
poll loop internally.
As shipped, this is the chosen design, and the cadence is **no
longer fixed at 500 ms**: it is the 500 ms *default* of
`MxGateway:Alarms:PollIntervalMilliseconds` (range 100 ms 1 h),
which the gateway hands the worker through the
`MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable. The
per-fetch cap is likewise configurable
(`MxGateway:Alarms:MaxAlarmsPerFetch`, default 1024).
One snapshot rule matters when reading the capture below: a fetch
that returns exactly the cap is treated as **truncated**, and the
worker *merges* it into the retained snapshot instead of replacing
it. `GetXmlCurrentAlarms2` caps its reply with no "more available"
flag, so a capped reply is authoritative about presence only —
alarms it had no room to mention are retained rather than allowed
to vanish, because their disappearance is what the gateway's
reconcile pass reads as a clear. Only a sub-cap fetch replaces the
snapshot wholesale and can therefore clear alarms. See
`docs/DesignDecisions.md`, "Alarms — a capped snapshot fetch never
implies a clear".
2. **Hook AVEVA's internal window.** Discover AVEVA's own window
(`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or
`SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's
+120
View File
@@ -135,6 +135,76 @@ alarm state is gateway-wide, not session-scoped — every client wants the same
current set plus updates, and forcing each to own a worker would multiply AVEVA
polling load for no benefit.
### Alarms — a capped snapshot fetch never implies a clear
Decision (2026-08-15): when the worker's `GetXmlCurrentAlarms2` fetch comes back
holding exactly `MxGateway:Alarms:MaxAlarmsPerFetch` records, the worker treats
the snapshot as **truncated** and merges it into the retained snapshot instead
of replacing it. Alarms the capped reply did carry update normally; alarms it
had no room to mention are retained untouched.
The COM API caps its reply at `maxAlmCnt` and exposes no "more available" flag,
so a reply sitting exactly on the cap is indistinguishable from a galaxy that
happens to hold exactly that many active alarms. Both are treated as truncated,
because the two error directions are not symmetric.
Nothing in the worker emits a Clear transition. The clear is an **inference**:
`WnWrapAlarmConsumer.ComputeTransitions` produces no transition for an alarm
that disappears from the snapshot, and `GatewayAlarmMonitor.ApplyReconcile`
later diffs its cache against `SnapshotActiveAlarms()` and broadcasts a Clear
for every cached alarm the worker no longer reports. Before this decision, a
capped fetch shrank that snapshot, so every alarm past the cap was broadcast as
cleared while still standing — a silent, galaxy-wide false clear on exactly the
alarm floods where the cap is reached.
Consequences, and how this sits with the existing failover/reconcile design:
- **The suppression is an eviction guard, not a transition filter.** It lives in
the snapshot update inside `PollOnce`, not in `ComputeTransitions`, which was
never going to emit anything for a disappearance. The reconcile/dedup
machinery (`_clearedByReconcile` tombstones, the NEXT-03 duplicate-Clear
suppression) is untouched: it still sees the same shape of snapshot, only
with the truncated poll's unmentionable alarms still present.
- **It preserves at-least-once, idempotent application.** The failure mode
becomes bounded staleness — a genuinely cleared alarm can linger until the
first sub-cap fetch evicts it, and the reconcile then broadcasts its Clear
late. A late Clear is repaired by the next complete poll; a Clear that never
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
back. Consumers already apply transitions as "set this alarm to this state",
so a repeated or delayed Clear is absorbed.
- **Under *sustained* truncation, some intermediate history is lost — end state
is not.** For an alarm that stays outside the fetch window, a full
clear→re-raise cycle that begins and ends between two sightings emits **no
transitions at all**: the retained record is identical before and after, so
the diff sees nothing to report. Consumers that render current state are
correct; consumers that *count occurrences* lose an event. Likewise, an
operator acknowledgement of an out-of-window alarm does not reach the feed
until that alarm re-enters a fetch window, at which point the reconcile
repairs the acked state. This is a strictly better failure than the
pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
on every poll), but it is not lossless, and it is another reason a
persistently truncating deployment is a configuration defect to fix rather
than a mode to run in.
- **It does not synthesize anything.** Suppressing an inference is the opposite
of inventing an event; no transition is fabricated on a truncated poll.
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
`IMxAccessAlarmConsumer` is live; the guard is internal to the wnwrap
consumer's own snapshot bookkeeping and changes neither the failure counting
that triggers failover nor the subtag standby's snapshot, which is built from
a bounded watch-list and has no per-fetch cap to hit.
- **Operators get told, weakly.** A truncated poll logs a rate-limited (once
per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
counts, and the running truncated-fetch total — identifiers and counts only,
never tag names, values, limits, or comments. Be honest about its reach: it
goes to the worker's console/stderr, which is captured on dev hosts but is
not a metric, not a dashboard tile, and not part of any session-status or
alarm-feed payload, so a production deployment can truncate indefinitely
without anyone noticing. Surfacing truncation as a **structural** degraded
status (a field on the alarm-provider mode/status surface the dashboard and
`StreamAlarms` consumers already read) is filed as a follow-up; until it
lands, the log line is the only signal. A galaxy that truncates persistently
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
## Session-Resilience Epic Scope
Decision (2026-07-09, archreview TST-04; migrated here 2026-08-07 from the retired
@@ -228,6 +298,56 @@ Storage recommendation:
administrators.
- Require TLS when the gateway is reachable off-machine.
## Audit Pipeline
Decision: audit is asynchronous, bounded, and swept.
The canonical `IAuditWriter` contract has always been best-effort — a failed audit write is
logged and swallowed so it cannot abort the action that produced it. The registered writer is
`ChannelAuditWriter`, which makes the cost of that promise explicit: a producer enqueues onto a
4096-event bounded channel and returns, and `AuditDrainService` commits up to 64 buffered events
per transaction. This exists because constraint denials are emitted per denied tag inside bulk
RPC loops: a partially denied 1,000-tag request previously awaited 1,000 sequential SQLite
inserts — each re-running `CREATE TABLE IF NOT EXISTS` — against the same database file every
authenticated call reads. The schema bootstrap now runs once, from the drain's `StartAsync`.
When the channel is full the newest event is dropped and counted rather than blocking the
producer: a stalled audit database must cost audit completeness, not gateway availability. Drops
are logged once and reported in aggregate on each sweep. Shutdown drains what is buffered under a
2-second cap.
Every other failure mode degrades to synchronous writes rather than to silent loss. The writer
falls back to the direct path whenever nothing is draining: before the drain attaches, after it
detaches, where no hosted service runs at all (the `apikey` admin CLI), and when the channel has
been completed — so no attach/detach sequence can leave producers filling a buffer with no reader.
If the drain loop itself dies it detaches the writer on the way out, which reverts every producer
to the direct path. A batch that will not commit is retried one event at a time, so an unwritable
row costs only itself instead of the up-to-63 good events sharing its transaction.
**All** audit is channelled, including admin and CRUD records — dashboard key create/revoke/rotate,
session Close/Kill, and the library-forwarded API-key lifecycle entries. The alternative considered
was keeping those on the synchronous writer and channelling only high-volume denial audit. It was
rejected because a single dashboard key-create emits two records through two different seams (the
library's `create-key` via `IApiKeyAuditStore`, and the enriching `dashboard-create-key` via
`IAuditWriter`); splitting them across two durability regimes gives an auditor a per-producer
matrix to reason about instead of one rule. The residual exposure is explicit: **if the gateway
process dies between the enqueue and the batch commit, buffered audit events are lost.** The window
is bounded by drain latency — the drain wakes on every write and commits immediately, so it is
sub-millisecond under normal load — and it does not apply to the `apikey` CLI, which writes
synchronously. Audit is a best-effort record of what the gateway did, not a write-ahead log of what
it is about to do; a deployment that needs crash-durable admin audit should ship the events off-box
rather than rely on this table.
`MxGateway:Security:AuditRetentionDays` (default 90, minimum 1) bounds the table: the drain sweeps
at startup and hourly, deleting older rows. Retention cannot be configured off. The sweep compares
through SQLite's `datetime()` rather than on the stored ISO-8601 text. Text comparison is correct
only while every row is UTC-normalized — which the canonical model guarantees for rows written
through the store, but not for rows that entered the table any other way — and on a mixed-format
column it silently deletes live audit, because `2026-05-17T09:00:00-05:00` is two hours after a
`2026-05-17T12:00:00+00:00` cutoff yet sorts before it. Comparing instants is correct however the
text got there, and a timestamp `datetime()` cannot parse yields NULL, so undateable audit is kept
rather than swept.
## Authorization
Decision: start with scope checks by command category.
+55 -37
View File
@@ -84,42 +84,36 @@ The names match the MXAccess command list in `gateway.md` exactly. `Write` and `
### API key redaction
`RedactApiKey` is built around the `mxgw_` API key format issued by the gateway. It preserves the bearer scheme and the key id segment so that operators can correlate a log entry to a specific principal, but always strips the secret tail:
`RedactClientIdentity` is the single redaction path for identity-bearing values; `RedactApiKey` is a
name-preserving alias for it. Redaction **fails closed**: the only value that survives with any of its
content is a gateway-issued `mxgw_<key-id>_<secret>` key, whose key id is kept so operators can
correlate a log entry to a specific principal.
```csharp
public static string? RedactApiKey(string? authorizationHeader)
{
if (string.IsNullOrWhiteSpace(authorizationHeader))
{
return authorizationHeader;
}
| Input | Output | Why |
|-------|--------|-----|
| `Bearer mxgw_operator01_super-secret` | `Bearer mxgw_operator01_[redacted]` | Recognized gateway key; key id identifies the principal |
| `Bearer eyJhbGciOi…` (any foreign token) | `Bearer [redacted]` | Structure is unknown, so the whole credential goes |
| `Basic dXNlcjpwYXNz` | `Basic [redacted]` | Same, for any recognized scheme |
| `Bearer mxgw_operator01` (no secret separator) | `Bearer mxgw_[redacted]` | No trustworthy key-id boundary |
| `Bearer` (scheme only), `anonymous`, `some junk` | `[redacted]` | No scheme/credential split that can be trusted |
| `null`, `""`, whitespace | unchanged | Nothing to redact |
const string bearerPrefix = "Bearer ";
if (!authorizationHeader.StartsWith(bearerPrefix, StringComparison.OrdinalIgnoreCase))
{
return RedactedValue;
}
A scheme word survives only when it is one of the recognized authorization schemes (`Bearer`,
`Basic`, `Digest`, `Negotiate`, `NTLM`, `ApiKey`, `Token`). An unrecognized leading word is as likely
to be credential material as it is to be a scheme, so it is dropped along with the rest. The key id is
also dropped when it runs longer than 64 characters — a long run before the first `_` is more likely to
be secret material than an identifier. Neither key-creation path (`ApiKeyAdminCommandLineParser.IsValidKeyId`,
`DashboardApiKeyManagementService.ValidateKeyId`) enforces a length, so this is a redaction heuristic
rather than a guarantee: operators should keep key ids under 64 characters, or the id stops appearing
in logs and only the `mxgw_[redacted]` shape survives. The direction of the failure is deliberate —
losing an identifier is cheap, logging a secret is not.
string token = authorizationHeader[bearerPrefix.Length..].Trim();
The parse is span-based (no regex, no `Split` allocation): the value is split once at the first space,
and the key id is read up to the first `_` of the remainder.
if (!token.StartsWith("mxgw_", StringComparison.OrdinalIgnoreCase))
{
return $"{bearerPrefix}{RedactedValue}";
}
string[] tokenParts = token.Split('_', 3, StringSplitOptions.RemoveEmptyEntries);
if (tokenParts.Length < 2)
{
return $"{bearerPrefix}mxgw_{RedactedValue}";
}
return $"{bearerPrefix}mxgw_{tokenParts[1]}_{RedactedValue}";
}
```
The split uses `count: 3` because the secret portion may itself contain underscores; only the first two segments (`mxgw` and the key id) are kept verbatim. Authorization headers that are not bearer tokens are reduced to `[redacted]` rather than passed through, since the gateway cannot reason about their structure.
`RedactClientIdentity` is the entry point used by `GatewayLogScope` and `DashboardRedactor`. It only invokes `RedactApiKey` when the input contains the `mxgw_` marker, leaving non-key identities (for example, Windows account names) untouched.
The consequence for callers is that a non-key identity (for example a Windows account name) reaching
`RedactClientIdentity` is now replaced rather than passed through. `DashboardRedactor` routes only
values containing the `mxgw_` marker here, so dashboard display names are unaffected.
### Command value redaction
@@ -160,12 +154,12 @@ public static IApplicationBuilder UseGatewayRequestLoggingScope(this IApplicatio
{
ArgumentNullException.ThrowIfNull(app);
ILogger logger = app.ApplicationServices
.GetRequiredService<ILoggerFactory>()
.CreateLogger("MxGateway.Request");
return app.Use(async (context, next) =>
{
ILogger logger = context.RequestServices
.GetRequiredService<ILoggerFactory>()
.CreateLogger("ZB.MOM.WW.MxGateway.Request");
using IDisposable? scope = logger.BeginGatewayScope(new GatewayLogScope(
SessionId: ReadHeader(context, SessionIdHeaderName),
WorkerProcessId: ReadInt32Header(context, WorkerProcessIdHeaderName),
@@ -190,7 +184,7 @@ The scope is keyed off four custom headers and the standard `authorization` head
The numeric headers use `int.TryParse` and `ulong.TryParse`; missing or unparseable values become `null` and are dropped by `GatewayLogScope.ToDictionary`. This keeps the middleware tolerant of clients that do not yet emit every header, which matters because the earliest call in a session (`OpenSession`) has no `SessionId` to send.
The logger category is `ZB.MOM.WW.MxGateway.Request`, which lets operators filter the request scope events independently from per-component categories.
The logger category is `MxGateway.Request`, which lets operators filter the request scope events independently from per-component categories. The logger is resolved once at registration rather than per request: the category is fixed, so a per-request `IServiceProvider` resolve and `ILoggerFactory.CreateLogger` (which takes the factory lock) bought nothing. Scope construction itself stays unconditional — gating it on `ILogger.IsEnabled` would drop scope state for providers and scope consumers registered after startup.
### Pipeline ordering
@@ -217,6 +211,30 @@ The order matters: putting the logging scope first ensures that authentication f
- `DashboardRedactor.Redact` delegates to `RedactClientIdentity` for any value containing the `mxgw_` marker, then falls back to a marker-keyword check for fields like `password` or `token`. This keeps dashboard renders aligned with log redaction.
- `ZB.MOM.WW.MxGateway.Tests/Diagnostics/GatewayLogRedactorTests.cs` covers each redaction branch, including the assertion that `WriteSecured` values stay redacted even when `valueLoggingEnabled` is true.
## Health Checks
The shared `ZB.MOM.WW.Health` package maps three endpoints — `/healthz` (live), `/health/ready`, and
`/health/active` — and each registered check opts into a tier by tag. The gateway registers two:
| Check | Endpoint tier | Fails when |
|---|---|---|
| `auth-store` | `ready` | The SQLite auth store cannot be opened. Every gRPC call authenticates against it, so its reachability genuinely gates whether the process should receive traffic. |
| `mxaccess-sessions` | `active` | Sessions exist and their workers have faulted. Reports `total` / `ready` / `faulted` / `starting` / `closing` as entry `data`. |
**Zero sessions is Healthy, and the tier choice follows from that.** The gateway opens an MXAccess
session when a client asks for one and holds none otherwise, so an idle gateway is working normally,
not broken. A count threshold ("unhealthy below N") would sit red forever on a host nothing dials
yet, and a permanently red probe is one operators stop reading — which leaves them worse off than no
probe at all. `mxaccess-sessions` is therefore graded on whether the sessions that exist are usable:
- nothing faulted → **Healthy** (including no sessions at all)
- some faulted, some still ready or starting → **Degraded**
- every session faulted → **Unhealthy**
For the same reason it is tagged `active` rather than `ready`. Readiness decides whether the process
should be sent traffic, and a gateway with no sessions is ready to serve; failing readiness there
would pull a working gateway out of rotation over a condition its clients create.
## Related Documentation
- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
+8 -3
View File
@@ -91,7 +91,7 @@ Environment variables use the normal .NET double-underscore form. For example,
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Authentication:Mode` | `ApiKey` | Selects public gRPC authentication. Supported values are `ApiKey` and `Disabled`. `Disabled` bypasses API-key verification and is for local development only. |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host. `appsettings.json` no longer ships an explicit value (SEC-33): the removed Windows literal was byte-identical to the Windows code default, and a Windows-absolute literal is **not** rooted on a Unix host, so it would have resolved against the CWD there. Deployed hosts still override the path through the NSSM environment (`MxGateway__Authentication__SqlitePath`). |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host. `appsettings.json` no longer ships an explicit value (SEC-33): the removed Windows literal was byte-identical to the Windows code default, and a Windows-absolute literal is **not** rooted on a Unix host, so it would have resolved against the CWD there. Deployed hosts still override the path through the NSSM environment (`MxGateway__Authentication__SqlitePath`). The validator additionally rejects a path **inside the application content root**, even an absolute one: the upgrade procedure renames that directory to `Server.bak.*`, which takes the credential store with it and silently starts an empty one. That is not hypothetical — it happened on a production host on 2026-08-09 and no gRPC client could authenticate for two days. |
| `MxGateway:Authentication:PepperSecretName` | `MxGateway:ApiKeyPepper` | Configuration key used to read the HMAC pepper for API-key secret hashing. The dashboard effective configuration redacts this value. |
| `MxGateway:Authentication:RunMigrationsOnStartup` | `true` | Runs SQLite auth schema migrations at gateway startup when API-key authentication is enabled. |
@@ -115,6 +115,7 @@ launch CWD (SEC-01, SEC-33).
| `MxGateway:Worker:StartupProbeRetryDelayMilliseconds` | `250` | Delay between transient startup probe retry attempts. |
| `MxGateway:Worker:PipeConnectAttemptTimeoutMilliseconds` | `2000` | Per-attempt timeout used by the worker named-pipe connect retry path. The overall pipe connection still stays under the startup budget. |
| `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` | Bounded wait the worker holds a unary write reply (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`; bulk writes excluded) for the matching MXAccess `OnWriteComplete` callback, so the reply's `statuses` carry the real commit outcome. `0` disables the wait (pure fire-and-forget replies). Must be `>= 0`. The gateway conveys the value to the worker via the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable. Consumers that time their own writes must budget above this wait: OtOpcUa's GalaxyDriver wraps gateway writes in a 2 s Tier A resilience timeout, so a deployment raising this option past ~2000 must raise that driver `ResilienceConfig` write timeout in step or slow-but-successful commits surface as consumer-side failures. |
| `MxGateway:Worker:EventQueueCapacity` | `10000` | Capacity, in events, of the worker's outbound MXAccess event queue. Must be between `1000` and `1000000`. This is burst headroom, not a throttle: the queue has no drop policy, so filling it records a `QueueOverflow` worker fault and faults the session. Raise it for sessions whose subscription set can outrun the drain loop (large advise sets, slow event consumers); the backing queue pre-allocates its slots, so the ceiling keeps a mistyped value from committing the 32-bit worker to an outsized allocation. The gateway conveys the value to the worker via the `MXGATEWAY_EVENT_QUEUE_CAPACITY` environment variable; a missing or unusable value leaves the worker on the 10000 default rather than failing the session. |
| `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. |
| `MxGateway:Worker:HeartbeatIntervalSeconds` | `5` | Worker heartbeat send interval and gateway heartbeat check cadence input. |
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatIntervalSeconds`. |
@@ -254,6 +255,7 @@ dev/test GLAuth posture (`glauth.md`), not a production posture.
| `MxGateway:Ldap:UserNameAttribute` | `cn` | LDAP attribute holding the login user name. |
| `MxGateway:Ldap:DisplayNameAttribute` | `cn` | LDAP attribute holding the display name. |
| `MxGateway:Ldap:GroupAttribute` | `memberOf` | LDAP attribute enumerating group membership (mapped to dashboard roles via `MxGateway:Dashboard:GroupToRole`). |
| `MxGateway:Ldap:FallbackServers` | *(empty)* | Ordered backup LDAP endpoints tried when the primary fails with a system-side error (connect/TLS, service-account bind, or search) — **not** when a user's credentials are simply wrong. Each entry is `host` (adopting `Port`) or `host:port`. Empty leaves single-endpoint behaviour exactly as before. Endpoint preference is sticky: the last endpoint that answered keeps being used until it fails. The `Transport` / `AllowInsecure` policy applies to every endpoint — a fallback is not a way to downgrade TLS. Entries are parsed at startup and a malformed one fails the boot, so a typo'd backup DC cannot lie dormant until the outage it exists to survive. Requires ZB.MOM.WW.Auth 0.2.0+. |
When LDAP is enabled, `Server`, `SearchBase`, `ServiceAccountDn`,
`ServiceAccountPassword`, and the attribute names must be non-blank, and `Port`
@@ -291,7 +293,7 @@ section (a sibling of `MxGateway`, not nested under it):
| Option | Default | Description |
|--------|---------|-------------|
| `Secrets:SqlitePath` | `mxgateway-secrets.db` | Path to the encrypted secrets store, resolved relative to the app content root when not rooted. |
| `Secrets:SqlitePath` | `<CommonApplicationData>/MxGateway/mxgateway-secrets.db` | Path to the encrypted secrets store. The default is supplied in code when the key is unset (`C:\ProgramData\MxGateway\...` on Windows), not from `appsettings.json` — a store inside the application directory is renamed away by the upgrade procedure, taking the secrets with it. On non-Windows hosts the default location is usually not writable by a normal user, so a local run must set `Secrets__SqlitePath` explicitly. |
| `Secrets:MasterKey:Source` | `Environment` | Key-encryption-key (KEK) provider. `Environment` reads a base64-encoded 32-byte key from an env var; `Dpapi` uses a machine-bound key file instead (see below). |
| `Secrets:MasterKey:EnvVarName` | `ZB_SECRETS_MASTER_KEY` | Env var name the `Environment` provider reads the KEK from. |
@@ -392,6 +394,7 @@ model requires otherwise.
| `MxGateway:Security:ApiKeyFailureWindowSeconds` | `60` | Sliding-window length, in seconds, over which API-key verification failures are counted, for both the per-partition and the per-key-id aggregate layer. Must be greater than zero. |
| `MxGateway:Security:ApiKeyFailureAggregateLimit` | `30` | Failed verifications for one key id counted across **all** transport peers within `ApiKeyFailureWindowSeconds` before that key id enters probe mode. This second layer bounds a distributed or source-rotating sprayer that never trips any single `(peer, key id)` partition. `0` disables the aggregate layer, leaving only per-partition counting. Must be zero or greater. |
| `MxGateway:Security:ApiKeyFailureProbeIntervalSeconds` | `5` | Minimum interval, in seconds, between probe admissions for an over-limit partition or key-id aggregate. An over-limit state is a valve rather than a wall: one request per interval reaches the real verifier — exactly one, even when a burst arrives together at the interval boundary — so the holder of the correct secret always gets through and clears the state, while everything else is still refused before the store read. `0` blocks absolutely instead — **not recommended**, because an unauthenticated peer can then deny the key to its holder for the whole window. Must be zero or greater. |
| `MxGateway:Security:AuditRetentionDays` | `90` | Days of canonical audit history kept in the `audit_event` table. The audit drain sweeps once at startup and hourly thereafter, deleting rows older than this window; without it the table grows without bound inside the same SQLite file the authentication hot path reads. Rows whose timestamp SQLite cannot parse are never swept. Must be greater than zero — retention can be widened but not switched off. |
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
## Galaxy Options
@@ -402,7 +405,7 @@ model requires otherwise.
| `MxGateway:Galaxy:CommandTimeoutSeconds` | `60` | Per-command SQL timeout for all Galaxy browse RPCs. |
| `MxGateway:Galaxy:DashboardRefreshIntervalSeconds` | `30` | Interval between background refreshes of the dashboard Galaxy summary cache. SQL is hit at most once per interval regardless of dashboard render rate. |
| `MxGateway:Galaxy:PersistSnapshot` | `true` | Persists the latest successful Galaxy browse dataset to disk. When `true`, the cache reloads that snapshot at startup so clients can still browse last-known data while the Galaxy database is unreachable. The restored data is served with `Stale` status until a live query confirms it. |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). `appsettings.json` no longer ships an explicit value (SEC-33): the option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package, so the gateway supplies the `CommonApplicationData`-derived default when the bound value is blank and registers `GalaxyRepositoryOptionsValidator` to enforce that — when `PersistSnapshot` is `true` — the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup instead of resolving against the launch working directory (SEC-01, SEC-33). |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). `appsettings.json` no longer ships an explicit value (SEC-33): the option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package, so the gateway supplies the `CommonApplicationData`-derived default when the bound value is blank and registers `GalaxyRepositoryOptionsValidator` to enforce that — when `PersistSnapshot` is `true` — the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup instead of resolving against the launch working directory (SEC-01, SEC-33). The same validator also rejects a path **inside the application content root**, because the upgrade procedure renames that directory away and the cached snapshot would be discarded on every deploy. |
See [Galaxy Repository Browse](./GalaxyRepository.md) for the RPC surface and
behavior.
@@ -415,6 +418,8 @@ behavior.
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
`StreamAlarms` are session-less RPCs served by the monitor.
+228 -26
View File
@@ -98,6 +98,7 @@ ZB.MOM.WW.MxGateway.Server
StatusBadge.razor
FaultList.razor
DashboardSnapshotService.cs
DashboardSnapshotFeed.cs
DashboardAuthorizationHandler.cs
DashboardAuthenticator.cs
DashboardApiKeyAuthorization.cs
@@ -110,9 +111,15 @@ ZB.MOM.WW.MxGateway.Server
```
The dashboard exposes three named SignalR hubs in addition to Blazor Server's
internal circuit; pages connect to those hubs from within the circuit via the
`DashboardHubConnectionFactory` helper. The hubs publish snapshot, alarm, and
per-session event updates that the pages render in place of polling.
internal circuit. The hubs are the **remote** surface: they publish snapshot,
alarm, and per-session event updates to clients outside the gateway process.
Server-rendered Blazor pages do not use them. A page runs inside this process,
so it consumes the producing services directly through in-process seams —
`IDashboardSnapshotFeed`, `IDashboardSessionEventSubscriber`, and
`IGatewayAlarmService` — instead of opening a loopback WebSocket back into its
own heap. `DashboardHubConnectionFactory`, the helper a circuit used to open
those connections, stays registered for out-of-tree consumers, but no in-repo
page resolves it.
## Dashboard Data Source
@@ -159,37 +166,183 @@ gateway internals.
## Realtime Updates
Updates flow over three SignalR hubs, all guarded by the
Realtime data reaches two audiences over two seams:
- **in-process**, for the server-rendered Blazor pages, which run inside the
gateway process and read the producing services directly;
- **SignalR hubs**, for clients outside the process.
Pages originally took the hub path too, which put a loopback WebSocket, a
hub-token mint, and a serialize/deserialize round trip between a Blazor component
and an object already in its own heap. The in-process seams remove that hop. The
hubs stay for the audience that genuinely needs a wire.
### In-process page feeds
| Page | Seam | Producer |
|---|---|---|
| every page deriving from `DashboardPageBase` | `IDashboardSnapshotFeed.WatchAsync` | `DashboardSnapshotFeed` (singleton) multicasting one `IDashboardSnapshotService.WatchSnapshotsAsync` enumeration |
| `SessionDetailsPage` | `IDashboardSessionEventSubscriber.Subscribe(sessionId)` | `DashboardEventBroadcaster` — the same singleton the session mirror publishes to, registered behind both interfaces |
| `AlarmsPage` | `IGatewayAlarmService.StreamAsync` | the central alarm monitor, **provider status only**; the alarm rows still come from the 3 s `QueryAlarmsAsync` poll |
The snapshot feed multicasts rather than handing each page its own enumeration:
`WatchSnapshotsAsync` is not multicast on its own — each enumeration owns a timer
and builds its own snapshot per tick — so a subscription per page would multiply
the snapshot cost by the number of open pages. Each subscriber reads through a
capacity-1 drop-oldest channel, so a circuit that renders slowly skips snapshots
instead of buffering without bound or stalling the pump.
`DashboardPageBase` seeds `Snapshot` synchronously from
`IDashboardSnapshotService.GetSnapshot()` in `OnInitializedAsync` so the first
render is non-empty, then calls `InvokeAsync(StateHasChanged)` for every snapshot
the feed yields. On dispose it cancels the watch and waits at most **5 seconds**
for the loop to drain, logging a warning on timeout. The bound is deliberate: the
loop marshals renders through the renderer's dispatcher and disposal can run on
that same dispatcher, so an unconditional wait would hang on a wedged dispatcher.
The accepted cost is that an abandoned loop still holds its feed subscription — the
feed's idle gate stays open until it unwinds — and the warning is the operator's
only signal that a circuit teardown wedged.
`SessionDetailsPage` subscribes for the current session id and renders the most
recent N events (default 50) in a "Recent events" table. Its pump drains everything
queued and renders once per batch rather than once per event, and it re-checks
**inside the renderer dispatch** — where the subscription field is written, making
the check an unsynchronized read of dispatcher-owned state — that the batch's
subscription is still the live one. A batch read before a session switch would
otherwise render the previous session's events under the new session's heading.
Detaching cancels the pump, disposes the subscription (which releases the viewer
registration and completes the channel, so the pump has an exit even if
cancellation is missed), then drains under its own timeout.
### SignalR hubs (remote clients)
Updates for out-of-process clients flow over three SignalR hubs, all guarded by the
`MxGateway.Dashboard.HubClients` policy (cookie OR `MxGateway.Dashboard.HubToken`
bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
| Hub | Path | Producer | Payload | Routing |
|---|---|---|---|---|
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick; new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`. The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session hub ACL that would scope a Viewer to specific sessions is still outstanding (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session ACL that would scope a Viewer to specific sessions is still outstanding for this seam and the in-process one alike (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
`DashboardPageBase` opens a `DashboardSnapshotHub` connection via the connection
factory in `OnInitializedAsync`, seeds `Snapshot` synchronously from
`IDashboardSnapshotService.GetSnapshot()` so the first render is non-empty, and
calls `InvokeAsync(StateHasChanged)` on every `SnapshotUpdated` push. SignalR's
`WithAutomaticReconnect` handles transient disconnects.
### Default cadences
`SessionDetailsPage` additionally opens an `EventsHub` connection for the
current session id and renders the most recent N events (default 50) in a
"Recent events" table with a live/offline connection pill.
Default cadences:
Both seams consume the same producing services, so they share these cadences:
- snapshot service produces one snapshot per
`MxGateway:Dashboard:SnapshotIntervalMilliseconds` (default 1s);
- alarm publisher emits on each transition observed by the central monitor;
- event publisher emits per event fanned by the session's `SessionEventDistributor`
to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`).
to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`);
- the alarms page's provider-status badge resubscribes one second after its
`IGatewayAlarmService.StreamAsync` enumeration ends — the monitor completes a
subscriber's stream when it falls behind and again when it restarts, both
recoverable by resubscribing — and holds its last value in between. The page's
alarm rows are independent of that stream and refresh on the 3 s poll.
### Idle gating and snapshot cost
A snapshot is not free: each one takes a session-registry snapshot and sorts it,
copies the metrics dictionaries under the global metrics lock, and projects
sessions, workers, faults, and the Galaxy summary. Without gating that work ran
once a second for the life of the process even when nothing was watching.
Gating is two-tier, because the two seams have independent audiences and each must
be able to reach zero on its own.
**Hub tier.** `DashboardSnapshotHub` counts live connections into the singleton
`DashboardSnapshotHubConnectionCounter` (`OnConnectedAsync` / `OnDisconnectedAsync`,
clamped at zero). `DashboardSnapshotPublisher` reads that count before advancing the
snapshot enumerator: while it is zero the publisher does not call `MoveNextAsync` at
all, so the producing iterator stays suspended at its `yield` and builds nothing —
the gate removes the snapshot *build*, not just the broadcast. The publisher
re-checks once a second while idle, so the first client to connect resumes the tick
within roughly one snapshot interval, and `OnConnectedAsync` pushes the current
snapshot to that connection immediately. Now that no in-repo page connects to the
hub, this tier stays idle unless a remote client connects.
**In-process tier.** `DashboardSnapshotFeed` gates its own pump on its subscriber
list: the first subscriber starts the pump, the last one leaving cancels it and
awaits it, so a gateway with no page open runs no timer and builds no snapshots on
this seam either. Successive pumps are chained through the previous pump's task, so
an unsubscribe immediately followed by a resubscribe restarts a fresh pump without
ever running two enumerations at once. A page does not wait for the pump's first
tick — `DashboardPageBase` seeds its first render synchronously from
`IDashboardSnapshotService.GetSnapshot()`.
That gate is generation-scoped rather than a plain subscriber count. Each pump owns
a generation, each subscriber is tagged with the generation it joined under, a pump
ends its generation the instant its source faults or completes — before the possibly
slow enumerator disposal — and a dying pump only ever detaches its own generation's
subscribers. Two races motivate the extra state:
- a subscriber arriving mid-teardown must start a fresh generation rather than
attach to a pump that is about to detach everybody and leave nobody watching;
- an unsubscribe must compare its own generation against the live one before it
cancels anything. Subscribers of an ending generation linger in the list until
that pump's reset runs, so counting the whole list would let them hold the idle
gate open, and cancelling on their behalf would stop a live pump that other
viewers depend on.
Two per-tick costs inside the snapshot itself are bounded independently of either gate:
- the effective configuration (`EffectiveGatewayConfiguration`) is built once and
cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
at startup and never reloads, so rebuilding the whole option tree every tick
produced an identical object;
- the API key summaries are refreshed at most once every 15 seconds
(`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
read whose content changes only when an operator creates, rotates, or revokes a
key, so a key change reaches the dashboard within that interval. Only a
*successful* refresh restarts the interval, so a failed or timed-out read is
retried on the next tick and the previous summaries stay on screen.
Avoid pushing every MXAccess data-change event into a wider broadcast group.
The current design routes events strictly through `session:{id}` groups; the
snapshot hub continues to carry aggregate event counters and rates.
Events are routed strictly per session (`session:{id}` groups on the hub,
per-session subscriber lists in process); the snapshot seams continue to carry
aggregate event counters and rates.
### Mirror gating
Each session's dashboard-mirror subscriber calls
`DashboardEventBroadcaster.Publish` for every event the session produces,
independently of whether anything is watching that session. `Publish` returns
immediately when `EventsHubViewerRegistry.HasViewers(sessionId)` is false,
**before** the redaction clone. That matters because redaction is on by default
(`Dashboard:ShowTagValues` false), so the unwatched steady state — nobody on any
session-details page — previously paid a deep protobuf clone plus a send to an
empty group for every event of every session. Behaviour for a watched session is
unchanged.
The registry counts both audiences, which is what lets one gate serve both seams.
`EventsHub` mirrors its own `AddToGroup` / `RemoveFromGroup` calls into it —
SignalR does not expose group membership, so the broadcaster cannot ask whether
`session:{id}` is empty — and `OnDisconnectedAsync` releases every subscription a
dropped connection held, the only reliable signal for a browser tab that closes
without unsubscribing. An in-process subscription registers the same way under a
synthetic `inproc-`-prefixed connection id, which cannot collide with a SignalR
connection id and makes the origin obvious in a debugger; disposing it removes the
viewer and releases the synthetic connection, because that id never reconnects and
nothing else would ever release its per-connection entry. Both paths use the same
ordering — register before becoming a delivery target, deregister after ceasing to
be one — so the widest a race window opens is a redaction clone that reaches
nobody, never a dropped event that was owed to a live viewer.
Redaction happens once per event, not once per audience: `Publish` produces a
single redacted clone and hands that same instance to the in-process subscribers
and to the hub group. In-process delivery runs first and synchronously — it cannot
throw, and it must not be skipped by the guard clause around the hub send — into
per-subscriber bounded drop-oldest channels, so a page that falls behind loses its
oldest queued events rather than blocking the session's event pipeline.
The mirror subscriber itself is still registered on the `SessionEventDistributor`
for the session's whole lifetime; only the per-event work is gated. Starting and
stopping the mirror lease lazily with the first and last viewer was considered
and deliberately not done — it entangles the dashboard with distributor
subscribe/unsubscribe lifetime (and with the replay/sequence bookkeeping that
attaching a subscriber mid-stream implies) for no additional saving beyond the
clone and send this gate already removes.
## Pages
@@ -312,8 +465,9 @@ panel. The panel shows each subscribed tag's live value, MXAccess data type,
quality and source timestamp, refreshed every two seconds. The subscription
panel is the explicit opt-in tag-value surface: it always shows values
regardless of `Dashboard:ShowTagValues`, which governs the diagnostic
session/worker views and the per-session `EventsHub` mirror (values are
redacted from the mirrored events when the flag is false).
session/worker views and the per-session event mirror — both its hub and
in-process audiences (values are redacted from the mirrored events when the flag
is false).
### Alarms page
@@ -323,7 +477,11 @@ defaults to showing unacknowledged `Active` alarms; filters add acknowledged
alarms and narrow by area, severity range, and a reference/source/description
text search. Cleared alarms are not retained — the gateway holds no
alarm-history store, so the page reflects only the live active set. The page is
read-only; it does not acknowledge alarms. If `MxGateway:Alarms:Enabled` is
read-only; it does not acknowledge alarms. A provider-status badge tracks the
central monitor's health from `IGatewayAlarmService.StreamAsync` in process — the
alarm service is already a multi-subscriber fan-out, so the badge needs no SignalR
client, no loopback socket, and no hub token — while the alarm rows themselves
still come from the three-second poll. If `MxGateway:Alarms:Enabled` is
false the central monitor never starts, and the page says so instead of showing
an empty list with no explanation.
@@ -337,6 +495,31 @@ its lease expires. One session means one worker process backs every dashboard
circuit; all access is serialised so the worker sees one in-flight command at a
time. Tag reads go through `GatewaySession.SubscribeBulkAsync` / `ReadBulkAsync`.
The advise set that backs those reads is capped at 256 tags (one browse page plus
headroom) and evicted least-recently-read-first. Without the cap every tag any
viewer ever inspected stayed advised on the single dashboard worker until the
session faulted, so browsing a large galaxy accreted unbounded live MXAccess
subscriptions — and the event churn they feed — on one x86 process. Reading a tag
already in the set marks it most-recently-read; subscribing past the cap unadvises
the oldest entries with `GatewaySession.UnsubscribeBulkAsync` in one batch before
the new ones are advised. Tags read in the same call are never evicted to make
room for each other. A failed unadvise does not fail the read: the tags are
dropped from tracking anyway (they re-subscribe if read again), because the
session-invalidation path already handles gateway/worker drift.
The cap is per-read, not absolute. A read may never evict a tag it is itself about
to return, so one read of more distinct tags than the cap leaves the set that
large; what the eviction pass guarantees is
> after any read, the advise set holds at most `max(256, distinct tags in that read)`
> tags.
The overshoot is not sticky: the next read that subscribes anything measures the
overflow against the oversized set and evicts the whole excess in one pass (a
300-tag set plus one new tag evicts 45 and lands back at 256). A read that
subscribes nothing new evicts nothing, but neither can it grow the set. A browse
page requests far fewer tags than the cap, so in practice the set settles at 256.
The Alarms page does **not** use the dashboard session: alarm data comes from
the gateway's always-on central monitor. `QueryAlarmsAsync` reads
`IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the
@@ -454,6 +637,14 @@ Three authorization policies are registered:
cookie OR a `MxGateway.Dashboard.HubToken` bearer (used by WebSocket upgrades
where the cookie can't be forwarded).
The in-process page feeds carry no authentication of their own, and need none:
`MapRazorComponents<App>()` applies `RequireAuthorization(ViewerPolicy)` to the
component endpoints, so a page can only run inside a circuit whose principal is
already an authorized Viewer. The hub-token flow below therefore covers only the
remote hub surface. Neither seam scopes a Viewer to particular sessions — SEC-25
(the per-session ACL) is outstanding for both, and the mirror's value redaction
remains the near-term mitigation, unchanged by the move in-process.
Two environmental bypasses still apply, both scoped to **read-only** access:
`MxGateway:Authentication:Mode = Disabled` and `MxGateway:Dashboard:AllowAnonymousLocalhost`
(default `true`, loopback only) each satisfy a requirement that includes the Viewer
@@ -498,12 +689,15 @@ surface is affected. Never enable in production.
### Hub bearer flow
This flow serves remote hub clients only; in-process pages are authorized by the
component endpoint's `ViewerPolicy` and never mint a token.
SignalR connections cannot reuse the `__Host-` cookie when the JS client
upgrades to WebSocket — the cookie's `SameSite=Strict; Path=/` keeps it from
being forwarded by the browser's WebSocket layer in some edge cases. The
dashboard mints short-lived bearer tokens for the connection:
1. The cookie-authenticated Blazor page calls `GET /hubs/token`
1. The cookie-authenticated client calls `GET /hubs/token`
(gated by `ViewerPolicy`, cookie-only).
2. `HubTokenService.Issue(user)` serializes the user's name, NameIdentifier,
and role claims to JSON, encrypts with the ASP.NET Core data-protection
@@ -523,7 +717,9 @@ dashboard mints short-lived bearer tokens for the connection:
`DashboardHubConnectionFactory` (scoped to the Blazor circuit) wraps the
HubConnectionBuilder and supplies a fresh token via `AccessTokenProvider` on
every (re)connect, so the short 5-minute lifetime is transparent to clients.
every (re)connect, so the short 5-minute lifetime is transparent to whoever uses
it. It remains registered, but no in-repo page opens a hub connection any more;
external clients implement the equivalent refresh themselves.
Caveat — logout does not revoke outstanding tokens. Logout clears the dashboard
cookie, but hub bearer tokens are self-contained, data-protection-encrypted, and
@@ -618,8 +814,14 @@ Integration tests should verify:
- a user in a Viewer-mapped LDAP group can render every page but cannot
invoke the Admin-only management actions,
- a user with no mapped LDAP group cannot sign in at all,
- live snapshot updates when a fake session changes state are delivered
via the `/hubs/snapshot` push, not by polling.
- live snapshot updates when a fake session changes state reach a page through
the in-process `IDashboardSnapshotFeed` and reach a remote client through the
`/hubs/snapshot` push — neither by polling;
- the snapshot feed's idle gate: no subscribers means no pump, the last
subscriber of the live generation stops it, and a subscriber arriving
mid-teardown gets a fresh generation rather than a dead one;
- the event mirror's viewer gate counts in-process subscriptions as well as hub
connections, and a disposed in-process subscription releases its viewer count.
## Initial Implementation Slice
+22
View File
@@ -593,6 +593,28 @@ Pending command handling:
Timeouts should not assume the COM call stopped. A timed-out command may still
finish inside the worker.
On timeout the client also forwards a `WorkerCancel` carrying the abandoned
correlation id, best-effort: the gateway has stopped waiting, but the worker has
not stopped working, and the worker owns a single STA. `WorkerPipeSession` routes
the cancel to `CancelCommand`, which drops the correlation from the STA queue if
it has not started and replies `Canceled` for it. A cancel that arrives after the
command reached MXAccess is a no-op — there is no way to abort an in-flight COM
call — so this shortens the STA backlog rather than freeing a call already
running on it, and the rule above still holds. A command whose envelope is still
in the gateway's outbound queue needs no special handling: the queue is FIFO, so
the worker reads the command and then its cancel and drops it before execution.
Cancels ride the same outbound channel as commands, whose capacity is
`MaxPendingCommands + 4`: the reserve above the pending-command limit is what
absorbs them, so a burst of timeouts stays bounded and cannot deadlock the
enqueue path. Failing to send the cancel is logged at debug and never replaces
the `CommandTimeout` the caller is owed.
Cancellation outranks the deadline. When a caller's token is canceled around the
same time the timeout fires, the command is reported as canceled
(`GatewayShutdown`, `OperationCanceledException`), not as `CommandTimeout`, and
no cancel is forwarded.
## Fault Model
Fault categories:
+17
View File
@@ -554,6 +554,23 @@ windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`
suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures.
### Two more findings from the 2026-08-15 windev gate
- `SecretsStorePathGuardTests.CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt`
used to fail deterministically on Windows: creating the builder opens `secrets.db`,
`Microsoft.Data.Sqlite`'s connection pool kept the file handle alive past the test body, and
the cleanup's recursive directory delete hit a sharing violation Windows enforces and Unix
does not. The cleanup now clears the SQLite connection pool before deleting the temp
directory (the same pattern as `TempDatabaseDirectory` and `PreHostSecretExpansionTests`),
so the test passes on Windows and macOS alike — count it as a pass on both.
- The `StaWaitHelper` timing tests (`WaitForSignalOrMessages_*`) flake on a loaded box with a
signature that reads like a broken wait but is not: the helper wakes on *input being
present*, so a message posted to the test thread ends the wait early. That is the helper
doing exactly what the STA pump needs. The tests drain the queue with
`PumpPendingMessages()` first for that reason; a failure here means the box was busy enough
to queue a message mid-test, not that the wait stopped honouring its handle or its timeout.
Re-run the class on its own before treating it as real, per the load caveat above.
### The full-suite testhost hang was a zero-buffer named pipe (fixed)
For months a full-suite run on windev reported `855 passed, 0 failed` and then never
+3 -3
View File
@@ -72,7 +72,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
|------------|--------------|-------------|
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24). Incremented when the read loop stages an event, decremented when the consumer reads it, so a backlog stuck in the staging channel is visible rather than invisible. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepthSources` (summed on demand) | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24) — summed across every live client at collection time (GWC-30). Each client owns an interlocked counter incremented when the read loop stages an event and decremented when the consumer reads it, and registers it as a gauge source for its lifetime, so a backlog stuck in a staging channel is visible and concurrent sessions add up instead of overwriting one another. |
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
## Snapshot Shape
@@ -111,7 +111,7 @@ The scalar fields mirror the counters and gauges. The four dictionaries provide
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
- `RetryAttemptsByArea` keys by the resilience `area` tag, e.g. `worker_startup`.
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking.
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking. `CommandsStarted`, `CommandsSucceeded`, `CommandsFailed`, and `CommandFailuresByMethod` are read the same way: the command counters run two-to-three times per gRPC call, so they are recorded with `Interlocked` and a `ConcurrentDictionary` rather than under `_syncRoot` (GWC-30). The two queue depths are pulled from their registered sources before the lock is taken, since those delegates reach into subscriber channels and worker clients.
## Recording Sites
@@ -146,7 +146,7 @@ _metrics.RemoveSessionEvents(session.SessionId);
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
- `HeartbeatFailed(SessionId)` per missed heartbeat.
- `SetWorkerEventQueueDepth(queueDepth)` when the read loop stages an event and when the consumer reads one, so the gauge tracks staged + queued events.
- `RegisterWorkerEventQueueDepthSource(...)` once at construction, disposed in `DisposeAsync`. The client's own `_eventQueueDepth` is incremented when the read loop stages an event and decremented when the consumer reads one, so the gauge tracks staged + queued events without either hot-path step calling into `GatewayMetrics`.
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.
- `QueueOverflow("worker-events")` when the timed write into the bounded consumer channel exceeds `EventChannelFullModeTimeout`, and `QueueOverflow("worker-event-staging")` when the staging channel is full at its `2 × EventChannelCapacity` bound. The two labels distinguish a stalled consumer from one that merely drains too slowly; both fault the session with `ProtocolViolation`.
+134 -12
View File
@@ -261,6 +261,62 @@ is still responsive. Shutdown marks the runtime as closing, wakes the pump,
rejects new commands, cancels queued work, uninitializes COM on the STA, and
waits for the thread to exit.
### Inner Completion Waits
Two commands hold the STA while waiting for a COM event they just provoked: the
unary write path waits for its `OnWriteComplete`
(`MxAccessWriteCompletionCache.TryWaitForCompletion`), and `ReadBulk` waits per
tag for the first `OnDataChange` (`MxAccessValueCache.TryWaitForUpdate`). Both
run the same loop shape as the outer pump, for the same reason — the event they
are waiting for *is* a Windows message, so the thread must keep dispatching to
receive it:
```text
loop:
pumpStep() # PeekMessage / TranslateMessage / DispatchMessage
if cache entry newer than baseline: return it
if now >= deadline: return the timed-out shape
MsgWaitForMultipleObjectsEx(
cache_update_event,
min(remaining, 50 ms),
QS_ALLINPUT,
MWMO_INPUTAVAILABLE)
```
The idle slice is a Win32 wait (`StaWaitHelper.WaitForSignalOrMessages`), never
`Thread.Sleep`. A sleeping STA pumps no messages, so a sleep-polled loop could
only dispatch the awaited COM event at poll-tick granularity while stalling
*every other* event for the same tick — up to 1.5 s for a write completion and
up to `timeout_ms` per tag for `ReadBulk`. The Win32 wait returns the instant a
message needs pumping, so the apartment dispatches continuously for the whole
wait. Each cache also sets an `AutoResetEvent` from its update path (outside the
cache lock) so a cross-thread producer wakes the waiter immediately; in the live
worker the update arrives on the STA from inside `pumpStep` itself, and the
message wake is what carries it.
`MWMO_INPUTAVAILABLE` makes the drain contract load-bearing: the wait wakes on
input that is merely *present*, including input an earlier `PeekMessage` saw but
did not remove. A `pumpStep` that drains only part of the queue — or a no-op one
— therefore leaves a message that satisfies the wake condition forever, and the
loop spins at 100% CPU until its deadline (deadline and reply shape still hold;
it is a CPU fault, not a correctness one). Every `pumpStep` must drain to empty,
as `StaRuntime.PumpPendingMessages` does.
The wait slice is capped at 50 ms so `pumpStep` runs periodically even when
nothing wakes the wait — a process with no STA message queue (unit tests drive
these caches from ordinary threads, standing in for the STA by updating the
cache from a fake `pumpStep`) must not block for a full poll interval. Timeouts,
deadline math, and return values are unchanged by the wait mechanism: an expired
write wait still yields the empty-`statuses` unconfirmed reply, and an expired
per-tag `ReadBulk` wait still reports its own timeout.
The write wait's budget is `MxGateway:Worker:WriteCompletionWaitMilliseconds`
(default 1500). It is a bounded hold on the STA per unary write, so deployments
whose write workload is effectively fire-and-forget — no consumer reads the
reply's `statuses` — can lower it, or set `0` to skip the wait entirely and
reply on acceptance alone.
## COM Creation
The MXAccess analysis source at `C:\Users\dohertj2\Desktop\mxaccess` identifies
@@ -368,7 +424,11 @@ type on buffered events. `OperationComplete` is only emitted from the native
`MxAccessEventQueue` is the bounded outbound event queue for one worker
session. It assigns the monotonic `WorkerSequence` and `WorkerTimestamp` when an
event is accepted, preserving the order in which MXAccess handlers enqueue
events. The default capacity is `10000`. When the queue reaches capacity it
events. The capacity is `10000` by default and comes from
`MxGateway:Worker:EventQueueCapacity`, which the gateway stamps onto the worker
launch environment as `MXGATEWAY_EVENT_QUEUE_CAPACITY`; a missing, unparseable,
or out-of-range value (outside `1000``1000000`) leaves the worker on the
default rather than failing the session. When the queue reaches capacity it
records a `WorkerFaultCategory.QueueOverflow` fault and rejects further events.
The event handler catches conversion and enqueue failures, records the first
fault on the queue, and returns to the STA message pump instead of writing to
@@ -378,16 +438,30 @@ If event conversion throws, catch it inside the event handler, record a
structured `WorkerFault`, and keep the worker alive only if the fault policy
allows it.
The event drain loop streams queued events as `WorkerEvent` frames. A single
event whose envelope exceeds the negotiated frame maximum is **undeliverable end
to end** — the pipe maximum sits only the envelope-overhead reserve above the
public gRPC cap, so a frame the pipe rejects would also be rejected on the
client-facing stream. The session therefore faults on it rather than dropping it
(a silent drop makes the event stream unfaithful, and a synthesized placeholder
is barred by the no-synthesized-events rule), but the death is structured: the
worker logs the event's identity — family, handles, worker sequence, and sizes,
never the value — writes a `WorkerFault` with category `ProtocolViolation` and
command method `EventDrain` carrying the same identity, and only then exits.
The event drain loop streams queued events as `WorkerEvent` frames. It is
**signal-driven, not polled**: `MxAccessEventQueue` carries a wake signal that
`Enqueue` and `RecordFault` release (outside the queue lock, so the STA's enqueue
stays a lock acquire plus a non-blocking release), and a drain that comes back
empty waits on that signal rather than sleeping. The signal is capped at one
pending wake, so a burst coalesces into a single wake and the waiter re-drains
everything that arrived the loop must therefore re-check `DrainFault()` and
re-drain after every wait, never treat a wake as "exactly one event". The 25 ms
`EventDrainInterval` survives as the **fallback ceiling** on an unsignalled wait,
not as a latency floor: an event arriving at an idle worker is framed at signal
latency instead of waiting out a tick, an idle worker parks instead of waking 40
times a second, and the interval only bounds how long the loop may sleep if some
future path mutates the queue without signalling.
A single event whose envelope exceeds the negotiated frame maximum is
**undeliverable end to end** — the pipe maximum sits only the envelope-overhead
reserve above the public gRPC cap, so a frame the pipe rejects would also be
rejected on the client-facing stream. The session therefore faults on it rather
than dropping it (a silent drop makes the event stream unfaithful, and a
synthesized placeholder is barred by the no-synthesized-events rule), but the
death is structured: the worker logs the event's identity — family, handles,
worker sequence, and sizes, never the value — writes a `WorkerFault` with
category `ProtocolViolation` and command method `EventDrain` carrying the same
identity, and only then exits.
Operator remediation is configuration: raise `MxGateway:Worker:MaxMessageBytes`
for that workload. Other per-frame rejection codes keep their previous behavior
because they indicate worker bugs, not workload size.
@@ -467,7 +541,11 @@ is bounded on **two** axes because no diagnostics command may be session-fatal:
maximum less a 64 KiB envelope/reply-wrapper reserve, and the size decision
happens inside the event queue's lock, so an event is dequeued only once it is
known to fit. An event that does not fit stays at the head of the queue and is
never lost.
never lost. Each event's serialized size is *measured* once at enqueue, outside
that lock, and stored beside it: the drain only compares memoized numbers, so a
large drain never walks messages under the lock the STA needs to enqueue the
next COM callback. The memoized size cannot go stale because an enqueued event
is never mutated again (WRK-11).
Truncation is reported in the reply's existing `DiagnosticMessage`
("N events returned, M remain; repeat DrainEvents for the rest") rather than in a
@@ -809,6 +887,50 @@ Graceful shutdown sequence:
If shutdown wedges, the gateway kills the process. The worker should be written
so process kill does not corrupt other sessions.
### Ending the pipe read (net48)
Step 8 above cannot be done by cancellation. On .NET Framework 4.8
`NamedPipeClientStream.ReadAsync` accepts a `CancellationToken` and then never
wires it to the overlapped I/O, so a read parked waiting for gateway bytes stays
parked no matter what the worker cancels. Closing the handle is the only thing
that ends it.
`WorkerPipeSession.RunMessageLoopAsync` races one outstanding read against the
heartbeat and event-drain loops, so every fault exit — an event-drain fault, an
event too large to frame, a failed heartbeat write — unwinds while that read is
still pending. The session therefore owns the transport: `RunAsync`'s outermost
`finally` disposes the stream as its last teardown step and then awaits the read
that disposal unblocks. Disposal comes last because in the ordinary case every
frame the session will ever write is already complete by then — the frame writer
signals a write only after it has been written *and* flushed. It is not last
because that is guaranteed: the wait on the heartbeat and drain loops is
budgeted, and a stream write is genuinely uncancellable, so an overrunning write
can still be in flight against the stream being disposed. Disposal is
consequently exception-*total*, catching anything the handle close throws and
logging it, because nothing raised while releasing a handle is more actionable
than the terminal exception that ended the session, and nothing may displace it.
Observation is unconditional; only the *logging* of it is budgeted.
`ObserveBackgroundTaskStopAsync` waits `BackgroundTaskStopTimeout` for a task to
stop and logs what it saw, but when it gives up it hands the task a
fault-observing continuation before returning. Windows owes no deadline for a
completion torn off a closed handle, so a bounded await on its own would reopen
the very orphaning window it was added to close. The same helper — and so the
same guarantee — covers the abandoned read, the heartbeat loop, and the
event-drain loop. This matters because the worker installs no
`TaskScheduler.UnobservedTaskException` handler: an unheld faulted task would
otherwise surface only at finalization, still holding the reader's reused
length-prefix buffer and its pooled payload buffer.
Two invariants follow. Nothing may call `WorkerFrameReader.ReadAsync` again once
a read has been abandoned — a second read would race the first for those buffers
and could return a pooled buffer twice — which `Debug.Assert`s at both read-issue
sites guard. And `WorkerPipeClient`'s `using` on the pipe stays as a backstop for
the paths the session never reaches (a session factory that throws), not as the
primary owner; disposal is idempotent, so its second `Dispose` is a no-op.
Graceful shutdown leaves no pending read at all, so the observation step is a
no-op on that path.
`MxAccessStaSession.ShutdownGracefullyAsync` implements the current cleanup
path. It first calls `StaCommandDispatcher.RequestShutdown()` so new commands
are rejected and queued commands that have not started receive
+32 -10
View File
@@ -207,6 +207,20 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
#### Teardown parallelism
A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses`, a compile-time constant of `4` in `SessionManager`. Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
Splitting the phases moves the TOCTOU re-check earlier, and that is an accepted trade rather than an unchanged behavior: selection now flips **every** chosen session to `Closing` up front, before any teardown runs, whereas the sequential sweep re-checked session *N* only after sessions *1..N-1* had finished closing. A client that re-attaches a subscriber while the close phase is running therefore loses a race it could previously win — the eligibility snapshot is taken at one instant for the whole pass. Expiry evaluation itself is unaffected, because `now` is a parameter and is not re-read per session.
A close that throws no longer abandons the rest of the selected set: the sweep attempts every selected session, captures the first failure, and rethrows it once the pass is done so `SessionLeaseMonitorHostedService` still logs the sweep failure as before. Because the closes run concurrently, *which* failure surfaces when several fail in one pass is nondeterministic; the log line is the diagnostic, not the identity of the exception.
`ShutdownAsync` drains sessions with the same bounded fan-out and the same per-session catch → `KillWorkerAsync` fallback, with two rules that keep a stop deadline from turning into leaked workers. First, its body is **exception-total** — nothing escapes it — because `Parallel.ForEachAsync` cancels the token handed to the sibling bodies as soon as one body throws, which would abort in-flight graceful shutdowns *and* make their kill fallback fail instantly on the freshly cancelled token. Second, the drain loop is deliberately **not** bound to the caller's `CancellationToken` and the kill fallback runs on `CancellationToken.None`: a cancelled `ParallelOptions` token stops dispatching the remaining sessions entirely, so the untried tail would be neither closed nor killed. This **fixes a leak the sequential drain also had**, rather than restoring the sequential drain's behavior — there the kill fallback ran on the caller's already-cancelled token, and `KillWorkerAsync`'s entry `ThrowIfCancellationRequested` threw out of the loop on the very first session, producing zero kills. The token is passed to the graceful close only, so a host stop deadline turns the drain into a kill sweep rather than into a leak. This matters because nothing reattaches to a leaked worker — a restarted gateway terminates orphans (see [Design Decisions](DesignDecisions.md)).
The asymmetry with `CloseExpiredLeasesAsync` — whose `ParallelOptions` *is* token-bound — is intentional: the sweep is periodic maintenance, so a pass abandoned on cancellation loses nothing permanently (the next pass re-selects, and `ShutdownAsync` backstops it), whereas the shutdown drain is terminal and must not be abandoned partway.
**Stranded-`Closing` bound.** A sweep pass that is cancelled after selection leaves its unclosed selections in `Closing` with close already started. `IsFaultedReapableCore` requires `state == Faulted`, so a session selected under `FaultedReason` and stranded this way is not re-selected as faulted; it is swept only when its normal lease expires (up to `MxGateway:Sessions:DefaultLeaseSeconds`, default 1800 s), since `IsLeaseExpiredCore` and `IsDetachGraceExpiredCore` are state-agnostic. This bound is documented rather than closed with a re-selection clause: the sweep's only caller cancels on the host's `stoppingToken`, so the very next thing that runs is `ShutdownAsync`, which drains (or kills) the whole registry — and any worker that still survives that is terminated as an orphan on the next gateway start. Adding a "`Closing` and close-started" re-selection clause would also have to distinguish an abandoned close from one that is merely still in flight, which would weaken the single invariant that makes the parallel close phase safe.
#### Detach-grace retention
`MxGateway:Sessions:DetachGraceSeconds` (default 30) is a bounded retention window kept after a session's *last external (gRPC) event-stream subscriber* drops, so a client can reconnect to the same session instead of having it torn down on the first stream disconnect. While the window is open the session stays `Ready` and fully usable — worker commands continue to work and a reconnecting subscriber re-attaches normally. Because retention is keyed on the *external* subscriber count (`_activeEventSubscriberCount`), and the gateway-owned internal dashboard mirror registers directly on the distributor with `isInternal: true` and is therefore *not* counted, a session whose only remaining subscriber is the dashboard mirror still enters detach-grace.
@@ -276,12 +290,13 @@ If both graceful shutdown and the kill fall-back fail, the original and kill exc
## Shutdown Coordination
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. The shutdown loop catches per-session exceptions, calls `KillWorker`, and removes the session so that one stuck worker cannot block the rest of the host:
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. Sessions are drained with the same bounded fan-out the lease sweep uses (`MaxParallelSessionCloses`), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions — *every* exception, including from the fallback — calls `KillWorkerAsync` on an uncancellable token, and removes the session, so that neither one stuck worker nor one failing teardown can block or abort the rest of the host's drain:
```csharp
public async Task ShutdownAsync(CancellationToken cancellationToken)
{
foreach (GatewaySession session in _registry.Snapshot())
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
async (session, _) =>
{
try
{
@@ -293,17 +308,24 @@ public async Task ShutdownAsync(CancellationToken cancellationToken)
exception,
"Graceful shutdown failed for session {SessionId}; killing worker.",
session.SessionId);
if (_registry.TryGet(session.SessionId, out _))
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
&& registeredSession is not null)
{
session.KillWorker(GatewayShutdownReason);
await RemoveSessionAsync(session).ConfigureAwait(false);
try
{
// Not the caller's token: the kill is the last-resort orphan preventer.
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
}
catch (Exception killException)
{
_logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
}
}
}
}
}
}).ConfigureAwait(false);
```
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry inside the loop without throwing.
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.
## Dependency Injection
+57 -11
View File
@@ -88,7 +88,9 @@ priority order. A caller enqueues its frame into the control or event queue
under a lock, then contends for a single write lock; whichever caller wins
drains every frame queued at that moment, control frames first and each class
in FIFO order, so a command reply, fault, heartbeat, or shutdown
acknowledgement is never delayed behind a backlog of queued events. Priority
acknowledgement is never delayed behind a backlog of queued events — neither
in the bytes written nor in the flush that marks them delivered (see the
class-boundary flush under flush coalescing below). Priority
only reorders *which frame writes next* — it does not affect the sequence
value a frame receives (see below), so a caller cannot infer priority class
from the wire sequence.
@@ -117,13 +119,36 @@ Two failure shapes are distinguished during a drain pass:
and every frame still queued, then stops draining entirely so no caller
waits forever on a stream that will not recover.
Flushes are coalesced across a drained batch: each frame in the batch is
written to the stream without an individual flush, then one `FlushAsync`
runs after the whole batch, and only then does every successfully-written
frame's completion resolve — so a caller's `WriteAsync` still does not
complete until its bytes are both written *and* flushed, but a batch that
happened to contain several queued frames pays one flush instead of one per
frame. Note the ordering this implies at the peer: the frames reach the pipe
Flushes are coalesced across a *run of same-class frames* inside a drain
pass: each frame in the run is written to the stream without an individual
flush, then one `FlushAsync` runs — at the end of the pass, and additionally
at every control-to-event boundary — and only then does every
successfully-written frame of that run resolve its completion. A caller's
`WriteAsync` therefore still does not complete until its bytes are both
written *and* flushed; what changed is *when* that moment arrives
for a control frame that a pass writes ahead of queued events. It used to be
the end of the pass, so a heartbeat, command reply, fault, or shutdown
acknowledgement was written first but only counted as delivered after up to a
full event batch had been written and flushed behind it. The boundary flush
closes the control run out before the events are written, so the priority
class governs the frame's delivery point and not just its byte order. The
cost stays bounded: a pure-event pass — the event hot path — still pays
exactly one flush however many frames drain together, a run of control
frames still pays one for the whole run (never one per heartbeat, the
syscall-per-frame cost the coalescing removed), and only a pass that actually
mixes both classes pays a second.
One consequence of the boundary flush is worth stating: a control frame whose
run has already been flushed and completed is out of the drain's
written-but-unflushed set, so a *later* failure in the same pass — a broken
write, or a failed end-of-pass flush — no longer reaches back and fails it.
That is the honest outcome: its bytes were flushed, so it was delivered. A
failure of the boundary flush itself is treated exactly like a failed
end-of-pass flush, and additionally fails the event frame the drain had
already claimed off its queue (nothing else would ever complete it) along
with every frame still queued.
Note the ordering all of this implies at the peer: the frames reach the pipe
before the flush that follows them, so the gateway can read a whole batch
while the writer has not yet flushed it. Anything observing the flush itself
(a test counting flushes, for instance) must wait for the flush, not infer it
@@ -134,12 +159,22 @@ drains them together, so a burst of N events costs one flush rather than N —
the coalescing the batch machinery was built for now engages on the event hot
path, not only when independent producers happen to queue behind a blocked
write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
concurrently queued control frame is still drained ahead of the batch. A
per-frame rejection inside a batch (for example one oversized event) surfaces
from the batch's awaited completions as that frame's
concurrently queued control frame is still drained — and now flushed and
completed — ahead of the batch's remaining events, which is why a batch a
control frame cuts into pays one extra flush while an uninterrupted batch
still pays exactly one. A per-frame rejection inside a batch (for example one
oversized event) surfaces from the batch's awaited completions as that frame's
`WorkerFrameProtocolException`; the remaining completions are still observed
so none faults unobserved.
The completion is the frame's delivery point, not necessarily the instant its
caller returns. A caller that loses the race for the write lock only observes
its own completion after the winning drainer releases the lock, so its return
remains bounded by that drain pass even though its control frame was flushed
and completed at the class boundary inside it. The boundary flush is what
makes the delivery point honest; unparking a lock-race loser from the winner's
pass would be a separate change to the enqueue-then-contend shape.
Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
for the write lock when its token fires tombstones the queued frame: the
cancelled caller marks its frame under `_gate`, and the draining lock-holder's
@@ -164,6 +199,17 @@ class drains both queues to empty, and the heartbeat loop guarantees one
arrives within a heartbeat interval, so worst-case residency is a few envelope
references for seconds — not a leak.
## Pipe Buffers
The gateway creates each worker pipe with an explicit 128 KiB kernel buffer per
direction (`SessionWorkerClientFactory.PipeBufferSizeBytes`) rather than the zero
quota the short `NamedPipeServerStream` overloads request. A zero-quota byte-mode
pipe makes every write rendezvous with a pending read, so a writer with no reader
parked blocks until one arrives — the failure class behind the historical windev
full-suite wedge. A real quota decouples writer latency from reader scheduling and
lets the flush coalescing above actually pay off. On Unix hosts, where named pipes
are Unix domain sockets, the sizes are advisory.
## Verification
The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`,
@@ -350,3 +350,61 @@ margins). Bumped for family-pin alignment.
**Verification.** Build 0 warnings / 0 errors; suite **879/879**; `staticwebassets.build.json`
resolves `zb.mom.ww.theme/0.4.1`. No stale-HTTP-cache clear was needed — restore picked 0.4.1
directly.
## 8. Follow-up: role-gate the side rail's Secrets link (family-wide nav task)
Requested as a family-wide sweep: every app's UI should link to the Secrets management page, visible
to Administrator-role users only.
**Found state.** The link already existed — `MainLayout.razor`, Admin section, `/admin/secrets`. What
did not exist was any gate: the rail rendered every item for every visitor, including a Viewer and
the anonymous-localhost read-only identity. The premise that there was an "existing role-gated nav
pattern" to follow was false; the rail's only `AuthorizeView` was the footer's signed-in/signed-out
split, so this introduces the pattern rather than extending it.
Not an access hole — the mounted page carries `[Authorize(Policy = "secrets:manage")]`, so a Viewer
clicking through was denied. It was a dead link presented as a live one.
**Gate chosen: the policy, not the role.** `<AuthorizeView Policy="@SecretsAuthorization.ManagePolicy">`,
i.e. the same policy the page itself enforces, so nav visibility cannot drift from page access. The
sweep asked for a role literal (`DashboardRoles.Admin` = `"Administrator"`), and in this host the two
are equivalent: `GatewayOptionsValidator` constrains `Dashboard:GroupToRole` values to
`Administrator` or `Viewer`, so the shared library's other manage-granting roles (`secrets-manager`,
`secrets-reveal`) are unreachable here. The policy form was preferred because it stays correct if
that constraint ever relaxes — a role literal would then hide the link from users who can use the
page.
**Deliberate asymmetry — API Keys stays ungated.** Its sibling item looks like the same case and is
not. `ApiKeysPage` renders for a Viewer with write affordances hidden (`@if (CanManageApiKeys)`), so
hiding its nav item would remove legitimate read access. The secrets page has no read-only mode. The
rule is "gate the link when the page denies the role outright", not "gate everything under Admin".
**Coverage.** Three tests pin the policy's verdict per principal (Administrator admitted, Viewer
refused, unauthenticated refused) in `SecretsNavGateTests`, and `/admin/secrets` joins the canonical
route list in `GatewayApplicationTests` — it is the one nav destination mounted from an RCL rather
than declared here, so a routing regression could remove it without touching this repo's pages.
### 8a. Correction: the policy tests could not detect a deleted gate
The coverage above shipped with a stated rationale — that rendering was disproportionate because the
policy verdict "is the part that can actually be wrong". That rationale was wrong, and a review point
from the OtOpcUa session identified why: the policy is library code this repo did not author, while
the *wiring* is the only thing this change introduced. Worse, the check applies specifically to repos
where the link already existed before gating — "an Administrator still sees it" is identical to the
pre-change behaviour, so it cannot distinguish a working gate from an inert one. **Only the negative
observation proves a gate exists at all.**
`SecretsNavRenderTests` now renders `MainLayout` through the framework's static `HtmlRenderer` — no
component-testing package needed, since the assertion is about emitted markup, not interactivity —
and asserts the Secrets item is absent for a Viewer and for an anonymous caller, present for an
Administrator, and that the ungated API Keys sibling stays present for a Viewer (so a later
"consistency fix" that hides it fails loudly).
**Confirmed non-vacuous by mutation**, which is the only thing that makes the absence assertions
worth anything: with the `AuthorizeView` removed from the layout, `Rail_OmitsSecretsLink_ForViewer`
and `Rail_OmitsSecretsLink_ForAnonymous` both go red — **and all three original policy tests stay
green**, demonstrating the gap concretely rather than by argument. The Administrator case is retained
as the control: without it, a rail that rendered no nav at all would satisfy both absence assertions
and the suite would report a working gate over a blank page.
**Verification.** Build 0 warnings / 0 errors; suite **899/899** (895 + 4).
@@ -0,0 +1,310 @@
# Deferred-Findings Remediation Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development to implement this plan task-by-task (Opus implementers per the user's instruction).
**Goal:** Resolve the six findings the 2026-08-15 perf-review remediation explicitly deferred (`docs/plans/2026-08-15-perf-review-remediation.md:611-620`) plus the pre-existing Windows-only `SecretsStorePathGuardTests` failure, so the deferred table empties and windev returns to a clean 1046/1046 gateway suite.
**Architecture:** Two phases. Phase A is gateway-side (net10, fully verifiable on macOS): the secrets-test fix, the distributor dictionary swap, event-path iterator flattening, and the dashboard in-process refactor that removes the Blazor pages' loopback SignalR hop while preserving the idle gate, mirror viewer gating, and clone-then-redact invariants. Phase B is worker-side (net48 x86, verified on windev over ssh): control-frame completion decoupling in the two-class frame writer, pipe-read teardown restructuring, and value-cache clone removal per the completed aliasing audit.
**Tech Stack:** .NET 10 / ASP.NET Core / Blazor Server / System.Threading.Channels (gateway); .NET Framework 4.8 x86 (worker); xUnit; windev CI clone `C:\build\mxaccessgw-ci` via `ssh windev`.
**Branch:** `perf/deferred-remediation` off local `main` (`15f188e`).
---
## Ground rules for every implementer subagent
- Shared working tree at `/Users/dohertj2/Desktop/MxAccessGateway`. **NEVER run `git stash`, `git reset`, `git clean`, `git checkout <sha/branch>`, or any command that touches files outside your task's `Files:` list.** Commit with explicit pathspecs only (`git add <your files> && git commit`).
- Build/test lock: before `dotnet build` or `dotnet test`, acquire the lock with `mkdir /private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/f36938ae-bbca-4245-b5c9-fac512d69e22/scratchpad/buildlock` (retry loop with sleep until it succeeds); `rmdir` it in ALL exit paths.
- `TreatWarningsAsErrors=true`, `Nullable=enable` repo-wide. Follow `docs/style-guides/CSharpStyleGuide.md`: file-scoped namespaces, `sealed` by default, `Async` suffix, MXAccess-aligned names.
- Worker projects (`ZB.MOM.WW.MxGateway.Worker*`) are net48/x86 and DO NOT COMPILE on macOS. For Phase B tasks: edit carefully, self-review for net48 compatibility (target-typed `new` and file-scoped namespaces ARE valid — `LangVersion=latest`; but no `Span`-based BCL overloads, no `IAsyncDisposable` on BCL types, `Channel` comes from System.Threading.Channels package which the worker already references). Compilation and tests happen at the Task 13 windev gate.
- Update affected docs in the same commit as the source (repo rule), except the dashboard design doc which Task 8 consolidates (deliberate, to avoid parallel edits to one file).
- MXAccess parity: never synthesize events, never mutate an event already handed to the outbound queue or wire.
---
## Phase A — gateway (macOS-verifiable)
### Task 1: Windows-safe cleanup in SecretsStorePathGuardTests
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 2, Task 3, Task 4
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Tests/Configuration/SecretsStorePathGuardTests.cs`
- Modify: `docs/GatewayTesting.md` (lines ~557-565, the "fails deterministically on Windows" note)
**Why:** `CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt` (lines 84-105) fails deterministically on Windows: `GatewayApplication.CreateBuilder` migrates the secrets store through `SecretsSqliteConnectionFactory` (`Pooling = true`, WAL), disposal returns the connection to the Microsoft.Data.Sqlite pool with the native handle open, and the `finally`'s `Directory.Delete(directory, recursive: true)` (line 103) hits a sharing violation. macOS passes only because Unix unlinks open files. The repo fixes this pattern twice already: `TestSupport/../TempDatabaseDirectory.cs:57` and `Configuration/PreHostSecretExpansionTests.cs:130-153`.
**Spec:**
1. In the failing test's `finally`, before `Directory.Delete`: call `Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();` and wrap the delete in `try { ... } catch (IOException) { } catch (UnauthorizedAccessException) { }` (best-effort, mirroring `TempDatabaseDirectory.Dispose`). Add a comment mirroring the one in `PreHostSecretExpansionTests.cs:133-137` (WAL + pooling keeps the handle alive past dispose).
2. Leave the rejection test alone (the guard means its file is never created).
3. Update `docs/GatewayTesting.md`: replace the "subtract it from the expected pass count on Windows" paragraph with a short note that the test's cleanup now clears the SQLite pool first and the failure is fixed as of this branch.
**Steps:** edit → `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~SecretsStorePathGuardTests"` (expect 2/2 on macOS; the real proof is the Task 13 windev gate) → commit `fix(tests): clear the SQLite pool before deleting the secrets path-guard temp dir — Windows sharing violation`.
---
### Task 2: SessionEventDistributor `_subscribers` → plain `Dictionary`
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 1, Task 3, Task 4
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionEventDistributor.cs`
**Why:** All five access sites (`:365`, `:555`, `:771`, `:799`, `:813`) are inside `lock (_lifecycleLock)`; the lock-free hot path reads the copy-on-write `_subscriberSnapshot` array (`:958`, `:302`), never the dictionary. The concurrent type buys nothing. Audit confirmed no external/reflection access.
**Spec:** Change the field at `:107` to `Dictionary<long, Subscriber>`; `TryRemove(subscriber.Id, out _)` at `:799` becomes `Remove(subscriber.Id)`. Reword the type remarks at `:69-80`, `:111-123`, and `:298-300` where they name `ConcurrentDictionary` by design — the invariant to state is now: "the dictionary is only ever touched under `_lifecycleLock`; lock-free readers use `_subscriberSnapshot`."
**Steps:** edit → `dotnet test ... --filter "FullyQualifiedName~SessionEventDistributorTests"` (29 facts, expect all green) → commit `refactor(sessions): _subscribers to plain Dictionary — every access is under _lifecycleLock`.
---
### Task 3: Merge the session event-source pass-through iterator
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/GatewaySession.cs` (`MapWorkerEventsAsync` ~:767-776, `ReadEventsAsync` ~:1517-1530)
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/GatewaySessionTests.cs` (existing)
**Why:** The worker→distributor source chain nests three compiler-generated async iterators per event: `WorkerClient.ReadEventsCoreAsync``GatewaySession.ReadEventsAsync` (pure pass-through: `TouchClientActivity(); yield return`) → `GatewaySession.MapWorkerEventsAsync` (`yield return mapper.MapEvent(...)`). The pass-through layer is two extra `MoveNextAsync` state-machine hops per event for no semantic value.
**Spec:**
1. FIRST grep all callers of `ReadEventsAsync`. If `MapWorkerEventsAsync` is its only caller, inline it: `MapWorkerEventsAsync` calls `GetReadyWorkerClientAsync`, iterates `client.ReadEventsAsync(ct)` directly, calls `TouchClientActivity()` per event, and `yield return mapper.MapEvent(workerEvent)`. Delete `ReadEventsAsync`. If other callers exist, keep the method for them but make `MapWorkerEventsAsync` self-contained as above — do NOT change any caller outside this file; report the finding.
2. Behavior must be byte-identical: same activity-touch cadence (per event), same exception propagation (WorkerClientException flows to the distributor pump unchanged), no event synthesis, worker order preserved.
3. `WorkerClient.ReadEventsCoreAsync`'s single-reader claim (`_eventsReaderClaimed`) must still be exercised exactly once per attach — do not add a second call site.
**Steps:** grep callers → edit → `dotnet test ... --filter "FullyQualifiedName~GatewaySession"` and `--filter "FullyQualifiedName~SessionEventDistributorTests"` → commit `perf(sessions): fold the ReadEventsAsync pass-through into MapWorkerEventsAsync — one fewer iterator per event`.
---
### Task 4: EventStreamService direct channel reads in the live loop
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 3
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Grpc/EventStreamService.cs`
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs` (existing 17 facts — must pass unchanged)
**Why:** The subscriber-side live loop materializes `subscriber.Reader.ReadAllAsync(ct).GetAsyncEnumerator(ct)` (`:109-111`) — a BCL async-iterator wrapper costing a state-machine hop per event on the hottest gateway path. Direct `ChannelReader` consumption (`WaitToReadAsync` + drain-with-`TryRead`) removes it.
**Spec:**
1. Replace the enumerator with direct reads: `while (await reader.WaitToReadAsync(ct)) { while (reader.TryRead(out MxEvent? mxEvent)) { ...existing per-event body... } }`; loop ends when `WaitToReadAsync` returns false (channel completed).
2. EVERY invariant in the current body survives, verbatim where possible:
- ReplayGap sentinel emitted exactly once, first, only when `replayGap` (`:133-139`) — untouched, it precedes the live loop.
- Replay batch stitching (`:141-150`) — untouched.
- Per-RPC dedup watermark `if (mxEvent.WorkerSequence <= afterWorkerSequence) continue;` (`:179-182`) — must apply to every live event.
- `WorkerClientException` catch → `session.MarkFaulted` → metrics → rethrow (`:164-174`): a completed-with-exception channel surfaces its exception from `WaitToReadAsync` — the catch must wrap the wait/read, preserving identical fault classification. Terminal `SessionManagerException(EventQueueOverflow)` propagates unchanged.
- `finally` ordering (`:192-200`): with no enumerator to dispose, the remaining order is backlog-gauge registration disposal → lease disposal → `metrics.StreamDisconnected("Detached")`. Keep the comments explaining why.
3. Cancellation: `WaitToReadAsync(ct)` throws `OperationCanceledException` on detach — must reach the same code path the enumerator's cancellation did (the gRPC layer treats it as client disconnect). Verify against `StreamEventsAsync_WhenCanceled_DetachesSubscriber`.
4. No public-surface change; `MxAccessGatewayService` (`:151-179`) is untouched.
**Steps:** edit → run the full `EventStreamServiceTests` class + `GatewayEndToEndReconnectReplayTests` + `GatewayEndToEndMultiSubscriberTests` → commit `perf(grpc): consume the subscriber channel directly in StreamEventsAsync — drops the ReadAllAsync iterator hop`.
---
### Task 5: In-process dashboard snapshot feed + page switch
**Classification:** high-risk
**Estimated implement time:** ~8 min (accepted overage; splitting further would split one invariant)
**Parallelizable with:** Task 6, Task 7
**Files:**
- Create: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/IDashboardSnapshotFeed.cs`
- Create: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardSnapshotFeed.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/DashboardPageBase.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardServiceCollectionExtensions.cs` (one `AddSingleton` line)
- Create: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardSnapshotFeedTests.cs`
**Why:** Eight pages inherit `DashboardPageBase` and each opens a loopback `/hubs/snapshot` HubConnection (`DashboardPageBase.cs:62`) — a WebSocket round trip back into the same process per circuit. `IDashboardSnapshotService.WatchSnapshotsAsync` exists but is NOT multicast (each enumeration = its own `PeriodicTimer` + snapshot build), so pages must not call it directly; a shared feed does one enumeration and fans out.
**Spec:**
1. `IDashboardSnapshotFeed` (singleton): `IAsyncEnumerable<DashboardSnapshot> WatchAsync(CancellationToken ct)`. Internally: per-subscriber `Channel<DashboardSnapshot>` with capacity 1 and `BoundedChannelFullMode.DropOldest` (a dashboard viewer only ever wants the latest snapshot; a slow circuit must never buffer unboundedly or stall others).
2. **Idle gating (the invariant this task must not lose):** the feed enumerates `IDashboardSnapshotService.WatchSnapshotsAsync` on a background task started when the subscriber count goes 0→1 and cancelled when it goes 1→0. While zero subscribers, the feed holds no timer and builds no snapshot. Guard subscriber add/remove with a plain lock; restart cleanly on resubscribe (mirror the start/stop discipline of `GatewayAlarmMonitor.StreamAsync` registration, `GatewayAlarmMonitor.cs:739-752`). If the underlying watch throws or completes, complete all subscriber channels with the error and reset so the next subscriber restarts it (mirror `DashboardSnapshotPublisher.ExecuteAsync`'s reconnect-after-delay posture, but per-feed).
3. `DashboardPageBase`: remove the HubConnection path (`:62` and the factory usage); keep the synchronous first render via `snapshotService.GetSnapshot()` (`:37`); then a background loop `await foreach (var s in feed.WatchAsync(_cts.Token)) { Snapshot = s; await InvokeAsync(StateHasChanged); }` started in `OnAfterRenderAsync(firstRender)` or `OnInitializedAsync` (match current lifecycle), cancelled + awaited in `DisposeAsync`. Update the class XML doc that narrates the hub subscription history (`:7-14`).
4. Hubs, `DashboardSnapshotPublisher`, `DashboardSnapshotHubConnectionCounter`, `DashboardHubConnectionFactory`, and `/hubs/token` all stay — they remain the remote/external surface. Do not touch them.
5. Auth: the pages are mapped behind `ViewerPolicy` (`DashboardEndpointRouteBuilderExtensions.cs:136`), which remains the gate for in-process consumption; add one comment on `WatchAsync` saying so.
6. Tests (`DashboardSnapshotFeedTests`): (a) zero subscribers → underlying service's `WatchSnapshotsAsync` never enumerated (fake service counts enumerations/`MoveNextAsync`); (b) first subscriber starts exactly one enumeration; two subscribers share it; (c) last unsubscribe cancels it; resubscribe restarts it; (d) slow subscriber observes latest-wins (push 3 snapshots, read 1, it is the newest) while a fast subscriber sees all; (e) underlying fault completes subscribers with the error and a fresh subscriber restarts.
**Steps:** write feed tests first (fail) → implement feed → page switch → `dotnet test ... --filter "FullyQualifiedName~DashboardSnapshotFeed"` then `--filter "FullyQualifiedName~Dashboard"` (whole dashboard test folder) → commit `feat(dashboard): in-process snapshot feed replaces the pages' loopback /hubs/snapshot hop`.
---
### Task 6: In-process session event subscription + SessionDetailsPage switch
**Classification:** high-risk
**Estimated implement time:** ~8 min
**Parallelizable with:** Task 5, Task 7
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHubViewerRegistry.cs` (only if a member is needed for synthetic connection ids; prefer reusing the existing API)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/SessionDetailsPage.razor` (the `/hubs/events` connection at `:271,297`)
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardEventBroadcasterTests.cs` (extend)
**Why:** `SessionDetailsPage` opens a loopback `/hubs/events` connection. The broadcaster already short-circuits on `!viewerRegistry.HasViewers(sessionId)` BEFORE the redaction deep clone (`DashboardEventBroadcaster.cs:51-56`) — the mirror viewer gating shipped last round. An in-process subscription must keep feeding that registry or every unwatched session pays `MxEvent.Clone()` per event again.
**Spec:**
1. Add to `DashboardEventBroadcaster` an in-process subscribe API: `IDashboardEventSubscription Subscribe(string sessionId)` returning a disposable that exposes `ChannelReader<MxEvent> Reader` (bounded, capacity ~256, `DropOldest` — this is a UI mirror, loss is acceptable and already documented for the hub path). On subscribe: register a synthetic connection id (e.g. `"inproc-" + Guid.NewGuid().ToString("N")`) with `EventsHubViewerRegistry.AddViewer(connectionId, sessionId)`; on dispose: `RemoveViewer` + `ReleaseConnection` in the order the hub uses (`EventsHub.cs:86,99`). Registry stays the single source of truth for `HasViewers`.
2. `Publish` (`:39-86`): after the existing `HasViewers` check and the clone-then-redact (`RedactValues` `:97-109`), `TryWrite` the SAME redacted clone to each in-process subscriber of that session, in addition to the hub group send. The source `MxEvent` is shared with the gRPC stream and replay ring — the existing never-mutate-in-place rule holds; in-process subscribers receive the redacted clone only.
3. `SessionDetailsPage`: replace the HubConnection + `SubscribeSession` invoke with `broadcaster.Subscribe(SessionId)` and a read loop marshalling to the renderer via `InvokeAsync(StateHasChanged)`; dispose the subscription in `DisposeAsync`. Keep the existing per-session ACL posture (any Viewer may watch any session — SEC-25 is tracked separately; do not widen or narrow it here).
4. Tests to add in `DashboardEventBroadcasterTests`: (a) in-process subscriber receives the redacted event when `ShowTagValues=false` and the source event is not mutated; (b) subscribing flips `HasViewers` so `Publish` stops short-circuiting (proves mirror gating integration); (c) disposing the last in-process subscriber restores the no-viewers short-circuit (no clone, no send — reuse the existing `Publish_WithNoRegisteredViewers_DoesNotCloneOrSend` fake pattern); (d) hub viewers and in-process viewers are independently counted.
**Steps:** tests first → implement → `dotnet test ... --filter "FullyQualifiedName~DashboardEventBroadcaster"` + `--filter "FullyQualifiedName~EventsHubViewerRegistry"` + `--filter "FullyQualifiedName~GatewaySessionDashboardMirror"` → commit `feat(dashboard): in-process session event subscription feeds the viewer registry — SessionDetailsPage drops its /hubs/events hop`.
---
### Task 7: AlarmsPage provider-status via IGatewayAlarmService
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 5, Task 6
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/AlarmsPage.razor` (`:194` HubConnection, `:281-304` poll loop untouched)
**Why:** `AlarmsPage` opens `/hubs/alarms` but only consumes `ProviderStatus` payloads from it (alarm rows come from the 3 s `QueryAlarmsAsync` poll). `IGatewayAlarmService.StreamAsync` (`GatewayAlarmMonitor.cs:724-777`) is already a true multi-subscriber in-process fan-out.
**Spec:** Replace the HubConnection with a background loop over `alarmService.StreamAsync(alarmFilterPrefix: null, ct)`, handling only `PayloadOneofCase.ProviderStatus` (skip snapshot/live alarm payloads — the poll stays authoritative for rows). The monitor's drop policy completes a lagging subscriber's channel (`:700-712`): on completion or fault, delay ~1 s and resubscribe (matching the hub path's `WithAutomaticReconnect` posture). Dispose via the page's existing cancellation. Leave the poll loop alone.
**Steps:** edit → `dotnet build src/ZB.MOM.WW.MxGateway.Server``dotnet test ... --filter "FullyQualifiedName~Alarms"` → commit `feat(dashboard): AlarmsPage reads provider status from IGatewayAlarmService in-process`.
---
### Task 8: Dashboard design-doc update (consolidated)
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (runs after 5, 6, 7 land)
**Files:**
- Modify: `docs/GatewayDashboardDesign.md` (sections at ~:112-114, :162-178, :190-217, :228-247, :535-541, :581-595)
**Spec:** Rewrite the affected sections to describe: pages consume in-process seams (`IDashboardSnapshotFeed`, `DashboardEventBroadcaster.Subscribe`, `IGatewayAlarmService.StreamAsync`); the three hubs and `/hubs/token` remain as the remote/external surface; idle gating is now two-tier (hub connection counter gates the hub publisher; feed subscriber count gates the in-process pump — while nobody watches, neither builds a snapshot); mirror gating counts hub viewers AND in-process viewers through the one registry; clone-then-redact still happens once in the broadcaster before any delivery; ViewerPolicy on the component endpoint is the in-process auth gate; SEC-25 per-session ACL gap unchanged. Present tense, why-not-what, no marketing.
**Commit:** `docs(dashboard): in-process page feeds, two-tier idle gating, hubs as the external surface`
---
### Task 9: Phase A gate — full gateway suite on macOS
**Classification:** trivial (verification only)
**Parallelizable with:** none (after Tasks 1-8)
Run `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (expect 0 warnings) and the full `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj` (expect ≥1046 passed, 0 failed; new feed/broadcaster tests raise the count). Fix-forward any failure before Phase B.
---
## Phase B — worker (net48 x86, verified on windev)
### Task 10: Control-frame completion decoupling in WorkerFrameWriter
**Classification:** high-risk
**Estimated implement time:** ~6 min
**Parallelizable with:** Task 11, Task 12
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs`
- Modify: `docs/WorkerFrameProtocol.md` (~:120-131 completion-semantics paragraph)
**Why:** Wire ordering is already correct — `DequeueNext` (`:383-413`) re-checks `_controlFrames` before every frame. The coupling is completion latency: `DrainQueuedFramesAsync` (`:304-361`) defers the single `FlushAsync` and ALL `TrySetResult` calls to after the whole drain pass, so a heartbeat/command-reply/fault/shutdown-ack `Task` awaited by its writer does not resolve until up to 128 event frames behind it are written and flushed. The XML docs claim "never delayed behind an event backlog" — true of bytes, false of the awaited completion.
**Spec:**
1. Record the priority class on `PendingFrame` (`:23-48`), set at construction in `WriteAsync` (`:109`) and `WriteBatchAsync` (`:192`).
2. In `DrainQueuedFramesAsync`: when `DequeueNext` returns an `Event` frame while `written` contains one or more not-yet-completed `Control` frames, first `FlushAsync` + complete + clear `written`, then continue draining. Exit-path flush at `:339-360` unchanged. Net effect: a control frame's completion never waits on an event frame dequeued after it; the pure-event 128-batch hot path still pays exactly one flush (guarded by the existing `WriteAsync_WhenBatchDrainedTogether_FlushesOnce` and `EventBurst_DrainLoopCoalescesFlushes`); a pure-control burst still pays one flush. Do NOT flush per control frame unconditionally — that reintroduces the pre-WRK-12 syscall-per-heartbeat cost.
3. Failure handling: `FailFrames(written, ...)` / `FailAllQueued` (`:327-336`) operate on the current `written` list; after an early flush+complete+clear, frames already completed must not be failable — verify the clear ordering makes that structurally true, and extend the fault-injection tests if the early-flush path adds a new failure window (a `FlushAsync` fault with a partially-completed pass).
4. New test (use the existing `GatedWriteStream` harness ~`:880`): queue a control frame behind N gated event frames within one drain pass; assert the control frame's `WriteAsync` task completes before the last event write is released. Keep all 9 existing writer tests green — sequence stamping (`:431-483`), claim/tombstone interlock (`:244-274`), and wire order must be untouched.
5. `docs/WorkerFrameProtocol.md`: update the completion-semantics paragraph — completion now resolves at the class-transition flush, still meaning "written AND flushed".
**Commit:** `perf(worker): control-frame completions resolve at the class-transition flush, not after the event batch`
---
### Task 11: Worker pipe-read teardown — dispose-to-unblock and observe the abandoned read
**Classification:** high-risk
**Estimated implement time:** ~8 min
**Parallelizable with:** Task 10, Task 12
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs` (`RunMessageLoopAsync` `:267-310`, ctor `:55-68`)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeClient.cs` (`:143-159`) — only if ownership must move; prefer not
- Modify: `src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionTests.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameReader.cs` (comment only)
**Why:** On net48, `NamedPipeClientStream.ReadAsync` ignores its `CancellationToken` (`WorkerFrameReader.cs:109-111`). Fault-path exits (event-drain fault, oversized event, watchdog, heartbeat write failure) leave `readTask` pending; it is unblocked only when `WorkerPipeClient`'s `using` disposes the pipe, at which point it faults with `ObjectDisposedException`/`IOException` on a Task nobody observes (the finally at `:303-309` awaits only heartbeat and drain). The frame-pooling change (GWC-30) makes this sharper: the abandoned read owns the per-instance `_lengthPrefix` buffer and possibly a rented ArrayPool payload — the reader's single-consumer invariant holds today only because nothing ever reads again after abandonment.
**Spec — constraints, implementer designs within them:**
1. **No unobserved faulted Task.** After the stream is disposed, `readTask`'s fault must be awaited/observed (reuse `ObserveBackgroundTaskStopAsync`'s timeout-and-log shape, `:312-348`) before `WorkerPipeClient.RunAsync` returns.
2. **Ordering: final writes complete before disposal.** The shutdown ack (`WriteShutdownAckAsync` `:1064-1069`) and fault frames (`TryWriteFaultAsync` `:1164+`) are written after the message loop exits on some paths — trace every exit path and place the stream disposal AFTER the last possible write on each. The clean design: `WorkerPipeSession` keeps a reference to the ctor `Stream`; `RunAsync`'s outermost finally (after runtime-session disposal and any fault write, `:133-145`) disposes the stream and then observes `readTask` (stored in a field by `RunMessageLoopAsync`). `WorkerPipeClient`'s `using` then double-disposes harmlessly. If the trace shows a fault write that happens in `WorkerPipeClient` after `session.RunAsync` returns (there is none known), fall back to moving observation into `WorkerPipeClient`.
3. **Never a second read.** After abandonment, no code path may call `_reader.ReadAsync` again (pooled-buffer use-after-return). The message loop already guarantees this (`return` before reassignment on the graceful path); keep it structurally true and assert it in a comment on `_lengthPrefix` (`WorkerFrameReader.cs:23-25`).
4. **Graceful path unchanged:** `WorkerShutdown`/`ShutdownWorker` exits have no pending read; disposal+observation must be a no-op there (observe a completed/absent task).
5. Document the net48 token-ignoring fact where the read is issued (`RunMessageLoopAsync` and/or `ReadExactlyOrThrowAsync`) — the research found zero comments acknowledging it.
6. Tests (net48 project, real `PipePair` harness `:2433-2485`): (a) fault-path exit (reuse the `RunAsync_EventFrameTooLarge_...` shape `:868`) — assert `RunAsync` completes within the existing 5 s bound AND, via a `TaskScheduler.UnobservedTaskException` hook armed in the test with a forced GC, that no unobserved exception leaks; (b) graceful shutdown still completes with no pending read; (c) the session disposes the stream (harness observes the gateway-side stream faulting its own pending read promptly rather than at `PipePair.Dispose`).
**Commit:** `fix(worker): session-owned stream disposal unblocks and observes the net48 pipe read at teardown`
---
### Task 12: Value-cache clone removal per the aliasing audit
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10, Task 11
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessValueCache.cs` (`Set` `:82,83,97`; `CachedValue` `:275`)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker.Tests/MxAccess/MxAccessValueCacheTests.cs` (rewrite the `:58` test)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker.Tests/MxAccess/MxAccessCommandExecutorTests.cs` (add cached-read test)
**Why (audit result):** All three clones in `Set``Value.Clone()` (deep, recursive for arrays), `SourceTimestamp.Clone()`, `Statuses.Clone()` (container + N proxies) — are removable. The event is fully stamped BEFORE `Set` runs (`Enqueue` at `MxAccessBaseEventSink.cs:263` precedes `postPublish` at `:288`; sequence/timestamp stamped inside `Enqueue`, `MxAccessEventQueue.cs:269-270`) and the queue's ownership invariant forbids later mutation. The alias already exists on the read side: `SucceededRead` (`MxAccessSession.cs:1086,1091,1096`) hands the cache's own `Value`/`SourceTimestamp` instances into every `BulkReadResult`, which downstream only wraps and serializes. Worker↔gateway is a process boundary — no gateway consumer can alias.
**Spec:**
1. Remove all three clones; `CachedValue` stores the event's own references.
2. Ownership contract comment on `Set` and on `CachedValue`: the cache holds borrowed references into an enqueued, write-once `MxEvent`; consumers may read and serialize, never mutate; mutation would additionally invalidate `QueuedEvent.Size` — the enqueue-time memoized serialized size that the byte-budgeted `Drain` charges (`MxAccessEventQueue.cs:499-506`), so a grown message could overshoot the negotiated frame max and fault the session via `MessageTooLarge`.
3. Rewrite `Set_StoresIndependentSnapshot_UnaffectedByLaterEventMutation` (`:58` — it codifies the invariant being reversed) into the aliasing contract: `Set` then `TryGet` returns the same `Value`/`SourceTimestamp`/`Statuses`-element instances (`Assert.Same`), with the doc comment explaining the write-once borrow.
4. Add the missing cached-read-path test in `MxAccessCommandExecutorTests`: seed the cache, dispatch a `ReadBulk` that hits `TryGetCachedReadFor` → assert `WasCached == true` and `result.Value` is reference-equal to the cached instance (closing the coverage gap the audit found — nothing today exercises `WasCached == true` end-to-end in the worker).
5. `MxAccessWriteCompletionCache.Record`'s parallel `statuses.Clone()` (`:76`) is left AS-IS deliberately (different lifecycle, not in the finding) — add one cross-reference comment there pointing at the value-cache ownership contract.
**Commit:** `perf(worker): value cache borrows the write-once event's instances — three clones per OnDataChange removed`
---
### Task 13: Phase B gate — windev full verification
**Classification:** trivial (verification only)
**Parallelizable with:** none (after Tasks 10-12; Phase A gate must be green)
Push the branch to origin, then on windev (`ssh windev`, clone `C:\build\mxaccessgw-ci`): fetch + checkout the branch; `dotnet build src/ZB.MOM.WW.MxGateway.slnx` (0 warnings); `dotnet build src/ZB.MOM.WW.MxGateway.Worker/... -p:Platform=x86`; `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/... -p:Platform=x86` (expect 501+ passed, 0 failed — new tests raise the count); `dotnet test src/ZB.MOM.WW.MxGateway.Tests/...` (expect **0 failed including SecretsStorePathGuardTests** — the Task 1 proof). Known caveat: the reconnect-replay test is load-sensitive on windev; re-run isolated before treating it as a regression (documented in `docs/GatewayTesting.md`).
---
### Task 14: Wrap-up — deferred table closure, docs sweep, final review
**Classification:** small
**Parallelizable with:** none (last)
- Append a closure note to `docs/plans/2026-08-15-perf-review-remediation.md`'s deferred table (one line: resolved by this plan, date, branch).
- Sweep: `gateway.md` / `docs/WorkerFrameProtocol.md` / `docs/GatewayDashboardDesign.md` / `docs/GatewayTesting.md` consistency with as-built behavior; record any accepted deviations in THIS plan's "As-built notes" section (add it).
- Update `.tasks.json` statuses; update auto-memory (`perf-remediation-branch.md` or successor) with the branch state.
- Dispatch the final integration code review (Opus) over `git diff main..perf/deferred-remediation` before reporting done. Merge remains the user's decision.
---
## Explicitly out of scope
| Item | Why |
|---|---|
| wnwrap alarm GUID identity semantics; `ALARM_RECORDS/@COUNT` probe | Need live alarms on windev — external state this plan cannot provide. Still tracked in the prior plan's follow-ups. |
| Structural alarm-truncation degraded-status signal | Contract-level design (proto change candidate) — separate effort. |
| SEC-25 per-session dashboard event ACL | Security roadmap item; Task 6 deliberately preserves the current posture. |
| `MxAccessWriteCompletionCache` clone | Different lifecycle than the value cache; consciously kept (Task 12.5). |
@@ -0,0 +1,20 @@
{
"planPath": "docs/plans/2026-08-15-deferred-remediation.md",
"tasks": [
{ "id": 1, "subject": "Task 1: Windows-safe cleanup in SecretsStorePathGuardTests", "status": "completed" },
{ "id": 2, "subject": "Task 2: SessionEventDistributor _subscribers to plain Dictionary", "status": "completed" },
{ "id": 3, "subject": "Task 3: Merge the session event-source pass-through iterator", "status": "completed" },
{ "id": 4, "subject": "Task 4: EventStreamService direct channel reads in the live loop", "status": "completed" },
{ "id": 5, "subject": "Task 5: In-process dashboard snapshot feed + page switch", "status": "completed" },
{ "id": 6, "subject": "Task 6: In-process session event subscription + SessionDetailsPage switch", "status": "completed" },
{ "id": 7, "subject": "Task 7: AlarmsPage provider-status via IGatewayAlarmService", "status": "completed" },
{ "id": 8, "subject": "Task 8: Dashboard design-doc update (consolidated)", "status": "completed", "blockedBy": [5, 6, 7] },
{ "id": 9, "subject": "Task 9: Phase A gate — full gateway suite on macOS", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] },
{ "id": 10, "subject": "Task 10: Control-frame completion decoupling in WorkerFrameWriter", "status": "completed", "blockedBy": [9] },
{ "id": 11, "subject": "Task 11: Worker pipe-read teardown — dispose-to-unblock and observe", "status": "completed", "blockedBy": [9] },
{ "id": 12, "subject": "Task 12: Value-cache clone removal per the aliasing audit", "status": "completed", "blockedBy": [9] },
{ "id": 13, "subject": "Task 13: Phase B gate — windev full verification", "status": "pending", "blockedBy": [10, 11, 12] },
{ "id": 14, "subject": "Task 14: Wrap-up — deferred table closure, docs sweep, final review", "status": "pending", "blockedBy": [13] }
],
"lastUpdated": "2026-08-15T00:00:00Z"
}
@@ -0,0 +1,627 @@
# Performance Review Remediation Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task (or superpowers-extended-cc:subagent-driven-development when executing in-session).
**Goal:** Resolve every actionable finding from the 2026-08-15 architectural performance review — six High findings, the Medium tier, and the worthwhile Low/hygiene items — without changing any MXAccess parity behavior or public contract.
**Architecture:** Two phases. Phase A is gateway-side (.NET 10, builds and tests locally on macOS via `NonWindows.slnx`); Phase B is worker-side (.NET Framework 4.8 x86, which does **not** compile on this Mac — Phase B tasks are edited here and verified in one consolidated pass on the windev box via the `psbridge` skill, Task 24). No `.proto` changes anywhere in this plan, so no client regeneration is needed. All work happens on branch `perf/review-remediation`.
**Tech Stack:** ASP.NET Core gRPC, System.Threading.Channels, SignalR, Microsoft.Data.Sqlite, .NET Framework 4.8 STA/COM interop, protobuf (Google.Protobuf).
---
## Ground rules for every implementer (read before your task)
- **Build gate:** `TreatWarningsAsErrors=true`, `Nullable=enable`, analyzers at latest. New warnings fail the build — fix them, never suppress.
- **Style:** follow `docs/style-guides/CSharpStyleGuide.md` — file-scoped namespaces, `sealed` by default, `Async` suffix, MXAccess-aligned names. Match the comment density and idiom of the file you're editing.
- **Parity is sacred:** do not change MXAccess-visible semantics (event ordering, `OperationComplete` behavior, write-completion reply shape, per-tag ReadBulk timeout meaning). These tasks change *mechanics* (waits, locks, allocations), never observable protocol behavior, except where a task explicitly says otherwise.
- **Never synthesize events.** Nothing in this plan may fabricate an `MxEvent`.
- **Docs in the same commit:** when a task changes configuration, event mechanics, security behavior, or lifecycle rules, the named docs must be updated in that task's commit.
- **Worker code (Phase B) does not compile on this machine.** `LangVersion=latest` applies, so modern syntax is fine, but only net48-era BCL APIs exist (no `Span`-taking stream overloads, no `ArgumentNullException.ThrowIfNull` — check what the file already uses). Match the existing worker idioms exactly. Verification is Task 24.
- **Tests:** gateway tests use the FakeWorkerHarness (`src/ZB.MOM.WW.MxGateway.Tests`), no MXAccess needed. Run only your task's filter, not the full suite (full suite runs once per phase).
- **Commit after every task**, message style: `perf(<area>): <what>` (or `fix(...)` for the two correctness bugs).
Verification commands used throughout:
```bash
# Gateway build (macOS-safe)
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
# Targeted gateway tests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~<TestClass>"
```
---
# Phase A — Gateway (local verification)
### Task 1: Named-pipe buffer sizes
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Tasks 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionWorkerClientFactory.cs` (`CreatePipe`, ~line 157)
- Modify: `docs/WorkerFrameProtocol.md` (add a short "Pipe buffers" note)
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/` (existing factory/e2e tests must stay green; no new test — buffer size isn't observable through the .NET API)
**Why:** the current 5-arg `NamedPipeServerStream` overload passes `inBufferSize: 0, outBufferSize: 0`. A zero-quota byte-mode pipe forces every write to rendezvous with a pending read — lock-step IPC, and the exact failure class behind the historical windev suite wedge.
**Step 1: Change the overload**
```csharp
private const int PipeBufferSizeBytes = 128 * 1024;
private static NamedPipeServerStream CreatePipe(string pipeName)
{
return new NamedPipeServerStream(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous,
inBufferSize: PipeBufferSizeBytes,
outBufferSize: PipeBufferSizeBytes);
}
```
Add a comment stating *why* (zero-quota rendezvous behavior; reference the windev wedge). Note: on Unix these sizes are advisory (Unix domain socket), which is fine — the fix targets Windows production.
**Step 2:** `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` → 0 errors.
**Step 3:** `dotnet test ... --filter "FullyQualifiedName~GatewayEndToEndFakeWorkerSmokeTests"` → PASS.
**Step 4:** Update `docs/WorkerFrameProtocol.md` with a 34 line "Pipe buffers" paragraph. Commit: `perf(ipc): give worker pipes real OS buffers instead of zero-quota rendezvous`
---
### Task 2: Metrics — pull-gauge for worker queue depth, lock-free command counters
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 1, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14 (NOT Task 7 — both edit `WorkerClient.cs`)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Metrics/GatewayMetrics.cs` (`SetWorkerEventQueueDepth` ~290; `CommandStarted/Succeeded/Failed` ~202247; gauge wiring ~91; snapshot ~461492)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs` (call sites ~303 and ~602)
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Metrics/` (extend the existing GatewayMetrics test class)
**Why:** `SetWorkerEventQueueDepth` takes the process-wide `_syncRoot` twice per event for every session, and the single scalar makes the gauge last-writer-wins across sessions (a correctness bug). The command counters take the same global lock 23× per RPC.
**Step 1 (failing test):** add a test that registers two worker-queue-depth sources reporting 3 and 4 and asserts the snapshot/gauge reports 7; add a test that `CommandStarted`×N from parallel tasks yields exactly N with no lock (behavioral: just correctness of count).
**Step 2 (implement):**
- Mirror the existing GWC-15 pattern verbatim: add `RegisterWorkerEventQueueDepthSource(Func<int> depth)` returning an `IDisposable` handle, a `ConcurrentDictionary<long, Func<int>>` of sources, and make `GetWorkerEventQueueDepth` sum the sources (clamp negatives). Delete `SetWorkerEventQueueDepth` and the `_workerEventQueueDepth` field.
- `WorkerClient`: at construction (or first use), register a source returning its staged+channel depth via `Volatile.Read` of a field the stage/consume paths maintain with `Interlocked` — the hot path does **no** metrics call at all anymore. Dispose the registration in `DisposeAsync`.
- Command counters: `_commandsStarted/_commandsSucceeded/_commandsFailed` become `long` updated with `Interlocked.Increment`; `_commandFailuresByMethod` becomes `ConcurrentDictionary<string, long>` (follow the existing `EventReceived` pattern in the same file). Snapshot reads with `Interlocked.Read`.
**Step 3:** run the Metrics test filter → PASS. **Step 4:** grep the repo for `SetWorkerEventQueueDepth` — zero hits outside tests you updated.
**Step 5:** Commit: `perf(metrics): pull-model worker queue gauge (fixes last-writer-wins), Interlocked command counters`
---
### Task 3: Distributor — copy-on-write subscriber snapshot
**Classification:** high-risk (core event fan-out concurrency)
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 1, 2, 4, 5, 6, 8, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionEventDistributor.cs` (pump loop ~600; register/unregister paths; the "snapshot-free enumerator" remark ~71)
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/` (existing SessionEventDistributor tests must stay green; add one test if a register-during-pump race test doesn't already exist)
**Why:** `_subscribers.Values` (the property) locks the whole `ConcurrentDictionary` and materializes a snapshot list **per event**, contradicting the adjacent comment.
**Step 1 (implement):** maintain a `volatile Subscriber[] _subscriberSnapshot` rebuilt inside the existing registration lock on every register/unregister (the set is tiny and mutates rarely). The pump iterates the array. Keep the dictionary if other paths use keyed lookup; the array is purely the fan-out view. Update the ~71 remark to describe the actual mechanism. Semantics to preserve exactly: a subscriber registered mid-iteration may miss the in-flight event ("late subscribers see events after they register") — the array snapshot preserves this naturally.
**Step 2:** run the distributor/replay test filters (`FullyQualifiedName~SessionEventDistributor`, `~Replay`) → PASS. The replay-handoff atomicity tests are the critical gate here.
**Step 3:** Commit: `perf(events): copy-on-write subscriber snapshot in fan-out pump`
---
### Task 4: Dashboard event mirror — viewer gating
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 1, 2, 3, 5, 6, 8, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs`
- Create: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHubViewerRegistry.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs` (Publish, ~39)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardServiceCollectionExtensions.cs` (register the registry)
- Modify: `docs/GatewayDashboardDesign.md` (mirror gating paragraph)
- Test: create `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/EventsHubViewerRegistryTests.cs` + extend the existing DashboardEventBroadcaster tests
**Why:** with `ShowTagValues=false` (default), `Publish` deep-clones every event and dispatches to a SignalR group that is empty in the steady state. No viewer gate exists anywhere on the path.
**Step 1 (failing test):** broadcaster with zero registered viewers for the session performs **no clone and no send** (assert via a counting fake hub-clients/`IHubContext` seam, matching however the existing broadcaster tests fake SignalR); with one viewer, behavior is unchanged (redacted clone sent).
**Step 2 (implement):**
- `EventsHubViewerRegistry` (singleton): `ConcurrentDictionary<string, int>` session→viewer count, `Increment(sessionId)`, `Decrement(sessionId)`, `HasViewers(sessionId)`. Track per-connection subscribed sessions in a `ConcurrentDictionary<string, ConcurrentDictionary<string,byte>>` keyed by connection id so `OnDisconnectedAsync` can decrement everything that connection held.
- `EventsHub`: `SubscribeSession`/`UnsubscribeSession` update the registry alongside the group add/remove; override `OnDisconnectedAsync` to release the connection's sessions. Keep the existing SEC-25 remark intact.
- `DashboardEventBroadcaster.Publish`: first line after the null-guards becomes `if (!viewerRegistry.HasViewers(sessionId)) { return; }` — before the redact/clone.
- Do **not** attempt lazy mirror-lease start in this task (it interacts with distributor lifecycle); the gate above removes ~all of the waste already. Note this decision in the doc paragraph.
**Step 3:** run Dashboard test filter → PASS. **Step 4:** update `docs/GatewayDashboardDesign.md`. Commit: `perf(dashboard): gate event mirror on live viewers — no clone, no send for unwatched sessions`
---
### Task 5: Snapshot pipeline — idle gating, cached config, keyed refresh cadence
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 1, 2, 3, 4, 6, 8, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardSnapshotPublisher.cs` (~6983)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardSnapshotHub.cs` (connection counting)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardSnapshotService.cs` (~103 config rebuild, ~163164 + ~267 API-key refresh)
- Modify: `docs/GatewayDashboardDesign.md`
- Test: extend existing snapshot service/publisher tests under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/`
**Why:** the 1 Hz tick runs an API-key SQLite read, a registry sort, a metrics snapshot, and a rebuild of the *static* effective-configuration record, broadcast to `Clients.All`, forever, with zero viewers.
**Step 1 (failing tests):** (a) effective configuration object is reference-identical across two snapshot builds; (b) API-key summaries refresh at most once per configured interval (inject `TimeProvider`, follow the file's existing time idiom); (c) publisher with zero connections does not enumerate the snapshot source (fake the hub context; count pulls).
**Step 2 (implement):**
- Cache `EffectiveGatewayConfiguration` in a field on first build (it's startup-static; add a comment saying so).
- `RefreshApiKeySummariesAsync`: skip unless `RefreshInterval` (new private constant, 15 s) has elapsed since the last successful refresh.
- `DashboardSnapshotHub`: `OnConnectedAsync`/`OnDisconnectedAsync` maintain an `int` connection count on a small singleton (or reuse the Task 4 registry class with a well-known key — implementer's choice, keep it simple). Publisher checks the count each tick: zero connections → `await Task.Delay(interval)` and skip both the snapshot build and the broadcast. First connection after idle gets a fresh snapshot on its next tick (≤1 interval of staleness — acceptable; pages also seed from `IDashboardSnapshotService` directly on load).
**Step 3:** dashboard test filter → PASS. Docs paragraph. Commit: `perf(dashboard): idle-gate the snapshot tick; cache static config; bound key-list refresh`
---
### Task 6: Reply ownership transfer in `MapCommandReply`
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Tasks 1, 2, 3, 4, 5, 8, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGrpcMapper.cs` (~74)
- Test: existing mapper/service tests under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/`
**Why:** every `WorkerCommandReply` is parsed fresh from one pipe frame and completed to exactly one awaiter; the gRPC handler is its only consumer. Events already got this treatment under GWC-07 — replies still deep-copy, which doubles the largest hot-path message on bulk reads.
**Step 1 (verify precondition, in-code):** confirm (grep) that no caller of `WorkerClient.InvokeAsync` retains `reply.Reply` after mapping — the review found the Invoke path clean; `GatewayAlarmMonitor` and `DashboardLiveDataService` own their separate replies. If you find a second consumer, STOP and surface it — that's a plan defect.
**Step 2 (implement):** `return reply.Reply.Clone();``return reply.Reply;` with a GWC-07-style ownership comment: the worker reply object is single-consumer by construction (one frame → one `PendingCommand` completion → one mapper call); the mapper transfers ownership to the gRPC response.
**Step 3:** run `FullyQualifiedName~MxAccessGrpcMapper` + the fake-worker smoke filter → PASS. Commit: `perf(grpc): transfer reply ownership instead of deep-cloning every worker reply`
---
### Task 7: WorkerClient — pooled-timer timeout, single sizing pass, `WorkerCancel` on timeout
**Classification:** high-risk (IPC concurrency + protocol behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 3, 4, 5, 8, 10, 11, 12, 13, 14 (NOT Task 2 — both edit `WorkerClient.cs`; run after Task 2)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs` (InvokeAsync ~226270; timeout path)
- Modify: `docs/GatewayProcessDesign.md` (command timeout → cancel-forwarding note)
- Test: extend `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/` worker-client tests (the fakes there already exercise timeout paths)
**Why:** each Invoke churns a linked CTS + `Task.Delay` timer + `WhenAny`; `CalculateSize` runs twice (protobuf doesn't memoize); and on timeout the gateway never tells the worker, so a timed-out COM call keeps occupying the STA and an envelope still queued gets written anyway.
**Step 1 (failing test):** on command timeout, the client enqueues a `WorkerCancel` envelope carrying the timed-out correlation id (assert via the fake connection's written-frame log).
**Step 2 (implement):**
- Replace the CTS/Delay/WhenAny block with `await pendingCommand.Task.WaitAsync(timeout, cancellationToken)` wrapped in a `try/catch (TimeoutException)` / `(OperationCanceledException)` mapping to the exact same `WorkerClientErrorCode`s and messages as today (tests depend on them).
- On the timeout path, after `RemovePendingCommandAsFailed`, best-effort enqueue a `WorkerCancel` envelope for the correlation id (fire-and-forget with a swallow-and-log; never let cancel failure mask the timeout exception). The worker already handles `WorkerCancel` (`WorkerPipeSession``CancelCommand`).
- Thread the already-computed `envelopeSize` into the frame write path if the writer API allows passing a known size; if the writer's public surface would have to change more than trivially, skip this sub-item and leave a `// PERF:` note — the timer and cancel fixes carry the task.
**Step 3:** worker-client test filter → PASS, including existing timeout tests unchanged. Docs note. Commit: `perf(ipc): WaitAsync command timeouts + forward WorkerCancel so a timed-out COM call frees the STA`
---
### Task 8: Audit pipeline — startup bootstrap, background writer, retention
**Classification:** high-risk (security/audit semantics)
**Estimated implement time:** ~5 min (split if it runs long: 8a writer, 8b retention)
**Parallelizable with:** Tasks 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Audit/SqliteCanonicalAuditStore.cs` (per-op `EnsureTableAsync` ~5254, ~94, ~131136)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Audit/CanonicalAuditWriter.cs` (~35)
- Create: `src/ZB.MOM.WW.MxGateway.Server/Security/Audit/ChannelAuditWriter.cs` (bounded channel + hosted drain)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/AuthStoreServiceCollectionExtensions.cs` (DI wiring + hosted service)
- Modify: `docs/DesignDecisions.md` (audit is asynchronous best-effort, bounded, with retention)
- Test: create `src/ZB.MOM.WW.MxGateway.Tests/Security/Audit/ChannelAuditWriterTests.cs`
**Why:** constraint denials await a SQLite insert inline per denied tag inside bulk RPC loops — sequential round-trips into the same DB file the auth store uses, each with a redundant `CREATE TABLE IF NOT EXISTS`, into a table with no retention.
**Step 1 (failing tests):** (a) `WriteAsync` returns without touching the store (enqueue-only) and the event lands in the store shortly after (drain); (b) when the bounded channel (capacity 4096) is full, `WriteAsync` drops (oldest or newest — pick drop-write/newest for simplicity) and increments a counter, never blocks; (c) retention sweep deletes rows older than the configured window.
**Step 2 (implement):**
- `ChannelAuditWriter : ICanonicalAuditWriter` (or whatever the current writer interface is named — read `CanonicalAuditWriter.cs` first): bounded `Channel<CanonicalAuditEvent>` (`BoundedChannelFullMode.DropWrite`), a `BackgroundService` drain that batches up to 64 events into one transaction per drain pass. The audit contract is already documented best-effort — say so in the class doc.
- Table bootstrap: run `EnsureTableAsync` once from the drain service's `StartAsync` (and from the store's first list call via a `Lazy`/latch); remove the per-insert and per-list calls.
- Retention: in the same drain service, once per hour, `DELETE FROM audit_event WHERE timestamp < now - RetentionDays` (new `SecurityOptions`/audit option, default 90 days, validated ≥1 in `GatewayOptionsValidator`); document in `docs/GatewayConfiguration.md`.
- Wire DI so `ConstraintEnforcer.RecordDenialAsync` transparently goes through the channel writer — **no signature changes** at the enforcer/service layer.
- Flush-on-shutdown: drain the channel in `StopAsync` with a 2 s cap.
**Step 3:** audit test filter + `FullyQualifiedName~ConstraintEnforcer` → PASS. Docs (`DesignDecisions.md`, `GatewayConfiguration.md`). Commit: `perf(audit): bounded async audit writer with batched inserts, one-time bootstrap, retention sweep`
---
### Task 9: Parallel session teardown in sweep and shutdown
**Classification:** high-risk (lifecycle concurrency)
**Estimated implement time:** ~4 min
**Parallelizable with:** Tasks 10, 11, 12, 13, 14 (edits only `SessionManager.cs` + docs; run any time)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs` (`CloseExpiredLeasesAsync` ~256296, `ShutdownAsync` ~301329)
- Modify: `docs/Sessions.md` (teardown parallelism note)
- Test: extend `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/` session-manager tests
**Why:** both loops `await CloseSessionCoreAsync` strictly sequentially, each bounded by the 10 s worker-shutdown timeout — a mass expiry with hung workers stalls slot reclamation, and 50-session shutdown exceeds any host stop-timeout.
**Step 1 (failing test):** two sessions whose fake worker shutdowns each take T complete a sweep in ~T, not ~2T (the fake harness supports delayed shutdown; if not, add a delay knob to the fake).
**Step 2 (implement):** wrap both loops in `Parallel.ForEachAsync` with `MaxDegreeOfParallelism = 4` (named constant, comment why: bounded so a mass expiry can't stampede worker teardown). `TryBeginCloseIfExpired` already makes per-session close idempotent/exclusive — state that in a comment; that's the invariant making this safe. Preserve the existing sweep precedence (lease-expiry → faulted → detach-grace) by keeping the *selection* phase sequential and parallelizing only the close calls on the selected set.
**Step 3:** session-manager filter → PASS. Docs. Commit: `perf(sessions): bounded-parallel teardown in lease sweep and shutdown`
---
### Task 10: Dashboard live-data subscription cap
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Tasks 19, 11, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardLiveDataService.cs` (~6170, `_subscribed`)
- Modify: `docs/GatewayDashboardDesign.md`
- Test: extend existing live-data tests under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/`
**Why:** every tag any viewer ever inspected stays advised on the shared worker session forever.
**Step 1 (failing test):** subscribing tag #257 when the cap is 256 unsubscribes the least-recently-read tag first (assert the fake session sees an `UnsubscribeBulk`/equivalent for the evicted tag).
**Step 2 (implement):** replace `_subscribed` (set) with an LRU: `Dictionary<string, LinkedListNode<string>>` + `LinkedList<string>` under the existing `_gate` (already serialized — no new locking). Cap at 256 (named constant; comment the sizing rationale: one browse page of tags plus headroom). On read of an already-subscribed tag, move to front. On insert past cap, evict from the back and call the session's unsubscribe for the evicted batch. On `InvalidateSession`, clear both structures (existing behavior).
**Step 3:** dashboard filter → PASS. Docs. Commit: `perf(dashboard): LRU cap on the shared live-read session's advised set`
---
### Task 11: Alarm monitor — cached `CurrentAlarms` projection
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Tasks 110, 12, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs` (~9099 + every mutation site under `_sync`)
- Test: extend `src/ZB.MOM.WW.MxGateway.Tests/Alarms/` monitor tests
**Why:** `CurrentAlarms` clones the full alarm set under the broadcast lock on every call.
**Step 1 (failing test):** two consecutive `CurrentAlarms` calls with no intervening transition return the same cached array instance; a transition invalidates it.
**Step 2 (implement):** add `private IReadOnlyList<ActiveAlarmSnapshot>? _currentAlarmsCache;``CurrentAlarms` builds it (still cloning, still under `_sync`) only when null; every mutation path that touches the alarm dictionary (`ApplyTransition`, reconcile apply, clear) nulls it under `_sync`. Callers already treat the result as read-only.
**Step 3:** alarms filter → PASS. Commit: `perf(alarms): memoize CurrentAlarms projection, invalidate on mutation`
---
### Task 12: Request-logging middleware — hoisted logger, bearer redaction fix
**Classification:** small (contains a security fix)
**Estimated implement time:** ~4 min
**Parallelizable with:** Tasks 111, 13, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/GatewayRequestLoggingMiddlewareExtensions.cs` (~2938)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/GatewayLogRedactor.cs` (~5477)
- Test: extend `src/ZB.MOM.WW.MxGateway.Tests/Diagnostics/` redactor tests
**Why:** `CreateLogger` (factory lock + DI resolve) per request; and — the security half — `RedactClientIdentity` passes any bearer credential that doesn't contain `mxgw_` through **unredacted** into log scope, violating the "never log secrets" convention.
**Step 1 (failing test):** `RedactClientIdentity("Bearer eyJhbGciOi...")` (a non-mxgw token) returns a redacted form (e.g. `Bearer [redacted]`), never the raw token. Keep the existing mxgw-shaped redaction (`mxgw_<id>_***`) intact — those tests must still pass.
**Step 2 (implement):**
- Redactor: any `authorization`-style value that is not recognized as an mxgw key redacts to a fixed `"[redacted]"` (preserve scheme word only). This is fail-closed.
- Middleware: resolve the `ILogger` once outside the per-request lambda (category-keyed, not request-keyed) via the app's `ILoggerFactory` at `Use...` registration time; keep the scope construction as-is (it carries per-request fields the log pipeline consumes — do not conditionalize it on log level in this task; note as considered-and-skipped since scope consumers may be added at runtime).
**Step 3:** diagnostics filter → PASS. Commit: `fix(logging): fail-closed bearer redaction; hoist per-request logger creation`
---
### Task 13: Auth-path hygiene — span token parse, limiter partition keys
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Tasks 112, 14
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs` (~153 `TryResolveKeyId`)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/CachingApiKeyVerifier.cs` (~229 `TryParseKeyId`)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/ApiKeyFailureLimiter.cs` (~244269, ~398)
- Test: existing auth tests under `src/ZB.MOM.WW.MxGateway.Tests/Security/` must stay green; add parse-equivalence cases
**Step 1 (failing test):** parse-equivalence table test: for a set of tokens (well-formed, missing `_`, empty, extra `_`), the new span parser returns exactly what `Split('_')` logic returned.
**Step 2 (implement):** replace `Split('_')` in both parsers with `IndexOf('_')` twice over a `ReadOnlySpan<char>`/string (no arrays, no substrings until the final key-id slice). In the limiter, compute the composite partition key once per RPC and pass it to both `Check` and `Reset` (or add an overload taking the precomputed key) instead of concatenating twice.
**Step 3:** security filter → PASS. Commit: `perf(auth): allocation-free token parsing; single partition-key build per RPC`
---
### Task 14: Bulk constraint loops, caches, and per-call hygiene
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 113 (NOT Task 6 if the mapper edit collides — it doesn't; different files)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGatewayService.cs` (bulk loops ~466troughs at 494/551/612/680; double session resolve ~104/126)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/ConstraintEnforcer.cs` (~214215 LINQ; expose `HasReadConstraints`/`HasWriteConstraints` if not present)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/GatewayApiKeyIdentityMapper.cs` (~3942 cache cliff)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SparseArrayExpander.cs` (~123126 capacity hints)
- Test: existing constraint/service tests under `src/ZB.MOM.WW.MxGateway.Tests/` + one new eviction test
**Step 1 (failing test):** constraint-blob cache: inserting entry `MaxCachedConstraintBlobs + 1` evicts the oldest instead of refusing to cache (FIFO like `GalaxyGlobMatcher` — copy its idiom).
**Step 2 (implement):**
- Bulk loops: hoist a single `identity has no read/write constraints` check before each per-item loop → unconstrained keys take an O(1) fast path (no per-item async interface dispatch, no denial bookkeeping allocation).
- Glob matching: replace the two `.Any(lambda)` calls with `for` loops over the glob lists.
- Denied-path double clone: build the filtered command directly (new message, copy allowed entries in) instead of `command.Clone()` then clear-and-refill; `MapCommand`'s own clone stays (that one is the load-bearing no-aliasing copy).
- Session double-resolve: add/`use` a `SessionManager` overload accepting the already-resolved `GatewaySession` (or have the service pass the session it resolved); keep the not-found exception behavior identical.
- `SparseArrayExpander`: set `RepeatedField.Capacity = length` (per element type) before the fill loops.
**Step 3:** run `FullyQualifiedName~ConstraintEnforcer`, `~MxAccessGatewayService`, `~SparseArray` filters → PASS. Commit: `perf(grpc): O(1) unconstrained bulk fast path, direct filtered-command build, cache eviction, capacity hints`
---
### Task 15: Phase A gate — full gateway suite
**Classification:** trivial (verification only)
**Estimated implement time:** ~5 min wall (suite runtime)
**Parallelizable with:** none (runs after Tasks 114)
Run, in order:
```bash
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```
Expected: 0 build errors, full suite green, clean process exit (0 surviving testhost). Fix anything red before Phase B. Commit only if fixes were needed.
---
# Phase B — Worker (.NET Framework 4.8; verified on windev in Task 24)
> Phase B implementers: you cannot compile. Be conservative — minimal diffs, match file idioms, net48 BCL only. Every task here lands as an unverified commit that Task 24 builds and tests remotely; keep commits clean so a failure bisects trivially.
### Task 16: Event drain loop — wake signal instead of 25 ms poll
**Classification:** high-risk (event path liveness)
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 18, 19, 21, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs` (add wake handle; `Enqueue` sets it)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs` (~18 `EventDrainInterval`, ~345372 drain loop)
- Modify: `docs/MxAccessWorkerInstanceDesign.md` (drain-loop paragraph ~381)
- Test: extend `src/ZB.MOM.WW.MxGateway.Worker.Tests/MxAccess/` event-queue tests + `Ipc/` pipe-session tests (they run on windev)
**Why:** the drain loop polls at 25 ms with no wake from `Enqueue` — a 25 ms latency floor on every burst from idle, 40 wakeups/s per idle worker, and less burst absorption before the 10k queue faults the session.
**Implement:**
- `MxAccessEventQueue`: add a `SemaphoreSlim _signal = new(0, 1)` (or an `AsyncAutoResetEvent`-shaped helper if the codebase has one — check first). `Enqueue` releases it (cap at 1, swallow `SemaphoreFullException`). Expose `Task WaitForEventsAsync(TimeSpan timeout, CancellationToken ct)`.
- Drain loop: when a drain returns empty, `await queue.WaitForEventsAsync(EventDrainInterval, ct)` instead of `Task.Delay` — the 25 ms becomes a *fallback* ceiling, not the floor; a signaled wait returns immediately. Loop structure otherwise unchanged (fault handling, batch size).
- Doc paragraph: drain is signal-driven with a 25 ms fallback tick.
- Tests: enqueue-after-idle results in a drain without waiting for the fallback interval (windev-run; write it now).
Commit: `perf(worker): signal-driven event drain — removes the 25 ms latency floor and idle wakeups`
---
### Task 17: Event queue capacity — launcher-configurable
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 18, 19, 21, 22, 23 (NOT Task 16 — both edit `MxAccessEventQueue.cs`/`WorkerPipeSession.cs`; run after 16)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs` (+`EventQueueCapacity`, default 10000) and `GatewayOptionsValidator.cs` (≥1000, ≤1_000_000)
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs` (new env var, mirror the `WorkerWriteCompletionWaitEnvironmentVariableName` pattern at ~2529 and ~186187 exactly)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerOptionsParser.cs` / `WorkerOptions.cs` / `EnvironmentVariableWorkerEnvironment.cs` (read it, following the write-completion variable's path)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs` (~52: pass capacity to `new MxAccessEventQueue(...)`)
- Modify: `docs/GatewayConfiguration.md` (+`MxGateway:Worker:EventQueueCapacity`), `docs/MxAccessWorkerInstanceDesign.md` (capacity paragraph ~371)
- Test: gateway side — validator test + launcher env-var test (these run locally); worker side — parser test (windev)
**Why:** the 10,000 default is headroom-critical (overflow faults the session) but not configurable without a rebuild.
**Implement:** copy the `WriteCompletionWaitMilliseconds` plumbing end to end under a new name (`MXGW_EVENT_QUEUE_CAPACITY` shaped like the existing variable's naming). Absent/invalid env value → default 10000 (never crash the worker on a bad value; log and default).
> **As-built note (1358332):** shipped as silent default without logging, matching the alarm-resolver precedent — no `ILogger` is reachable from the static resolve site without new plumbing; the silent fallback is disclosed in `GatewayConfiguration.md`. The Bootstrap parser files listed above were correctly NOT touched — the established env-var pattern reads `Environment.GetEnvironmentVariable` at the resolve site.
Note the gateway-side files here don't overlap Phase A tasks — safe after Task 15.
Commit: `perf(worker): launcher-configurable event queue capacity`
---
### Task 18: STA completion waits — message-driven, not sleep-polled
**Classification:** high-risk (STA/pump semantics)
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 16, 17, 19, 21, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessWriteCompletionCache.cs` (~97118 wait loop)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessValueCache.cs` (~135150 wait loop)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Sta/StaMessagePump.cs` (if it doesn't already expose a bounded "pump until signaled or timeout" primitive)
- Modify: `docs/MxAccessWorkerInstanceDesign.md`
- Test: extend `src/ZB.MOM.WW.MxGateway.Worker.Tests/Sta/` + `MxAccess/` cache tests (windev)
**Why:** both waits run `pumpStep(); ...; Thread.Sleep(5)` on the STA — during each 5 ms sleep no messages pump, so COM event dispatch stalls in 5 ms bites for up to 1.5 s (writes) / 1 s per tag (ReadBulk).
**Implement:**
- Add a wake to both caches: the update path (`OnWriteComplete` recording a completion / `OnDataChange` recording a value) signals a Win32 auto-reset event (`AutoResetEvent` is fine — it wraps one).
- Replace `Thread.Sleep(pollIntervalMs)` with a pump-integrated wait: `MsgWaitForMultipleObjectsEx(1, [waitHandle], remainingMs-capped-at-50, QS_ALLINPUT, MWMO_INPUTAVAILABLE)`; on `WAIT_OBJECT_0 + 1` (message arrived) run `pumpStep()` and re-check; on `WAIT_OBJECT_0` (signaled) re-check the entry immediately. The existing `StaMessagePump`/`StaRuntime` already use exactly this Win32 pattern (~`StaRuntime.cs:255261`) — reuse/extract their P/Invoke declarations, do not duplicate.
- **Semantics unchanged:** timeouts, deadline math, return values, and the unconfirmed-empty-statuses reply shape stay byte-identical. Only the *waiting mechanism* changes: latency to observe a completion drops from ≤5 ms granularity to immediate, and the pump keeps running throughout the wait.
- **Do not** change the plain-`Write` completion-wait default in this task. The 1.5 s default is a documented OtOpcUa contract (`MxGateway:Worker:WriteCompletionWaitMilliseconds` is already configurable). Leave a doc note that operators with pure fire-and-forget write workloads can lower it.
Commit: `perf(worker): message-driven completion waits — the STA pumps continuously while waiting`
---
### Task 19: Handle registry — reverse index, cached views, O(1) removals
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 16, 17, 18, 21, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessHandleRegistry.cs` (`ItemHandles`/`ServerHandles`/`AdviceHandles` properties ~1426; `RemoveAdviceHandles` ~137148; `UnregisterServerHandle` ~4665)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessSession.cs` (`TryGetCachedReadFor` ~9881000)
- Test: extend `src/ZB.MOM.WW.MxGateway.Worker.Tests/MxAccess/` registry tests (windev)
**Why:** the sorted list properties re-sort and copy the whole table on **every access**, `TryGetCachedReadFor` reads `ItemHandles` once per ReadBulk tag (O(tags × items·log items)), and advice/server removals do full LINQ scans (O(n²) bulk teardown).
**Implement:**
- Reverse index: `Dictionary<long, Dictionary<string, int>>` server→(tagAddress→itemHandle) — or flat `Dictionary<(int,int-packed + tag)>` — maintained on register/unregister. `TryGetCachedReadFor` becomes two dictionary probes (the file's own comment already asks for this).
- Cached materialization: memoize each sorted array with a version stamp bumped on any mutation; property returns the cached array when the version matches. Registry is STA-confined (verify: no locking in the file today ⇒ single-threaded by contract — state it in a comment), so no locking needed.
- Removals: secondary index advice-by-item (`Dictionary<long, List<advice>>` keyed on the packed `(serverHandle, itemHandle)` the item table already uses) so `RemoveAdviceHandles`/`UnregisterServerHandle` stop scanning.
Commit: `perf(worker): reverse tag index + memoized views + indexed removals in the handle registry`
---
### Task 20: Event conversion — exact-format timestamps, compiled status accessors
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 16, 17, 18, 19, 21, 22, 23 (different files)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventMapper.cs` (~360377 timestamp parse)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Conversion/MxStatusProxyConverter.cs` (~96109 reflection reads)
- Test: extend `src/ZB.MOM.WW.MxGateway.Worker.Tests/Conversion/` (windev) — these files have solid existing tests; add exact-format cases
**Implement:**
- Timestamps: try `DateTime.TryParseExact` against a small cached array of the observed MXAccess formats (`M/d/yyyy h:mm:ss.fff tt` and its zero-padded/24 h siblings — derive the list from the existing tests' fixture strings) **first**, falling back to the existing two-stage `TryParse` chain so behavior never regresses on an unexpected locale. Order: exact formats → current-culture → invariant (today's chain).
- Status fields: replace the per-read `field.GetValue` with delegates compiled once per field via `Expression.Lambda<Func<object, T>>` (net48-safe) cached alongside the existing `FieldInfo` cache. Same values out, no boxing per event.
Commit: `perf(worker): exact-format timestamp parse and compiled status-field accessors on the event path`
---
### Task 21: Event queue drain — size memoized at enqueue
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Tasks 18, 19, 20, 22, 23 (NOT 16/17 — same file; run after them)
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs` (~249269 byte-budgeted `Drain`; enqueue path ~152170)
- Test: extend the windev event-queue tests: budget math unchanged for a mixed-size batch
**Why:** `Drain(maxEvents, maxTotalBytes)` calls `CalculateSize()` per event **inside** the queue lock the STA needs to enqueue — a large drain stalls COM callbacks.
**Implement:** compute `CalculateSize()` once at enqueue time (outside any lock — the caller owns the event exclusively there) and store it on the queue's node/wrapper alongside the event; `Drain` uses the memoized size. The WRK-21 never-strand-the-head guarantee is untouched (same comparisons, precomputed operand). Events are never mutated after enqueue (WRK-11 no-clone contract) so the memoized size cannot go stale — say so in a comment.
Commit: `perf(worker): memoize event frame size at enqueue; drain stops sizing under the STA's lock`
---
### Task 22: Worker frame writer/reader — pooled buffers
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Tasks 16, 17, 18, 19, 20, 21, 23
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs` (~467 per-frame `new byte[]`)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameReader.cs` (~33 per-frame prefix buffer)
- Test: windev `Ipc/` frame tests must stay green (they're thorough — rely on them)
**Why:** the worker side allocates a fresh frame buffer + prefix buffer per frame while the gateway side already pools (`ArrayPool`, GWC-30) — the fix was applied on one side only. `System.Buffers` is already referenced by the worker (its reader uses `ArrayPool.Shared`).
**Implement:** mirror the gateway codec: rent the frame buffer from `ArrayPool<byte>.Shared`, write prefix+payload into it, return in a `finally`; hoist the 4-byte prefix buffer to an instance field on the reader (single-reader by contract — copy the gateway reader's comment). Exact same wire bytes.
Commit: `perf(worker): pooled frame buffers — brings the net48 codec up to the gateway side's GWC-30 pattern`
---
### Task 23: Alarm consumer — cheap parse, truncation detection, configurable cadence
**Classification:** high-risk (alarm correctness)
**Estimated implement time:** ~5 min (split 23a parse / 23b truncation+config if long)
**Parallelizable with:** Tasks 16, 17, 19, 20, 21, 22
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs` (~402437 parse; ~50 `DefaultMaxAlarmsPerFetch`; ~323330 snapshot rebuild; `ComputeTransitions` absence rule ~356)
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs` (~22 hard-coded 500 ms)
- Modify: `docs/GatewayConfiguration.md`, `docs/DesignDecisions.md` (alarm sections)
- Test: extend windev `MxAccess/` alarm-consumer tests — the truncation test is the important one
**Implement (three independent sub-changes):**
1. **Parse cost:** in the per-alarm extraction, replace the ~14 `SelectSingleNode(child)` XPath calls with one pass over `alarmNode.ChildNodes` switching on `Name` (same fields, same defaults for absent children). Keep `XmlDocument` (an `XmlReader` rewrite is a bigger change than the win justifies once XPath is gone). Reuse the snapshot dictionary across polls (clear-and-refill → swap two dictionaries) only if trivially safe; otherwise skip — the XPath removal is the payload.
2. **Truncation cliff (correctness fix):** when the fetch returns exactly `maxAlarmsPerFetch` records, treat the snapshot as **truncated**: log a warning (rate-limited, identifiers only) and suppress the absence-implies-Clear inference in `ComputeTransitions` for that poll (present alarms still update; nothing is cleared on the evidence of a capped fetch). Add the test: 1024-record fetch + a known alarm missing from it → no Clear transition emitted, warning logged.
3. **Cadence + cap configurable:** plumb `MxGateway:Alarms:PollIntervalMilliseconds` (default 500, min 100) and `MaxAlarmsPerFetch` (default 1024) through the existing env-var pattern (as in Task 17). Gateway-side option + validator + launcher env, worker-side parse.
Commit: `fix(alarms): truncation-safe transitions; perf: single-pass alarm parse; configurable poll cadence`
---
### Task 24: Phase B verification on windev (psbridge)
**Classification:** high-risk (this is the gate for every Phase B commit)
**Estimated implement time:** ~10 min wall
**Parallelizable with:** none (after all Phase B tasks)
**Steps:**
1. Invoke the `psbridge` skill and follow it (it covers exec/push/deploy against the Windows box).
2. Push/pull the branch to windev (whatever the skill's established flow is — the repo has a remote the Windows box shares; `git pull` the branch there).
3. On windev, run in order and capture output:
```powershell
dotnet build src/ZB.MOM.WW.MxGateway.slnx
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```
4. Any failure: fix on the Mac, commit, re-run the failed leg. Bisect by commit if the failure isn't obvious — Phase B commits are deliberately one-task-each.
5. If psbridge is unreachable: STOP and report — Phase B remains "edited, unverified"; do not merge.
Live MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, `WorkerLiveMxAccessSmokeTests`) if provider state is available on windev; otherwise record why skipped, per `docs/GatewayTesting.md`.
---
### Task 25: Wrap-up — docs sweep, umbrella index, review deltas
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (last)
**Files:**
- Verify each task's doc edits landed (`gateway.md`, `docs/Sessions.md`, `docs/GatewayConfiguration.md`, `docs/GatewayDashboardDesign.md`, `docs/DesignDecisions.md`, `docs/MxAccessWorkerInstanceDesign.md`, `docs/WorkerFrameProtocol.md`)
- Modify: `../scadaproj/CLAUDE.md`**only if** a fact the umbrella index records changed (new `MxGateway:Worker:EventQueueCapacity` / alarm options are config, not indexed facts; expected outcome: no umbrella change needed — verify, don't assume)
- Check: no `.proto` diffs (`git diff main -- '*.proto'` must be empty)
Commit anything found: `docs: remediation plan doc sweep`
---
## Explicitly deferred (decided, not forgotten)
| Finding | Why deferred |
|---|---|
| Value-cache triple clone per `OnDataChange` | Removing the defensive copies needs a GWC-07-style aliasing audit across cache consumers; risk outweighs the win until profiled. |
| net48 pipe-read cancellation | Benign in practice (worker exits after shutdown); a correct fix means restructuring stream teardown for a path that only fires at exit. |
| Control-frame completion coupled to event batch drain | Documented, bounded (≤128 frames) behavior of the two-class writer design; revisit only if heartbeat latency shows up in metrics. |
| Blazor pages' loopback SignalR hop | Works correctly; in-process `WatchSnapshotsAsync` consumption is a dashboard refactor with payoff only at viewer counts the product doesn't target. |
| Event-path triple async-iterator flattening | LOW-rated; touches the most invariant-dense code in the gateway for two `MoveNextAsync` hops per event. Reconsider after Tasks 3/4 land and if profiling still shows it. |
| `SessionEventDistributor._subscribers` `ConcurrentDictionary` → plain `Dictionary` (Task 25 / Task 3 review) | Every mutation is already inside `_lifecycleLock`, so the concurrent type buys nothing. Behavior-neutral refactor with no measurable win, proposed after the windev gate was already green — not worth re-running the verification matrix for. Comment cleanups from the same review landed; this swap did not. |
## Execution notes for the orchestrator
- Branch: `git checkout -b perf/review-remediation` before Task 1.
- Implementer subagents run on **Opus** per the user's instruction; reviewer chain per each task's Classification.
- Parallel dispatch waves (no file overlap): **Wave 1:** 1, 3, 4, 5, 6, 8 · **Wave 2:** 2, 9, 10, 11, 12, 13, 14 · then 7 (after 2) · then 15 · **Wave 3 (Phase B):** 16, 18, 19, 20, 22, 23 · then 17, 21 (after 16) · then 24 · then 25. (Waves are a suggestion; the per-task `Parallelizable with` fields are the contract.)
- Each implementer gets: its full task text, the ground rules block, and nothing else — the `Files:` block is the scope contract.
@@ -0,0 +1,188 @@
{
"planPath": "docs/plans/2026-08-15-perf-review-remediation.md",
"tasks": [
{
"id": 1,
"subject": "Task 1: Named-pipe buffer sizes",
"status": "completed"
},
{
"id": 2,
"subject": "Task 2: Metrics pull-gauge + Interlocked counters",
"status": "completed"
},
{
"id": 3,
"subject": "Task 3: Distributor copy-on-write subscriber snapshot",
"status": "completed"
},
{
"id": 4,
"subject": "Task 4: Dashboard event mirror viewer gating",
"status": "completed"
},
{
"id": 5,
"subject": "Task 5: Snapshot pipeline idle gating + cached config",
"status": "completed"
},
{
"id": 6,
"subject": "Task 6: Reply ownership transfer in MapCommandReply",
"status": "completed"
},
{
"id": 7,
"subject": "Task 7: WorkerClient WaitAsync timeout + WorkerCancel",
"status": "completed",
"blockedBy": [
2
]
},
{
"id": 8,
"subject": "Task 8: Audit pipeline background writer + retention",
"status": "completed"
},
{
"id": 9,
"subject": "Task 9: Parallel session teardown",
"status": "completed"
},
{
"id": 10,
"subject": "Task 10: Dashboard live-data subscription cap",
"status": "completed"
},
{
"id": 11,
"subject": "Task 11: Alarm monitor cached CurrentAlarms",
"status": "completed"
},
{
"id": 12,
"subject": "Task 12: Logging middleware hoist + bearer redaction fix",
"status": "completed"
},
{
"id": 13,
"subject": "Task 13: Auth-path span parsing + limiter keys",
"status": "completed"
},
{
"id": 14,
"subject": "Task 14: Bulk constraint loops, caches, hygiene",
"status": "completed"
},
{
"id": 15,
"subject": "Task 15: Phase A gate \u2014 full gateway suite",
"status": "completed",
"blockedBy": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14
]
},
{
"id": 16,
"subject": "Task 16: Event drain wake signal",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 17,
"subject": "Task 17: Event queue capacity env plumbing",
"status": "completed",
"blockedBy": [
16
]
},
{
"id": 18,
"subject": "Task 18: STA message-driven completion waits",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 19,
"subject": "Task 19: Handle registry reverse index + O(1) removals",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 20,
"subject": "Task 20: Event conversion TryParseExact + compiled accessors",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 21,
"subject": "Task 21: Drain size memoized at enqueue",
"status": "completed",
"blockedBy": [
16,
17
]
},
{
"id": 22,
"subject": "Task 22: Worker frame writer/reader pooled buffers",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 23,
"subject": "Task 23: Alarm consumer parse + truncation + cadence",
"status": "completed",
"blockedBy": [
15
]
},
{
"id": 24,
"subject": "Task 24: Phase B verification on windev (psbridge)",
"status": "completed",
"blockedBy": [
16,
17,
18,
19,
20,
21,
22,
23
]
},
{
"id": 25,
"subject": "Task 25: Wrap-up docs sweep + follow-ups",
"status": "completed",
"blockedBy": [
24
]
}
],
"lastUpdated": "2026-08-15T18:55:00Z"
}
+50 -3
View File
@@ -20,6 +20,20 @@ failure text was stamped as the source revision. The observed form is:
0.1.2+fatal: cannot change to ...
```
The full string recovered from wonder's 2026-08-09 server binary (read 2026-08-12) shows the whole
failure, including the mismatched `'``"` that caused it:
```
0.1.2+fatal: cannot change to 'C:\build\mxgw-deploy\src" rev-parse --short HEAD': Invalid argument
```
**Do not read the leading `0.1.2` as provenance.** It is the static base `<Version>` every build
carries, not a truncated SHA. The hazard is a false positive rather than a blank: `0.1.2+fatal:…`
reads like a version that succeeded and then picked up noise, when in fact there is no usable
identity anywhere in the string. For a binary built in this window the commit is **not recoverable
from the binary at all** — so finding nothing is the expected result, not evidence against a SHA
established another way.
`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
a short-SHA shape so no future git failure text can become the revision either.
@@ -48,13 +62,28 @@ In rough order of cost:
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
stamped. This is what settled the 2026-08-11 investigation.
stamped. **Do not read this as *the* technique on its own.** What settled the 2026-08-11
investigation was two independent derivations agreeing: a PDB source-hash match, and a
contemporaneous deploy record written the same evening that named the same two commits. The
convergence is the result's strength, not either method alone — a hash match tells you which
sources a binary was built from, but not that the build was intentional or which host it went to.
A reader with only one of the two available should weight it accordingly and look for a second
line of evidence.
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
directories carry operator-chosen labels naming the work (for example
`Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
accidental or off-book deploy tends not to follow them.
4. **The host's own backup directories, read as a chain.** Each `Server.bak.<timestamp>` holds the
exe that deploy *replaced*, so a sweep of `VersionInfo` across them reconstructs the host's deploy
history from the host itself, with no repo access and no deploy record. A backup stamped
`20260811T060739` containing an exe written 2026-08-09 is the 08-09 build being displaced — the
backup's timestamp dates the *next* deploy, not the build inside it. Reading a file's version is
non-destructive, unlike opening a SQLite store in a backup directory, which mutates it. This is
what established that wonder's `b948e69` and `0a9715d` were two deploys two days apart rather
than two competing claims about one binary.
Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
the server and a later one for the worker.
@@ -65,11 +94,29 @@ the server and a later one for the worker.
|---|---|---|---|
| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
| 2026-08-11 | `wonder-app-vd03` | `0a9715d` (this deploy wrote `Server.bak.20260811T060739`, holding the displaced 08-09 build) | *carried forward* |
| 2026-08-12 | `wonder-app-vd03` | `55f2889` (this deploy wrote `Server.bak.20260812T040122`, holding `0a9715d`) | *carried forward* |
The two wonder rows after 08-09 are **server swaps**; their worker cells are carried forward from the
08-09 entry rather than re-verified, so treat the worker SHA there as unconfirmed. Their server SHAs
come from the backup-chain read described above (technique 4), except `55f2889`, which was read
directly from the live exe's stamp — trustworthy because it postdates `0152180`.
`b948e69` is **confirmed by PDB source-hash match plus the contemporaneous record, never by a version
stamp** — that build falls in the broken-stamp window and its stamp is structurally unavailable (see
the first section). `0a9715d` is the first wonder build to stamp cleanly, since `0152180` landed
before it.
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that
day are the second swap's fingerprint, not redundancy.
the latter two minutes after `53f69cd` merged at 19:48, with a matching service stop/start at
19:50:29/34.
That reading is confirmable from artifacts still on disk, without trusting the narrative: windev
carries **two** worker backup directories from that day (`Worker.bak-20260809` and
`Worker.bak-20260809-planwrites`), and wonder carries `Worker.bak.20260809-planwrites`. Two backups
because there were two worker operations. This is easy to misread as redundancy — it is the second
swap's fingerprint.
## The 2026-08-09 deploy returned the worker to mainline
@@ -22,8 +22,8 @@
(IntegrationTests-028).
-->
<ItemGroup>
<PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.2.1" />
<PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.2.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="10.0.7" />
</ItemGroup>
@@ -34,6 +34,13 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
private readonly Dictionary<string, ActiveAlarmSnapshot> _alarms = new(StringComparer.Ordinal);
private readonly List<Subscriber> _subscribers = [];
// Memoized CurrentAlarms projection, guarded by _sync: the cloned, read-only view of _alarms
// handed to the dashboard and the QueryActiveAlarms RPC. Cloning the whole set per read held
// _sync — the broadcast lock — for the length of the copy, so a polled dashboard stalled every
// ApplyTransition/Broadcast behind it. Null means "not built for the current generation":
// every path that writes _alarms must null this under _sync, or readers keep a stale set.
private ActiveAlarmSnapshot[]? _currentAlarmsProjection;
// NEXT-03 dedup tombstones, guarded by _sync: alarm instances whose Clear was synthesized by
// the most recent reconcile pass, keyed by reference with the instance's original raise
// timestamp as the identity marker. A buffered live Clear for the same instance is a duplicate
@@ -93,7 +100,12 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
lock (_sync)
{
return _alarms.Values.Select(alarm => alarm.Clone()).ToArray();
// Same clone semantics as an uncached read — callers still get instances no
// mutation can leak back into the cache — but built once per alarm-set
// generation instead of once per caller.
return _currentAlarmsProjection ??= _alarms.Values
.Select(alarm => alarm.Clone())
.ToArray();
}
}
}
@@ -422,6 +434,11 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
if (transition.TransitionKind == AlarmTransitionKind.Clear)
{
bool wasKnown = _alarms.Remove(reference);
if (wasKnown)
{
_currentAlarmsProjection = null;
}
if (!wasKnown && IsDuplicateOfReconcileClear(reference, transition))
{
return;
@@ -433,6 +450,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
bool duplicate = _alarms.TryGetValue(reference, out ActiveAlarmSnapshot? existing)
&& IsDuplicateOfCachedState(existing, snapshot);
_alarms[reference] = snapshot;
_currentAlarmsProjection = null;
if (duplicate)
{
return;
@@ -650,6 +668,8 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
_alarms[incoming.Key] = incoming.Value;
}
_currentAlarmsProjection = null;
}
}
@@ -696,6 +716,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
lock (_sync)
{
_alarms.Clear();
_currentAlarmsProjection = null;
}
}
@@ -46,6 +46,33 @@ public sealed class AlarmsOptions
/// </summary>
public int ReconcileIntervalSeconds { get; init; } = 30;
/// <summary>
/// Cadence at which the worker's STA polls the AVEVA alarm consumer
/// (<c>GetXmlCurrentAlarms2</c>) for the current active-alarm snapshot.
/// Default 500 ms; must be between 100 ms and 3,600,000 ms (one hour).
/// Every poll is a COM call plus an XML parse on the STA that also
/// serves reads and writes, so driving it below 100 ms starves the
/// command path; above an hour the cadence stops being a cadence and
/// silently disables alarm polling. Conveyed to the worker through the
/// <c>MXGATEWAY_ALARM_POLL_INTERVAL_MS</c> environment variable.
/// </summary>
public int PollIntervalMilliseconds { get; init; } = 500;
/// <summary>
/// Cap the worker passes to <c>GetXmlCurrentAlarms2</c>'s
/// <c>maxAlmCnt</c> argument. Default 1024; must be between 64 and
/// 65,536 — the worker is a 32-bit process that materializes each
/// fetch as one BSTR plus a full XmlDocument, so an unbounded cap
/// faults the STA rather than merely slowing it. A fetch that comes
/// back holding exactly this many records is treated as truncated: the
/// worker keeps the alarms the capped fetch could not mention in its
/// snapshot rather than letting their absence read as a clear. Raise it
/// on galaxies whose steady-state active-alarm count approaches the
/// cap. Conveyed to the worker through the
/// <c>MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH</c> environment variable.
/// </summary>
public int MaxAlarmsPerFetch { get; init; } = 1024;
/// <summary>
/// Configuration for the alarm-manager ↔ subtag fallback mechanism:
/// operating mode, failure-detection thresholds, discovery, and subtag
@@ -11,4 +11,5 @@ public sealed record EffectiveLdapConfiguration(
string ServiceAccountPassword,
string UserNameAttribute,
string DisplayNameAttribute,
string GroupAttribute);
string GroupAttribute,
IReadOnlyList<string> FallbackServers);
@@ -13,6 +13,33 @@ namespace ZB.MOM.WW.MxGateway.Server.Configuration;
/// </summary>
public sealed class GalaxyRepositoryOptionsValidator : OptionsValidatorBase<GalaxyRepositoryOptions>
{
// See GatewayOptionsValidator for why this is nullable and what null means.
private readonly string? _contentRootPath;
/// <summary>
/// Initializes a new instance of the <see cref="GalaxyRepositoryOptionsValidator"/> class for
/// the dependency-injection path, taking the content root from the host environment.
/// </summary>
/// <param name="environment">The host environment.</param>
public GalaxyRepositoryOptionsValidator(IHostEnvironment environment)
{
ArgumentNullException.ThrowIfNull(environment);
_contentRootPath = environment.ContentRootPath;
}
/// <summary>
/// Initializes a new instance of the <see cref="GalaxyRepositoryOptionsValidator"/> class for
/// unit tests and non-DI callers.
/// </summary>
/// <param name="contentRootPath">
/// Content root to test the snapshot path against; <see langword="null"/> leaves the
/// content-root rule inactive.
/// </param>
internal GalaxyRepositoryOptionsValidator(string? contentRootPath = null)
{
_contentRootPath = contentRootPath;
}
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GalaxyRepositoryOptions options)
{
@@ -37,5 +64,10 @@ public sealed class GalaxyRepositoryOptionsValidator : OptionsValidatorBase<Gala
options.SnapshotCachePath,
"MxGateway:Galaxy:SnapshotCachePath must be an absolute (rooted) path so the Galaxy snapshot never lands in the launch working directory.",
builder);
GatewayConfigPathRules.AddIfUnderContentRoot(
options.SnapshotCachePath,
_contentRootPath,
$"MxGateway:Galaxy:SnapshotCachePath must not be inside the application directory ({_contentRootPath}). The upgrade procedure renames that directory, so the cached snapshot is discarded on every deploy and the gateway starts cold.",
builder);
}
}
@@ -53,25 +53,102 @@ internal static class GatewayConfigPathRules
return;
}
try
{
_ = Path.GetFullPath(value);
}
catch (ArgumentException)
{
builder.Add(message);
}
catch (NotSupportedException)
{
builder.Add(message);
}
catch (PathTooLongException)
{
builder.Add(message);
}
catch (IOException)
if (!TryGetFullPath(value, out _))
{
builder.Add(message);
}
}
/// <summary>
/// Fails validation when <paramref name="value"/> resolves to a location inside
/// <paramref name="contentRoot"/> — the directory the application runs from.
/// </summary>
/// <remarks>
/// <para>
/// <b>Rooted is not the same as safe, and this is the rule that closes the gap.</b>
/// <see cref="AddIfNotRooted"/> stops a store drifting with the working directory, but an
/// absolute path <em>inside the app directory</em> passes it cleanly — and that is what failed
/// in production on 2026-08-09. The upgrade procedure renames the app directory to
/// <c>Server.bak.*</c> and unpacks a new one; a store living there is renamed away with it, the
/// process then creates a fresh empty one at the same path, and nothing reports an error. All
/// API keys were lost and no gRPC client could authenticate for two days. The deploy itself was
/// executed correctly — the binaries were the point of the rename, and the store was collateral.
/// </para>
/// <para>
/// The same shape catches the dev-side symptom: a store under the content root lands in the
/// source tree, which is how <c>mxgateway-secrets.db</c> once tripped the repository's
/// tree-hygiene test.
/// </para>
/// <para>
/// Comparison is case-insensitive only on Windows. On a case-insensitive macOS volume this can
/// miss a violation that differs only in case, which is a missed warning in dev; assuming
/// case-insensitivity on Linux would instead reject a legitimate path, and a false startup
/// abort is the worse failure.
/// </para>
/// </remarks>
/// <param name="value">The configured path value.</param>
/// <param name="contentRoot">The application content root to test against.</param>
/// <param name="message">The failure message to record when the value is under the content root.</param>
/// <param name="builder">The validation builder accumulating failures.</param>
public static void AddIfUnderContentRoot(
string? value,
string? contentRoot,
string message,
ValidationBuilder builder)
{
if (string.IsNullOrWhiteSpace(value) || string.IsNullOrWhiteSpace(contentRoot))
{
return;
}
// A malformed path is AddIfInvalidPath's message to report; staying silent here keeps one
// bad value from producing two failures that say different things about the same mistake.
if (!TryGetFullPath(value, out string fullValue) || !TryGetFullPath(contentRoot, out string fullRoot))
{
return;
}
fullRoot = fullRoot.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
StringComparison comparison = OperatingSystem.IsWindows()
? StringComparison.OrdinalIgnoreCase
: StringComparison.Ordinal;
// The separator is load-bearing: a bare prefix test would also match a sibling directory
// whose name merely starts with the root's ("/srv/app" against "/srv/app-data").
if (string.Equals(fullValue, fullRoot, comparison)
|| fullValue.StartsWith(fullRoot + Path.DirectorySeparatorChar, comparison))
{
builder.Add(message);
}
}
private static bool TryGetFullPath(string value, out string fullPath)
{
try
{
fullPath = Path.GetFullPath(value);
return true;
}
catch (ArgumentException)
{
fullPath = string.Empty;
return false;
}
catch (NotSupportedException)
{
fullPath = string.Empty;
return false;
}
catch (PathTooLongException)
{
fullPath = string.Empty;
return false;
}
catch (IOException)
{
fullPath = string.Empty;
return false;
}
}
}
@@ -30,7 +30,8 @@ public sealed class GatewayConfigurationProvider(IOptions<GatewayOptions> option
ServiceAccountPassword: RedactedValue,
UserNameAttribute: value.Ldap.UserNameAttribute,
DisplayNameAttribute: value.Ldap.DisplayNameAttribute,
GroupAttribute: value.Ldap.GroupAttribute),
GroupAttribute: value.Ldap.GroupAttribute,
FallbackServers: value.Ldap.FallbackServers),
Worker: new EffectiveWorkerConfiguration(
ExecutablePath: value.Worker.ExecutablePath,
WorkingDirectory: value.Worker.WorkingDirectory,
@@ -10,20 +10,33 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
private const int MinimumMaxMessageBytes = 1024;
private const int MaximumMaxMessageBytes = 256 * 1024 * 1024;
// Bounds on the worker's outbound event-queue capacity. The floor keeps enough headroom that a
// normal subscription burst cannot overflow the queue (an overflow faults the whole session);
// the ceiling keeps a mistyped value from committing the x86 worker to an unbounded backlog.
private const int MinimumWorkerEventQueueCapacity = 1000;
private const int MaximumWorkerEventQueueCapacity = 1_000_000;
// Whether the host is running in the Production environment. Drives the production-only
// hard-stops (dashboard login disabled, plaintext LDAP transport) that must abort startup
// rather than merely warn. Non-production hosts keep the permissive dev posture.
private readonly bool _isProduction;
// The application content root. Store paths must not live under it — see
// GatewayConfigPathRules.AddIfUnderContentRoot. Null for non-DI callers that supply no
// environment, which skips the rule rather than inventing a root to test against.
private readonly string? _contentRootPath;
/// <summary>
/// Initializes a new instance of the <see cref="GatewayOptionsValidator"/> class for the
/// dependency-injection path, deriving the production posture from the host environment.
/// dependency-injection path, deriving the production posture and content root from the host
/// environment.
/// </summary>
/// <param name="environment">The host environment.</param>
public GatewayOptionsValidator(IHostEnvironment environment)
{
ArgumentNullException.ThrowIfNull(environment);
_isProduction = environment.IsProduction();
_contentRootPath = environment.ContentRootPath;
}
/// <summary>
@@ -32,15 +45,20 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
/// hard-stops do not fire; pass <see langword="true"/> to exercise them.
/// </summary>
/// <param name="isProduction">Whether to treat the host as running in Production.</param>
internal GatewayOptionsValidator(bool isProduction = false)
/// <param name="contentRootPath">
/// Content root to test store paths against; <see langword="null"/> leaves the content-root
/// rule inactive, which is what a caller with no real host wants.
/// </param>
internal GatewayOptionsValidator(bool isProduction = false, string? contentRootPath = null)
{
_isProduction = isProduction;
_contentRootPath = contentRootPath;
}
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GatewayOptions options)
{
ValidateAuthentication(options.Authentication, builder);
ValidateAuthentication(options.Authentication, _contentRootPath, builder);
ValidateLdap(options.Ldap, builder, _isProduction);
ValidateWorker(options.Worker, builder);
ValidateSessions(options.Sessions, builder);
@@ -88,6 +106,13 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
"MxGateway:Security:ApiKeyFailureTrackedPeers must be greater than zero.",
builder);
// Retention must be at least one day: 0 would sweep the audit table on every pass, which
// is a way to silently disable auditing rather than an expression of intent.
AddIfNotPositive(
options.AuditRetentionDays,
"MxGateway:Security:AuditRetentionDays must be greater than zero (at least one day of audit history is retained).",
builder);
// The two-layer limiter knobs (SEC-31) accept 0 as "disable this layer": a zero aggregate
// limit turns off cross-peer counting, and a zero probe interval restores absolute blocking.
// Negatives express no intent.
@@ -101,7 +126,10 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
builder);
}
private static void ValidateAuthentication(AuthenticationOptions options, ValidationBuilder builder)
private static void ValidateAuthentication(
AuthenticationOptions options,
string? contentRootPath,
ValidationBuilder builder)
{
if (!Enum.IsDefined(options.Mode))
{
@@ -123,6 +151,11 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
options.SqlitePath,
"MxGateway:Authentication:SqlitePath must be an absolute (rooted) path so the credential store never lands in the launch working directory.",
builder);
AddIfUnderContentRoot(
options.SqlitePath,
contentRootPath,
$"MxGateway:Authentication:SqlitePath must not be inside the application directory ({contentRootPath}). The upgrade procedure renames that directory, which abandons the credential store and silently starts an empty one — every API key is lost and no client can authenticate.",
builder);
AddIfBlank(
options.PepperSecretName,
"MxGateway:Authentication:PepperSecretName is required when API-key authentication is enabled.",
@@ -248,6 +281,12 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
"MxGateway:Worker:HeartbeatGraceSeconds must be greater than or equal to HeartbeatIntervalSeconds.");
}
if (options.EventQueueCapacity is < MinimumWorkerEventQueueCapacity or > MaximumWorkerEventQueueCapacity)
{
builder.Add(
$"MxGateway:Worker:EventQueueCapacity must be between {MinimumWorkerEventQueueCapacity} and {MaximumWorkerEventQueueCapacity}.");
}
if (options.MaxMessageBytes is < MinimumMaxMessageBytes or > MaximumMaxMessageBytes)
{
builder.Add(
@@ -387,8 +426,38 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
private static readonly string[] ValidAlarmFallbackModes = ["Auto", "ForceAlarmManager", "ForceSubtag"];
private const int MinimumAlarmPollIntervalMilliseconds = 100;
// One hour. Above this the cadence stops being a cadence: int.MaxValue
// milliseconds is ~24 days, which silently disables alarm polling instead
// of reporting the misconfiguration.
private const int MaximumAlarmPollIntervalMilliseconds = 3_600_000;
private const int MinimumMaxAlarmsPerFetch = 64;
// The worker is a 32-bit process and materializes each fetch as one BSTR
// plus a full XmlDocument over it, so an unbounded cap is an out-of-memory
// fault on the STA rather than a slow poll.
private const int MaximumMaxAlarmsPerFetch = 65_536;
private static void ValidateAlarms(AlarmsOptions options, ValidationBuilder builder)
{
// Validated regardless of Enabled: both values are stamped onto every
// worker launch environment, so a bad value is a misconfiguration even
// before the central monitor is switched on.
if (options.PollIntervalMilliseconds is < MinimumAlarmPollIntervalMilliseconds
or > MaximumAlarmPollIntervalMilliseconds)
{
builder.Add(
$"MxGateway:Alarms:PollIntervalMilliseconds must be between {MinimumAlarmPollIntervalMilliseconds} and {MaximumAlarmPollIntervalMilliseconds}.");
}
if (options.MaxAlarmsPerFetch is < MinimumMaxAlarmsPerFetch or > MaximumMaxAlarmsPerFetch)
{
builder.Add(
$"MxGateway:Alarms:MaxAlarmsPerFetch must be between {MinimumMaxAlarmsPerFetch} and {MaximumMaxAlarmsPerFetch}.");
}
if (!options.Enabled)
{
return;
@@ -555,4 +624,14 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
private static void AddIfInvalidPath(string? value, string message, ValidationBuilder builder)
=> GatewayConfigPathRules.AddIfInvalidPath(value, message, builder);
// Rooted is not the same as safe: an absolute path inside the app directory passes
// AddIfNotRooted and is still renamed away by the upgrade procedure. See
// GatewayConfigPathRules.AddIfUnderContentRoot.
private static void AddIfUnderContentRoot(
string? value,
string? contentRoot,
string message,
ValidationBuilder builder)
=> GatewayConfigPathRules.AddIfUnderContentRoot(value, contentRoot, message, builder);
}
@@ -68,4 +68,19 @@ public sealed class LdapOptions
/// <summary>Gets the LDAP attribute name for group membership.</summary>
public string GroupAttribute { get; init; } = "memberOf";
/// <summary>
/// Gets the ordered fallback LDAP endpoints (<c>"host"</c> or <c>"host:port"</c>) the shared
/// provider walks when the primary fails with a system-side error. Empty (the default) leaves
/// single-endpoint behaviour unchanged. Mirrors
/// <see cref="ZB.MOM.WW.Auth.Abstractions.Ldap.LdapOptions.FallbackServers"/>, added in
/// ZB.MOM.WW.Auth 0.2.0.
/// <para>
/// Carried here only so the effective-config display does not hide a configured backup DC —
/// nothing on the gateway side reads it. Entry syntax is validated at boot by the shared
/// <c>LdapOptionsValidator</c>, which owns the (internal) parser; re-validating here would
/// mean a second, drifting copy of that grammar.
/// </para>
/// </summary>
public IReadOnlyList<string> FallbackServers { get; init; } = [];
}
@@ -88,4 +88,13 @@ public sealed class SecurityOptions
/// ceiling of twice this value. Default is 4096.
/// </summary>
public int ApiKeyFailureTrackedPeers { get; init; } = 4096;
/// <summary>
/// Gets how many days of canonical audit history the gateway keeps. The audit drain sweeps
/// <c>audit_event</c> once at startup and hourly thereafter, deleting rows older than this
/// window; without it the table grows without bound in the same SQLite file the
/// authentication hot path reads. Must be greater than zero — audit retention cannot be
/// disabled by configuration, only widened. Default is 90 days.
/// </summary>
public int AuditRetentionDays { get; init; } = 90;
}
@@ -33,6 +33,18 @@ public sealed class WorkerOptions
/// </summary>
public int WriteCompletionWaitMilliseconds { get; init; } = 1500;
/// <summary>
/// Capacity of the worker's outbound MXAccess event queue, in events.
/// Default 10,000; must be between 1,000 and 1,000,000. This is
/// headroom, not a throttle: the queue has no drop policy, so a burst
/// that fills it faults the session with a <c>QueueOverflow</c> worker
/// fault. Raise it for sessions whose subscription set can outrun the
/// drain loop (large advise sets, slow event consumers). Conveyed to
/// the worker through the <c>MXGATEWAY_EVENT_QUEUE_CAPACITY</c>
/// environment variable.
/// </summary>
public int EventQueueCapacity { get; init; } = 10000;
/// <summary>The maximum time in seconds for graceful shutdown.</summary>
public int ShutdownTimeoutSeconds { get; init; } = 10;
@@ -1,80 +1,118 @@
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.SignalR.Client;
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Components;
/// <summary>
/// Base class for Blazor dashboard pages that watch gateway metrics
/// snapshots. The previous implementation polled
/// <see cref="IDashboardSnapshotService.WatchSnapshotsAsync"/> directly; we
/// now subscribe to <see cref="DashboardSnapshotHub"/> so updates are
/// pushed and disconnects survive reconnects via SignalR's
/// auto-reconnect.
/// Base class for Blazor dashboard pages that watch gateway metrics snapshots.
/// Pages subscribe to the in-process <see cref="IDashboardSnapshotFeed"/>, which
/// multicasts a single <see cref="IDashboardSnapshotService.WatchSnapshotsAsync"/>
/// enumeration to every circuit. An earlier implementation had each page open its
/// own SignalR connection to <c>/hubs/snapshot</c> — a loopback WebSocket back into
/// this same process, per page. The snapshot hub and its publisher remain for
/// external (non-circuit) clients; server-rendered pages no longer use them.
/// </summary>
public abstract class DashboardPageBase : ComponentBase, IAsyncDisposable
{
private HubConnection? _hub;
/// <summary>
/// Upper bound on waiting for the watch loop while disposing. The loop marshals
/// renders through the renderer's dispatcher and disposal can run on that same
/// dispatcher, so the wait is bounded rather than unconditional.
/// </summary>
private static readonly TimeSpan WatchDrainTimeout = TimeSpan.FromSeconds(5);
/// <summary>Snapshot service used to seed the initial render before the hub connects.</summary>
private readonly CancellationTokenSource _watchCancellation = new();
private Task? _watchTask;
/// <summary>Snapshot service used to seed the initial render before the first feed update.</summary>
[Inject]
protected IDashboardSnapshotService SnapshotService { get; set; } = null!;
/// <summary>Factory that builds the SignalR connection (mints the hub bearer token).</summary>
/// <summary>Shared in-process snapshot feed this page renders from.</summary>
[Inject]
protected DashboardHubConnectionFactory HubFactory { get; set; } = null!;
protected IDashboardSnapshotFeed SnapshotFeed { get; set; } = null!;
/// <summary>Logger used to report a snapshot subscription that ended or would not drain.</summary>
[Inject]
protected ILogger<DashboardPageBase>? Logger { get; set; }
/// <summary>
/// The most recent gateway metric snapshot. Synchronously seeded from
/// <see cref="IDashboardSnapshotService.GetSnapshot"/> for the very
/// first render, then refreshed by hub push.
/// <see cref="IDashboardSnapshotService.GetSnapshot"/> for the very first
/// render, then refreshed from the feed.
/// </summary>
protected DashboardSnapshot? Snapshot { get; private set; }
/// <inheritdoc />
protected override async Task OnInitializedAsync()
protected override Task OnInitializedAsync()
{
Snapshot = SnapshotService.GetSnapshot();
await ConnectHubAsync().ConfigureAwait(false);
// Deliberately not awaited: the watch loop runs for the lifetime of the page
// and is cancelled and drained by DisposeAsync.
_watchTask = WatchSnapshotsAsync(_watchCancellation.Token);
return Task.CompletedTask;
}
/// <summary>Disposes the SignalR hub connection created for this page, tolerating disposal-time errors.</summary>
/// <summary>Cancels the snapshot subscription created for this page, tolerating disposal-time errors.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
if (_hub is not null)
{
try
{
await _hub.DisposeAsync().ConfigureAwait(false);
}
catch
{
// Disposal-time errors are best-effort.
}
}
GC.SuppressFinalize(this);
}
private async Task ConnectHubAsync()
{
_hub = HubFactory.Create("/hubs/snapshot");
_hub.On<DashboardSnapshot>(DashboardSnapshotHub.SnapshotMessage, async snapshot =>
{
Snapshot = snapshot;
await InvokeAsync(StateHasChanged).ConfigureAwait(false);
});
try
{
await _hub.StartAsync().ConfigureAwait(false);
await _watchCancellation.CancelAsync().ConfigureAwait(false);
if (_watchTask is not null)
{
await _watchTask.WaitAsync(WatchDrainTimeout).ConfigureAwait(false);
}
}
catch (TimeoutException)
{
// Accepted limitation: the abandoned loop still holds its feed subscription, so
// the feed's idle gate stays open until it does unwind. There is no way to force
// a detach — the loop is parked on a dispatcher that is not draining — so the
// warning is the operator's only signal that a circuit teardown wedged.
Logger?.LogWarning(
"Dashboard page {Page} did not release its snapshot subscription within {Timeout}; "
+ "the shared snapshot feed stays active until it unwinds.",
GetType().Name,
WatchDrainTimeout);
}
catch
{
// Hub is best-effort; the initial GetSnapshot() seed remains
// valid and the snapshot service keeps populating its cache for
// the next reconnect cycle.
// Other disposal-time errors are best-effort.
}
_watchCancellation.Dispose();
GC.SuppressFinalize(this);
}
private async Task WatchSnapshotsAsync(CancellationToken cancellationToken)
{
try
{
await foreach (DashboardSnapshot snapshot in SnapshotFeed
.WatchAsync(cancellationToken)
.ConfigureAwait(false))
{
Snapshot = snapshot;
await InvokeAsync(StateHasChanged).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
{
// The page is going away.
}
catch (Exception error)
{
// The feed is best-effort: the last rendered snapshot stays on screen and the
// snapshot service keeps serving GetSnapshot() for the next page load. Logged
// once here, on the way out of the loop — never per snapshot.
Logger?.LogWarning(
error,
"Live snapshot updates ended for dashboard page {Page}; it keeps the last rendered snapshot.",
GetType().Name);
}
}
}
@@ -1,4 +1,5 @@
@inherits LayoutComponentBase
@using ZB.MOM.WW.Secrets.Ui
@* Thin layout: delegates the side-rail chassis (hamburger, brand, responsive
collapse) to the shared ZB.MOM.WW.Theme <ThemeShell>. The nav is reproduced
@@ -19,7 +20,20 @@
</NavRailSection>
<NavRailSection Title="Admin" Key="admin">
<NavRailItem Href="/apikeys" Text="API Keys" />
<NavRailItem Href="/admin/secrets" Text="Secrets" />
@* Gated on the SAME policy the mounted /admin/secrets page enforces, not on a role
literal, so nav visibility cannot drift from page access. In this host the two are
equivalent — GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting roles
(secrets-manager, secrets-reveal) are unreachable here — but the policy form stays
correct if that ever relaxes. Deliberately NOT applied to the API Keys item above:
that page renders read-only for Viewers, so hiding its link would remove legitimate
read access, whereas the secrets page denies a Viewer outright and its link would be
a dead end. *@
<AuthorizeView Policy="@SecretsAuthorization.ManagePolicy">
<Authorized>
<NavRailItem Href="/admin/secrets" Text="Secrets" />
</Authorized>
</AuthorizeView>
<NavRailItem Href="/settings" Text="Settings" />
</NavRailSection>
</Nav>
@@ -1,10 +1,9 @@
@page "/alarms"
@implements IAsyncDisposable
@using Microsoft.AspNetCore.SignalR.Client
@using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs
@using ZB.MOM.WW.MxGateway.Server.Alarms
@inject IDashboardLiveDataService LiveData
@inject IOptions<GatewayOptions> GatewayOptions
@inject DashboardHubConnectionFactory HubFactory
@inject IGatewayAlarmService AlarmService
<PageTitle>Dashboard Alarms</PageTitle>
@@ -173,13 +172,13 @@
private Task? _pollTask;
private DashboardAlarmProviderStatus _providerStatus = DashboardAlarmProviderStatus.Healthy;
private HubConnection? _alarmsHub;
private Task? _providerStatusTask;
/// <inheritdoc />
protected override void OnInitialized()
{
_pollTask = PollLoopAsync();
_ = AttachAlarmsHubAsync();
_providerStatusTask = ProviderStatusLoopAsync();
}
private string? ProviderStatusTitle()
@@ -189,26 +188,51 @@
: null;
}
private async Task AttachAlarmsHubAsync()
// The badge tracks the central monitor directly rather than looping back through
// /hubs/alarms: the alarm service is an in-process multi-subscriber fan-out, so a
// server-rendered page needs no SignalR client, no loopback socket and no auth token.
// Alarm rows still come from the 3-second poll below — this loop only feeds the badge.
private async Task ProviderStatusLoopAsync()
{
_alarmsHub = HubFactory.Create("/hubs/alarms");
_alarmsHub.On<AlarmFeedMessage>(AlarmsHub.AlarmMessage, async message =>
while (!_cts.IsCancellationRequested)
{
if (message.PayloadCase == AlarmFeedMessage.PayloadOneofCase.ProviderStatus)
try
{
_providerStatus = DashboardAlarmProviderStatus.FromFeed(message);
await InvokeAsync(StateHasChanged).ConfigureAwait(false);
}
});
await foreach (AlarmFeedMessage message in AlarmService
.StreamAsync(alarmFilterPrefix: null, _cts.Token)
.ConfigureAwait(false))
{
if (message.PayloadCase != AlarmFeedMessage.PayloadOneofCase.ProviderStatus)
{
continue;
}
try
{
await _alarmsHub.StartAsync(_cts.Token).ConfigureAwait(false);
}
catch
{
// The badge is best-effort; it stays at the healthy default until
// the hub reconnects and delivers a fresh provider-status message.
await InvokeAsync(() =>
{
_providerStatus = DashboardAlarmProviderStatus.FromFeed(message);
StateHasChanged();
}).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
{
return;
}
catch
{
// The monitor completes a subscriber's stream when it falls behind, and
// again when the monitor restarts. Both are recoverable by resubscribing;
// the badge holds its last value in the meantime.
}
try
{
await Task.Delay(TimeSpan.FromSeconds(1), _cts.Token).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return;
}
}
}
@@ -312,18 +336,6 @@
{
await _cts.CancelAsync();
if (_alarmsHub is not null)
{
try
{
await _alarmsHub.DisposeAsync();
}
catch
{
// Disposal-time errors are best-effort.
}
}
if (_pollTask is not null)
{
try
@@ -335,6 +347,17 @@
}
}
if (_providerStatusTask is not null)
{
try
{
await _providerStatusTask;
}
catch (OperationCanceledException)
{
}
}
_cts.Dispose();
GC.SuppressFinalize(this);
}
@@ -1,11 +1,11 @@
@page "/sessions/{SessionId}"
@inherits DashboardPageBase
@implements IAsyncDisposable
@using Microsoft.AspNetCore.SignalR.Client
@using ZB.MOM.WW.MxGateway.Contracts.Proto
@using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs
@inject AuthenticationStateProvider AuthenticationStateProvider
@inject IDashboardSessionAdminService SessionAdminService
@inject IDashboardSessionEventSubscriber EventSubscriber
<PageTitle>Dashboard Session</PageTitle>
@@ -157,7 +157,18 @@ else
private DashboardSessionSummary? CurrentSession => Snapshot?.Sessions.FirstOrDefault(session =>
string.Equals(session.SessionId, SessionId, StringComparison.Ordinal));
private HubConnection? _eventsHub;
// Upper bound on waiting for the event pump while detaching, mirroring
// DashboardPageBase's snapshot-watch drain: the pump marshals renders through the
// renderer's dispatcher and a detach can run on that same dispatcher, so the wait
// is bounded rather than unconditional.
private static readonly TimeSpan EventPumpDrainTimeout = TimeSpan.FromSeconds(5);
// Written only on the renderer's dispatcher (the lifecycle methods below), and read
// on it from inside the pump's dispatched callback — that pairing is what makes the
// stale-batch guard in PumpEventsAsync reliable.
private IDashboardEventSubscription? _eventSubscription;
private CancellationTokenSource? _eventPumpCancellation;
private Task? _eventPumpTask;
private bool _eventsConnected;
private string? _subscribedSessionId;
private readonly LinkedList<MxEvent> _recentEvents = new();
@@ -183,8 +194,11 @@ else
{
if (!string.Equals(_subscribedSessionId, SessionId, StringComparison.Ordinal))
{
await DetachEventsHubAsync().ConfigureAwait(false);
await AttachEventsHubAsync().ConfigureAwait(false);
// Deliberately no ConfigureAwait(false): the resumption must stay on the
// renderer's dispatcher so the new subscription is published to
// _eventSubscription from the same thread the pump's guard reads it on.
await DetachEventsAsync();
AttachEvents();
}
}
@@ -261,68 +275,123 @@ else
string ConfirmButtonClass,
Func<System.Security.Claims.ClaimsPrincipal, Task<DashboardSessionAdminResult>> Action);
private async Task AttachEventsHubAsync()
// The dashboard runs in the same process as the event mirror, so this page reads
// the session's mirrored events straight from it. It used to open a loopback
// SignalR connection to /hubs/events — mint a hub token, negotiate, hold a
// WebSocket, serialize every event — to reach data already sitting in memory.
// IDashboardSessionEventSubscriber resolves to the same singleton that serves
// IDashboardEventBroadcaster, and the subscription registers with
// EventsHubViewerRegistry, so the "nobody is watching" gate keeps working for
// both audiences.
// ACL posture is unchanged from the hub path: any dashboard Viewer may watch
// any session (SEC-25 tracks the per-session ACL for both seams).
private void AttachEvents()
{
if (string.IsNullOrWhiteSpace(SessionId))
{
return;
}
_eventsHub = HubFactory.Create("/hubs/events");
_eventsHub.On<MxEvent>(EventsHub.EventMessage, async mxEvent =>
{
_recentEvents.AddFirst(mxEvent);
while (_recentEvents.Count > MaxRecentEvents)
{
_recentEvents.RemoveLast();
}
_eventSubscription = EventSubscriber.Subscribe(SessionId);
_eventPumpCancellation = new CancellationTokenSource();
_eventsConnected = true;
_subscribedSessionId = SessionId;
await InvokeAsync(StateHasChanged).ConfigureAwait(false);
});
_eventsHub.Closed += _ =>
{
_eventsConnected = false;
return InvokeAsync(StateHasChanged);
};
_eventsHub.Reconnected += _ =>
{
_eventsConnected = true;
return InvokeAsync(StateHasChanged);
};
// Deliberately not awaited: the pump runs for as long as the page watches this
// session and is cancelled and drained by DetachEventsAsync.
_eventPumpTask = PumpEventsAsync(_eventSubscription, _eventPumpCancellation.Token);
}
private async Task PumpEventsAsync(IDashboardEventSubscription subscription, CancellationToken cancellationToken)
{
try
{
await _eventsHub.StartAsync().ConfigureAwait(false);
await _eventsHub.SendAsync("SubscribeSession", SessionId).ConfigureAwait(false);
_eventsConnected = true;
_subscribedSessionId = SessionId;
while (await subscription.Reader.WaitToReadAsync(cancellationToken).ConfigureAwait(false))
{
// Drain what is queued and render once: a burst costs one render pass,
// not one per event. Reading past the display cap would be wasted work,
// and anything left queued is picked up on the next pass.
List<MxEvent> batch = new();
while (batch.Count < MaxRecentEvents && subscription.Reader.TryRead(out MxEvent? mxEvent))
{
batch.Add(mxEvent);
}
if (batch.Count == 0)
{
continue;
}
await InvokeAsync(() =>
{
// The batch was read before this callback was dispatched, and a
// session switch can land in between. Rendering it then would show
// the previous session's events under the new session's heading, so
// a batch whose subscription is no longer the live one is dropped.
// Safe as an unsynchronized read: _eventSubscription is written on
// this same dispatcher.
if (!ReferenceEquals(_eventSubscription, subscription))
{
return;
}
foreach (MxEvent mxEvent in batch)
{
_recentEvents.AddFirst(mxEvent);
}
while (_recentEvents.Count > MaxRecentEvents)
{
_recentEvents.RemoveLast();
}
StateHasChanged();
}).ConfigureAwait(false);
}
}
catch
catch (OperationCanceledException)
{
_eventsConnected = false;
// The page navigated to another session or was disposed.
}
catch (ObjectDisposedException)
{
// Either the renderer went away mid-dispatch, or the drain below timed out
// and disposed the cancellation source this loop is still reading.
}
}
private async Task DetachEventsHubAsync()
private async Task DetachEventsAsync()
{
HubConnection? hub = _eventsHub;
_eventsHub = null;
IDashboardEventSubscription? subscription = _eventSubscription;
CancellationTokenSource? cancellation = _eventPumpCancellation;
Task? pump = _eventPumpTask;
_eventSubscription = null;
_eventPumpCancellation = null;
_eventPumpTask = null;
_eventsConnected = false;
_subscribedSessionId = null;
_recentEvents.Clear();
if (hub is not null)
// Cancel and drop the subscription before draining. Disposing it releases the
// viewer registration — the whole point of the gate — and completes the channel,
// so the pump has an exit even if cancellation is missed.
cancellation?.Cancel();
subscription?.Dispose();
try
{
try
if (pump is not null)
{
await hub.DisposeAsync().ConfigureAwait(false);
}
catch
{
// Disposal-time errors are best-effort.
await pump.WaitAsync(EventPumpDrainTimeout);
}
}
catch
{
// Detach-time errors (including a drain timeout) are best-effort.
}
// Disposed after the drain so the pump is no longer reading the token.
cancellation?.Dispose();
}
private static string EventStatusLabel(MxEvent evt)
@@ -334,7 +403,7 @@ else
public new async ValueTask DisposeAsync()
{
await DetachEventsHubAsync().ConfigureAwait(false);
await DetachEventsAsync();
await base.DisposeAsync().ConfigureAwait(false);
}
}
@@ -26,6 +26,21 @@ else
<tr><th scope="row">Run migrations</th><td>@Snapshot.Configuration.Authentication.RunMigrationsOnStartup</td></tr>
<tr><th scope="row">LDAP enabled</th><td>@Snapshot.Configuration.Ldap.Enabled</td></tr>
<tr><th scope="row">LDAP server</th><td>@Snapshot.Configuration.Ldap.Server:@Snapshot.Configuration.Ldap.Port</td></tr>
<tr>
<th scope="row">LDAP fallback servers</th>
@* Rendered even when empty: "none" is the operationally interesting answer
on a host someone believes has a backup DC configured. *@
<td>
@if (Snapshot.Configuration.Ldap.FallbackServers.Count == 0)
{
<span class="text-muted">none</span>
}
else
{
<code>@string.Join(", ", Snapshot.Configuration.Ldap.FallbackServers)</code>
}
</td>
</tr>
<tr><th scope="row">LDAP transport</th><td>@Snapshot.Configuration.Ldap.Transport</td></tr>
<tr><th scope="row">LDAP search base</th><td><code>@Snapshot.Configuration.Ldap.SearchBase</code></td></tr>
<tr><th scope="row">LDAP service account</th><td><code>@Snapshot.Configuration.Ldap.ServiceAccountDn</code></td></tr>
@@ -15,13 +15,36 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
{
private const string BackendName = "Galaxy";
private const string ClientName = "mxgateway-dashboard";
// One browse page of tags plus headroom. Bounds the standing advise load the
// single dashboard worker carries — and the event churn that advise set feeds —
// however much of a galaxy an operator browses through in one sitting.
//
// The bound is per-read, not absolute: a read may never evict a tag it is itself
// about to return, so a single read of more distinct tags than the cap leaves the
// set that large. The invariant EvictForAsync actually maintains is
//
// |advise set| after a read <= max(MaxSubscribedTags, distinct tags in that read)
//
// and any overshoot is squeezed back out by the next read that subscribes a tag
// (see EvictForAsync). A browse page requests far fewer tags than the cap, so in
// practice the set settles at MaxSubscribedTags.
private const int MaxSubscribedTags = 256;
private static readonly TimeSpan ReadTimeout = TimeSpan.FromSeconds(5);
private readonly ISessionManager _sessionManager;
private readonly IGatewayAlarmService _alarmService;
private readonly ILogger<DashboardLiveDataService> _logger;
private readonly SemaphoreSlim _gate = new(1, 1);
private readonly HashSet<string> _subscribed = new(StringComparer.OrdinalIgnoreCase);
// Least-recently-read-last advise set: the list holds every currently advised
// tag ordered most- to least-recently read, the dictionary indexes into it.
// Both are only ever touched under _gate, which already serialises all viewers.
private readonly Dictionary<string, LinkedListNode<SubscribedTag>> _subscribed =
new(StringComparer.OrdinalIgnoreCase);
private readonly LinkedList<SubscribedTag> _recency = new();
private GatewaySession? _session;
private int _serverHandle;
@@ -58,15 +81,15 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
(GatewaySession session, int serverHandle) = await EnsureReadyAsync(cancellationToken)
.ConfigureAwait(false);
string[] toSubscribe = tagAddresses.Where(tag => !_subscribed.Contains(tag)).ToArray();
string[] toSubscribe = TouchAndCollectNewTags(tagAddresses, out int justReadCount);
if (toSubscribe.Length > 0)
{
await session.SubscribeBulkAsync(serverHandle, toSubscribe, cancellationToken)
await EvictForAsync(session, serverHandle, toSubscribe.Length, justReadCount, cancellationToken)
.ConfigureAwait(false);
foreach (string tag in toSubscribe)
{
_subscribed.Add(tag);
}
IReadOnlyList<SubscribeResult> subscribeResults = await session
.SubscribeBulkAsync(serverHandle, toSubscribe, cancellationToken)
.ConfigureAwait(false);
TrackSubscribed(toSubscribe, subscribeResults);
}
IReadOnlyList<BulkReadResult> results = await session
@@ -107,6 +130,148 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
return Task.FromResult(new DashboardAlarmQueryResult(alarms, error, _alarmService.WorkerProcessId));
}
// Promotes every already-advised tag in this read to the front of the recency
// list and returns the tags that still need subscribing (distinct, in request
// order). `justReadCount` is how many distinct tags of this read were already
// advised — they now occupy the front of the list and must never be evicted to
// make room for the same read's new tags. Callers must hold _gate.
//
// Every tag of one read is equally recently read; the recency list needs a total
// order anyway, so the whole service uses one tie-break: later in the request wins.
// Promoting in request order gives that here, and TrackSubscribed inserts new tags
// the same way.
private string[] TouchAndCollectNewTags(IReadOnlyCollection<string> tagAddresses, out int justReadCount)
{
int touched = 0;
List<string> toSubscribe = [];
HashSet<string> seen = new(StringComparer.OrdinalIgnoreCase);
foreach (string tag in tagAddresses)
{
if (_subscribed.TryGetValue(tag, out LinkedListNode<SubscribedTag>? node))
{
if (!ReferenceEquals(node, _recency.First))
{
_recency.Remove(node);
_recency.AddFirst(node);
}
if (seen.Add(tag))
{
touched++;
}
}
else if (seen.Add(tag))
{
toSubscribe.Add(tag);
}
}
justReadCount = touched;
return [.. toSubscribe];
}
// Drops least-recently-read tags off the back of the advise set until the
// incoming tags fit under MaxSubscribedTags, unadvising them on the worker in
// one batch. A failed unadvise must not fail the read: the tags are dropped
// from tracking regardless, and the session-invalidation path already handles
// gateway/worker drift. Callers must hold _gate.
//
// Eviction stops at the tags this read just touched (`justReadCount`), so a read
// whose own distinct tags outnumber the cap ends over it — see MaxSubscribedTags
// for the exact invariant. That overshoot is not sticky: the next read that
// subscribes anything computes `overflow` against the oversized set and evicts the
// whole excess in one pass (a 300-tag set plus one new tag evicts 45 and lands
// back at the cap). A read that subscribes nothing new evicts nothing, but it also
// cannot grow the set.
//
// Cancellation mid-eviction follows this file's policy: OperationCanceledException
// is deliberately not caught here or in ReadAsync, so it propagates with the tags
// already dropped from tracking — the same end state as a failed unadvise.
private async Task EvictForAsync(
GatewaySession session,
int serverHandle,
int incomingCount,
int justReadCount,
CancellationToken cancellationToken)
{
int overflow = _subscribed.Count + incomingCount - MaxSubscribedTags;
int evictable = _subscribed.Count - justReadCount;
int evictCount = Math.Min(overflow, evictable);
if (evictCount <= 0)
{
return;
}
List<int> evictedHandles = new(evictCount);
for (int i = 0; i < evictCount && _recency.Last is { } oldest; i++)
{
_recency.RemoveLast();
_subscribed.Remove(oldest.Value.TagAddress);
if (oldest.Value.ItemHandle != 0)
{
evictedHandles.Add(oldest.Value.ItemHandle);
}
}
_logger.LogDebug(
"Dashboard advise set hit its cap of {Cap}; evicted {EvictedCount} least-recently-read tags.",
MaxSubscribedTags,
evictCount);
if (evictedHandles.Count == 0)
{
return;
}
try
{
await session.UnsubscribeBulkAsync(serverHandle, evictedHandles, cancellationToken)
.ConfigureAwait(false);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
_logger.LogDebug(
exception,
"Unadvising {EvictedCount} evicted dashboard tags failed; they stay dropped from tracking.",
evictedHandles.Count);
}
}
// Records the freshly advised tags as the most recently read, keeping each
// tag's item handle so eviction can unadvise it. Tags the worker failed to
// advise are still tracked (matching the pre-cap behaviour of not retrying
// them on every read) but carry no handle, so eviction just forgets them.
// Callers must hold _gate.
private void TrackSubscribed(IReadOnlyList<string> tagAddresses, IReadOnlyList<SubscribeResult> results)
{
Dictionary<string, int> handles = new(results.Count, StringComparer.OrdinalIgnoreCase);
foreach (SubscribeResult result in results)
{
if (result.WasSuccessful && !string.IsNullOrEmpty(result.TagAddress))
{
handles[result.TagAddress] = result.ItemHandle;
}
}
// Request order, so the read's last tag ends up most recent — the same
// tie-break TouchAndCollectNewTags applies to the tags it promotes.
foreach (string tag in tagAddresses)
{
handles.TryGetValue(tag, out int itemHandle);
_subscribed[tag] = _recency.AddFirst(new SubscribedTag(tag, itemHandle));
}
}
// Forgets the whole advise set without unadvising: every call site is one where
// the backing session (and with it every item handle) is already gone.
// Callers must hold _gate.
private void ClearSubscriptions()
{
_subscribed.Clear();
_recency.Clear();
}
// Returns a Ready session + its Register server handle, opening a fresh
// session when none exists or the current one is no longer usable. Callers
// must hold _gate.
@@ -132,7 +297,7 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
await CloseQuietlyAsync(existing.SessionId).ConfigureAwait(false);
}
_subscribed.Clear();
ClearSubscriptions();
_session = null;
GatewaySession session = await _sessionManager.OpenSessionAsync(
@@ -178,7 +343,7 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
{
_session = null;
_serverHandle = 0;
_subscribed.Clear();
ClearSubscriptions();
}
private async Task CloseQuietlyAsync(string sessionId)
@@ -212,4 +377,8 @@ public sealed class DashboardLiveDataService : IDashboardLiveDataService, IAsync
_gate.Dispose();
}
// One entry of the advise set. ItemHandle is the handle the worker bound for
// the tag, or 0 when the subscribe failed and there is nothing to unadvise.
private readonly record struct SubscribedTag(string TagAddress, int ItemHandle);
}
@@ -38,6 +38,7 @@ public static class DashboardServiceCollectionExtensions
services.AddZbLdapAuth(configuration, "MxGateway:Ldap");
services.AddSingleton<IDashboardSnapshotService, DashboardSnapshotService>();
services.AddSingleton<IDashboardSnapshotFeed, DashboardSnapshotFeed>();
services.AddSingleton<IDashboardLiveDataService, DashboardLiveDataService>();
services.AddSingleton<IDashboardAuthenticator, DashboardAuthenticator>();
services.AddSingleton<IGroupRoleMapper<string>, DashboardGroupRoleMapper>();
@@ -47,7 +48,21 @@ public static class DashboardServiceCollectionExtensions
services.AddSingleton<HubTokenService>();
services.AddScoped<Hubs.DashboardHubConnectionFactory>();
services.AddScoped<IDashboardBrowseService, DashboardBrowseService>();
services.AddSingleton<Hubs.IDashboardEventBroadcaster, Hubs.DashboardEventBroadcaster>();
// Singleton: EventsHub instances are transient (one per hub invocation), so the
// subscriber bookkeeping they share with the broadcaster must outlive them.
services.AddSingleton<Hubs.EventsHubViewerRegistry>();
// One instance behind two interfaces, registered concretely and forwarded: the
// publish side (IDashboardEventBroadcaster, driven by the session pipeline) and
// the in-process subscribe side (IDashboardSessionEventSubscriber, used by the
// session-details page) share subscriber bookkeeping, so resolving them to two
// instances would leave the page subscribed to a mirror nobody publishes to.
services.AddSingleton<Hubs.DashboardEventBroadcaster>();
services.AddSingleton<Hubs.IDashboardEventBroadcaster>(
static provider => provider.GetRequiredService<Hubs.DashboardEventBroadcaster>());
services.AddSingleton<Hubs.IDashboardSessionEventSubscriber>(
static provider => provider.GetRequiredService<Hubs.DashboardEventBroadcaster>());
services.AddSingleton<Hubs.DashboardSnapshotHubConnectionCounter>();
services.AddHostedService<Hubs.DashboardSnapshotPublisher>();
services.AddHostedService<Hubs.AlarmsHubPublisher>();
services.AddHttpContextAccessor();
@@ -0,0 +1,372 @@
using System.Runtime.CompilerServices;
using System.Threading.Channels;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>
/// Fans one <see cref="IDashboardSnapshotService.WatchSnapshotsAsync"/> enumeration out to
/// every dashboard circuit. The underlying watch is not multicast — each enumeration owns a
/// timer and builds its own snapshot per tick — so subscribing per page would multiply the
/// snapshot cost by the number of open pages.
/// </summary>
/// <remarks>
/// <para>
/// The pump is idle-gated: it starts when the first subscriber arrives and is cancelled and
/// awaited when the last subscriber <em>of the live generation</em> leaves, so an unwatched
/// gateway runs no timer and builds no snapshots. Successive pumps are chained through
/// <c>_pumpTask</c>, so a rapid
/// unsubscribe/resubscribe restarts a fresh pump without ever running two enumerations at
/// once.
/// </para>
/// <para>
/// Every subscriber is tagged with the pump generation it joined under, and a dying pump only
/// ever detaches its own generation. A pump ends its generation the instant its source fails
/// or completes — before the (possibly slow) enumerator disposal — so a subscriber arriving
/// while a pump unwinds starts a fresh generation instead of silently attaching to a dead
/// pump that is about to detach everybody and leave nobody watching.
/// </para>
/// </remarks>
public sealed class DashboardSnapshotFeed : IDashboardSnapshotFeed
{
/// <summary>Generation value meaning "no pump is accepting subscribers".</summary>
private const long NoGeneration = 0;
private readonly IDashboardSnapshotService _snapshotService;
private readonly ILogger<DashboardSnapshotFeed> _logger;
private readonly object _gate = new();
private readonly List<Subscription> _subscribers = [];
/// <summary>
/// The most recent pump, completed while idle. A starting pump awaits its predecessor
/// before enumerating, which is what guarantees a single live enumeration.
/// </summary>
private Task _pumpTask = Task.CompletedTask;
/// <summary>Cancellation for the live pump; null when no generation is accepting subscribers.</summary>
private CancellationTokenSource? _pumpCancellation;
/// <summary>The generation new subscribers join, or <see cref="NoGeneration"/> when no pump is live.</summary>
private long _generation = NoGeneration;
/// <summary>Last generation handed out; only ever incremented under <c>_gate</c>.</summary>
private long _lastGeneration = NoGeneration;
/// <summary>Initializes a new instance of the <see cref="DashboardSnapshotFeed"/> class.</summary>
/// <param name="snapshotService">Snapshot source to multicast.</param>
/// <param name="logger">Optional logger for pump faults.</param>
public DashboardSnapshotFeed(
IDashboardSnapshotService snapshotService,
ILogger<DashboardSnapshotFeed>? logger = null)
{
_snapshotService = snapshotService ?? throw new ArgumentNullException(nameof(snapshotService));
_logger = logger ?? NullLogger<DashboardSnapshotFeed>.Instance;
}
/// <inheritdoc />
public async IAsyncEnumerable<DashboardSnapshot> WatchAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
// Capacity 1 + DropOldest: a viewer only ever wants the latest snapshot, so a
// circuit that renders slowly neither buffers without bound nor blocks the pump
// (TryWrite always succeeds) — it just skips the snapshots it was too slow for.
Channel<DashboardSnapshot> channel = Channel.CreateBounded<DashboardSnapshot>(
new BoundedChannelOptions(1)
{
FullMode = BoundedChannelFullMode.DropOldest,
SingleReader = true,
SingleWriter = false,
});
Subscription subscription = Subscribe(channel);
try
{
await foreach (DashboardSnapshot snapshot in channel.Reader
.ReadAllAsync(cancellationToken)
.ConfigureAwait(false))
{
yield return snapshot;
}
}
finally
{
// Untokened on purpose: teardown must run to completion even when this
// subscriber is unwinding because its own token fired.
await UnsubscribeAsync(subscription).ConfigureAwait(false);
}
}
private Subscription Subscribe(Channel<DashboardSnapshot> channel)
{
lock (_gate)
{
// A live generation is joined; otherwise this subscriber starts one. Keying on
// "is a generation live" rather than "is this the first subscriber" is what makes
// a subscriber arriving while a pump unwinds start a fresh pump for itself.
long generation = _pumpCancellation is null ? StartPumpLocked() : _generation;
Subscription subscription = new(channel, generation);
_subscribers.Add(subscription);
return subscription;
}
}
private async Task UnsubscribeAsync(Subscription subscription)
{
CancellationTokenSource? cancellation;
Task pump;
lock (_gate)
{
if (!_subscribers.Remove(subscription))
{
// The pump already detached this subscription (it completed or faulted).
return;
}
if (subscription.Generation != _generation)
{
// This viewer belonged to a generation that has already ended. The live
// pump — if there is one — serves other viewers and must not be cancelled
// on their behalf; the dying pump is stopping under its own steam.
return;
}
if (HasSubscribersLocked(_generation))
{
// Other viewers are still watching the live generation. Counting the whole
// list here would be wrong: subscribers of an ending generation linger in it
// until that pump's Reset runs, and they must not hold the idle gate open.
return;
}
cancellation = _pumpCancellation;
_pumpCancellation = null;
_generation = NoGeneration;
pump = _pumpTask;
}
try
{
cancellation?.Cancel();
}
catch (ObjectDisposedException)
{
// The pump ended on its own and disposed its cancellation source first.
}
try
{
await pump.ConfigureAwait(false);
}
catch (Exception)
{
// A pump fault has already been reported to the subscribers it had; the
// unsubscribing caller is only waiting for the enumeration to stop.
}
}
/// <summary>Reports whether any subscriber is still being served by a generation.</summary>
/// <param name="generation">The generation to look for.</param>
/// <returns>True when at least one subscriber carries that generation.</returns>
private bool HasSubscribersLocked(long generation)
{
foreach (Subscription subscriber in _subscribers)
{
if (subscriber.Generation == generation)
{
return true;
}
}
return false;
}
/// <summary>
/// Starts a pump generation. Must be called while holding <c>_gate</c>; the caller adds
/// the subscribers that belong to the returned generation.
/// </summary>
/// <returns>The new generation identifier.</returns>
private long StartPumpLocked()
{
long generation = ++_lastGeneration;
CancellationTokenSource cancellation = new();
Task previous = _pumpTask;
_generation = generation;
_pumpCancellation = cancellation;
// Task.Run, not a direct call: an async iterator runs synchronously up to its
// first suspension, and the first pull of the underlying watch can read the API
// key table. That must not run on the subscribing circuit's thread, let alone
// while this lock is held.
_pumpTask = Task.Run(() => PumpAsync(generation, previous, cancellation, cancellation.Token));
return generation;
}
private async Task PumpAsync(
long generation,
Task previous,
CancellationTokenSource cancellation,
CancellationToken cancellationToken)
{
try
{
// Never overlap with the enumeration this pump replaces.
await previous.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
// Enumerated by hand rather than with await foreach so the generation can be
// ended the moment the source fails or completes — await foreach would run the
// enumerator's disposal first, and a subscriber arriving during that disposal
// would join a generation that is already doomed.
IAsyncEnumerator<DashboardSnapshot> snapshots = _snapshotService
.WatchSnapshotsAsync(cancellationToken)
.GetAsyncEnumerator(cancellationToken);
try
{
while (true)
{
bool moved;
try
{
moved = await snapshots.MoveNextAsync().ConfigureAwait(false);
}
catch
{
EndGeneration(generation);
throw;
}
if (!moved)
{
EndGeneration(generation);
break;
}
Broadcast(generation, snapshots.Current);
}
}
finally
{
await snapshots.DisposeAsync().ConfigureAwait(false);
}
// The source completed on its own; hand the completion to this generation's
// subscribers and re-arm so the next one starts a fresh enumeration.
Reset(generation, error: null);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// The production DashboardSnapshotService swallows cancellation and yield-breaks,
// so normal teardown exits through the fall-through above (with an ownership-checked
// Reset that finds no subscribers); an implementation that propagates the token
// instead exits here. Both shapes end the generation exactly once.
EndGeneration(generation);
}
catch (Exception error)
{
_logger.LogWarning(error, "Dashboard snapshot feed stopped; the next subscriber restarts it.");
Reset(generation, error);
}
finally
{
cancellation.Dispose();
}
}
private void Broadcast(long generation, DashboardSnapshot snapshot)
{
lock (_gate)
{
foreach (Subscription subscriber in _subscribers)
{
if (subscriber.Generation != generation)
{
continue;
}
// Bounded/DropOldest: always accepted unless the channel is completed.
subscriber.Channel.Writer.TryWrite(snapshot);
}
}
}
/// <summary>
/// Stops handing <paramref name="generation"/> to new subscribers. Called the instant a
/// pump's source fails or completes, before its enumerator is disposed.
/// </summary>
/// <param name="generation">The generation that has ended.</param>
private void EndGeneration(long generation)
{
lock (_gate)
{
EndGenerationLocked(generation);
}
}
/// <summary>Clears the live-pump state if <paramref name="generation"/> still owns it.</summary>
/// <param name="generation">The generation that has ended.</param>
private void EndGenerationLocked(long generation)
{
if (_generation != generation)
{
return;
}
_generation = NoGeneration;
_pumpCancellation = null;
}
/// <summary>
/// Detaches the subscribers of a finished generation and re-arms the feed. Subscribers of
/// any other generation are left alone — they belong to a pump that is still running (or
/// about to), so a dying pump must not take them down with it.
/// </summary>
/// <param name="generation">The generation whose subscribers are being detached.</param>
/// <param name="error">Failure to surface, or null when the source completed cleanly.</param>
private void Reset(long generation, Exception? error)
{
List<Channel<DashboardSnapshot>> detached = [];
lock (_gate)
{
for (int index = _subscribers.Count - 1; index >= 0; index--)
{
if (_subscribers[index].Generation != generation)
{
continue;
}
detached.Add(_subscribers[index].Channel);
_subscribers.RemoveAt(index);
}
EndGenerationLocked(generation);
if (_subscribers.Count > 0 && _pumpCancellation is null)
{
// Belt and braces: subscribers left with no live pump would be frozen for
// good, because only a subscriber that finds no generation starts one.
long restarted = StartPumpLocked();
foreach (Subscription subscriber in _subscribers)
{
subscriber.Generation = restarted;
}
}
}
foreach (Channel<DashboardSnapshot> channel in detached)
{
channel.Writer.TryComplete(error);
}
}
/// <summary>One viewer's delivery channel plus the pump generation serving it.</summary>
/// <param name="channel">Delivery channel for this viewer.</param>
/// <param name="generation">Pump generation this viewer joined under.</param>
private sealed class Subscription(Channel<DashboardSnapshot> channel, long generation)
{
/// <summary>Gets the viewer's delivery channel.</summary>
public Channel<DashboardSnapshot> Channel { get; } = channel;
/// <summary>Gets or sets the pump generation currently serving this viewer.</summary>
public long Generation { get; set; } = generation;
}
}
@@ -16,6 +16,17 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
{
private const string HealthyStatus = "Healthy";
/// <summary>
/// Minimum spacing between API key list reads. The list is a SQLite query whose
/// content only changes when an operator creates, rotates, or revokes a key, so
/// refreshing it on every ~1s snapshot tick buys nothing; the dashboard still sees
/// a key change within this interval.
/// </summary>
private static readonly TimeSpan ApiKeySummaryRefreshInterval = TimeSpan.FromSeconds(15);
/// <summary>Sentinel for "the API key summaries have never been refreshed".</summary>
private const long NeverRefreshedTicks = long.MinValue;
private readonly ISessionRegistry _sessionRegistry;
private readonly GatewayMetrics _metrics;
private readonly IGatewayConfigurationProvider _configurationProvider;
@@ -30,6 +41,13 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
private readonly ILogger<DashboardSnapshotService> _logger;
private readonly SemaphoreSlim _apiKeySummaryRefreshGate = new(1, 1);
private IReadOnlyList<DashboardApiKeySummary> _apiKeySummaries = Array.Empty<DashboardApiKeySummary>();
private long _apiKeySummariesRefreshedAtTicks = NeverRefreshedTicks;
// The effective configuration is built from IOptions<GatewayOptions> and is startup-static:
// the gateway binds options once at boot and never reloads them, so this projection cannot
// change for the process lifetime. Build it once instead of re-projecting the whole option
// tree on every snapshot tick. A racing first build is harmless — the projection is pure,
// so either winner stores equivalent content.
private EffectiveGatewayConfiguration? _effectiveConfiguration;
// Memoizes ONLY the O(N) template/category breakdown against the cache sequence. The shared
// library bumps Sequence only on a heavy refresh that replaces the object set, so an unchanged
// sequence means the breakdown is unchanged and can be reused — keeping the ~1s snapshot tick
@@ -100,10 +118,23 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
Metrics: CreateMetricSummaries(metricsSnapshot),
Faults: CreateFaultSummaries(sessions, generatedAt),
ApiKeys: Volatile.Read(ref _apiKeySummaries),
Configuration: _configurationProvider.GetEffectiveConfiguration(),
Configuration: ResolveEffectiveConfiguration(),
Galaxy: ResolveGalaxySummary());
}
private EffectiveGatewayConfiguration ResolveEffectiveConfiguration()
{
EffectiveGatewayConfiguration? cached = Volatile.Read(ref _effectiveConfiguration);
if (cached is not null)
{
return cached;
}
EffectiveGatewayConfiguration configuration = _configurationProvider.GetEffectiveConfiguration();
Volatile.Write(ref _effectiveConfiguration, configuration);
return configuration;
}
private DashboardGalaxySummary ResolveGalaxySummary()
{
GalaxyHierarchyCacheEntry entry = _galaxyHierarchyCache.Current;
@@ -255,6 +286,20 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
private async Task RefreshApiKeySummariesAsync(CancellationToken cancellationToken)
{
DateTimeOffset now = _timeProvider.GetUtcNow();
long lastRefreshedAtTicks = Interlocked.Read(ref _apiKeySummariesRefreshedAtTicks);
if (lastRefreshedAtTicks != NeverRefreshedTicks
&& now.UtcTicks - lastRefreshedAtTicks < ApiKeySummaryRefreshInterval.Ticks)
{
// Inside the refresh window: reuse the cached summaries rather than
// re-reading the API key table on this tick. Only a *successful* refresh
// moves the timestamp, so a failed read is retried on the next tick.
// This check is deliberately outside the refresh gate, so it races
// benignly: if two callers both read a stale timestamp, the zero-timeout
// gate below admits one and the other returns without touching the store.
return;
}
if (!await _apiKeySummaryRefreshGate.WaitAsync(0, cancellationToken).ConfigureAwait(false))
{
return;
@@ -278,6 +323,7 @@ public sealed class DashboardSnapshotService : IDashboardSnapshotService
.ToArray();
Volatile.Write(ref _apiKeySummaries, summaries);
Interlocked.Exchange(ref _apiKeySummariesRefreshedAtTicks, now.UtcTicks);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
@@ -1,3 +1,4 @@
using System.Threading.Channels;
using Microsoft.AspNetCore.SignalR;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
@@ -6,11 +7,13 @@ using ZB.MOM.WW.MxGateway.Server.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Broadcasts MxEvents to <see cref="EventsHub"/> clients subscribed to the
/// session's group. Fire-and-forget: we hand the send to the hub context
/// and return immediately so the source gRPC stream is never blocked.
/// Errors are logged once and dropped — keeping the SignalR mirror best-effort
/// preserves the gRPC contract that exists today.
/// Broadcasts MxEvents to the two dashboard audiences for a session: remote
/// <see cref="EventsHub"/> clients subscribed to the session's group, and
/// in-process subscribers opened through
/// <see cref="IDashboardSessionEventSubscriber.Subscribe"/>. Fire-and-forget: we
/// hand the send to the hub context and return immediately so the source gRPC
/// stream is never blocked. Errors are logged once and dropped — keeping the
/// SignalR mirror best-effort preserves the gRPC contract that exists today.
/// </summary>
/// <remarks>
/// When <c>MxGateway:Dashboard:ShowTagValues</c> is false (the default), tag
@@ -21,13 +24,49 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// the mirror independently of the still-outstanding per-session hub ACL
/// (see <see cref="EventsHub"/>).
/// </remarks>
/// <param name="hubContext">Hub context used to send to the session's group.</param>
/// <param name="viewerRegistry">
/// Live-subscriber registry consulted before any per-event work is done. Both
/// audiences register here — hub connections by their SignalR connection id,
/// in-process subscriptions by a synthetic one — so the gate stays a single
/// source of truth.
/// </param>
/// <param name="options">Gateway options supplying <c>Dashboard:ShowTagValues</c>.</param>
/// <param name="logger">Logger for best-effort mirror failures.</param>
public sealed class DashboardEventBroadcaster(
IHubContext<EventsHub> hubContext,
EventsHubViewerRegistry viewerRegistry,
IOptions<GatewayOptions> options,
ILogger<DashboardEventBroadcaster> logger) : IDashboardEventBroadcaster
ILogger<DashboardEventBroadcaster> logger) : IDashboardEventBroadcaster, IDashboardSessionEventSubscriber
{
/// <summary>
/// Queue depth per in-process subscriber. The consumer is a Blazor page
/// rendering the newest handful of events, so a burst it cannot keep up with
/// is dropped oldest-first rather than allowed to grow — same best-effort
/// contract the SignalR mirror already has.
/// </summary>
private const int InProcessQueueCapacity = 256;
private readonly bool _showTagValues = options.Value.Dashboard.ShowTagValues;
private readonly object _syncRoot = new();
/// <summary>
/// In-process subscribers per session. Values are treated as immutable once
/// stored: a subscribe or dispose swaps in a new array under
/// <see cref="_syncRoot"/>, so <see cref="Publish"/> can grab the reference
/// and write to it after releasing the lock.
/// </summary>
private readonly Dictionary<string, InProcessSubscription[]> _inProcessSubscribers =
new(StringComparer.Ordinal);
/// <summary>
/// Total live in-process subscribers, read without the lock so the common
/// case — nobody has a session-details page open — never contends on it.
/// Written only under <see cref="_syncRoot"/>.
/// </summary>
private int _inProcessSubscriberCount;
/// <inheritdoc />
public void Publish(string sessionId, MxEvent mxEvent)
{
@@ -36,8 +75,22 @@ public sealed class DashboardEventBroadcaster(
return;
}
// Every session's dashboard-mirror subscriber calls Publish for every event,
// whether or not a browser is on that session's page. Without this gate the
// steady state — no dashboard viewer at all — still paid a deep protobuf
// clone (redaction is on by default) plus a send to an empty SignalR group
// per event. Bail before both.
if (!viewerRegistry.HasViewers(sessionId))
{
return;
}
MxEvent outbound = _showTagValues ? mxEvent : RedactValues(mxEvent);
// In-process delivery first: it is synchronous, cannot throw, and must not be
// skipped by the early return the hub send's guard clause takes.
DeliverInProcess(sessionId, outbound);
// Wrap the Task acquisition in a try/catch so a hypothetical synchronous throw
// from SendAsync (e.g. an implementation that throws before returning the Task)
// cannot escape Publish. The interface contract is never-throw; fire-and-forget.
@@ -68,6 +121,117 @@ public sealed class DashboardEventBroadcaster(
}
}
/// <inheritdoc />
public IDashboardEventSubscription Subscribe(string sessionId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(sessionId);
// A synthetic connection id keeps the registry's per-connection bookkeeping
// usable for a subscriber that has no SignalR connection behind it. The
// "inproc-" prefix cannot collide with a SignalR connection id and makes the
// origin obvious in a debugger.
string connectionId = "inproc-" + Guid.NewGuid().ToString("N");
InProcessSubscription subscription = new(this, sessionId, connectionId, InProcessQueueCapacity);
// Register before the subscriber becomes a delivery target, exactly as
// EventsHub.SubscribeSession registers before joining the group: the reverse
// order would leave a window in which this subscriber is a delivery target but
// Publish's gate still reports the session unwatched, silently dropping events
// it should receive. The cost of this order is at worst a redaction clone that
// reaches nobody for the width of the window.
viewerRegistry.AddViewer(connectionId, sessionId);
lock (_syncRoot)
{
_inProcessSubscribers[sessionId] =
_inProcessSubscribers.TryGetValue(sessionId, out InProcessSubscription[]? existing)
? [.. existing, subscription]
: [subscription];
Volatile.Write(ref _inProcessSubscriberCount, _inProcessSubscriberCount + 1);
}
return subscription;
}
/// <summary>
/// Hands the already-redacted event to every in-process subscriber of the
/// session. Writes are non-blocking and lossy by construction, so this never
/// stalls the caller's event pipeline.
/// </summary>
/// <param name="sessionId">Session the event belongs to.</param>
/// <param name="outbound">The event as the dashboard should see it.</param>
private void DeliverInProcess(string sessionId, MxEvent outbound)
{
// The gate above admits hub-only viewers too, so check for in-process
// subscribers before touching the lock at all.
if (Volatile.Read(ref _inProcessSubscriberCount) == 0)
{
return;
}
InProcessSubscription[] subscribers;
lock (_syncRoot)
{
if (!_inProcessSubscribers.TryGetValue(sessionId, out InProcessSubscription[]? found))
{
return;
}
subscribers = found;
}
// The array is never mutated in place, so the writes happen outside the lock.
foreach (InProcessSubscription subscriber in subscribers)
{
subscriber.TryWrite(outbound);
}
}
/// <summary>
/// Removes a disposed subscription from the delivery map and releases its
/// viewer registration. Called at most once per subscription.
/// </summary>
/// <param name="subscription">The subscription being disposed.</param>
private void Unsubscribe(InProcessSubscription subscription)
{
// Drop the delivery target first and deregister after, mirroring
// EventsHub.UnsubscribeSession: the mirror stays enabled for the brief overlap
// rather than dropping events still owed to the session's other subscribers.
lock (_syncRoot)
{
if (_inProcessSubscribers.TryGetValue(subscription.SessionId, out InProcessSubscription[]? existing))
{
InProcessSubscription[] remaining =
[.. existing.Where(candidate => !ReferenceEquals(candidate, subscription))];
// Equal lengths mean it was never in this bucket, so the counter it
// would decrement is not its own to release.
if (remaining.Length != existing.Length)
{
if (remaining.Length == 0)
{
// Drop the key so the map does not grow one entry per session ever viewed.
_inProcessSubscribers.Remove(subscription.SessionId);
}
else
{
_inProcessSubscribers[subscription.SessionId] = remaining;
}
Volatile.Write(ref _inProcessSubscriberCount, _inProcessSubscriberCount - 1);
}
}
}
viewerRegistry.RemoveViewer(subscription.ConnectionId, subscription.SessionId);
// The synthetic connection id is used once and never reconnects, so nothing
// else will ever call ReleaseConnection for it; without this the registry
// would retain an empty per-connection entry per subscription ever opened.
viewerRegistry.ReleaseConnection(subscription.ConnectionId);
}
/// <summary>
/// Produces a deep clone of <paramref name="source"/> with every tag-value
/// field cleared, leaving tag reference, quality, status, and timestamps
@@ -90,4 +254,75 @@ public sealed class DashboardEventBroadcaster(
return redacted;
}
/// <summary>
/// One in-process subscriber's feed: a bounded, drop-oldest channel plus the
/// registry bookkeeping that keeps <see cref="Publish"/>'s viewer gate honest
/// while the feed is live.
/// </summary>
private sealed class InProcessSubscription : IDashboardEventSubscription
{
private readonly DashboardEventBroadcaster _owner;
private readonly Channel<MxEvent> _channel;
private int _disposed;
/// <summary>Initializes a new instance of the <see cref="InProcessSubscription"/> class.</summary>
/// <param name="owner">Broadcaster to deregister from on disposal.</param>
/// <param name="sessionId">Session whose events this subscription carries.</param>
/// <param name="connectionId">Synthetic connection id registered with the viewer registry.</param>
/// <param name="capacity">Queue depth before the oldest queued event is dropped.</param>
internal InProcessSubscription(
DashboardEventBroadcaster owner,
string sessionId,
string connectionId,
int capacity)
{
_owner = owner;
SessionId = sessionId;
ConnectionId = connectionId;
_channel = Channel.CreateBounded<MxEvent>(new BoundedChannelOptions(capacity)
{
// DropOldest, not Wait: a write must never block the gRPC event
// pipeline that calls Publish, and the newest events are the ones a
// live view wants.
FullMode = BoundedChannelFullMode.DropOldest,
SingleReader = true,
SingleWriter = false,
});
}
/// <inheritdoc />
public ChannelReader<MxEvent> Reader => _channel.Reader;
/// <summary>Gets the session this subscription is watching.</summary>
internal string SessionId { get; }
/// <summary>Gets the synthetic connection id held in the viewer registry.</summary>
internal string ConnectionId { get; }
/// <summary>
/// Queues an event for the subscriber, dropping the oldest queued event when
/// the reader has fallen behind. Never blocks and never throws.
/// </summary>
/// <param name="mxEvent">The event to queue.</param>
internal void TryWrite(MxEvent mxEvent) => _channel.Writer.TryWrite(mxEvent);
/// <summary>
/// Deregisters the subscription and completes its channel so a reader's
/// loop ends. Idempotent — a second call does nothing, so it can never
/// release a viewer count that a sibling subscription owns.
/// </summary>
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0)
{
return;
}
_owner.Unsubscribe(this);
_channel.Writer.TryComplete();
}
}
}
@@ -9,8 +9,15 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// immediately via <see cref="OnConnectedAsync"/>; subsequent refreshes are
/// broadcast by <see cref="DashboardSnapshotPublisher"/>.
/// </summary>
/// <remarks>
/// Connections are counted into <see cref="DashboardSnapshotHubConnectionCounter"/>
/// so <see cref="DashboardSnapshotPublisher"/> can stop building and broadcasting
/// snapshots while nobody is watching.
/// </remarks>
[Authorize(Policy = DashboardAuthenticationDefaults.HubClientsPolicy)]
public sealed class DashboardSnapshotHub(IDashboardSnapshotService snapshotService) : Hub
public sealed class DashboardSnapshotHub(
IDashboardSnapshotService snapshotService,
DashboardSnapshotHubConnectionCounter connectionCounter) : Hub
{
/// <summary>Method name used to push snapshot updates to clients.</summary>
public const string SnapshotMessage = "SnapshotUpdated";
@@ -18,7 +25,17 @@ public sealed class DashboardSnapshotHub(IDashboardSnapshotService snapshotServi
/// <inheritdoc />
public override async Task OnConnectedAsync()
{
// Count the viewer before seeding it, so the publisher resumes its tick
// no later than the first snapshot this connection renders.
connectionCounter.Increment();
await Clients.Caller.SendAsync(SnapshotMessage, snapshotService.GetSnapshot()).ConfigureAwait(false);
await base.OnConnectedAsync().ConfigureAwait(false);
}
/// <inheritdoc />
public override async Task OnDisconnectedAsync(Exception? exception)
{
connectionCounter.Decrement();
await base.OnDisconnectedAsync(exception).ConfigureAwait(false);
}
}
@@ -0,0 +1,52 @@
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Process-wide count of live <see cref="DashboardSnapshotHub"/> connections.
/// Registered as a singleton and read by <see cref="DashboardSnapshotPublisher"/>
/// to idle-gate the snapshot tick: with no dashboard connected there is nothing
/// to broadcast to, so no snapshot is built.
/// </summary>
public sealed class DashboardSnapshotHubConnectionCounter
{
private int _count;
/// <summary>Gets the number of live snapshot hub connections.</summary>
public int Count => Volatile.Read(ref _count);
/// <summary>Records a new snapshot hub connection.</summary>
/// <returns>The connection count after the increment.</returns>
public int Increment()
{
return Interlocked.Increment(ref _count);
}
/// <summary>
/// Records a snapshot hub disconnection, clamped at zero: SignalR can invoke
/// <c>OnDisconnectedAsync</c> for a connection whose <c>OnConnectedAsync</c>
/// faulted, and a negative count would idle-gate the publisher while viewers
/// are still attached.
/// </summary>
/// <remarks>
/// The clamp is applied inside the compare-and-swap rather than as a repair
/// afterwards. Decrementing first and then correcting a negative result races:
/// two unmatched decrements from zero would both plan a repair, a real
/// connection could increment in between, and the stale repair would then
/// overwrite that live connection's increment — freezing a real viewer's
/// dashboard behind the idle gate. Reading, clamping, and publishing as one
/// atomic step means a lost race simply retries against the fresh value.
/// </remarks>
/// <returns>The connection count after the decrement.</returns>
public int Decrement()
{
int current;
int next;
do
{
current = Volatile.Read(ref _count);
next = current > 0 ? current - 1 : 0;
}
while (Interlocked.CompareExchange(ref _count, next, current) != current);
return next;
}
}
@@ -9,6 +9,7 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// gateway process; clients listen via the hub.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="ExecuteAsync"/> wraps the snapshot subscription in
/// a reconnect loop with a configurable retry delay (5s by default,
/// mirroring <see cref="AlarmsHubPublisher"/>). A transient failure inside
@@ -16,44 +17,67 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// one-time logger-init failure or a transient SQL error from the Galaxy
/// summary projection — would otherwise end the BackgroundService with no
/// reconnect, taking the dashboard offline until process restart.
/// </para>
/// <para>
/// The loop is idle-gated on <see cref="DashboardSnapshotHubConnectionCounter"/>.
/// Each snapshot costs a session-registry snapshot and sort, a metrics snapshot
/// that copies dictionaries under the global metrics lock, and (periodically) a
/// SQLite read of the API key table — work with no consumer when no dashboard is
/// connected. While the count is zero the publisher does not advance the snapshot
/// enumerator at all, so the producing iterator stays suspended and builds nothing.
/// </para>
/// </remarks>
public sealed class DashboardSnapshotPublisher : BackgroundService
{
private static readonly TimeSpan DefaultReconnectDelay = TimeSpan.FromSeconds(5);
private static readonly TimeSpan DefaultIdlePollInterval = TimeSpan.FromSeconds(1);
private readonly IDashboardSnapshotService _snapshotService;
private readonly IHubContext<DashboardSnapshotHub> _hubContext;
private readonly DashboardSnapshotHubConnectionCounter _connectionCounter;
private readonly ILogger<DashboardSnapshotPublisher> _logger;
private readonly TimeSpan _reconnectDelay;
private readonly TimeSpan _idlePollInterval;
/// <summary>Initializes a new instance of the DashboardSnapshotPublisher class.</summary>
/// <param name="snapshotService">The snapshot service to subscribe to.</param>
/// <param name="hubContext">The SignalR hub context for broadcasting.</param>
/// <param name="connectionCounter">Live snapshot hub connection count used to idle-gate the tick.</param>
/// <param name="logger">The logger instance.</param>
public DashboardSnapshotPublisher(
IDashboardSnapshotService snapshotService,
IHubContext<DashboardSnapshotHub> hubContext,
DashboardSnapshotHubConnectionCounter connectionCounter,
ILogger<DashboardSnapshotPublisher> logger)
: this(snapshotService, hubContext, logger, DefaultReconnectDelay)
: this(snapshotService, hubContext, connectionCounter, logger, DefaultReconnectDelay, DefaultIdlePollInterval)
{
}
/// <summary>Initializes a new instance of the DashboardSnapshotPublisher class with custom reconnect delay.</summary>
/// <remarks>Internal hook for testing: tests inject a very short reconnect delay so assertions don't wait full 5s.</remarks>
/// <summary>Initializes a new instance of the DashboardSnapshotPublisher class with custom cadences.</summary>
/// <remarks>
/// Internal hook for testing: tests inject a very short reconnect delay so assertions
/// don't wait the full 5s, and a short idle poll so the resume-from-idle path is fast.
/// </remarks>
/// <param name="snapshotService">The snapshot service to subscribe to.</param>
/// <param name="hubContext">The SignalR hub context for broadcasting.</param>
/// <param name="connectionCounter">Live snapshot hub connection count used to idle-gate the tick.</param>
/// <param name="logger">The logger instance.</param>
/// <param name="reconnectDelay">The delay before reconnecting after a subscription failure.</param>
/// <param name="idlePollInterval">How often the idle publisher re-checks for a connected viewer.</param>
internal DashboardSnapshotPublisher(
IDashboardSnapshotService snapshotService,
IHubContext<DashboardSnapshotHub> hubContext,
DashboardSnapshotHubConnectionCounter connectionCounter,
ILogger<DashboardSnapshotPublisher> logger,
TimeSpan reconnectDelay)
TimeSpan reconnectDelay,
TimeSpan idlePollInterval)
{
_snapshotService = snapshotService;
_hubContext = hubContext;
_connectionCounter = connectionCounter;
_logger = logger;
_reconnectDelay = reconnectDelay;
_idlePollInterval = idlePollInterval;
}
/// <inheritdoc />
@@ -66,15 +90,31 @@ public sealed class DashboardSnapshotPublisher : BackgroundService
{
try
{
await foreach (DashboardSnapshot snapshot in _snapshotService
// Enumerated by hand rather than with await foreach: the snapshot is
// built inside the producer's MoveNextAsync, so not calling MoveNextAsync
// is what makes the idle gate skip the build and not just the broadcast.
await using IAsyncEnumerator<DashboardSnapshot> snapshots = _snapshotService
.WatchSnapshotsAsync(stoppingToken)
.ConfigureAwait(false))
.GetAsyncEnumerator(stoppingToken);
while (!stoppingToken.IsCancellationRequested)
{
if (stoppingToken.IsCancellationRequested)
if (_connectionCounter.Count == 0)
{
// Nobody is watching: leave the producer suspended and re-check
// shortly. The first viewer to connect resumes the tick, and is
// seeded directly by the hub's OnConnectedAsync meanwhile.
await Task.Delay(_idlePollInterval, stoppingToken).ConfigureAwait(false);
continue;
}
if (!await snapshots.MoveNextAsync().ConfigureAwait(false))
{
break;
}
DashboardSnapshot snapshot = snapshots.Current;
try
{
await _hubContext.Clients
@@ -9,8 +9,14 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// session; <see cref="DashboardEventBroadcaster"/> sends messages to
/// <c>session:{id}</c> as events arrive from the live gRPC stream.
/// </summary>
/// <remarks>
/// Group membership is mirrored into <see cref="EventsHubViewerRegistry"/>
/// because SignalR does not expose it, and the broadcaster consults the
/// registry to skip all mirror work for sessions nobody is watching.
/// </remarks>
/// <param name="viewerRegistry">Registry tracking which sessions have live subscribers.</param>
[Authorize(Policy = DashboardAuthenticationDefaults.HubClientsPolicy)]
public sealed class EventsHub : Hub
public sealed class EventsHub(EventsHubViewerRegistry viewerRegistry) : Hub
{
/// <summary>Method name used to push individual <c>MxEvent</c> values to clients.</summary>
public const string EventMessage = "MxEvent";
@@ -55,19 +61,43 @@ public sealed class EventsHub : Hub
return Task.CompletedTask;
}
// Register before joining the group: the reverse order would leave a window
// in which this connection is a group member but the broadcaster's gate still
// reports the session unwatched, silently dropping events it should receive.
viewerRegistry.AddViewer(Context.ConnectionId, sessionId);
return Groups.AddToGroupAsync(Context.ConnectionId, GroupName(sessionId));
}
/// <summary>Unsubscribes the calling SignalR connection from the per-session events group.</summary>
/// <param name="sessionId">Session id to unsubscribe the caller from.</param>
/// <returns>A task representing the unsubscription operation.</returns>
public Task UnsubscribeSession(string sessionId)
public async Task UnsubscribeSession(string sessionId)
{
if (string.IsNullOrWhiteSpace(sessionId))
{
return Task.CompletedTask;
return;
}
return Groups.RemoveFromGroupAsync(Context.ConnectionId, GroupName(sessionId));
// Leave the group first, deregister after — the mirror stays enabled for the
// brief overlap rather than dropping events still owed to other subscribers.
await Groups.RemoveFromGroupAsync(Context.ConnectionId, GroupName(sessionId)).ConfigureAwait(false);
viewerRegistry.RemoveViewer(Context.ConnectionId, sessionId);
}
/// <summary>
/// Releases every session subscription the dropped connection held. A browser
/// tab that closes never calls <see cref="UnsubscribeSession"/>, so without
/// this the session would look watched forever and the mirror would keep
/// cloning and sending events to an empty group.
/// </summary>
/// <param name="exception">The exception that terminated the connection, if any.</param>
/// <returns>A task representing the disconnect handling.</returns>
public override Task OnDisconnectedAsync(Exception? exception)
{
viewerRegistry.ReleaseConnection(Context.ConnectionId);
return base.OnDisconnectedAsync(exception);
}
}
@@ -0,0 +1,152 @@
using System.Collections.Concurrent;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// Tracks which sessions currently have at least one live <see cref="EventsHub"/>
/// subscriber, so <see cref="DashboardEventBroadcaster"/> can skip the redaction
/// clone and the group send for sessions nobody is watching.
/// </summary>
/// <remarks>
/// SignalR does not expose group membership, so the hub mirrors its own
/// <c>AddToGroup</c>/<c>RemoveFromGroup</c> calls here. In the steady state no
/// browser is on a session-details page, yet every session's dashboard-mirror
/// subscriber still called <c>Publish</c> for every event — a deep protobuf
/// clone (values are redacted by default) plus a send to an empty group, per
/// event, thrown away. This registry is the cheap gate in front of that work.
/// <para>
/// Per-connection subscriptions are tracked as well, because a browser tab that
/// simply goes away never calls <c>UnsubscribeSession</c>; the hub's
/// <c>OnDisconnectedAsync</c> releases everything the connection held.
/// </para>
/// </remarks>
public sealed class EventsHubViewerRegistry
{
private readonly ConcurrentDictionary<string, int> _viewersBySession = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, ConcurrentDictionary<string, byte>> _sessionsByConnection =
new(StringComparer.Ordinal);
/// <summary>
/// Records that <paramref name="connectionId"/> is watching
/// <paramref name="sessionId"/>. Repeat calls for the same pair are
/// idempotent, so one <see cref="RemoveViewer"/> always clears them.
/// </summary>
/// <param name="connectionId">SignalR connection id of the subscriber.</param>
/// <param name="sessionId">Session id being watched.</param>
public void AddViewer(string connectionId, string sessionId)
{
if (string.IsNullOrWhiteSpace(connectionId) || string.IsNullOrWhiteSpace(sessionId))
{
return;
}
ConcurrentDictionary<string, byte> sessions = _sessionsByConnection.GetOrAdd(
connectionId,
static _ => new ConcurrentDictionary<string, byte>(StringComparer.Ordinal));
// The per-connection set is the source of truth for the count: only a
// subscription that was genuinely new increments the session's viewers.
if (!sessions.TryAdd(sessionId, 0))
{
return;
}
_viewersBySession.AddOrUpdate(sessionId, 1, static (_, count) => count + 1);
}
/// <summary>
/// Records that <paramref name="connectionId"/> stopped watching
/// <paramref name="sessionId"/>. A removal with no matching
/// <see cref="AddViewer"/> is a no-op, so the count cannot go negative.
/// </summary>
/// <param name="connectionId">SignalR connection id of the subscriber.</param>
/// <param name="sessionId">Session id no longer being watched.</param>
public void RemoveViewer(string connectionId, string sessionId)
{
if (string.IsNullOrWhiteSpace(connectionId) || string.IsNullOrWhiteSpace(sessionId))
{
return;
}
if (!_sessionsByConnection.TryGetValue(connectionId, out ConcurrentDictionary<string, byte>? sessions)
|| !sessions.TryRemove(sessionId, out _))
{
return;
}
ReleaseSession(sessionId);
}
/// <summary>
/// Releases every subscription held by <paramref name="connectionId"/>.
/// Called from the hub's disconnect callback, which is the only reliable
/// signal for a browser tab that closed without unsubscribing.
/// </summary>
/// <param name="connectionId">SignalR connection id that dropped.</param>
public void ReleaseConnection(string connectionId)
{
if (string.IsNullOrWhiteSpace(connectionId))
{
return;
}
// Detaching the set is safe against a SubscribeSession that arrives after the disconnect
// only because SignalR dispatches a connection's hub invocations sequentially by default
// (MaximumParallelInvocationsPerClient = 1): OnDisconnectedAsync cannot overlap an
// AddViewer for the same connection, so no late add can re-create the entry and leak a
// count that nothing will ever release. Raising that option would break this.
if (!_sessionsByConnection.TryRemove(connectionId, out ConcurrentDictionary<string, byte>? sessions))
{
return;
}
foreach (string sessionId in sessions.Keys)
{
// TryRemove, not a bare enumeration: a concurrent RemoveViewer on the
// same detached set must not let the session be decremented twice.
if (sessions.TryRemove(sessionId, out _))
{
ReleaseSession(sessionId);
}
}
}
/// <summary>Gets a value indicating whether any hub connection is watching the session.</summary>
/// <param name="sessionId">Session id to test.</param>
/// <returns><see langword="true"/> when at least one connection is subscribed.</returns>
public bool HasViewers(string sessionId) =>
!string.IsNullOrEmpty(sessionId)
&& _viewersBySession.TryGetValue(sessionId, out int count)
&& count > 0;
/// <summary>
/// Decrements the session's viewer count, dropping the entry entirely at
/// zero so the dictionary does not grow one key per session ever viewed.
/// The compare-and-swap loop keeps the decrement correct against a
/// concurrent <see cref="AddViewer"/> on the same session.
/// </summary>
/// <param name="sessionId">Session id whose count is released.</param>
private void ReleaseSession(string sessionId)
{
while (true)
{
if (!_viewersBySession.TryGetValue(sessionId, out int count))
{
return;
}
if (count <= 1)
{
if (_viewersBySession.TryRemove(new KeyValuePair<string, int>(sessionId, count)))
{
return;
}
}
else if (_viewersBySession.TryUpdate(sessionId, count - 1, count))
{
return;
}
}
}
}
@@ -0,0 +1,26 @@
using System.Threading.Channels;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// A live in-process feed of one session's dashboard-mirrored MxEvents, handed
/// out by <see cref="IDashboardSessionEventSubscriber.Subscribe"/>. Server-side
/// Blazor components read it directly instead of looping back through
/// <see cref="EventsHub"/> over a loopback SignalR connection.
/// </summary>
/// <remarks>
/// Events are delivered exactly as a hub client would see them — the same
/// redacted clone the group send carries, so <c>MxGateway:Dashboard:ShowTagValues</c>
/// governs both paths identically. The feed is a bounded, lossy queue: a
/// consumer that falls behind loses the oldest queued events, matching the
/// best-effort contract the SignalR mirror already has. Disposing the
/// subscription deregisters it, which is what lets the broadcaster go back to
/// skipping all mirror work for a session nobody is watching — so callers must
/// dispose. Dispose is idempotent.
/// </remarks>
public interface IDashboardEventSubscription : IDisposable
{
/// <summary>Gets the reader delivering this session's mirrored events.</summary>
ChannelReader<MxEvent> Reader { get; }
}
@@ -0,0 +1,34 @@
namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
/// <summary>
/// In-process subscription seam on the dashboard event mirror. Implemented by
/// <see cref="DashboardEventBroadcaster"/> alongside
/// <see cref="IDashboardEventBroadcaster"/>.
/// </summary>
/// <remarks>
/// The interactive-server dashboard runs in the same process as the broadcaster,
/// so a session-details page has no reason to open a loopback SignalR connection
/// back to <see cref="EventsHub"/> — mint a hub token, negotiate, hold a
/// WebSocket, and serialize every event — just to read events the broadcaster
/// already holds. It subscribes here instead. The registry gate stays honest
/// either way: an in-process subscription registers a synthetic connection id
/// with <see cref="EventsHubViewerRegistry"/> exactly as the hub registers a real
/// one, so <see cref="IDashboardEventBroadcaster.Publish"/> keeps skipping the
/// redaction clone for sessions nobody is watching.
/// <para>
/// It is a separate interface rather than a member of
/// <see cref="IDashboardEventBroadcaster"/> because publishing and consuming are
/// different roles: the session pipeline only ever publishes, and its test
/// doubles should not have to implement a subscription feed.
/// </para>
/// </remarks>
public interface IDashboardSessionEventSubscriber
{
/// <summary>Opens an in-process feed of the session's mirrored events.</summary>
/// <param name="sessionId">Session id whose events the caller wants.</param>
/// <returns>
/// The subscription. Dispose it to stop the feed and release the viewer
/// registration that keeps the mirror enabled for this session.
/// </returns>
IDashboardEventSubscription Subscribe(string sessionId);
}
@@ -0,0 +1,26 @@
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
/// <summary>
/// In-process multicast over <see cref="IDashboardSnapshotService.WatchSnapshotsAsync"/>.
/// One enumeration of the underlying watch is fanned out to every subscriber, so N
/// dashboard circuits cost one snapshot build per tick instead of N — and while nobody
/// subscribes, nothing runs at all.
/// </summary>
/// <remarks>
/// There is no authentication or authorization gate here: the feed is reached only from
/// Blazor dashboard components, whose endpoints already require
/// <see cref="DashboardAuthenticationDefaults.ViewerPolicy"/>, so every caller is a circuit
/// authorized as Viewer. Remote (non-circuit) consumers still go through
/// <c>/hubs/snapshot</c>, which applies the hub authorization policy itself.
/// </remarks>
public interface IDashboardSnapshotFeed
{
/// <summary>
/// Watches the shared snapshot stream. Each caller gets the snapshots produced while
/// it is subscribed; a caller that reads slowly sees only the newest snapshot rather
/// than a backlog, and never delays the other subscribers.
/// </summary>
/// <param name="cancellationToken">Token that ends this caller's subscription.</param>
/// <returns>An asynchronous stream of dashboard snapshots.</returns>
IAsyncEnumerable<DashboardSnapshot> WatchAsync(CancellationToken cancellationToken);
}
@@ -15,6 +15,28 @@ public static class GatewayLogRedactor
"WriteSecured2"
};
/// <summary>
/// Authorization schemes whose name may survive redaction. Anything outside this list is
/// dropped whole: an unrecognized leading word is as likely to be credential material as it
/// is to be a scheme, so it is not worth the leak.
/// </summary>
private static readonly string[] KnownAuthorizationSchemes =
[
"Bearer",
"Basic",
"Digest",
"Negotiate",
"NTLM",
"ApiKey",
"Token",
];
/// <summary>Prefix identifying a gateway-issued API key.</summary>
private const string GatewayKeyPrefix = "mxgw_";
/// <summary>Upper bound on a key id kept in the clear; a longer run is treated as secret material.</summary>
private const int MaxKeyIdLength = 64;
/// <summary>
/// Determines whether a command method bears credentials.
/// </summary>
@@ -27,44 +49,24 @@ public static class GatewayLogRedactor
}
/// <summary>
/// Redacts the API key secret portion of a Bearer authorization header.
/// Redacts the credential portion of an authorization header value.
/// </summary>
/// <param name="authorizationHeader">The authorization header value to redact.</param>
/// <returns>The header with the secret portion redacted, or the original value when it is null, blank, or not a Bearer header.</returns>
/// <returns>The header with the credential redacted, or the original value when it is null or blank.</returns>
public static string? RedactApiKey(string? authorizationHeader)
{
if (string.IsNullOrWhiteSpace(authorizationHeader))
{
return authorizationHeader;
}
const string bearerPrefix = "Bearer ";
if (!authorizationHeader.StartsWith(bearerPrefix, StringComparison.OrdinalIgnoreCase))
{
return RedactedValue;
}
string token = authorizationHeader[bearerPrefix.Length..].Trim();
if (!token.StartsWith("mxgw_", StringComparison.OrdinalIgnoreCase))
{
return $"{bearerPrefix}{RedactedValue}";
}
string[] tokenParts = token.Split('_', 3, StringSplitOptions.RemoveEmptyEntries);
if (tokenParts.Length < 2)
{
return $"{bearerPrefix}mxgw_{RedactedValue}";
}
return $"{bearerPrefix}mxgw_{tokenParts[1]}_{RedactedValue}";
return RedactClientIdentity(authorizationHeader);
}
/// <summary>
/// Redacts the client identity if it contains an API key.
/// Redacts the credential carried by a client identity. Redaction fails closed: only a
/// gateway-issued API key keeps its <c>mxgw_&lt;key-id&gt;_</c> shape (so operators can tell keys
/// apart in logs), and only a recognized scheme keeps its name. Every other value — a foreign
/// bearer token, a scheme-less string, junk — is replaced whole, because nothing that reaches
/// this method is known to be safe to log.
/// </summary>
/// <param name="clientIdentity">The client identity string to redact.</param>
/// <returns>The redacted client identity, or the original value when it contains no API key.</returns>
/// <returns>The redacted client identity, or the original value when it is null or blank.</returns>
public static string? RedactClientIdentity(string? clientIdentity)
{
if (string.IsNullOrWhiteSpace(clientIdentity))
@@ -72,9 +74,61 @@ public static class GatewayLogRedactor
return clientIdentity;
}
return clientIdentity.Contains("mxgw_", StringComparison.OrdinalIgnoreCase)
? RedactApiKey(clientIdentity)
: clientIdentity;
ReadOnlySpan<char> value = clientIdentity.AsSpan().Trim();
int separatorIndex = value.IndexOf(' ');
if (separatorIndex < 0)
{
// A single token carries no scheme, so the token itself is the credential.
return RedactedValue;
}
ReadOnlySpan<char> scheme = value[..separatorIndex];
ReadOnlySpan<char> credential = value[(separatorIndex + 1)..].Trim();
if (credential.IsEmpty || !IsKnownAuthorizationScheme(scheme))
{
return RedactedValue;
}
return credential.StartsWith(GatewayKeyPrefix, StringComparison.OrdinalIgnoreCase)
? $"{scheme} {GatewayKeyPrefix}{RedactKeyId(credential)}"
: $"{scheme} {RedactedValue}";
}
/// <summary>
/// Renders the trailing portion of a gateway API key: the key id when the key is well formed,
/// otherwise nothing but the placeholder.
/// </summary>
/// <param name="credential">The credential, known to start with the gateway key prefix.</param>
/// <returns>The <c>&lt;key-id&gt;_[redacted]</c> tail, or just the placeholder.</returns>
private static string RedactKeyId(ReadOnlySpan<char> credential)
{
ReadOnlySpan<char> remainder = credential[GatewayKeyPrefix.Length..];
int secretIndex = remainder.IndexOf('_');
// No separator means no secret boundary to trust, so the whole remainder is treated as secret.
return secretIndex is <= 0 or > MaxKeyIdLength
? RedactedValue
: $"{remainder[..secretIndex]}_{RedactedValue}";
}
/// <summary>
/// Determines whether a leading word is a recognized authorization scheme.
/// </summary>
/// <param name="scheme">The candidate scheme word.</param>
/// <returns><see langword="true"/> when the word may survive redaction; otherwise <see langword="false"/>.</returns>
private static bool IsKnownAuthorizationScheme(ReadOnlySpan<char> scheme)
{
foreach (string knownScheme in KnownAuthorizationSchemes)
{
if (scheme.Equals(knownScheme, StringComparison.OrdinalIgnoreCase))
{
return true;
}
}
return false;
}
/// <summary>
@@ -24,12 +24,16 @@ public static class GatewayRequestLoggingMiddlewareExtensions
{
ArgumentNullException.ThrowIfNull(app);
// Resolved once at registration: the logger is keyed by category, not by request, so the
// per-request DI resolve and logger-factory lock bought nothing.
ILogger logger = app.ApplicationServices
.GetRequiredService<ILoggerFactory>()
.CreateLogger("MxGateway.Request");
return app.Use(async (context, next) =>
{
ILogger logger = context.RequestServices
.GetRequiredService<ILoggerFactory>()
.CreateLogger("MxGateway.Request");
// Scope construction is deliberately unconditional: gating it on IsEnabled would drop
// scope state for providers (and scope consumers) registered after startup.
using IDisposable? scope = logger.BeginGatewayScope(new GatewayLogScope(
SessionId: ReadHeader(context, SessionIdHeaderName),
WorkerProcessId: ReadInt32Header(context, WorkerProcessIdHeaderName),
@@ -0,0 +1,114 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Sessions;
namespace ZB.MOM.WW.MxGateway.Server.Diagnostics;
/// <summary>
/// Reports how many MXAccess sessions are healthy. Each session is one worker process holding one
/// MXAccess COM instance — a live connection into a Galaxy — so this is the "how many Galaxy
/// connections are healthy" probe, expressed in the vocabulary the code actually uses.
/// </summary>
/// <remarks>
/// <para>
/// <b>Zero sessions is healthy, deliberately.</b> The gateway is a server: it opens a session when
/// a client asks and holds none otherwise, so idle-with-no-clients is the normal steady state, not
/// a fault. A count-based rule ("unhealthy below N") would sit red forever on a host nothing dials
/// yet, and a probe that is permanently red is one people learn to ignore — which costs more than
/// having no probe. The status here is therefore false only when a session exists and its worker
/// has actually failed.
/// </para>
/// <para>
/// This is tagged <c>active</c> rather than <c>ready</c> for the same reason. Readiness gates
/// whether the process should receive traffic, and a gateway with no sessions is legitimately ready
/// to serve — unlike the auth store, which every call depends on (see
/// <see cref="AuthStoreHealthCheck"/>). Failing readiness on session state would take a working
/// gateway out of rotation for a condition its own clients cause.
/// </para>
/// </remarks>
public sealed class SessionHealthCheck : IHealthCheck
{
private readonly ISessionRegistry _sessionRegistry;
/// <summary>Initializes a new instance of the <see cref="SessionHealthCheck"/> class.</summary>
/// <param name="sessionRegistry">Registry holding the live sessions.</param>
public SessionHealthCheck(ISessionRegistry sessionRegistry) =>
_sessionRegistry = sessionRegistry ?? throw new ArgumentNullException(nameof(sessionRegistry));
/// <summary>Buckets the live sessions by state and grades the result.</summary>
/// <param name="context">The health check context.</param>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <returns>
/// Healthy when nothing is faulted (including when no sessions are open), Degraded when some
/// sessions are faulted but others are still usable, and Unhealthy when every session is
/// faulted.
/// </returns>
public Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default)
{
int ready = 0;
int faulted = 0;
int starting = 0;
int closing = 0;
foreach (GatewaySession session in _sessionRegistry.Snapshot())
{
switch (session.State)
{
case SessionState.Ready:
ready++;
break;
case SessionState.Faulted:
faulted++;
break;
case SessionState.Closing:
case SessionState.Closed:
// Counted but excluded from the verdict: a session on its way out is an
// expected lifecycle stage, not a failure, and Snapshot() still returns
// Closed sessions until they are removed from the registry.
closing++;
break;
default:
// Creating / StartingWorker / WaitingForPipe / Handshaking /
// InitializingWorker — mid-startup, not yet usable but not wrong.
// Unspecified lands here too; it is the proto zero value and should not occur.
starting++;
break;
}
}
int total = ready + faulted + starting + closing;
int usable = ready + starting;
Dictionary<string, object> data = new(StringComparer.Ordinal)
{
["total"] = total,
["ready"] = ready,
["faulted"] = faulted,
["starting"] = starting,
["closing"] = closing,
};
HealthCheckResult result = (faulted, usable) switch
{
(0, _) => HealthCheckResult.Healthy(Describe(total, ready, faulted), data),
(_, 0) => HealthCheckResult.Unhealthy(Describe(total, ready, faulted), data: data),
_ => HealthCheckResult.Degraded(Describe(total, ready, faulted), data: data),
};
return Task.FromResult(result);
}
private static string Describe(int total, int ready, int faulted)
{
if (total == 0)
{
return "No MXAccess sessions are open.";
}
return faulted == 0
? $"{ready} of {total} MXAccess sessions ready."
: $"{ready} of {total} MXAccess sessions ready, {faulted} faulted.";
}
}
@@ -70,15 +70,27 @@ public static class GatewayApplication
});
StaticWebAssetsLoader.UseStaticWebAssets(builder.Environment, builder.Configuration);
ApplyDefaultSecretsStorePath(builder.Configuration);
// Resolve ${secret:...} references in configuration BEFORE any config consumer (TLS, Kestrel,
// GatewayOptions/Ldap/Galaxy validators) reads a value, using a standalone secrets provider
// (envelope-decrypted via the master key). A token referencing a missing secret fails fast
// here (SecretNotFoundException); config with no tokens is untouched (no-op), so this is safe
// to always run. CreateBuilder is synchronous and single-shot at bootstrap, so the two awaits
// are driven via GetAwaiter().GetResult() (no sync-context deadlock risk during host startup).
// The content root is passed explicitly because this container is a throwaway
// ServiceCollection with no IHostEnvironment in it. Without it the library cannot tell
// "no content root exists" from "no host is registered", so it skips the
// under-content-root rule — and the migrator below CREATES the store before the real host
// ever validates. The boot then fails a moment later, having already left an empty
// database with its -wal/-shm siblings at the very path the rule rejects. That artifact is
// what made the 2026-08-09 outage read as "the database is there, it's just empty".
// DO NOT simplify this to the 3-argument overload: it still compiles, the app still boots
// when the path is correct, and the guard silently stops running at the one moment that
// matters.
#pragma warning disable ASP0000 // deliberate throwaway container, disposed here, shares no singletons
using (var secretsProvider = new ServiceCollection()
.AddZbSecrets(builder.Configuration, "Secrets")
.AddZbSecrets(builder.Configuration, "Secrets", builder.Environment.ContentRootPath)
.BuildServiceProvider())
#pragma warning restore ASP0000
{
@@ -106,7 +118,13 @@ public static class GatewayApplication
.AddTypeActivatedCheck<AuthStoreHealthCheck>(
"auth-store",
failureStatus: null,
tags: new[] { ZbHealthTags.Ready });
tags: new[] { ZbHealthTags.Ready })
// Active, not Ready: a gateway holding no sessions is legitimately ready to serve.
// See SessionHealthCheck for why zero sessions is healthy.
.AddTypeActivatedCheck<SessionHealthCheck>(
"mxaccess-sessions",
failureStatus: null,
tags: new[] { ZbHealthTags.Active });
builder.Services.AddSingleton<GatewayMetrics>();
builder.AddZbTelemetry(o =>
{
@@ -180,6 +198,60 @@ public static class GatewayApplication
});
}
/// <summary>
/// Supplies the default location of the encrypted secrets store when nothing configured one.
/// </summary>
/// <remarks>
/// <para>
/// The store used to default to a bare relative <c>mxgateway-secrets.db</c>, which resolves
/// against the working directory and therefore normally lands inside the application directory.
/// That is the shape that lost every API key on a production host: the upgrade procedure renames
/// the application directory away, the store goes with it, and a fresh empty one appears in its
/// place with no error. In development the same default writes a database into the source tree.
/// </para>
/// <para>
/// This sets a default for an <em>unset</em> key; it never relocates a value someone configured.
/// That distinction matters — <see cref="Configuration.GatewayConfigPathRules"/> deliberately
/// rejects bad configured paths rather than quietly moving them, because silently relocating a
/// credential store is worse than a boot error. Choosing where to put a value nobody specified
/// is a different act from overriding one they did.
/// </para>
/// <para>
/// The location mirrors <c>AuthenticationOptions.SqlitePath</c> so both gateway stores sit
/// together, and the mechanism is the one SEC-33 already used for
/// <c>MxGateway:Galaxy:SnapshotCachePath</c> below — same problem, same fix, same file. It also
/// matches what <c>docs/GatewayConfiguration.md</c> already tells operators to
/// pass to the <c>secret</c> CLI — an absolute default also removes the CLI/gateway divergence
/// that a working-directory-relative path can cause. On non-Windows hosts
/// <see cref="Environment.SpecialFolder.CommonApplicationData"/> is typically not writable by a
/// normal user, so a local run there must set <c>Secrets__SqlitePath</c> explicitly, exactly as
/// it already must for the auth store.
/// </para>
/// <para>
/// <b>This deliberately differs from the <c>ZB.MOM.WW.Secrets</c> library default</b>, which is
/// <see cref="Environment.SpecialFolder.LocalApplicationData"/>-derived so the family's
/// cross-platform apps still boot locally without an override. The gateway keeps
/// <c>CommonApplicationData</c> because it runs as a machine-wide Windows service and its other
/// two stores — the auth database and the Galaxy snapshot — already live there; splitting them
/// would be the greater inconsistency. The value set here always wins, so the library default is
/// unreachable in this app. Do not "fix" the difference by deleting this method: that would
/// silently move the store, which is the failure this whole rule exists to prevent.
/// </para>
/// </remarks>
/// <param name="configuration">The configuration to supply the default into.</param>
private static void ApplyDefaultSecretsStorePath(IConfiguration configuration)
{
if (!string.IsNullOrWhiteSpace(configuration["Secrets:SqlitePath"]))
{
return;
}
configuration["Secrets:SqlitePath"] = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"MxGateway",
"mxgateway-secrets.db");
}
private static void ConfigureSelfSignedTls(WebApplicationBuilder builder)
{
if (!Security.Tls.KestrelTlsInspector.RequiresGeneratedCertificate(builder.Configuration))
@@ -1,4 +1,5 @@
using System.Runtime.CompilerServices;
using System.Threading.Channels;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Configuration;
@@ -37,7 +38,7 @@ public sealed class EventStreamService(
// non-blocking. When this subscriber's channel is full the pump applies the per-subscriber
// backpressure policy and completes this subscriber's channel with a SessionManagerException
// (SessionManagerErrorCode.EventQueueOverflow). That terminal fault surfaces here when the
// reader's MoveNextAsync throws, and it propagates to the gRPC client unchanged. The overflow
// reader's WaitToReadAsync throws, and it propagates to the gRPC client unchanged. The overflow
// metric, and (in the legacy single-subscriber FailFast case) the session fault + fault metric,
// are recorded by the distributor's overflow handler so the session, the pump, and other
// subscribers are isolated from this subscriber's slowness.
@@ -106,9 +107,14 @@ public sealed class EventStreamService(
options.Value.Sessions.MaxEventSubscribersPerSession);
}
IAsyncEnumerator<MxEvent> reader = subscriber.Reader
.ReadAllAsync(cancellationToken)
.GetAsyncEnumerator(cancellationToken);
// Consume the subscriber channel directly (WaitToReadAsync + an inner TryRead drain)
// rather than through ReadAllAsync's IAsyncEnumerable wrapper. This is the hottest
// per-event path in the gateway and the wrapper added a second async state machine hop
// per event for no behavioral benefit: WaitToReadAsync observes cancellation and a
// faulted completion exactly as MoveNextAsync did, and TryRead drains what is already
// buffered without allocating a wait. StreamEventsAsync itself stays an async iterator —
// its `yield return` is what feeds the gRPC writer.
ChannelReader<MxEvent> reader = subscriber.Reader;
// GWC-15: register this subscriber's channel as a live backlog source instead of
// reconciling the queue-depth gauge on every event. The gauge previously read the
@@ -151,15 +157,14 @@ public sealed class EventStreamService(
while (true)
{
MxEvent mxEvent;
bool hasMore;
try
{
if (!await reader.MoveNextAsync().ConfigureAwait(false))
{
break;
}
mxEvent = reader.Current;
// A cleanly completed channel returns false here (end of stream); a channel
// completed WITH a fault rethrows that fault from the wait once the buffer
// is drained — the same surface MoveNextAsync presented, so the terminal
// SessionManagerException(EventQueueOverflow) still propagates unchanged.
hasMore = await reader.WaitToReadAsync(cancellationToken).ConfigureAwait(false);
}
catch (WorkerClientException workerException)
{
@@ -173,24 +178,36 @@ public sealed class EventStreamService(
throw;
}
// Per-RPC filter stays at the subscriber boundary: each request may resume
// from a different AfterWorkerSequence, so the shared pump fans raw events and
// this loop drops the ones at or below the caller's watermark.
if (mxEvent.WorkerSequence <= afterWorkerSequence)
if (!hasMore)
{
continue;
break;
}
// The queue-depth gauge is maintained lazily via the backlog registration above
// (GWC-15): the metric reads this subscriber's channel Count only when scraped,
// so there is no per-event gauge bookkeeping on this hot path.
yield return mxEvent;
// Drain everything already buffered before waiting again. TryRead never throws;
// a fault left on the channel is observed by the next WaitToReadAsync above.
while (reader.TryRead(out MxEvent? mxEvent))
{
// Per-RPC filter stays at the subscriber boundary: each request may resume
// from a different AfterWorkerSequence, so the shared pump fans raw events
// and this loop drops the ones at or below the caller's watermark. It
// applies to every live event, drained or awaited alike.
if (mxEvent.WorkerSequence <= afterWorkerSequence)
{
continue;
}
// The queue-depth gauge is maintained lazily via the backlog registration
// above (GWC-15): the metric reads this subscriber's channel Count only when
// scraped, so there is no per-event gauge bookkeeping on this hot path.
yield return mxEvent;
}
}
}
finally
{
await reader.DisposeAsync().ConfigureAwait(false);
// Nothing to dispose for the reader: consuming the ChannelReader directly means
// there is no enumerator wrapper holding the cancellation registration.
//
// Remove this subscriber's live backlog contribution before disposing the lease so
// the gauge stops counting a channel that is about to be completed; after this the
// gauge reflects only the remaining subscribers (zero when none remain).
@@ -101,6 +101,12 @@ public sealed class MxAccessGatewayService(
try
{
requestValidator.ValidateInvoke(request);
// PERF(followup): this resolve and the sessionManager.InvokeAsync below look the same
// session up twice (a dictionary hit each, so measured cost is negligible). Collapsing
// them needs a SessionManager overload taking an already-resolved GatewaySession, which
// would duplicate InvokeAsync's fault mapping (SessionNotFound / state checks / metrics)
// at a second entry point — deliberately not worth it until a profile says otherwise.
GatewaySession session = ResolveSession(request.SessionId);
MxCommand command = request.Command;
BulkConstraintPlan? bulkConstraintPlan = await ApplyConstraintsAsync(
@@ -461,6 +467,14 @@ public sealed class MxAccessGatewayService(
string? correlationId,
CancellationToken cancellationToken)
{
// An identity with no read constraints allows every tag, so the per-item enforcer call below
// can only answer "allowed" — the whole loop (and the plan it would build) is dead work.
// Returning null is exactly what the denied.Count == 0 exit below returns.
if (!constraintEnforcer.HasReadConstraints(identity))
{
return null;
}
Dictionary<int, SubscribeResult> denied = [];
List<string> allowed = [];
for (int index = 0; index < tagAddresses.Count; index++)
@@ -491,16 +505,23 @@ public sealed class MxAccessGatewayService(
return null;
}
MxCommand filtered = command.Clone();
if (filtered.Kind == MxCommandKind.AddItemBulk)
// Build the filtered command directly instead of cloning the original and clearing it:
// the clone deep-copied every denied address only to drop it. The payload's other fields
// (server_handle) are copied across explicitly. Nothing aliases the request here — these
// bulk payloads carry only strings — and the worker-bound graph is still the unaliased copy
// MapCommand makes.
MxCommand filtered = new() { Kind = command.Kind };
if (command.Kind == MxCommandKind.AddItemBulk)
{
filtered.AddItemBulk.TagAddresses.Clear();
filtered.AddItemBulk.TagAddresses.Add(allowed);
AddItemBulkCommand payload = new() { ServerHandle = command.AddItemBulk.ServerHandle };
payload.TagAddresses.Add(allowed);
filtered.AddItemBulk = payload;
}
else
{
filtered.SubscribeBulk.TagAddresses.Clear();
filtered.SubscribeBulk.TagAddresses.Add(allowed);
SubscribeBulkCommand payload = new() { ServerHandle = command.SubscribeBulk.ServerHandle };
payload.TagAddresses.Add(allowed);
filtered.SubscribeBulk = payload;
}
return new SubscribeBulkConstraintPlan(filtered, tagAddresses.Count, denied, allowed.Count > 0);
@@ -517,6 +538,11 @@ public sealed class MxAccessGatewayService(
// Mirrors FilterTagBulkAsync but produces BulkReadResult denial entries
// so the reply payload merges into BulkReadReply.Results, not
// BulkSubscribeReply.Results.
if (!constraintEnforcer.HasReadConstraints(identity))
{
return null;
}
Dictionary<int, BulkReadResult> denied = [];
List<string> allowed = [];
for (int index = 0; index < tagAddresses.Count; index++)
@@ -548,9 +574,14 @@ public sealed class MxAccessGatewayService(
return null;
}
MxCommand filtered = command.Clone();
filtered.ReadBulk.TagAddresses.Clear();
filtered.ReadBulk.TagAddresses.Add(allowed);
MxCommand filtered = new() { Kind = command.Kind };
ReadBulkCommand payload = new()
{
ServerHandle = command.ReadBulk.ServerHandle,
TimeoutMs = command.ReadBulk.TimeoutMs,
};
payload.TagAddresses.Add(allowed);
filtered.ReadBulk = payload;
return new ReadBulkConstraintPlan(filtered, tagAddresses.Count, denied, allowed.Count > 0);
}
@@ -572,6 +603,11 @@ public sealed class MxAccessGatewayService(
// Parameterising on TEntry + getItemHandle keeps a single filter
// routine for all four and avoids duplicating CheckWriteHandleAsync
// calls.
if (!constraintEnforcer.HasWriteConstraints(identity))
{
return null;
}
Dictionary<int, BulkWriteResult> denied = [];
List<TEntry> allowed = [];
for (int index = 0; index < entries.Count; index++)
@@ -609,33 +645,74 @@ public sealed class MxAccessGatewayService(
return null;
}
MxCommand filtered = command.Clone();
ReplaceWriteBulkEntries(filtered, allowed);
return new WriteBulkConstraintPlan(filtered, entries.Count, denied, allowed.Count > 0);
return new WriteBulkConstraintPlan(
BuildFilteredWriteBulkCommand(command, allowed),
entries.Count,
denied,
allowed.Count > 0);
}
private static void ReplaceWriteBulkEntries<TEntry>(MxCommand command, IReadOnlyList<TEntry> allowed)
/// <summary>
/// Builds the allowed-only bulk-write command. The allowed entries are carried over by
/// reference rather than deep-cloned: the caller only reads this command (TrackCommandReply),
/// and the copy the worker mutates and owns is the one <c>MapCommand</c> clones — the same
/// no-aliasing boundary as before. Cloning the whole command here and clearing it copied
/// every denied entry's payload (including <c>WriteSecured</c> values) for nothing.
/// </summary>
/// <typeparam name="TEntry">The per-family bulk-write entry message type.</typeparam>
/// <param name="command">The original command, read for its kind and payload scalars.</param>
/// <param name="allowed">The entries that survived constraint filtering, in original order.</param>
/// <returns>A command of the same kind carrying only the allowed entries.</returns>
private static MxCommand BuildFilteredWriteBulkCommand<TEntry>(MxCommand command, IReadOnlyList<TEntry> allowed)
where TEntry : class
{
MxCommand filtered = new() { Kind = command.Kind };
switch (command.Kind)
{
case MxCommandKind.WriteBulk:
command.WriteBulk.Entries.Clear();
command.WriteBulk.Entries.Add((IEnumerable<WriteBulkEntry>)allowed);
{
WriteBulkCommand payload = new() { ServerHandle = command.WriteBulk.ServerHandle };
payload.Entries.Add((IEnumerable<WriteBulkEntry>)allowed);
filtered.WriteBulk = payload;
break;
}
case MxCommandKind.Write2Bulk:
command.Write2Bulk.Entries.Clear();
command.Write2Bulk.Entries.Add((IEnumerable<Write2BulkEntry>)allowed);
{
Write2BulkCommand payload = new() { ServerHandle = command.Write2Bulk.ServerHandle };
payload.Entries.Add((IEnumerable<Write2BulkEntry>)allowed);
filtered.Write2Bulk = payload;
break;
}
case MxCommandKind.WriteSecuredBulk:
command.WriteSecuredBulk.Entries.Clear();
command.WriteSecuredBulk.Entries.Add((IEnumerable<WriteSecuredBulkEntry>)allowed);
{
WriteSecuredBulkCommand payload = new() { ServerHandle = command.WriteSecuredBulk.ServerHandle };
payload.Entries.Add((IEnumerable<WriteSecuredBulkEntry>)allowed);
filtered.WriteSecuredBulk = payload;
break;
}
case MxCommandKind.WriteSecured2Bulk:
command.WriteSecured2Bulk.Entries.Clear();
command.WriteSecured2Bulk.Entries.Add((IEnumerable<WriteSecured2BulkEntry>)allowed);
{
WriteSecured2BulkCommand payload = new() { ServerHandle = command.WriteSecured2Bulk.ServerHandle };
payload.Entries.Add((IEnumerable<WriteSecured2BulkEntry>)allowed);
filtered.WriteSecured2Bulk = payload;
break;
}
default:
// Only the four bulk-write kinds above reach FilterWriteBulkAsync, so this is
// unreachable. It throws rather than falling back to the unmodified command,
// because that fallback failed OPEN: a fifth bulk-write kind added upstream
// without a case here would silently ship the DENIED entries to the worker while
// still reporting them denied to the caller. Failing loud on a kind nobody can
// reach today is strictly safer than a constraint bypass nobody would notice.
throw new UnreachableException(
$"Command kind {command.Kind} reached bulk-write constraint filtering without a filter case.");
}
return filtered;
}
private async Task<BulkConstraintPlan?> FilterHandleBulkAsync(
@@ -647,6 +724,11 @@ public sealed class MxAccessGatewayService(
string? correlationId,
CancellationToken cancellationToken)
{
if (!constraintEnforcer.HasReadConstraints(identity))
{
return null;
}
Dictionary<int, SubscribeResult> denied = [];
List<int> allowed = [];
for (int index = 0; index < itemHandles.Count; index++)
@@ -677,9 +759,10 @@ public sealed class MxAccessGatewayService(
return null;
}
MxCommand filtered = command.Clone();
filtered.AdviseItemBulk.ItemHandles.Clear();
filtered.AdviseItemBulk.ItemHandles.Add(allowed);
MxCommand filtered = new() { Kind = command.Kind };
AdviseItemBulkCommand payload = new() { ServerHandle = command.AdviseItemBulk.ServerHandle };
payload.ItemHandles.Add(allowed);
filtered.AdviseItemBulk = payload;
return new SubscribeBulkConstraintPlan(filtered, itemHandles.Count, denied, allowed.Count > 0);
}
@@ -71,7 +71,17 @@ public sealed class MxAccessGrpcMapper
};
}
return reply.Reply.Clone();
// GWC-07 / IPC-05: ownership transfer, not a deep clone — the same rule MapEvent follows,
// applied to the other (and larger, on bulk reads) hot-path message. The enclosing
// WorkerCommandReply is parsed fresh from a single pipe frame in WorkerClient's read loop
// and is single-consumer by construction: CompleteCommand's TryRemove hands it to exactly
// one PendingCommand awaiter, that awaiter is the gRPC Invoke handler, and the handler's
// one call is this mapping. Nothing else aliases or reads reply.Reply afterwards — the
// enclosing WorkerCommandReply is discarded here. We therefore move the inner
// MxCommandReply into the gRPC response instead of copying it; the handler owning it
// outright is also what makes BulkConstraintPlan.MergeDeniedInto's in-place splice safe.
// If a second consumer of the same WorkerCommandReply is ever added, restore a .Clone().
return reply.Reply;
}
/// <summary>
@@ -28,7 +28,9 @@ public sealed class GatewayMetrics : IDisposable
private readonly Histogram<double> _workerStartupLatencyHistogram;
private readonly Histogram<double> _commandLatencyHistogram;
private readonly Histogram<double> _eventStreamSendLatencyHistogram;
private readonly Dictionary<string, long> _commandFailuresByMethod = new(StringComparer.OrdinalIgnoreCase);
// Concurrent (not Dictionary + _syncRoot) because CommandFailed runs on every failing gRPC call:
// the command counters are recorded outside the lock, so their breakdown map must be too.
private readonly ConcurrentDictionary<string, long> _commandFailuresByMethod = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<string, long> _eventsByFamily = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<string, long> _eventsBySession = new(StringComparer.Ordinal);
private readonly Dictionary<string, long> _retryAttemptsByArea = new(StringComparer.OrdinalIgnoreCase);
@@ -41,9 +43,16 @@ public sealed class GatewayMetrics : IDisposable
private readonly ConcurrentDictionary<long, Func<int>> _eventStreamBacklogSources = new();
private long _nextEventStreamBacklogSourceId;
// GWC-30: the same pull model for the worker event queue depth. It replaces a pushed scalar that
// every WorkerClient wrote twice per event (staged, consumed) under _syncRoot — a process-wide
// lock on the hottest path, and last-writer-wins across sessions, so the gauge reported one
// arbitrary session's backlog instead of the gateway's. Each client registers a source returning
// its own undelivered depth; the gauge sums them at collection time only.
private readonly ConcurrentDictionary<long, Func<int>> _workerEventQueueDepthSources = new();
private long _nextWorkerEventQueueDepthSourceId;
private int _openSessions;
private int _workersRunning;
private int _workerEventQueueDepth;
private int _alarmProviderMode;
private long _sessionsOpened;
private long _sessionsClosed;
@@ -201,10 +210,10 @@ public sealed class GatewayMetrics : IDisposable
/// <param name="method">Name of the command method.</param>
public void CommandStarted(string method)
{
lock (_syncRoot)
{
_commandsStarted++;
}
// GWC-30: the three command counters run two-to-three times per gRPC call, so they use
// Interlocked rather than _syncRoot — the same idiom as EventReceived. Nothing here needs a
// consistent multi-field view; GetSnapshot reads each with Interlocked.Read.
Interlocked.Increment(ref _commandsStarted);
_commandsStartedCounter.Add(1, new KeyValuePair<string, object?>("method", method));
}
@@ -216,10 +225,7 @@ public sealed class GatewayMetrics : IDisposable
/// <param name="duration">Elapsed time to complete the command.</param>
public void CommandSucceeded(string method, TimeSpan duration)
{
lock (_syncRoot)
{
_commandsSucceeded++;
}
Interlocked.Increment(ref _commandsSucceeded);
KeyValuePair<string, object?> methodTag = new("method", method);
_commandsSucceededCounter.Add(1, methodTag);
@@ -234,11 +240,8 @@ public sealed class GatewayMetrics : IDisposable
/// <param name="duration">Elapsed time before command failed.</param>
public void CommandFailed(string method, string category, TimeSpan duration)
{
lock (_syncRoot)
{
_commandsFailed++;
Increment(_commandFailuresByMethod, method);
}
Interlocked.Increment(ref _commandsFailed);
Increment(_commandFailuresByMethod, method);
KeyValuePair<string, object?> methodTag = new("method", method);
KeyValuePair<string, object?> categoryTag = new("category", category);
@@ -275,29 +278,24 @@ public sealed class GatewayMetrics : IDisposable
}
/// <summary>
/// Sets the worker event queue depth; delegates to SetWorkerEventQueueDepth.
/// Registers a live depth source for the worker event queue-depth gauge and returns a handle
/// that removes it when disposed. Each <c>WorkerClient</c> registers once and reports its own
/// undelivered (staged + queued) event count, so the gauge is the gateway-wide sum instead of
/// the last value any one session happened to push (GWC-30).
/// </summary>
/// <param name="depth">Queue depth value.</param>
public void SetEventQueueDepth(int depth)
/// <param name="depth">
/// Returns this worker client's current undelivered event count. Invoked only at collection
/// time; must be cheap and non-blocking (a <see cref="Volatile.Read(ref int)"/> of an
/// interlocked counter). Negative readings — which a racing decrement can produce — are
/// clamped to zero when summed.
/// </param>
/// <returns>A handle whose disposal unregisters the source. Safe to dispose more than once.</returns>
public IDisposable RegisterWorkerEventQueueDepthSource(Func<int> depth)
{
SetWorkerEventQueueDepth(depth);
}
/// <summary>
/// Sets the worker event queue depth to the given value.
/// </summary>
/// <param name="depth">Queue depth value.</param>
public void SetWorkerEventQueueDepth(int depth)
{
if (depth < 0)
{
throw new ArgumentOutOfRangeException(nameof(depth), depth, "Queue depth cannot be negative.");
}
lock (_syncRoot)
{
_workerEventQueueDepth = depth;
}
ArgumentNullException.ThrowIfNull(depth);
long id = Interlocked.Increment(ref _nextWorkerEventQueueDepthSourceId);
_workerEventQueueDepthSources[id] = depth;
return new GaugeSourceRegistration(_workerEventQueueDepthSources, id);
}
/// <summary>
@@ -318,7 +316,7 @@ public sealed class GatewayMetrics : IDisposable
ArgumentNullException.ThrowIfNull(backlog);
long id = Interlocked.Increment(ref _nextEventStreamBacklogSourceId);
_eventStreamBacklogSources[id] = backlog;
return new EventStreamBacklogRegistration(this, id);
return new GaugeSourceRegistration(_eventStreamBacklogSources, id);
}
/// <summary>
@@ -460,21 +458,23 @@ public sealed class GatewayMetrics : IDisposable
/// <returns>The current metrics snapshot.</returns>
public GatewayMetricsSnapshot GetSnapshot()
{
// Compute the live gRPC stream backlog outside _syncRoot: the sources are the subscriber
// channels' Count (their own locks) and must not run under this lock. GWC-15.
// Compute the live queue depths outside _syncRoot: the sources are the subscriber channels'
// Count (their own locks) and the worker clients' interlocked counters, neither of which may
// run under this lock. GWC-15, GWC-30.
int workerEventQueueDepth = GetWorkerEventQueueDepth();
int grpcEventStreamQueueDepth = GetGrpcEventStreamQueueDepth();
lock (_syncRoot)
{
return new GatewayMetricsSnapshot(
OpenSessions: _openSessions,
WorkersRunning: _workersRunning,
WorkerEventQueueDepth: _workerEventQueueDepth,
WorkerEventQueueDepth: workerEventQueueDepth,
GrpcEventStreamQueueDepth: grpcEventStreamQueueDepth,
SessionsOpened: _sessionsOpened,
SessionsClosed: _sessionsClosed,
CommandsStarted: _commandsStarted,
CommandsSucceeded: _commandsSucceeded,
CommandsFailed: _commandsFailed,
CommandsStarted: Interlocked.Read(ref _commandsStarted),
CommandsSucceeded: Interlocked.Read(ref _commandsSucceeded),
CommandsFailed: Interlocked.Read(ref _commandsFailed),
EventsReceived: Interlocked.Read(ref _eventsReceived),
QueueOverflows: _queueOverflows,
Faults: _faults,
@@ -521,22 +521,19 @@ public sealed class GatewayMetrics : IDisposable
}
}
private int GetWorkerEventQueueDepth()
{
lock (_syncRoot)
{
return _workerEventQueueDepth;
}
}
// Sums the undelivered event backlog across every live worker client (GWC-30).
private int GetWorkerEventQueueDepth() => SumSources(_workerEventQueueDepthSources);
// Sums the live backlog across every registered event-stream subscriber. Runs at collection
// time (ObservableGauge scrape) or when GetSnapshot projects the value — never on the
// per-event path. Enumerating ConcurrentDictionary.Values never throws on concurrent
// Sums the live backlog across every registered event-stream subscriber.
private int GetGrpcEventStreamQueueDepth() => SumSources(_eventStreamBacklogSources);
// Runs at collection time (ObservableGauge scrape) or when GetSnapshot projects the value —
// never on a per-event path. Enumerating ConcurrentDictionary.Values never throws on concurrent
// register/unregister; a source removed mid-enumeration simply drops from this sample.
private int GetGrpcEventStreamQueueDepth()
private static int SumSources(ConcurrentDictionary<long, Func<int>> sources)
{
int total = 0;
foreach (Func<int> source in _eventStreamBacklogSources.Values)
foreach (Func<int> source in sources.Values)
{
int value = source();
if (value > 0)
@@ -548,11 +545,6 @@ public sealed class GatewayMetrics : IDisposable
return total;
}
private void UnregisterEventStreamBacklogSource(long id)
{
_eventStreamBacklogSources.TryRemove(id, out _);
}
private int GetAlarmProviderMode()
{
lock (_syncRoot)
@@ -572,9 +564,10 @@ public sealed class GatewayMetrics : IDisposable
values.AddOrUpdate(key, 1, static (_, currentValue) => currentValue + 1);
}
// Handle returned by RegisterEventStreamBacklogSource. Disposal (once) removes the source
// from the gauge's live sum. Idempotent so a double dispose from a stream teardown is safe.
private sealed class EventStreamBacklogRegistration(GatewayMetrics metrics, long id) : IDisposable
// Handle returned by the pull-model gauge registrations. Disposal (once) removes the source from
// that gauge's live sum. Idempotent so a double dispose from a stream or worker-client teardown
// is safe, and shared by both gauges so the two registrations cannot drift apart.
private sealed class GaugeSourceRegistration(ConcurrentDictionary<long, Func<int>> sources, long id) : IDisposable
{
private int _disposed;
@@ -582,7 +575,7 @@ public sealed class GatewayMetrics : IDisposable
{
if (Interlocked.Exchange(ref _disposed, 1) == 0)
{
metrics.UnregisterEventStreamBacklogSource(id);
sources.TryRemove(id, out _);
}
}
}
@@ -0,0 +1,281 @@
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.MxGateway.Server.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Security.Audit;
/// <summary>
/// Drains <see cref="ChannelAuditWriter"/> onto the durable <see cref="IAuditEventSink"/>,
/// owns the one-time schema bootstrap, and sweeps audit rows past their retention window.
/// </summary>
/// <remarks>
/// Batching is the point: up to <see cref="MaxBatchSize"/> buffered events are committed in a
/// single transaction, so a burst of constraint denials costs a handful of commits instead of
/// one per denied tag. The bootstrap runs here — before the writer starts enqueueing — so no
/// audit write ever pays a <c>CREATE TABLE IF NOT EXISTS</c> round-trip.
/// <para>
/// Every failure mode ends in synchronous audit rather than silent loss: a batch that will not
/// commit is retried one event at a time so only the offending row is dropped, and a drain loop
/// that dies detaches the writer, which reverts every producer to the direct write path.
/// </para>
/// </remarks>
/// <param name="writer">The channel writer whose buffered events are drained.</param>
/// <param name="sink">The durable sink events are committed to.</param>
/// <param name="security">Security options carrying the audit retention window.</param>
/// <param name="timeProvider">Clock used for the retention cutoff and sweep interval.</param>
/// <param name="logger">Logger for bootstrap, drain and sweep diagnostics.</param>
public sealed class AuditDrainService(
ChannelAuditWriter writer,
IAuditEventSink sink,
SecurityOptions security,
TimeProvider timeProvider,
ILogger<AuditDrainService> logger) : BackgroundService
{
/// <summary>Maximum number of audit events committed in one transaction per drain pass.</summary>
public const int MaxBatchSize = 64;
/// <summary>How often the retention sweep runs while the gateway is up.</summary>
public static readonly TimeSpan RetentionSweepInterval = TimeSpan.FromHours(1);
/// <summary>Upper bound on how long shutdown waits for the remaining buffered events.</summary>
private static readonly TimeSpan ShutdownDrainCap = TimeSpan.FromSeconds(2);
/// <summary>
/// Bootstraps the audit table, runs one retention sweep, then attaches the drain so the
/// writer switches from synchronous write-through to enqueueing.
/// </summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public override async Task StartAsync(CancellationToken cancellationToken)
{
try
{
await sink.EnsureInitializedAsync(cancellationToken).ConfigureAwait(false);
await SweepRetentionAsync(cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
// Audit is best-effort: a bootstrap failure must not take the gateway down. The
// sink's own latch will retry the schema check on the first write.
logger.LogWarning(exception, "Audit store bootstrap failed; audit writes will retry the schema check.");
}
writer.AttachDrain();
await base.StartAsync(cancellationToken).ConfigureAwait(false);
}
/// <summary>
/// Detaches the drain (so late writes go straight to the sink) and gives the buffered
/// events a bounded window to reach the store.
/// </summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public override async Task StopAsync(CancellationToken cancellationToken)
{
writer.DetachDrain();
writer.CompleteWriting();
await base.StopAsync(cancellationToken).ConfigureAwait(false);
using CancellationTokenSource drainCap = new(ShutdownDrainCap);
try
{
await DrainPendingAsync(drainCap.Token).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
logger.LogWarning(
"Shutdown drain exceeded {CapSeconds}s; remaining buffered audit events were not persisted.",
ShutdownDrainCap.TotalSeconds);
}
}
/// <summary>
/// Commits every event currently buffered, in batches of at most <see cref="MaxBatchSize"/>.
/// </summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The number of events persisted.</returns>
/// <exception cref="OperationCanceledException">
/// The drain was cancelled — at shutdown this is the 2-second cap expiring, which the caller
/// reports as unpersisted audit rather than as a store fault.
/// </exception>
public async Task<int> DrainPendingAsync(CancellationToken cancellationToken)
{
int persisted = 0;
List<AuditEvent> batch = new(MaxBatchSize);
while (!cancellationToken.IsCancellationRequested)
{
batch.Clear();
while (batch.Count < MaxBatchSize && writer.Reader.TryRead(out AuditEvent? auditEvent))
{
batch.Add(auditEvent);
}
if (batch.Count == 0)
{
break;
}
try
{
await sink.InsertBatchAsync(batch, cancellationToken).ConfigureAwait(false);
persisted += batch.Count;
}
catch (OperationCanceledException)
{
// Cancellation is the shutdown cap, not a store fault: surface it so StopAsync
// reports unpersisted audit instead of misreporting it as a failed write.
throw;
}
catch (Exception exception)
{
logger.LogWarning(
exception,
"Failed to commit a batch of {Count} audit events; retrying them individually.",
batch.Count);
persisted += await InsertIndividuallyAsync(batch, cancellationToken).ConfigureAwait(false);
}
}
return persisted;
}
/// <summary>
/// Deletes audit rows older than <c>MxGateway:Security:AuditRetentionDays</c>, and reports the
/// running total of audit events dropped by channel pressure since startup.
/// </summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task SweepRetentionAsync(CancellationToken cancellationToken)
{
DateTimeOffset cutoff = timeProvider.GetUtcNow() - TimeSpan.FromDays(security.AuditRetentionDays);
try
{
int deleted = await sink.DeleteOlderThanAsync(cutoff, cancellationToken).ConfigureAwait(false);
if (deleted > 0)
{
logger.LogInformation(
"Audit retention sweep removed {Deleted} events older than {Cutoff:o} ({RetentionDays} days).",
deleted,
cutoff,
security.AuditRetentionDays);
}
}
catch (Exception exception)
{
logger.LogWarning(exception, "Audit retention sweep failed; it will be retried on the next interval.");
}
long dropped = writer.DroppedCount;
if (dropped > 0)
{
logger.LogWarning(
"{Dropped} audit events have been dropped since startup because the audit channel was full.",
dropped);
}
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await Task.WhenAll(
DrainLoopAsync(stoppingToken),
RetentionLoopAsync(stoppingToken)).ConfigureAwait(false);
}
// Re-inserts a failed batch one event at a time so a single unwritable row costs only itself
// rather than the up-to-MaxBatchSize good events that happened to share its transaction.
private async Task<int> InsertIndividuallyAsync(
IReadOnlyList<AuditEvent> batch,
CancellationToken cancellationToken)
{
int persisted = 0;
foreach (AuditEvent auditEvent in batch)
{
try
{
await sink.InsertAsync(auditEvent, cancellationToken).ConfigureAwait(false);
persisted++;
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception exception)
{
logger.LogWarning(
exception,
"Dropped audit event {EventId} (action {Action}); it could not be persisted individually.",
auditEvent.EventId,
auditEvent.Action);
}
}
logger.LogWarning(
"Recovered {Persisted} of {Count} audit events from a failed batch.",
persisted,
batch.Count);
return persisted;
}
private async Task DrainLoopAsync(CancellationToken stoppingToken)
{
try
{
while (await writer.Reader.WaitToReadAsync(stoppingToken).ConfigureAwait(false))
{
await DrainPendingAsync(stoppingToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
{
// Shutdown; StopAsync performs the final bounded drain.
}
catch (Exception exception)
{
// A dead drain loop would silently discard every later audit write, because producers
// keep enqueueing into a channel nobody reads. Detaching (below) reverts them to the
// synchronous path, so audit degrades in latency rather than disappearing.
logger.LogError(exception, "Audit drain loop failed; reverting to synchronous audit writes.");
}
finally
{
writer.DetachDrain();
// Detaching alone leaves a racer that already passed the attached check enqueueing into
// a channel this loop will never read again — those events would sit in the buffer until
// StopAsync's final drain. Completing the writer as well makes that racer's TryWrite
// return false, which is the write-through branch, so the event reaches the store now.
// TryComplete is idempotent, so StopAsync's own CompleteWriting stays safe either way.
writer.CompleteWriting();
}
}
private async Task RetentionLoopAsync(CancellationToken stoppingToken)
{
try
{
using PeriodicTimer timer = new(RetentionSweepInterval, timeProvider);
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
{
await SweepRetentionAsync(stoppingToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
{
// Shutdown.
}
catch (Exception exception)
{
// Retention is unbounded growth if it stops: say so loudly rather than letting the
// audit table grow forever behind a silently dead timer loop.
logger.LogError(exception, "Audit retention loop failed; expired audit rows will no longer be swept.");
}
}
}
@@ -3,24 +3,26 @@ using ZB.MOM.WW.Audit;
namespace ZB.MOM.WW.MxGateway.Server.Security.Audit;
/// <summary>
/// Best-effort <see cref="IAuditWriter"/> over the MxGateway-owned
/// <see cref="SqliteCanonicalAuditStore"/>. It honours the canonical
/// Best-effort, <em>synchronous</em> <see cref="IAuditWriter"/> over the MxGateway-owned
/// <see cref="IAuditEventSink"/>. It honours the canonical
/// <see cref="IAuditWriter"/> contract: a failed audit write is swallowed and logged
/// rather than propagated, so it can never abort the user-facing action that produced it.
/// </summary>
/// <remarks>
/// This is the single sink through which ALL MxGateway audit flows — the library admin
/// verbs (via <see cref="CanonicalForwardingApiKeyAuditStore"/>) and the gateway's own
/// dashboard / constraint-denial producers, which write canonical events directly. The
/// best-effort wrapping here also closes the gap that the library's
/// This is the durable bottom of the audit pipeline. Callers reach it two ways: through
/// <see cref="ChannelAuditWriter"/> — the registered <see cref="IAuditWriter"/>, which
/// enqueues and lets <see cref="AuditDrainService"/> batch events onto the sink — and
/// directly, when there is no drain to batch behind (the <c>apikey</c> CLI, and any host
/// shutdown window), where writing through immediately is the only way the event survives.
/// The best-effort wrapping here also closes the gap that the library's
/// <c>SqliteApiKeyAuditStore.AppendAsync</c> propagated exceptions.
/// </remarks>
public sealed class CanonicalAuditWriter(
SqliteCanonicalAuditStore store,
IAuditEventSink sink,
ILogger<CanonicalAuditWriter> logger) : IAuditWriter
{
/// <summary>
/// Persists a canonical audit event to the underlying <see cref="SqliteCanonicalAuditStore"/>.
/// Persists a canonical audit event to the underlying <see cref="IAuditEventSink"/>.
/// Any failure is caught, logged, and swallowed rather than propagated to the caller.
/// </summary>
/// <param name="auditEvent">The canonical audit event to persist.</param>
@@ -32,7 +34,7 @@ public sealed class CanonicalAuditWriter(
try
{
await store.InsertAsync(auditEvent, cancellationToken).ConfigureAwait(false);
await sink.InsertAsync(auditEvent, cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
@@ -0,0 +1,139 @@
using System.Threading.Channels;
using ZB.MOM.WW.Audit;
namespace ZB.MOM.WW.MxGateway.Server.Security.Audit;
/// <summary>
/// Bounded, non-blocking <see cref="IAuditWriter"/>: <see cref="WriteAsync"/> enqueues onto a
/// fixed-capacity channel and returns, leaving <see cref="AuditDrainService"/> to batch the
/// events onto the durable <see cref="IAuditEventSink"/>.
/// </summary>
/// <remarks>
/// The canonical <see cref="IAuditWriter"/> contract is already best-effort — a failed audit
/// write is swallowed rather than propagated. The channel makes the <em>bound</em> on that
/// promise explicit: audit can cost the calling RPC at most one enqueue, never a SQLite
/// round-trip, and at most <see cref="ChannelCapacity"/> events of memory. This matters on the
/// constraint-denial path, where a partially denied bulk RPC previously awaited one insert per
/// denied tag, serially, against the same database file the authentication hot path reads.
/// <para>
/// When the channel is full the newest write is dropped (<see cref="BoundedChannelFullMode.DropWrite"/>)
/// and counted in <see cref="DroppedCount"/>. Dropping is the deliberate choice over blocking:
/// a stalled audit database must degrade audit completeness, not stall the gateway. Drops are
/// logged, and <see cref="AuditDrainService"/> reports the running total on its sweep.
/// </para>
/// <para>
/// Until a drain attaches (<see cref="AttachDrain"/>), and again after it detaches, writes go
/// straight through to <see cref="CanonicalAuditWriter"/>. Enqueueing into a channel nobody will
/// ever read would silently discard audit in the processes that have no hosted services — the
/// <c>apikey</c> admin CLI and the DI-only tests — so those keep the original synchronous path.
/// The same fallback covers a completed channel, so no combination of attach/detach can leave
/// producers writing into a buffer that will never be read.
/// </para>
/// </remarks>
public sealed class ChannelAuditWriter : IAuditWriter
{
/// <summary>
/// Maximum number of audit events buffered before writes start being dropped. Sized to
/// absorb a fully denied bulk RPC (the gateway's bulk request cap) plus headroom, so a
/// realistic burst is buffered rather than lost.
/// </summary>
public const int ChannelCapacity = 4096;
private readonly CanonicalAuditWriter _directWriter;
private readonly ILogger<ChannelAuditWriter> _logger;
private readonly Channel<AuditEvent> _channel;
private long _droppedCount;
private int _drainAttached;
private int _dropLogged;
/// <summary>Creates the writer and its bounded buffer.</summary>
/// <param name="directWriter">The synchronous writer used when no drain is attached.</param>
/// <param name="logger">Logger for drop diagnostics.</param>
public ChannelAuditWriter(CanonicalAuditWriter directWriter, ILogger<ChannelAuditWriter> logger)
{
_directWriter = directWriter;
_logger = logger;
// DropWrite discards the incoming item and still reports success to the producer, so the
// itemDropped callback is the only place a drop can be observed and counted.
_channel = Channel.CreateBounded<AuditEvent>(
new BoundedChannelOptions(ChannelCapacity)
{
FullMode = BoundedChannelFullMode.DropWrite,
SingleReader = true,
SingleWriter = false,
},
itemDropped: RecordDrop);
}
/// <summary>Gets the number of audit events dropped because the channel was full.</summary>
public long DroppedCount => Interlocked.Read(ref _droppedCount);
/// <summary>Gets the reader the drain service consumes buffered events from.</summary>
public ChannelReader<AuditEvent> Reader => _channel.Reader;
/// <summary>
/// Marks a drain as running, so subsequent writes enqueue instead of writing through.
/// Called by <see cref="AuditDrainService"/> once its one-time bootstrap has completed.
/// </summary>
public void AttachDrain() => Volatile.Write(ref _drainAttached, 1);
/// <summary>
/// Marks the drain as no longer running, so writes revert to the synchronous path. Called at
/// shutdown, and whenever the drain loop dies, so late audit is still persisted rather than
/// buffered into a channel with no reader.
/// </summary>
public void DetachDrain() => Volatile.Write(ref _drainAttached, 0);
/// <summary>
/// Enqueues a canonical audit event for the drain to persist. Never blocks and never throws.
/// It also never touches the store on the caller's thread while a drain is attached — with one
/// exception: once the channel has been completed (shutdown, or a drain loop that died), the
/// enqueue fails and this falls through to the synchronous write, which is what keeps the event
/// rather than stranding it in a buffer nobody reads.
/// </summary>
/// <param name="auditEvent">The canonical audit event to persist.</param>
/// <param name="cancellationToken">Token honoured only by the direct write-through path.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public Task WriteAsync(AuditEvent auditEvent, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(auditEvent);
if (Volatile.Read(ref _drainAttached) == 0)
{
return _directWriter.WriteAsync(auditEvent, cancellationToken);
}
// Under DropWrite a full channel still reports SUCCESS — the discard surfaces through the
// itemDropped callback. So a false here does not mean "full", it means the channel has
// been completed and no drain will ever read it again (shutdown, or a re-attach onto a
// dead channel). Writing through is the only outcome that keeps the event.
if (!_channel.Writer.TryWrite(auditEvent))
{
return _directWriter.WriteAsync(auditEvent, cancellationToken);
}
return Task.CompletedTask;
}
/// <summary>Signals that no further events will be enqueued, so the drain loop can finish.</summary>
public void CompleteWriting() => _channel.Writer.TryComplete();
private void RecordDrop(AuditEvent auditEvent)
{
Interlocked.Increment(ref _droppedCount);
// Log the first drop only; the running total is reported on the drain's periodic sweep,
// so a sustained overload cannot turn audit pressure into a log flood.
if (Interlocked.Exchange(ref _dropLogged, 1) == 0)
{
_logger.LogWarning(
"Audit channel is full ({Capacity} events); dropping audit event {EventId} (action {Action}). "
+ "Audit is best-effort and bounded; further drops are reported in aggregate.",
ChannelCapacity,
auditEvent.EventId,
auditEvent.Action);
}
}
}
@@ -0,0 +1,38 @@
using ZB.MOM.WW.Audit;
namespace ZB.MOM.WW.MxGateway.Server.Security.Audit;
/// <summary>
/// Durable sink the audit pipeline persists canonical <see cref="AuditEvent"/>s through.
/// It exists so the write path (<see cref="CanonicalAuditWriter"/>) and the batching drain
/// (<see cref="AuditDrainService"/>) depend on the storage contract rather than on the
/// concrete <see cref="SqliteCanonicalAuditStore"/>.
/// </summary>
public interface IAuditEventSink
{
/// <summary>
/// Bootstraps the backing storage. Called once at startup so no write path pays a schema
/// round-trip; implementations must be idempotent and safe to call concurrently.
/// </summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task EnsureInitializedAsync(CancellationToken cancellationToken);
/// <summary>Persists a single canonical audit event.</summary>
/// <param name="auditEvent">The canonical event to persist.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task InsertAsync(AuditEvent auditEvent, CancellationToken cancellationToken);
/// <summary>Persists a batch of canonical audit events as one unit of work.</summary>
/// <param name="auditEvents">The canonical events to persist.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task InsertBatchAsync(IReadOnlyList<AuditEvent> auditEvents, CancellationToken cancellationToken);
/// <summary>Deletes every audit row that occurred strictly before <paramref name="cutoffUtc"/>.</summary>
/// <param name="cutoffUtc">The retention cutoff; rows older than this are removed.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The number of rows deleted.</returns>
Task<int> DeleteOlderThanAsync(DateTimeOffset cutoffUtc, CancellationToken cancellationToken);
}
@@ -18,11 +18,27 @@ namespace ZB.MOM.WW.MxGateway.Server.Security.Audit;
/// <c>IApiKeyAuditStore</c> registration is overridden by
/// <see cref="CanonicalForwardingApiKeyAuditStore"/>, which forwards onto this store via
/// <see cref="CanonicalAuditWriter"/>. The library's <c>schema_version</c> /
/// <c>api_key_audit</c> tables are not touched here; the <c>audit_event</c> table is
/// created idempotently (<c>CREATE TABLE IF NOT EXISTS</c>) on each write so it
/// self-bootstraps regardless of migration ordering.
/// <c>api_key_audit</c> tables are not touched here.
/// <para>
/// The <c>audit_event</c> table is created idempotently, but the <c>CREATE TABLE IF NOT
/// EXISTS</c> is <em>latched</em>: <see cref="AuditDrainService"/> runs
/// <see cref="EnsureInitializedAsync"/> once at startup, and every later insert/list/delete
/// then skips the DDL round-trip. Keeping the (now free) check on each path rather than
/// dropping it means the store still self-bootstraps for callers that use it without the
/// hosted drain — the <c>apikey</c> CLI and the DI-only tests — regardless of migration
/// ordering. The latch is deliberately racy: a lost race merely re-runs an idempotent
/// <c>CREATE TABLE IF NOT EXISTS</c>, and a failure leaves the latch open so the next call
/// retries.
/// </para>
/// </remarks>
public sealed class SqliteCanonicalAuditStore(AuthSqliteConnectionFactory connectionFactory)
/// <param name="connectionFactory">Factory for connections to the shared auth database file.</param>
/// <param name="logger">
/// Optional logger for row-level read diagnostics. Optional because the store is also constructed
/// directly by the <c>apikey</c> CLI path and by DI-free tests, which have no logger to hand.
/// </param>
public sealed class SqliteCanonicalAuditStore(
AuthSqliteConnectionFactory connectionFactory,
ILogger<SqliteCanonicalAuditStore>? logger = null) : IAuditEventSink
{
private const string CreateTableSql =
"""
@@ -40,14 +56,111 @@ public sealed class SqliteCanonicalAuditStore(AuthSqliteConnectionFactory connec
);
""";
/// <summary>Inserts a canonical audit event into the <c>audit_event</c> table.</summary>
/// <param name="auditEvent">The canonical event to persist.</param>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task InsertAsync(AuditEvent auditEvent, CancellationToken cancellationToken)
private const string InsertSql =
"""
INSERT INTO audit_event
(event_id, occurred_at_utc, actor, action, outcome,
category, target, source_node, correlation_id, details_json)
VALUES
($event_id, $occurred_at_utc, $actor, $action, $outcome,
$category, $target, $source_node, $correlation_id, $details_json);
""";
/// <summary>0 until the <c>audit_event</c> table has been created at least once by this instance.</summary>
private int _tableEnsured;
/// <inheritdoc />
public async Task EnsureInitializedAsync(CancellationToken cancellationToken)
{
await using SqliteConnection connection =
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
await EnsureTableAsync(connection, cancellationToken).ConfigureAwait(false);
}
/// <inheritdoc />
public Task InsertAsync(AuditEvent auditEvent, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(auditEvent);
return InsertBatchAsync([auditEvent], cancellationToken);
}
/// <inheritdoc />
/// <remarks>
/// One connection, one transaction and one prepared command for the whole batch: the drain
/// pays a single commit for up to <see cref="AuditDrainService.MaxBatchSize"/> events rather
/// than one round-trip per event.
/// </remarks>
public async Task InsertBatchAsync(IReadOnlyList<AuditEvent> auditEvents, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(auditEvents);
if (auditEvents.Count == 0)
{
return;
}
await using SqliteConnection connection =
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
await EnsureTableAsync(connection, cancellationToken).ConfigureAwait(false);
await using SqliteTransaction transaction =
(SqliteTransaction)await connection.BeginTransactionAsync(cancellationToken).ConfigureAwait(false);
await using (SqliteCommand command = connection.CreateCommand())
{
command.Transaction = transaction;
command.CommandText = InsertSql;
SqliteParameter eventId = command.Parameters.Add("$event_id", SqliteType.Text);
SqliteParameter occurredAtUtc = command.Parameters.Add("$occurred_at_utc", SqliteType.Text);
SqliteParameter actor = command.Parameters.Add("$actor", SqliteType.Text);
SqliteParameter action = command.Parameters.Add("$action", SqliteType.Text);
SqliteParameter outcome = command.Parameters.Add("$outcome", SqliteType.Text);
SqliteParameter category = command.Parameters.Add("$category", SqliteType.Text);
SqliteParameter target = command.Parameters.Add("$target", SqliteType.Text);
SqliteParameter sourceNode = command.Parameters.Add("$source_node", SqliteType.Text);
SqliteParameter correlationId = command.Parameters.Add("$correlation_id", SqliteType.Text);
SqliteParameter detailsJson = command.Parameters.Add("$details_json", SqliteType.Text);
foreach (AuditEvent auditEvent in auditEvents)
{
ArgumentNullException.ThrowIfNull(auditEvent);
eventId.Value = auditEvent.EventId.ToString();
occurredAtUtc.Value = auditEvent.OccurredAtUtc.ToString("O", CultureInfo.InvariantCulture);
actor.Value = auditEvent.Actor;
action.Value = auditEvent.Action;
outcome.Value = auditEvent.Outcome.ToString();
category.Value = (object?)auditEvent.Category ?? DBNull.Value;
target.Value = (object?)auditEvent.Target ?? DBNull.Value;
sourceNode.Value = (object?)auditEvent.SourceNode ?? DBNull.Value;
correlationId.Value = (object?)auditEvent.CorrelationId?.ToString() ?? DBNull.Value;
detailsJson.Value = (object?)auditEvent.DetailsJson ?? DBNull.Value;
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
}
await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
}
/// <inheritdoc />
/// <remarks>
/// The comparison goes through SQLite's <c>datetime()</c> rather than comparing the stored
/// ISO-8601 text directly. Text comparison is only correct while every row is UTC-normalized
/// ISO-8601 — which <see cref="AuditEvent.OccurredAtUtc"/> guarantees for rows written through
/// this store, but not for rows that entered the table any other way (a repair script, an
/// older schema, a future producer). On a mixed-format column a text comparison silently
/// deletes live audit: <c>2026-05-17T09:00:00-05:00</c> is two hours AFTER a
/// <c>2026-05-17T12:00:00+00:00</c> cutoff yet sorts before it. Comparing instants is correct
/// regardless of how the text got there, and anything <c>datetime()</c> cannot parse yields
/// NULL and is therefore never deleted — audit that cannot be dated is kept, not swept.
/// </remarks>
public async Task<int> DeleteOlderThanAsync(DateTimeOffset cutoffUtc, CancellationToken cancellationToken)
{
await using SqliteConnection connection =
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
@@ -56,25 +169,14 @@ public sealed class SqliteCanonicalAuditStore(AuthSqliteConnectionFactory connec
await using SqliteCommand command = connection.CreateCommand();
command.CommandText =
"""
INSERT INTO audit_event
(event_id, occurred_at_utc, actor, action, outcome,
category, target, source_node, correlation_id, details_json)
VALUES
($event_id, $occurred_at_utc, $actor, $action, $outcome,
$category, $target, $source_node, $correlation_id, $details_json);
DELETE FROM audit_event
WHERE datetime(occurred_at_utc) < datetime($cutoff);
""";
command.Parameters.AddWithValue("$event_id", auditEvent.EventId.ToString());
command.Parameters.AddWithValue("$occurred_at_utc", auditEvent.OccurredAtUtc.ToString("O", CultureInfo.InvariantCulture));
command.Parameters.AddWithValue("$actor", auditEvent.Actor);
command.Parameters.AddWithValue("$action", auditEvent.Action);
command.Parameters.AddWithValue("$outcome", auditEvent.Outcome.ToString());
command.Parameters.AddWithValue("$category", (object?)auditEvent.Category ?? DBNull.Value);
command.Parameters.AddWithValue("$target", (object?)auditEvent.Target ?? DBNull.Value);
command.Parameters.AddWithValue("$source_node", (object?)auditEvent.SourceNode ?? DBNull.Value);
command.Parameters.AddWithValue("$correlation_id", (object?)auditEvent.CorrelationId?.ToString() ?? DBNull.Value);
command.Parameters.AddWithValue("$details_json", (object?)auditEvent.DetailsJson ?? DBNull.Value);
command.Parameters.AddWithValue(
"$cutoff",
cutoffUtc.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture));
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
return await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
/// <summary>Returns the most recent canonical audit events, newest first.</summary>
@@ -113,7 +215,7 @@ public sealed class SqliteCanonicalAuditStore(AuthSqliteConnectionFactory connec
events.Add(new AuditEvent
{
EventId = Guid.Parse(reader.GetString(0)),
OccurredAtUtc = ParseUtc(reader.GetString(1)),
OccurredAtUtc = ParseUtcOrMinValue(reader.GetString(1), reader.GetString(0)),
Actor = reader.GetString(2),
Action = reader.GetString(3),
Outcome = Enum.Parse<AuditOutcome>(reader.GetString(4)),
@@ -128,13 +230,42 @@ public sealed class SqliteCanonicalAuditStore(AuthSqliteConnectionFactory connec
return events;
}
private static async Task EnsureTableAsync(SqliteConnection connection, CancellationToken cancellationToken)
// Latched bootstrap: after the first success this is a single volatile read, so the DDL
// round-trip is paid once per process rather than once per audit write.
private async Task EnsureTableAsync(SqliteConnection connection, CancellationToken cancellationToken)
{
if (Volatile.Read(ref _tableEnsured) == 1)
{
return;
}
await using SqliteCommand command = connection.CreateCommand();
command.CommandText = CreateTableSql;
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
Volatile.Write(ref _tableEnsured, 1);
}
private static DateTimeOffset ParseUtc(string value) =>
DateTimeOffset.Parse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);
// Reading is defensive where writing is not: an insert always round-trips "O", but the table is
// append-only shared state that an operator (or a future migration) can put an unparseable
// timestamp into, and the retention sweep deliberately keeps such a row — SQLite's datetime()
// yields NULL for it, so the DELETE's comparison is never true. A throwing Parse here would let
// that single row take out the dashboard's whole recent-audit view. MinValue instead sorts the
// row to the far past and keeps every other column readable, which is what an operator looking
// at the view actually needs.
private DateTimeOffset ParseUtcOrMinValue(string value, string eventId)
{
if (DateTimeOffset.TryParse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind, out DateTimeOffset parsed))
{
return parsed;
}
// Debug, not warning: the row is still returned and the timestamp text itself is not logged
// (audit rows are not secrets, but the value is attacker-influenceable in the worst case).
logger?.LogDebug(
"Audit event {EventId} has an unparseable occurred_at_utc; reporting it as DateTimeOffset.MinValue.",
eventId);
return DateTimeOffset.MinValue;
}
}
@@ -97,18 +97,47 @@ public static class AuthStoreServiceCollectionExtensions
sp.GetService<TimeProvider>() ?? TimeProvider.System));
DecorateVerifierWithCache(services, security);
// GetService, not GetRequiredService, for the same reason the writer registration below
// gives: the DI-only unit tests build a bare ServiceCollection with no AddLogging(). The
// store's logger is optional and only carries row-level read diagnostics.
services.AddSingleton(sp =>
new SqliteCanonicalAuditStore(sp.GetRequiredService<AuthSqliteConnectionFactory>()));
new SqliteCanonicalAuditStore(
sp.GetRequiredService<AuthSqliteConnectionFactory>(),
sp.GetService<ILogger<SqliteCanonicalAuditStore>>()));
services.AddSingleton<IAuditEventSink>(sp => sp.GetRequiredService<SqliteCanonicalAuditStore>());
// Resolve the logger defensively: the production host always registers ILogger<T>, but the
// DI-only auth/CLI/dashboard unit tests build a bare ServiceCollection without AddLogging().
// Fall back to NullLogger there so the audit writer (and the IApiKeyAuditStore override that
// depends on it) still resolve. The write path is best-effort regardless.
services.AddSingleton<IAuditWriter>(sp =>
services.AddSingleton(sp =>
new CanonicalAuditWriter(
sp.GetRequiredService<SqliteCanonicalAuditStore>(),
sp.GetRequiredService<IAuditEventSink>(),
sp.GetService<ILogger<CanonicalAuditWriter>>()
?? Microsoft.Extensions.Logging.Abstractions.NullLogger<CanonicalAuditWriter>.Instance));
// The registered IAuditWriter is the bounded, asynchronous one: audit producers — above
// all IConstraintEnforcer.RecordDenialAsync, which fires once per denied tag inside bulk
// RPC loops — enqueue and return instead of awaiting a SQLite insert each. No producer
// signature changes; the seam is entirely here. AuditDrainService batches the buffered
// events onto the sink, owns the one-time schema bootstrap and sweeps expired rows. Where
// no hosted service runs (the `apikey` CLI, the DI-only tests) the channel writer falls
// back to CanonicalAuditWriter's synchronous path, so audit is never silently buffered
// into a channel nobody drains.
services.AddSingleton(sp =>
new ChannelAuditWriter(
sp.GetRequiredService<CanonicalAuditWriter>(),
sp.GetService<ILogger<ChannelAuditWriter>>()
?? Microsoft.Extensions.Logging.Abstractions.NullLogger<ChannelAuditWriter>.Instance));
services.AddSingleton<IAuditWriter>(sp => sp.GetRequiredService<ChannelAuditWriter>());
services.AddSingleton(sp => new AuditDrainService(
sp.GetRequiredService<ChannelAuditWriter>(),
sp.GetRequiredService<IAuditEventSink>(),
security,
sp.GetService<TimeProvider>() ?? TimeProvider.System,
sp.GetService<ILogger<AuditDrainService>>()
?? Microsoft.Extensions.Logging.Abstractions.NullLogger<AuditDrainService>.Instance));
services.AddHostedService(sp => sp.GetRequiredService<AuditDrainService>());
// OVERRIDE the library's IApiKeyAuditStore (AddZbApiKeyAuth registered the library's
// SqliteApiKeyAuditStore via TryAddSingleton) with an adapter that canonicalizes every
// library-emitted ApiKeyAuditEntry onto AuditEvent and forwards it through IAuditWriter.
@@ -213,7 +213,10 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
// DashboardApiKeyManagementService.ValidateKeyId each restrict a key id to
// char.IsAsciiLetterOrDigit || '.' || '-'. Key ids are never library-generated, so no path can
// mint one containing '_'.
private static string? TryParseKeyId(string? authorizationHeader)
//
// Internal rather than private so the parse rules can be pinned directly by test: the guard's
// correctness depends on this returning the full key id.
internal static string? TryParseKeyId(string? authorizationHeader)
{
if (string.IsNullOrEmpty(authorizationHeader))
{
@@ -226,15 +229,29 @@ public sealed class CachingApiKeyVerifier : IApiKeyVerifier, IApiKeyCacheInvalid
? header[bearer.Length..].Trim()
: header;
string[] parts = token.ToString().Split('_');
if (parts.Length < 3
|| !string.Equals(parts[0], TokenPrefix, StringComparison.Ordinal)
|| parts[1].Length == 0)
// Scanned rather than split, for the same reason as the interceptor's copy: Split would
// allocate a token copy, an array and a string per segment on every cache miss to produce
// one key id. Two IndexOf scans allocate only that key id.
//
// The '_' checked immediately after the prefix is what makes "mxgw" the whole first segment
// (so "mxgwabc_..." is still rejected), and the second separator must exist because the
// split form required three segments — a token with no secret delimiter is not a key token.
if (!token.StartsWith(TokenPrefix, StringComparison.Ordinal)
|| token.Length <= TokenPrefix.Length
|| token[TokenPrefix.Length] != '_')
{
return null;
}
return parts[1];
ReadOnlySpan<char> afterPrefix = token[(TokenPrefix.Length + 1)..];
int separator = afterPrefix.IndexOf('_');
if (separator <= 0)
{
// -1 is a token with no second separator; 0 is an empty key id.
return null;
}
return new string(afterPrefix[..separator]);
}
private void IndexCacheKey(string keyId, string cacheKey)
@@ -19,10 +19,35 @@ namespace ZB.MOM.WW.MxGateway.Server.Security.Authentication;
/// </remarks>
public static class GatewayApiKeyIdentityMapper
{
private const int MaxCachedConstraintBlobs = 1024;
/// <summary>
/// Maximum number of parsed constraint blobs retained in <see cref="ConstraintCache"/>.
/// Blobs are admin-controlled (one per API key), so the cap is only a memory backstop for a
/// store with an unusually large number of distinct constrained keys.
/// </summary>
internal const int MaxCachedConstraintBlobs = 1024;
/// <summary>
/// Bounded parsed-constraints cache keyed by the raw constraints JSON. The blob is parsed
/// once per authenticated RPC otherwise, so this keeps the JSON parse off the hot path.
/// Beyond <see cref="MaxCachedConstraintBlobs"/> entries the oldest insertion is evicted
/// rather than the cache refusing new entries — a hard stop at the cap would leave every key
/// admitted after it re-parsing its blob on every RPC for the process lifetime. Eviction is
/// approximate (FIFO over insertion order, not true LRU) because only the bound matters.
/// </summary>
private static readonly ConcurrentDictionary<string, ApiKeyConstraints> ConstraintCache =
new(StringComparer.Ordinal);
/// <summary>
/// Insertion-order queue used to evict the oldest cache entry once the cache exceeds
/// <see cref="MaxCachedConstraintBlobs"/>. Keeping it separate leaves
/// <see cref="ConstraintCache"/> reads lock-free; the lock guards only the eviction path.
/// </summary>
private static readonly ConcurrentQueue<string> InsertionOrder = new();
private static readonly object EvictionLock = new();
/// <summary>Current cache size, exposed for tests asserting the cap is honoured.</summary>
internal static int CurrentCacheSize => ConstraintCache.Count;
private static ApiKeyConstraints DeserializeConstraints(string? constraintsJson)
{
if (string.IsNullOrWhiteSpace(constraintsJson))
@@ -36,12 +61,36 @@ public static class GatewayApiKeyIdentityMapper
}
ApiKeyConstraints parsed = ApiKeyConstraintSerializer.Deserialize(constraintsJson);
if (ConstraintCache.Count < MaxCachedConstraintBlobs)
// GetOrAdd returns whichever instance is in the cache after the call, so concurrent parsers
// of the same blob converge on one instance; it also avoids the TryAdd-then-read race where
// the key could be evicted between a failed TryAdd and the read back.
ApiKeyConstraints result = ConstraintCache.GetOrAdd(constraintsJson, parsed);
if (ReferenceEquals(result, parsed))
{
ConstraintCache.TryAdd(constraintsJson, parsed);
// We were the inserter — track for FIFO eviction and bound the cache.
InsertionOrder.Enqueue(constraintsJson);
EvictIfOverCapacity();
}
return parsed;
return result;
}
private static void EvictIfOverCapacity()
{
if (ConstraintCache.Count <= MaxCachedConstraintBlobs)
{
return;
}
// Serialize eviction so two threads do not race past the cap together.
lock (EvictionLock)
{
while (ConstraintCache.Count > MaxCachedConstraintBlobs && InsertionOrder.TryDequeue(out string? oldest))
{
ConstraintCache.TryRemove(oldest, out _);
}
}
}
/// <summary>
@@ -127,16 +127,31 @@ public sealed class ApiKeyFailureLimiter
/// <summary>Decides whether an authentication attempt may reach the verifier.</summary>
/// <param name="partition">The throttle partition derived from the request.</param>
/// <returns>The admission decision for this attempt.</returns>
public ApiKeyThrottleDecision Check(ApiKeyThrottlePartition partition)
public ApiKeyThrottleDecision Check(ApiKeyThrottlePartition partition) => Check(partition, out _);
/// <summary>
/// Decides whether an authentication attempt may reach the verifier, handing back the storage
/// key it resolved so a caller that goes on to <see cref="Reset(ApiKeyThrottlePartition, PartitionResolution)"/>
/// the same request does not resolve — and rebuild the composite key string — a second time.
/// </summary>
/// <param name="partition">The throttle partition derived from the request.</param>
/// <param name="resolution">
/// The resolved storage key, or <see cref="PartitionResolution.Unresolved"/> when the limiter is
/// disabled and never resolved one.
/// </param>
/// <returns>The admission decision for this attempt.</returns>
internal ApiKeyThrottleDecision Check(ApiKeyThrottlePartition partition, out PartitionResolution resolution)
{
string peer = RequirePeer(partition);
if (_limit <= 0)
{
resolution = PartitionResolution.Unresolved;
return ApiKeyThrottleDecision.Allowed;
}
long now = _clock.GetUtcNow().UtcTicks;
(string partitionKey, string? effectiveKeyId) = ResolvePartitionKey(peer, partition.KeyId, mint: false);
resolution = new PartitionResolution(partitionKey, effectiveKeyId);
WindowState? peerState = _partitions.TryGetValue(partitionKey, out WindowState? tracked) ? tracked : null;
WindowState? aggregateState = null;
@@ -221,10 +236,25 @@ public sealed class ApiKeyFailureLimiter
/// <summary>Clears both limiter layers for the partition after a successful verification.</summary>
/// <param name="partition">The throttle partition derived from the request.</param>
public void Reset(ApiKeyThrottlePartition partition)
public void Reset(ApiKeyThrottlePartition partition) => Reset(partition, PartitionResolution.Unresolved);
/// <summary>
/// Clears both limiter layers for the partition after a successful verification, reusing the
/// storage key <see cref="Check(ApiKeyThrottlePartition, out PartitionResolution)"/> already
/// resolved for this request.
/// </summary>
/// <param name="partition">The throttle partition derived from the request.</param>
/// <param name="resolution">
/// The resolution handed back by <c>Check</c>; <see cref="PartitionResolution.Unresolved"/>
/// resolves here instead. Reusing the check-time resolution is deliberate: it is the partition
/// this request was admitted against, so the reset clears exactly what the check consulted.
/// </param>
internal void Reset(ApiKeyThrottlePartition partition, PartitionResolution resolution)
{
string peer = RequirePeer(partition);
(string partitionKey, string? effectiveKeyId) = ResolvePartitionKey(peer, partition.KeyId, mint: false);
(string partitionKey, string? effectiveKeyId) = resolution.IsResolved
? (resolution.PartitionKey!, resolution.EffectiveKeyId)
: ResolvePartitionKey(peer, partition.KeyId, mint: false);
// Clear only a partition this caller actually owns. When its key id was squeezed into the
// address's shared fallback bucket by the per-peer cap, that bucket also holds failures
@@ -542,6 +572,25 @@ public sealed class ApiKeyFailureLimiter
public long ProbeVersion;
}
/// <summary>
/// A partition's resolved storage key, carried from the check to the reset of the same request.
/// The composite key is a fresh string per build, so resolving once per RPC rather than once per
/// call keeps the successful auth path (check, then reset) to a single allocation.
/// </summary>
/// <param name="PartitionKey">The storage key, or <see langword="null"/> when unresolved.</param>
/// <param name="EffectiveKeyId">
/// The key id that actually earned a partition, or <see langword="null"/> when the token carried
/// none or the per-peer cap collapsed it onto the transport-peer fallback.
/// </param>
internal readonly record struct PartitionResolution(string? PartitionKey, string? EffectiveKeyId)
{
/// <summary>Gets the sentinel for "not resolved yet"; the receiving call resolves it itself.</summary>
internal static PartitionResolution Unresolved => default;
/// <summary>Gets a value indicating whether this carries a resolved storage key.</summary>
internal bool IsResolved => PartitionKey is not null;
}
/// <summary>A probe slot reservation: what to restore, and the stamp proving it is still ours.</summary>
/// <param name="PreviousProbeAtTicks">The slot value replaced when the claim was made.</param>
/// <param name="Version">The <see cref="WindowState.ProbeVersion"/> stamped by this claim.</param>
@@ -16,6 +16,14 @@ public sealed class ConstraintEnforcer(
IGalaxyHierarchyCache cache,
IAuditWriter auditWriter) : IConstraintEnforcer
{
/// <inheritdoc />
public bool HasReadConstraints(ApiKeyIdentity? identity) =>
identity?.EffectiveConstraints.HasReadConstraints ?? false;
/// <inheritdoc />
public bool HasWriteConstraints(ApiKeyIdentity? identity) =>
identity?.EffectiveConstraints.HasWriteConstraints ?? false;
/// <inheritdoc />
public Task<ConstraintFailure?> CheckReadTagAsync(
ApiKeyIdentity? identity,
@@ -211,7 +219,25 @@ public sealed class ConstraintEnforcer(
return true;
}
return subtreeGlobs.Any(glob => GalaxyGlobMatcher.IsMatch(containedPath, glob))
|| tagGlobs.Any(glob => GalaxyGlobMatcher.IsMatch(tagAddress, glob));
// Plain index loops rather than Any(lambda): this runs once per item of every bulk
// read/write, and the closures the lambdas capture (containedPath / tagAddress) allocate a
// display class plus a delegate per call. Same short-circuit order, same result.
for (int i = 0; i < subtreeGlobs.Count; i++)
{
if (GalaxyGlobMatcher.IsMatch(containedPath, subtreeGlobs[i]))
{
return true;
}
}
for (int i = 0; i < tagGlobs.Count; i++)
{
if (GalaxyGlobMatcher.IsMatch(tagAddress, tagGlobs[i]))
{
return true;
}
}
return false;
}
}
@@ -77,8 +77,13 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
// aggregate for the key id. An over-limit state still admits one probe per interval, so the
// holder of the correct secret always reaches the verifier and resets the state.
// ResourceExhausted signals throttling without revealing whether any secret was valid.
//
// The check hands back the storage key it resolved so the reset below reuses it instead of
// rebuilding the composite (peer, key id) string a second time on every successful RPC.
ApiKeyThrottlePartition throttlePartition = ResolveThrottlePartition(authorizationHeader, context);
ApiKeyThrottleDecision decision = failureLimiter.Check(throttlePartition);
ApiKeyThrottleDecision decision = failureLimiter.Check(
throttlePartition,
out ApiKeyFailureLimiter.PartitionResolution throttleResolution);
if (decision is ApiKeyThrottleDecision.ThrottledByPeer or ApiKeyThrottleDecision.ThrottledByAggregate)
{
metrics.RecordAuthThrottled(
@@ -107,7 +112,7 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
// fat-fingered a few attempts is not penalised once it recovers — and, because the check
// above admits a probe rather than blocking absolutely, this reset stays reachable while the
// key is under an active spray.
failureLimiter.Reset(throttlePartition);
failureLimiter.Reset(throttlePartition, throttleResolution);
ApiKeyIdentity identity = GatewayApiKeyIdentityMapper.ToGatewayIdentity(verification.Identity);
@@ -137,7 +142,11 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
// before a key-id partition is minted so a spray of invented tokens cannot mint one tracked
// partition each and flush the limiter's bounded map (SEC-32). Anything that fails the check
// falls back to the sender's transport-peer partition.
private static string? TryResolveKeyId(string? authorizationHeader)
//
// Internal rather than private so the parse rules can be pinned directly by test; the shape
// check is a security boundary (SEC-32) and is worth asserting without routing every case
// through a full RPC.
internal static string? TryResolveKeyId(string? authorizationHeader)
{
if (string.IsNullOrWhiteSpace(authorizationHeader))
{
@@ -150,22 +159,36 @@ public sealed class GatewayGrpcAuthorizationInterceptor(
? header[bearer.Length..].Trim()
: header;
string[] parts = token.ToString().Split('_');
if (parts.Length < 3)
// Scanned rather than split: this runs on every authenticated RPC, and Split would copy the
// token out of the header and allocate an array plus a string per segment to reach a key id
// that is then usually a dictionary-lookup miss. Two IndexOf scans reach the same answer and
// allocate only the key id itself.
const string prefix = AuthStoreServiceCollectionExtensions.TokenPrefix;
if (!token.StartsWith(prefix, StringComparison.Ordinal)
|| token.Length <= prefix.Length
|| token[prefix.Length] != '_')
{
// Guards the whole first segment, not just its start: the '_' immediately after the
// prefix is what makes "mxgw" the entire segment, so "mxgwabc_..." is still rejected.
return null;
}
ReadOnlySpan<char> afterPrefix = token[(prefix.Length + 1)..];
int separator = afterPrefix.IndexOf('_');
if (separator <= 0 || separator > MaxKeyIdLength)
{
// -1 is a token with no second separator (too few segments); 0 is an empty key id.
return null;
}
// The third segment must be non-empty, which the split form expressed as parts[2].Length: it
// ends at the NEXT separator, so a secret beginning with '_' fails the same way it always did.
ReadOnlySpan<char> afterKeyId = afterPrefix[(separator + 1)..];
if (afterKeyId.IsEmpty || afterKeyId[0] == '_')
{
return null;
}
if (!string.Equals(parts[0], AuthStoreServiceCollectionExtensions.TokenPrefix, StringComparison.Ordinal))
{
return null;
}
if (parts[1].Length == 0 || parts[1].Length > MaxKeyIdLength || parts[2].Length == 0)
{
return null;
}
return parts[1];
return new string(afterPrefix[..separator]);
}
}
@@ -5,6 +5,30 @@ namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
public interface IConstraintEnforcer
{
/// <summary>
/// Gets a value indicating whether any read constraint applies to an identity at all, so a
/// bulk caller can hoist the question out of its per-item loop.
/// </summary>
/// <param name="identity">The API key identity.</param>
/// <returns><see langword="true"/> when at least one read constraint applies; otherwise <see langword="false"/>.</returns>
/// <remarks>
/// Every per-item <see cref="CheckReadTagAsync"/> / <see cref="CheckReadHandleAsync"/> call for
/// an unconstrained identity allows the item, so skipping the loop removes work without
/// changing a decision. The default implementation answers <see langword="true"/> — an
/// implementation that does not model constraints (test doubles, allow-all enforcers) keeps
/// being consulted per item rather than being silently bypassed.
/// </remarks>
bool HasReadConstraints(ApiKeyIdentity? identity) => true;
/// <summary>
/// Gets a value indicating whether any write constraint applies to an identity at all, the
/// write-side counterpart of <see cref="HasReadConstraints"/>.
/// </summary>
/// <param name="identity">The API key identity.</param>
/// <returns><see langword="true"/> when at least one write constraint applies; otherwise <see langword="false"/>.</returns>
/// <remarks>The same conservative default as <see cref="HasReadConstraints"/> applies.</remarks>
bool HasWriteConstraints(ApiKeyIdentity? identity) => true;
/// <summary>Checks whether a read constraint is satisfied for a tag address.</summary>
/// <param name="identity">The API key identity.</param>
/// <param name="tagAddress">Tag address to check.</param>
@@ -764,11 +764,24 @@ public sealed class GatewaySession
// The distributor's single event source. Drains the worker event stream once (the
// distributor guarantees a single consumer) and maps each frame to the public MxEvent,
// preserving worker order. Mirrors the former ProduceEventsAsync mapping exactly.
//
// This deliberately duplicates the three lines of ReadEventsAsync rather than enumerating
// it: every worker event crosses this source, and routing it through a second pure
// pass-through iterator cost two extra MoveNextAsync state-machine hops per event for no
// semantic value. ReadEventsAsync stays for ISessionManager.ReadEventsAsync; keep the two
// bodies in step. Only one of them may run per attach — WorkerClient.ReadEventsAsync
// single-reader-claims the event channel and throws on a second consumer — and on the
// distributor path that one consumer is this method.
private async IAsyncEnumerable<MxEvent> MapWorkerEventsAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
MxAccessGrpcMapper mapper = _eventStreaming.Mapper;
await foreach (WorkerEvent workerEvent in ReadEventsAsync(cancellationToken)
IWorkerClient workerClient = await GetReadyWorkerClientAsync(cancellationToken).ConfigureAwait(false);
TouchClientActivity(_eventStreaming.TimeProvider.GetUtcNow());
await foreach (WorkerEvent workerEvent in workerClient
.ReadEventsAsync(cancellationToken)
.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
yield return mapper.MapEvent(workerEvent);
@@ -1513,6 +1526,13 @@ public sealed class GatewaySession
/// Reads events from the worker as an asynchronous enumerable stream.
/// </summary>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <remarks>
/// Backs <c>ISessionManager.ReadEventsAsync</c>. The distributor does <em>not</em> come
/// through here — <c>MapWorkerEventsAsync</c> inlines this body to save a per-event
/// iterator hop, so changes made here belong there too. The two are mutually exclusive
/// per attach: <see cref="IWorkerClient.ReadEventsAsync"/> claims the worker event
/// channel for a single reader and throws on the second consumer.
/// </remarks>
/// <returns>An asynchronous stream of worker events.</returns>
public async IAsyncEnumerable<WorkerEvent> ReadEventsAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
@@ -68,12 +68,25 @@ public interface ISessionManager
/// <param name="now">The current time to evaluate expiration against.</param>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <returns>The number of sessions closed.</returns>
/// <remarks>
/// A close that fails does not abandon the rest of the pass: every session selected by this
/// sweep is attempted, and the first failure is then rethrown so the caller still observes
/// (and logs) that the sweep failed. Which failure surfaces is nondeterministic when several
/// closes fail in the same pass, because the closes run concurrently.
/// </remarks>
Task<int> CloseExpiredLeasesAsync(
DateTimeOffset now,
CancellationToken cancellationToken);
/// <summary>Shuts down all sessions and the session manager.</summary>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <param name="cancellationToken">
/// Token that <em>degrades</em> the drain rather than cancelling it. It is passed only to each
/// session's graceful close; the drain loop and the kill fallback are not bound to it, so
/// cancelling turns the drain into a kill sweep instead of abandoning the untried sessions as
/// leaked workers. The call therefore overruns a cancelled token by a bounded amount —
/// roughly <c>ceil(sessionCount / 4)</c> batches of the worker shutdown timeout in the worst
/// case, where 4 is <c>MaxParallelSessionCloses</c>.
/// </param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task ShutdownAsync(CancellationToken cancellationToken);
}
@@ -1,4 +1,3 @@
using System.Collections.Concurrent;
using System.Threading.Channels;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
@@ -66,12 +65,25 @@ public delegate void SubscriberOverflowHandler(bool isOnlySubscriber, bool isInt
/// <c>EventStreamService.ProduceEventsAsync</c> ordering.
/// </para>
/// <para>
/// <b>Concurrency.</b> The subscriber set is a
/// <see cref="ConcurrentDictionary{TKey, TValue}"/> keyed by a monotonic id.
/// The pump iterates it with a snapshot-free enumerator (which never throws on
/// concurrent add/remove), and <see cref="Register"/> / lease disposal mutate it
/// without any lock held across an <c>await</c>. Each subscriber channel has a
/// single writer — the pump — so per-channel writes never race. MXAccess parity:
/// <b>Concurrency.</b> The subscriber set is a plain
/// <see cref="Dictionary{TKey, TValue}"/> keyed by a monotonic id, used for keyed
/// add/remove only. It needs no concurrent collection type because it is never
/// touched outside the <c>_lifecycleLock</c> critical section: every mutation
/// (<see cref="Register"/>, <see cref="RegisterWithReplay"/>, lease disposal,
/// overflow disconnect) and every read (the terminal completion sweep, the snapshot
/// rebuild) holds that lock, and each mutation rebuilds an immutable copy-on-write
/// <c>Subscriber[]</c> snapshot inside the same section. The lock-free readers see only that snapshot,
/// never the dictionary: the pump reads it once per event and
/// <see cref="SubscriberCount"/> reads its length. Fan-out therefore does NOT
/// enumerate the dictionary — it walks a captured array, with no dictionary
/// traversal and no per-event allocation on the hot path. The subscriber set is
/// tiny (one to a handful) and mutates rarely, so paying a full array rebuild per
/// registration to buy that is the right trade. No lock is held across an
/// <c>await</c>. Each subscriber channel has a single writer — the pump — so
/// per-channel writes never race. A subscriber registered after the pump captured
/// the array for the in-flight event misses that event, which matches "late
/// subscribers see events after they register"; the reconnect path closes that
/// window deliberately (see <see cref="RegisterWithReplay"/>). MXAccess parity:
/// events are fanned in the order received; the pump never reorders or
/// synthesizes events.
/// </para>
@@ -93,10 +105,29 @@ public sealed class SessionEventDistributor : IAsyncDisposable
private readonly TimeSpan _shutdownTimeout;
private readonly ILogger<SessionEventDistributor> _logger;
private readonly TimeProvider _timeProvider;
private readonly ConcurrentDictionary<long, Subscriber> _subscribers = new();
// Keyed subscriber set. Touched ONLY under _lifecycleLock (add in RegisterSubscriber and
// RegisterWithReplay, remove in RemoveSubscriber, read in CompleteAllSubscribers and
// RebuildSubscriberSnapshot), which is why a plain Dictionary suffices: lock-free readers
// never see this field, they read _subscriberSnapshot below.
private readonly Dictionary<long, Subscriber> _subscribers = [];
private readonly CancellationTokenSource _shutdownCts = new();
private readonly object _lifecycleLock = new();
// Copy-on-write fan-out snapshot of _subscribers.Values. Rebuilt (a whole new array)
// inside the _lifecycleLock section of every register/unregister; never mutated in
// place, so the pump can walk the array it captured with no lock and no allocation.
// Volatile.Write / Volatile.Read ORDER the access — they keep the publishing store from
// sinking past the lock release, and they keep a lock-free reader's load from being hoisted
// or cached. The pump is NOT that reader: its single capture point sits inside the
// _replayLock section of AppendToReplayBufferAndCaptureSubscribers, so the lock edge already
// orders it. The genuinely lock-free reader is SubscriberCount, which loads the field with no
// lock at all. They do NOT promise freshness, and nothing here needs them to: a reader
// may legitimately observe the previous array, which IS the documented "late subscribers
// see events after they register" window. Where visibility must be guaranteed — the
// RegisterWithReplay handoff — it comes from the _replayLock edge, not from Volatile.
// See the type remarks for why fan-out walks this array instead of enumerating _subscribers.
private Subscriber[] _subscriberSnapshot = [];
// Replay ring buffer. Appended on the pump thread and queried from arbitrary
// threads via TryGetReplayFrom, so every access is under _replayLock. Backed by a
// fixed-size circular array preallocated to the capacity so appending a retained
@@ -268,7 +299,13 @@ public sealed class SessionEventDistributor : IAsyncDisposable
/// <see cref="GatewaySession.ActiveEventSubscriberCount"/>, which tracks only external
/// (gRPC) subscribers and excludes the internal dashboard subscriber.
/// </summary>
public int SubscriberCount => _subscribers.Count;
/// <remarks>
/// Read from the copy-on-write snapshot rather than the dictionary, because this
/// property is a lock-free reader and the dictionary may only be touched under
/// <c>_lifecycleLock</c>. The snapshot is rebuilt in the same <c>_lifecycleLock</c>
/// section that mutates the dictionary, so the two never diverge.
/// </remarks>
public int SubscriberCount => Volatile.Read(ref _subscriberSnapshot).Length;
/// <summary>
/// Starts the background pump. Idempotent — a second call is a no-op.
@@ -332,6 +369,7 @@ public sealed class SessionEventDistributor : IAsyncDisposable
{
ObjectDisposedException.ThrowIf(_disposed, this);
_subscribers[subscriber.Id] = subscriber;
RebuildSubscriberSnapshot();
// Close the register-after-pump-completion window: if the pump already ran its
// final CompleteAllSubscribers (source completed/faulted) but the distributor is
@@ -416,27 +454,40 @@ public sealed class SessionEventDistributor : IAsyncDisposable
/// <para>
/// <b>Why this is atomic and the handoff is correct.</b> The replay snapshot and the
/// subscriber registration both run inside the SAME <c>_replayLock</c> critical
/// section. The pump appends each event to the replay buffer under <c>_replayLock</c>
/// <em>before</em> fanning it to subscribers (outside the lock). Therefore, relative
/// to this method's critical section, for every event E:
/// section. The pump appends each event to the replay buffer AND captures the
/// copy-on-write subscriber array in one <c>_replayLock</c> section, then fans the
/// event to that captured array outside the lock. Mutual exclusion therefore places
/// every event E strictly on one side of this method's critical section:
/// </para>
/// <list type="bullet">
/// <item>
/// If the pump appended E before this critical section, E is in
/// <paramref name="replayedEvents"/> (when newer than
/// <paramref name="afterSequence"/>). The pump's fan-out of E may race the
/// registration: if it writes E to this new channel too, E's sequence is
/// <c>&lt;= liveResumeSequence</c>, so the caller's live filter DROPS it — no
/// duplicate.
/// <paramref name="afterSequence"/>). The pump captured its subscriber array in
/// that same earlier section, so it cannot also fan E into this
/// not-yet-registered channel — no duplicate. Belt and braces: even if it did,
/// E's sequence is <c>&lt;= liveResumeSequence</c> and the caller's live filter
/// DROPS it.
/// </item>
/// <item>
/// If the pump appends E after this critical section, E is NOT in the snapshot,
/// but this subscriber is already registered, so the pump fans E into the live
/// channel with sequence <c>&gt; liveResumeSequence</c> — delivered as live, no
/// gap.
/// but this subscriber was registered — and the snapshot array republished —
/// before that section began, so the pump's capture includes it and E is fanned
/// into the live channel with sequence <c>&gt; liveResumeSequence</c> — delivered
/// as live, no gap.
/// </item>
/// </list>
/// <para>
/// Capturing the fan-out array inside the append's <c>_replayLock</c> section is what
/// makes the first bullet's "cannot" hold. It is defense in depth rather than a
/// correctness fix: a capture taken after that lock released could not drop an event
/// either (the lock edge orders it), it could only produce the duplicate the live
/// filter already discards. Doing it under the lock costs nothing and stops
/// no-duplicate from depending on every caller remembering to apply the filter —
/// which callers MUST still do, since <paramref name="liveResumeSequence"/> remains
/// part of this method's contract.
/// </para>
/// <para>
/// Lock ordering: this is the only path that holds both <c>_replayLock</c> and
/// <c>_lifecycleLock</c>; it always takes <c>_replayLock</c> first then
/// <c>_lifecycleLock</c>. No other path acquires both, so there is no inversion.
@@ -508,6 +559,7 @@ public sealed class SessionEventDistributor : IAsyncDisposable
{
ObjectDisposedException.ThrowIf(_disposed, this);
_subscribers[id] = subscriber;
RebuildSubscriberSnapshot();
// Same register-after-pump-completion guard as Register: a resume that races in
// after the source already ended still gets its retained replay batch (snapshot
@@ -591,13 +643,22 @@ public sealed class SessionEventDistributor : IAsyncDisposable
// Retain for replay BEFORE fan-out so a reconnecting subscriber that
// queries between fan-out and its own read still sees this event. Order
// is preserved: the pump is the single appender and events arrive in
// source order.
AppendToReplayBuffer(mxEvent);
// source order. The same call returns the subscriber array to fan to,
// captured under _replayLock — see the method for why the capture must
// share the append's critical section.
Subscriber[] subscribers = AppendToReplayBufferAndCaptureSubscribers(mxEvent);
// Enumerating a ConcurrentDictionary's Values never throws on concurrent
// add/remove; a subscriber registered mid-iteration may miss this event,
// which matches "late subscribers see events after they register".
foreach (Subscriber subscriber in _subscribers.Values)
// Walk the captured copy-on-write array: no dictionary enumeration, no
// per-event allocation. A subscriber registered after this capture misses
// this event, which matches "late subscribers see events after they
// register". A subscriber UNREGISTERED after the capture is still written to,
// and TryWrite on its completed channel returns false — from here that is
// indistinguishable from a real overflow. The window predates the
// copy-on-write array (enumerating the dictionary materialized its values up
// front too) and its outcome is NOT benign, so telling a graceful unregister
// apart from a genuine overflow is OnSubscriberOverflow's job, not this
// loop's.
foreach (Subscriber subscriber in subscribers)
{
// Non-blocking write: TryWrite never blocks the pump on a slow reader.
// A false return means this subscriber's bounded channel is full — the
@@ -631,14 +692,38 @@ public sealed class SessionEventDistributor : IAsyncDisposable
}
// Applies the per-subscriber backpressure policy when a subscriber's bounded channel is
// full. Runs on the pump thread. The offending subscriber is ALWAYS disconnected with an
// overflow fault and unregistered, so it can never wedge the pump again; the overflow
// handler decides the observable side effects (overflow metric, and — for legacy
// full — or, indistinguishably from the pump's side, already completed. A subscriber that
// really overflowed is ALWAYS disconnected with an overflow fault and unregistered, so it
// can never wedge the pump again; one that merely unregistered itself is dropped silently
// (see the discriminator below). Runs on the pump thread. The overflow handler decides the
// observable side effects (overflow metric, and — for legacy
// single-subscriber FailFast — faulting the owning session). Multi-subscriber FailFast
// intentionally degrades to a plain disconnect (see SubscriberOverflowHandler docs): one
// slow consumer must not fault a session shared by other healthy subscribers.
private void OnSubscriberOverflow(Subscriber subscriber, ulong workerSequence)
{
// Claim the disconnect FIRST, because a false TryWrite is ambiguous. It means either
// "channel full" (a genuine overflow) or "channel already completed" — which happens
// when the subscriber unregistered after the pump captured the fan-out array and is
// therefore a GRACEFUL close, not backpressure. RemoveSubscriber separates the two:
// every path that completes a channel during fan-out (lease disposal via Unregister,
// and this method) removes the subscriber from the set BEFORE completing it, so a
// completed channel implies the subscriber is already gone and RemoveSubscriber
// returns false. (CompleteAllSubscribers completes without removing, but only after the
// pump has left its loop, so it cannot be observed here — except on the DisposeAsync
// abandon path: a source factory that ignores cancellation past the 5 s shutdown timeout
// leaves the pump fanning while DisposeAsync completes subscribers, so a spurious overflow
// report is possible there. It is harmless, because the session is already being disposed.)
//
// Bailing out on false is what keeps a normal stream ending mid-traffic from emitting
// a bogus EventQueueOverflow metric and — under the default single-subscriber FailFast
// policy — faulting the whole session. Winning the removal also guarantees the side
// effects below run exactly once per subscriber.
if (!RemoveSubscriber(subscriber))
{
return;
}
// Decide whether FailFast may fault the whole session for this overflow. This is the
// "isOnlySubscriber" signal the legacy single-subscriber FailFast path keys on.
bool isOnlySubscriber = !subscriber.IsInternal && _singleSubscriberMode;
@@ -665,16 +750,15 @@ public sealed class SessionEventDistributor : IAsyncDisposable
subscriber.Id);
}
// Disconnect ONLY this subscriber: complete its channel with the overflow fault and
// remove it from the fan-out set. Its gRPC reader's MoveNextAsync then throws the
// SessionManagerException, which EventStreamService surfaces to the client exactly as
// the pre-epic per-RPC overflow did. The pump and every other subscriber are untouched.
if (_subscribers.TryRemove(subscriber.Id, out _))
{
subscriber.Channel.Writer.TryComplete(new SessionManagerException(
SessionManagerErrorCode.EventQueueOverflow,
$"Session {_sessionId} event stream queue overflowed."));
}
// Disconnect ONLY this subscriber: it is already out of the fan-out set (removed above),
// so complete its channel with the overflow fault. Its gRPC reader's MoveNextAsync then
// throws the SessionManagerException, which EventStreamService surfaces to the client
// exactly as the pre-epic per-RPC overflow did. The pump and every other subscriber are
// untouched. This runs even when the handler above threw — the subscriber must never be
// left attached with an un-completed channel.
subscriber.Channel.Writer.TryComplete(new SessionManagerException(
SessionManagerErrorCode.EventQueueOverflow,
$"Session {_sessionId} event stream queue overflowed."));
}
private void CompleteAllSubscribers(Exception? error)
@@ -699,12 +783,42 @@ public sealed class SessionEventDistributor : IAsyncDisposable
private void Unregister(Subscriber subscriber)
{
if (_subscribers.TryRemove(subscriber.Id, out _))
if (RemoveSubscriber(subscriber))
{
subscriber.Channel.Writer.TryComplete();
}
}
// Removes a subscriber from the fan-out set and republishes the copy-on-write snapshot.
// Returns true only for the caller that actually removed it, so the channel is completed
// exactly once however many disposal/overflow paths race. Completing the channel is left to
// that caller and happens OUTSIDE the lock: this lock guards set membership only.
//
// Remove-then-complete (never the reverse) is load-bearing, not incidental: it is what lets
// OnSubscriberOverflow read a false return as "this subscriber unregistered gracefully"
// rather than "this subscriber overflowed". Completing before removing would resurrect the
// spurious-session-fault bug.
private bool RemoveSubscriber(Subscriber subscriber)
{
lock (_lifecycleLock)
{
if (!_subscribers.Remove(subscriber.Id))
{
return false;
}
RebuildSubscriberSnapshot();
return true;
}
}
// Republishes the fan-out array from the current dictionary contents. MUST be called with
// _lifecycleLock held — holding that lock across the dictionary mutation and this rebuild is
// what keeps the array and the dictionary from diverging, and it is also what makes the plain
// (non-concurrent) Dictionary safe: this enumeration never races a mutation.
private void RebuildSubscriberSnapshot()
=> Volatile.Write(ref _subscriberSnapshot, [.. _subscribers.Values]);
/// <summary>
/// Returns the retained events with <see cref="MxEvent.WorkerSequence"/> strictly
/// greater than <paramref name="afterSequence"/>, in ascending sequence order, so a
@@ -791,7 +905,30 @@ public sealed class SessionEventDistributor : IAsyncDisposable
}
}
private void AppendToReplayBuffer(MxEvent mxEvent)
// Appends an event to the replay ring AND captures the fan-out array the pump will write it
// to, in ONE _replayLock section, making append+capture atomic with respect to
// RegisterWithReplay (which snapshots the ring and registers under that same lock). Each
// event therefore lands strictly on one side of a resume: replayed to that subscriber, or
// fanned to it live — never both.
//
// This is defense in depth, NOT a correctness fix; capturing after the lock released would
// also be correct. Monitor.Enter is an acquire (ECMA-335 I.12.6.5), so a later read cannot
// move above the append's lock acquisition, and a resume whose entire locked section
// (ring snapshot, registration, array republish) preceded the append is visible across that
// lock edge — no event can be silently dropped. What a late capture would allow is the
// benign case: an event both replayed AND written to the new subscriber's live channel, a
// duplicate the caller's liveResumeSequence filter discards. Capturing under the lock
// removes that duplicate at the source, so "no duplicate" no longer rests on the caller
// actually applying the filter — bought at zero cost, since the pump holds this lock anyway.
//
// Lock ordering: this helper only READS the already-published array, deliberately. The one
// permitted nesting in this type is RegisterWithReplay's _replayLock -> _lifecycleLock;
// every other path takes exactly one lock. Rebuilding here instead — an obvious-looking
// lock(_lifecycleLock) inside this _replayLock section — would drag the pump's hot path into
// that nesting and turn any future _lifecycleLock -> _replayLock path into a deadlock.
//
// Returns the array; the pump fans OUTSIDE the lock so a slow reader can never stall replay.
private Subscriber[] AppendToReplayBufferAndCaptureSubscribers(MxEvent mxEvent)
{
lock (_replayLock)
{
@@ -802,28 +939,30 @@ public sealed class SessionEventDistributor : IAsyncDisposable
}
// Capacity 0 disables retention: track the highest-seen sequence (so replay
// can still report a gap) but keep no events.
if (_replayBufferCapacity == 0)
// can still report a gap) but keep no events. The capture below still runs —
// retention being off says nothing about the fan-out set.
if (_replayBufferCapacity > 0)
{
return;
// Append at the logical tail. When the ring is full the oldest entry is
// overwritten in place (its slot becomes the new tail) and the head advances,
// so the newest _replayBufferCapacity events are retained with no allocation.
ReplayEntry entry = new(mxEvent, _timeProvider.GetUtcNow());
if (_replayCount < _replayBufferCapacity)
{
_replayBuffer[(_replayHead + _replayCount) % _replayBufferCapacity] = entry;
_replayCount++;
}
else
{
_replayBuffer[_replayHead] = entry;
_replayHead = (_replayHead + 1) % _replayBufferCapacity;
}
EvictAged();
}
// Append at the logical tail. When the ring is full the oldest entry is
// overwritten in place (its slot becomes the new tail) and the head advances,
// so the newest _replayBufferCapacity events are retained with no allocation.
ReplayEntry entry = new(mxEvent, _timeProvider.GetUtcNow());
if (_replayCount < _replayBufferCapacity)
{
_replayBuffer[(_replayHead + _replayCount) % _replayBufferCapacity] = entry;
_replayCount++;
}
else
{
_replayBuffer[_replayHead] = entry;
_replayHead = (_replayHead + 1) % _replayBufferCapacity;
}
EvictAged();
// Single capture point for both the retained and no-retention paths.
return Volatile.Read(ref _subscriberSnapshot);
}
}
@@ -1,4 +1,5 @@
using System.Diagnostics.CodeAnalysis;
using System.Runtime.ExceptionServices;
using System.Security.Cryptography;
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging;
@@ -20,6 +21,12 @@ public sealed class SessionManager : ISessionManager
public const string DetachGraceExpiredReason = "detach-grace-expired";
public const string FaultedReason = "faulted-reaped";
// Bounded so a mass expiry (or a host stop with a full registry) cannot stampede
// worker-process teardown: every concurrent close is one x86 worker being shut down or
// killed, and the point of the fan-out is to hide a few hung workers, not to tear the whole
// registry down at once.
private const int MaxParallelSessionCloses = 4;
private readonly ISessionRegistry _registry;
private readonly ISessionWorkerClientFactory _workerClientFactory;
private readonly GatewayMetrics _metrics;
@@ -252,7 +259,10 @@ public sealed class SessionManager : ISessionManager
DateTimeOffset now,
CancellationToken cancellationToken)
{
int closedCount = 0;
// Selection phase — deliberately sequential. Only the close calls below run in parallel:
// deciding WHICH sessions to close must stay a single ordered pass so the sweep-precedence
// rule and the TOCTOU re-check keep their meaning.
List<(GatewaySession Session, string Reason)> selected = [];
foreach (GatewaySession session in _registry.Snapshot())
{
// A session is swept when its normal lease has expired, it has FAULTED (a faulted
@@ -288,45 +298,124 @@ public sealed class SessionManager : ISessionManager
continue;
}
await CloseSessionCoreAsync(session, reason, cancellationToken).ConfigureAwait(false);
closedCount++;
selected.Add((session, reason));
}
if (selected.Count == 0)
{
return 0;
}
int closedCount = 0;
object failureSyncRoot = new();
ExceptionDispatchInfo? firstFailure = null;
// Close phase. Each close is bounded by the worker shutdown timeout (default 10 s), so a
// mass expiry with a few hung workers would serialize reaping and starve session slots.
// Parallel close is safe because TryBeginCloseIfExpired above already flipped every
// selected session to Closing under its own lock — that idempotent begin-close is the
// per-session exclusivity invariant, so no two teardowns can run against one session and
// a session selected here cannot be re-selected by a concurrent sweep.
await Parallel.ForEachAsync(
selected,
new ParallelOptions
{
MaxDegreeOfParallelism = MaxParallelSessionCloses,
CancellationToken = cancellationToken,
},
async (candidate, closeToken) =>
{
try
{
await CloseSessionCoreAsync(candidate.Session, candidate.Reason, closeToken).ConfigureAwait(false);
Interlocked.Increment(ref closedCount);
}
catch (Exception exception)
{
// The sequential sweep let a close failure propagate to the lease monitor,
// which logs it; that signal is preserved by rethrowing the first failure
// below. It is captured rather than thrown here so one failed (or hung)
// teardown does not abandon the rest of the already-selected set.
lock (failureSyncRoot)
{
firstFailure ??= ExceptionDispatchInfo.Capture(exception);
}
}
}).ConfigureAwait(false);
firstFailure?.Throw();
return closedCount;
}
/// <inheritdoc />
public async Task ShutdownAsync(CancellationToken cancellationToken)
{
foreach (GatewaySession session in _registry.Snapshot())
{
try
// Sessions are drained in parallel: at a worst-case worker shutdown timeout each, a
// one-at-a-time drain of a full registry outruns any host stop-timeout and leaves the
// tail to the orphan killer. Per-session exclusivity comes from GatewaySession.CloseAsync's
// own close gate, and each iteration touches only its own session plus thread-safe
// registry/metrics state.
//
// The body must be exception-TOTAL. Parallel.ForEachAsync cancels the token it hands the
// sibling bodies as soon as one body throws, so a single escaping exception would abort up
// to MaxParallelSessionCloses - 1 in-flight graceful shutdowns AND make their kill fallback
// throw immediately on the freshly cancelled token — sessions neither closed nor killed,
// i.e. leaked x86 workers that nothing reattaches to (a gateway restart terminates orphans
// rather than adopting them).
//
// For the same reason the loop itself is NOT bound to cancellationToken: a cancelled
// ParallelOptions token stops dispatching the remaining sessions entirely, so a stop
// deadline would leave the untried tail neither closed nor killed. Note this FIXES a leak
// the sequential drain also had rather than restoring its behavior: there the kill fallback
// ran on the caller's cancelled token, and KillWorkerAsync's entry
// ThrowIfCancellationRequested threw out of the loop on the first session — zero kills, not
// "fail fast and still kill". The token is passed to the graceful close only, and the kill
// runs on CancellationToken.None, so a host stop deadline turns the drain into a kill sweep
// rather than into a leak.
//
// The asymmetry with CloseExpiredLeasesAsync (whose ParallelOptions IS token-bound) is
// intentional: that sweep is periodic maintenance whose missed sessions are picked up by
// the next pass and, ultimately, by this drain. This drain is terminal — nothing runs after
// it — so it must not be abandoned partway.
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
async (session, _) =>
{
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
_logger.LogWarning(
exception,
"Graceful shutdown failed for session {SessionId}; killing worker.",
session.SessionId);
if (_registry.TryGet(session.SessionId, out _))
try
{
try
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
}
catch (Exception exception)
{
_logger.LogWarning(
exception,
"Graceful shutdown failed for session {SessionId}; killing worker.",
session.SessionId);
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
&& registeredSession is not null)
{
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
}
catch (SessionManagerException killException)
{
_logger.LogWarning(
killException,
"Worker kill fallback failed for session {SessionId}.",
session.SessionId);
try
{
// Deliberately NOT the caller's token: the kill is the last-resort orphan
// preventer, so it must still run when the host stop deadline (or a
// sibling body's failure) has already cancelled the drain. It is a
// synchronous Kill plus registry/dispose bookkeeping, not a wait on
// the worker, so it cannot extend the drain meaningfully.
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
}
catch (Exception killException)
{
_logger.LogWarning(
killException,
"Worker kill fallback failed for session {SessionId}.",
session.SessionId);
}
}
}
}
}
}).ConfigureAwait(false);
}
private async Task<SessionCloseResult> CloseSessionCoreAsync(
@@ -16,7 +16,14 @@ public sealed class SessionShutdownHostedService(
return Task.CompletedTask;
}
/// <summary>Shuts down all gateway sessions as the host stops, logging (without throwing) if the host's shutdown timeout cancels the operation first.</summary>
/// <summary>Shuts down all gateway sessions as the host stops.</summary>
/// <remarks>
/// The catch below is now effectively unreachable: <see cref="ISessionManager.ShutdownAsync"/>
/// no longer aborts on the host's shutdown timeout, it degrades to a kill sweep and logs a
/// per-session warning for each session that failed its graceful close. The clause is kept as
/// a cheap guard against that contract regressing, not as an expected path — the operator
/// signal for a timed-out shutdown is now those per-session warnings.
/// </remarks>
/// <param name="cancellationToken">Token that signals the host's shutdown timeout has elapsed.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task StopAsync(CancellationToken cancellationToken)
@@ -12,6 +12,18 @@ namespace ZB.MOM.WW.MxGateway.Server.Sessions;
/// <summary>Factory for creating worker clients and launching worker processes.</summary>
public sealed class SessionWorkerClientFactory : ISessionWorkerClientFactory
{
/// <summary>
/// Kernel buffer quota requested for each direction of a worker pipe. A zero quota — what the
/// short <see cref="NamedPipeServerStream"/> overloads request — makes every byte-mode write
/// rendezvous with a pending read, so writer latency is coupled to reader scheduling and a
/// writer with no reader parked blocks indefinitely. That is the failure class behind the
/// historical windev full-suite wedge (all tests reported, testhost never exiting). A real
/// quota lets a whole frame land in the kernel and the writer return. 128 KiB comfortably
/// holds the control traffic and typical event batches without reserving nonpaged pool per
/// session for the rare maximum-sized frame, which still streams through in chunks.
/// </summary>
private const int PipeBufferSizeBytes = 128 * 1024;
private readonly IWorkerProcessLauncher _workerProcessLauncher;
private readonly GatewayMetrics _metrics;
private readonly TimeProvider _timeProvider;
@@ -155,6 +167,11 @@ public sealed class SessionWorkerClientFactory : ISessionWorkerClientFactory
/// <summary>Creates a named pipe for worker communication.</summary>
/// <param name="pipeName">The pipe name.</param>
/// <returns>Named pipe server stream.</returns>
/// <remarks>
/// The buffer sizes are explicit so the pipe is not created with a zero quota; see
/// <see cref="PipeBufferSizeBytes"/>. On Unix hosts (the macOS test matrix, where named pipes
/// are Unix domain sockets) the sizes are advisory — the fix targets Windows production.
/// </remarks>
private static NamedPipeServerStream CreatePipe(string pipeName)
{
return new NamedPipeServerStream(
@@ -162,7 +179,9 @@ public sealed class SessionWorkerClientFactory : ISessionWorkerClientFactory
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
PipeOptions.Asynchronous,
inBufferSize: PipeBufferSizeBytes,
outBufferSize: PipeBufferSizeBytes);
}
/// <summary>Waits for a client to connect to the pipe.</summary>
@@ -120,6 +120,10 @@ internal static class SparseArrayExpander
case MxDataType.Boolean:
{
BoolArray values = new();
// Size the backing store once: the fill below adds exactly `length` elements,
// so without this the RepeatedField doubles its array log2(length) times.
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(false);
@@ -137,6 +141,7 @@ internal static class SparseArrayExpander
case MxDataType.Integer when UsesInt64(elements):
{
Int64Array values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(0L);
@@ -154,6 +159,7 @@ internal static class SparseArrayExpander
case MxDataType.Integer:
{
Int32Array values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(0);
@@ -171,6 +177,7 @@ internal static class SparseArrayExpander
case MxDataType.Float:
{
FloatArray values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(0f);
@@ -188,6 +195,7 @@ internal static class SparseArrayExpander
case MxDataType.Double:
{
DoubleArray values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(0d);
@@ -205,6 +213,7 @@ internal static class SparseArrayExpander
case MxDataType.String:
{
StringArray values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(string.Empty);
@@ -222,6 +231,7 @@ internal static class SparseArrayExpander
case MxDataType.Time:
{
TimestampArray values = new();
values.Values.Capacity = length;
for (int i = 0; i < length; i++)
{
values.Values.Add(new Timestamp { Seconds = 0, Nanos = 0 });
@@ -40,6 +40,12 @@ public sealed class WorkerClient : IWorkerClient
private readonly ConcurrentDictionary<string, PendingCommand> _pendingCommands = new(StringComparer.Ordinal);
private readonly SemaphoreSlim _pendingCommandSlots;
private readonly CancellationTokenSource _stopCts = new();
// GWC-30: this client's contribution to the gateway-wide worker queue-depth gauge. Registered
// once here and read only when the gauge is scraped, so staging and consuming an event cost an
// Interlocked on _eventQueueDepth and nothing else — previously each of those two hot-path steps
// called into GatewayMetrics and took its process-wide lock. Null when metrics are disabled.
private readonly IDisposable? _eventQueueDepthRegistration;
// Touched only by WriteLoopAsync — the single consumer of _outboundEnvelopes — so it needs no
// interlocking. See WriteLoopAsync for why the stamp happens there rather than at construction.
private ulong _nextSequence;
@@ -111,6 +117,8 @@ public sealed class WorkerClient : IWorkerClient
FullMode = BoundedChannelFullMode.Wait,
AllowSynchronousContinuations = false,
});
_eventQueueDepthRegistration = _metrics?.RegisterWorkerEventQueueDepthSource(
() => Volatile.Read(ref _eventQueueDepth));
_lastHeartbeatAt = _timeProvider.GetUtcNow();
}
@@ -224,6 +232,13 @@ public sealed class WorkerClient : IWorkerClient
// session. Command envelopes are the only gateway-authored outbound payload whose
// size the caller controls; checking here keeps a MessageTooLarge in the write loop a
// genuine desync signal.
//
// PERF(GWC-31): this size cannot be handed to WorkerFrameWriter to spare its own
// CalculateSize. WriteLoopAsync stamps envelope.Sequence immediately before the write
// (GWC-28), and a non-zero varint field grows the encoding — so the number computed here
// is a lower bound on the frame the writer actually emits, never the frame length. Passing
// it as a knownSize would under-length the prefix and desync the worker's framing. The
// pre-check stays a pre-check: it is conservative in the right direction.
int envelopeSize = commandEnvelope.CalculateSize();
if (envelopeSize > _connection.FrameOptions.MaxMessageBytes)
{
@@ -234,36 +249,74 @@ public sealed class WorkerClient : IWorkerClient
}
await EnqueueAsync(commandEnvelope, cancellationToken).ConfigureAwait(false);
using CancellationTokenSource timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
Task timeoutTask = Task.Delay(timeout, timeoutCts.Token);
Task<WorkerCommandReply> replyTask = pendingCommand.Task;
Task completedTask = await Task.WhenAny(replyTask, timeoutTask).ConfigureAwait(false);
if (completedTask == replyTask)
// GWC-31: one pooled timer instead of a linked CTS + Task.Delay + WhenAny per command.
// Task.WaitAsync arms a TimerQueueTimer on the shared timer queue and cancels it when the
// reply lands, so the steady-state cost of a command that replies in time is a single
// continuation — the old shape allocated a linked CancellationTokenSource, its
// registration, a delay Task, and the WhenAny Task on every invoke, and left the delay
// Task rooted until the cancel completed. WaitAsync raises TimeoutException for the
// deadline and OperationCanceledException for the caller's token — but unlike the old
// wait it races the two and reports whichever fired first, whereas the old code inspected
// cancellationToken.IsCancellationRequested BEFORE classifying a won delay as a timeout.
// The filter on the CommandTimeout clause restores that priority: a token canceled around
// the deadline is still classified as cancellation, never as CommandTimeout. Error codes
// and messages are unchanged.
try
{
await timeoutCts.CancelAsync().ConfigureAwait(false);
return await replyTask.ConfigureAwait(false);
return await pendingCommand.Task.WaitAsync(timeout, cancellationToken).ConfigureAwait(false);
}
catch (TimeoutException) when (!cancellationToken.IsCancellationRequested)
{
string timeoutMessage = $"Worker command {method} timed out after {timeout}.";
bool removed = RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.CommandTimeout,
timeoutMessage);
if (cancellationToken.IsCancellationRequested)
// The gateway has stopped waiting, but the worker has not stopped working: the
// correlation is still on its single STA queue and would execute (or keep executing)
// regardless. Tell it, so a queued-but-not-started command is dropped instead of
// occupying the STA behind a caller that is already gone. Gated on the removal so a
// reply that won the race — the pending entry is already gone and the caller is about
// to see it — never has a cancel chase it. Best-effort by design; the send cannot
// throw, so it can never replace the timeout the caller is owed.
if (removed)
{
TrySendCancelForTimedOutCommand(correlationId, method, timeout);
}
throw new WorkerClientException(
WorkerClientErrorCode.CommandTimeout,
timeoutMessage);
}
catch (OperationCanceledException)
{
RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.GatewayShutdown,
"Command wait was canceled.");
// WaitAsync surfaces TaskCanceledException; throwing through the token keeps the
// exception the caller observes exactly what the hand-rolled wait produced.
cancellationToken.ThrowIfCancellationRequested();
throw;
}
catch (TimeoutException)
{
// The deadline and the caller's cancellation raced and WaitAsync picked the timer.
// The old wait classified this as cancellation, so this clause — reached only when
// the filter above saw a canceled token — reproduces that treatment exactly.
RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.GatewayShutdown,
"Command wait was canceled.");
cancellationToken.ThrowIfCancellationRequested();
throw;
}
RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.CommandTimeout,
$"Worker command {method} timed out after {timeout}.");
throw new WorkerClientException(
WorkerClientErrorCode.CommandTimeout,
$"Worker command {method} timed out after {timeout}.");
}
catch
{
@@ -299,8 +352,8 @@ public sealed class WorkerClient : IWorkerClient
{
await foreach (WorkerEvent workerEvent in _events.Reader.ReadAllAsync(cancellationToken).ConfigureAwait(false))
{
int queueDepth = Math.Max(0, Interlocked.Decrement(ref _eventQueueDepth));
_metrics?.SetWorkerEventQueueDepth(queueDepth);
// No metrics call on the hot path: the gauge pulls _eventQueueDepth when scraped (GWC-30).
Interlocked.Decrement(ref _eventQueueDepth);
yield return workerEvent;
}
}
@@ -371,6 +424,9 @@ public sealed class WorkerClient : IWorkerClient
}
_disposed = true;
// Drop out of the worker queue-depth gauge before teardown: whatever this client still holds
// is about to be discarded, and the sum must not keep counting a client that is going away.
_eventQueueDepthRegistration?.Dispose();
KillOwnedProcess("Dispose");
_stopCts.Cancel();
_outboundEnvelopes.Writer.TryComplete();
@@ -598,8 +654,7 @@ public sealed class WorkerClient : IWorkerClient
{
// Counted here rather than at the _events write so the single gauge reports total
// undelivered events (staged + queued). ReadEventsCoreAsync decrements on consumer read.
int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
_metrics?.SetWorkerEventQueueDepth(queueDepth);
Interlocked.Increment(ref _eventQueueDepth);
return;
}
@@ -725,7 +780,12 @@ public sealed class WorkerClient : IWorkerClient
/// <param name="pendingCommand">The pending command.</param>
/// <param name="errorCode">Error code.</param>
/// <param name="message">Error message.</param>
private void RemovePendingCommandAsFailed(
/// <returns>
/// <c>true</c> when this call removed the pending entry and owns the failure; <c>false</c> when
/// the entry was already gone — a reply, fault, or shutdown got there first, so the caller must
/// not take any further action on behalf of that correlation.
/// </returns>
private bool RemovePendingCommandAsFailed(
string correlationId,
PendingCommand pendingCommand,
WorkerClientErrorCode errorCode,
@@ -733,13 +793,98 @@ public sealed class WorkerClient : IWorkerClient
{
if (!_pendingCommands.TryRemove(correlationId, out _))
{
return;
return false;
}
ReleasePendingCommandSlot();
TimeSpan duration = _timeProvider.GetElapsedTime(pendingCommand.StartTimestamp);
_metrics?.CommandFailed(pendingCommand.Method, errorCode.ToString(), duration);
pendingCommand.SetException(new WorkerClientException(errorCode, message));
return true;
}
/// <summary>
/// Forwards a <c>WorkerCancel</c> for a correlation the gateway has given up waiting for, so the
/// worker can drop it from its STA queue (<c>WorkerPipeSession</c> routes the envelope to
/// <c>CancelCommand</c>). A cancel that arrives after the command already reached the COM call
/// is a no-op — MXAccess offers no way to abort an in-flight call — so this shortens the STA
/// backlog rather than freeing a call already running on it.
/// <para>
/// A command whose envelope has not yet left <c>_outboundEnvelopes</c> is handled by the same
/// path rather than by pulling it back out: <see cref="Channel{T}"/> exposes no removal, and
/// the queue is FIFO, so the worker simply reads the command and then its cancel and drops the
/// correlation before it ever reaches the STA. Nothing is gained by dequeuing it here.
/// </para>
/// </summary>
/// <remarks>
/// The whole body sits under one catch-all that debug-logs, because the guarantee this method
/// owes its caller is structural, not incidental: the caller is on the throw path for the
/// timeout, so anything escaping here — an envelope that failed to build, a <c>TryWrite</c>
/// against a disposed channel, a scheduler refusing the detached task — would replace the
/// <see cref="WorkerClientErrorCode.CommandTimeout"/> the caller is owed with an unrelated
/// exception. Losing the cancel costs the worker one wasted command; losing the timeout
/// misreports why the call failed. The detached task carries its own handler for the same
/// reason: its failures (including the <see cref="ObjectDisposedException"/> from
/// <c>_stopCts</c> if the client is disposed underneath it) happen after this method returns
/// and would otherwise be unobserved. It is deliberately not tracked or awaited — it holds no
/// resource the shutdown path needs back, and the outbound channel is completed on close.
/// </remarks>
/// <param name="correlationId">Correlation id of the command that timed out.</param>
/// <param name="method">Command method name, for the cancel reason and diagnostics.</param>
/// <param name="timeout">The elapsed command timeout, for the cancel reason.</param>
private void TrySendCancelForTimedOutCommand(
string correlationId,
string method,
TimeSpan timeout)
{
try
{
WorkerEnvelope cancelEnvelope = CreateEnvelope(
correlationId,
envelope => envelope.WorkerCancel = new WorkerCancel
{
Reason = $"gateway command timeout after {timeout}",
});
if (_outboundEnvelopes.Writer.TryWrite(cancelEnvelope))
{
return;
}
_ = Task.Run(async () =>
{
try
{
await EnqueueAsync(cancelEnvelope, _stopCts.Token).ConfigureAwait(false);
}
catch (Exception exception)
{
LogCancelNotForwarded(exception, method, correlationId);
}
});
}
catch (Exception exception)
{
LogCancelNotForwarded(exception, method, correlationId);
}
}
/// <summary>Records a cancel that could not be forwarded for a timed-out command.</summary>
/// <param name="exception">The failure that stopped the cancel from being sent.</param>
/// <param name="method">Command method name of the timed-out command.</param>
/// <param name="correlationId">Correlation id of the timed-out command.</param>
private void LogCancelNotForwarded(
Exception exception,
string method,
string correlationId)
{
_logger.LogDebug(
exception,
"Could not forward a cancel for timed-out worker command {Method} on session {SessionId} "
+ "and correlation {CorrelationId}.",
method,
SessionId,
correlationId);
}
/// <summary>Reads and validates a handshake envelope.</summary>
@@ -29,11 +29,34 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
public const string WorkerWriteCompletionWaitEnvironmentVariableName =
"MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS";
/// <summary>
/// Conveys MxGateway:Alarms:PollIntervalMilliseconds to the worker: the
/// cadence at which the worker's STA polls the AVEVA alarm consumer.
/// </summary>
public const string WorkerAlarmPollIntervalEnvironmentVariableName =
"MXGATEWAY_ALARM_POLL_INTERVAL_MS";
/// <summary>
/// Conveys MxGateway:Alarms:MaxAlarmsPerFetch to the worker: the cap
/// passed to GetXmlCurrentAlarms2, which is also the record count at
/// which the worker treats a fetch as truncated.
/// </summary>
public const string WorkerMaxAlarmsPerFetchEnvironmentVariableName =
"MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH";
/// <summary>
/// Conveys MxGateway:Worker:EventQueueCapacity to the worker: the capacity
/// of the outbound MXAccess event queue, whose overflow faults the session.
/// </summary>
public const string WorkerEventQueueCapacityEnvironmentVariableName =
"MXGATEWAY_EVENT_QUEUE_CAPACITY";
private readonly IWorkerProcessFactory _processFactory;
private readonly IWorkerStartupProbe _startupProbe;
private readonly GatewayMetrics _metrics;
private readonly TimeProvider _timeProvider;
private readonly WorkerOptions _workerOptions;
private readonly AlarmsOptions _alarmsOptions;
private readonly ILogger<WorkerProcessLauncher> _logger;
/// <summary>
@@ -59,6 +82,7 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
ArgumentNullException.ThrowIfNull(metrics);
_workerOptions = gatewayOptions.Value.Worker;
_alarmsOptions = gatewayOptions.Value.Alarms;
_processFactory = processFactory;
_startupProbe = startupProbe;
_metrics = metrics;
@@ -185,6 +209,12 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
_workerOptions.PipeConnectAttemptTimeoutMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerWriteCompletionWaitEnvironmentVariableName] =
_workerOptions.WriteCompletionWaitMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerEventQueueCapacityEnvironmentVariableName] =
_workerOptions.EventQueueCapacity.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerAlarmPollIntervalEnvironmentVariableName] =
_alarmsOptions.PollIntervalMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerMaxAlarmsPerFetchEnvironmentVariableName] =
_alarmsOptions.MaxAlarmsPerFetch.ToString(System.Globalization.CultureInfo.InvariantCulture);
commandLine = new WorkerProcessCommandLine(executablePath, arguments);
@@ -10,20 +10,20 @@
<ItemGroup>
<PackageReference Include="Grpc.AspNetCore" Version="2.76.0" />
<PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.2.1" />
<PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.2.1" />
<PackageReference Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.2.1" />
<PackageReference Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.2.1" />
<PackageReference Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.Theme" Version="0.4.1" />
<PackageReference Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.Health" Version="0.2.0" />
<PackageReference Include="ZB.MOM.WW.Health" Version="0.3.0" />
<PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.GalaxyRepository" Version="0.2.0" />
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.6.2" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.2" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
<PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
@@ -12,7 +12,6 @@
},
"AllowedHosts": "*",
"Secrets": {
"SqlitePath": "mxgateway-secrets.db",
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30"
@@ -87,7 +86,9 @@
"Enabled": true,
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
"DefaultArea": "",
"ReconcileIntervalSeconds": 30
"ReconcileIntervalSeconds": 30,
"PollIntervalMilliseconds": 500,
"MaxAlarmsPerFetch": 1024
}
}
}
@@ -282,6 +282,55 @@ public sealed class GatewayAlarmMonitorAttachOrderTests
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>
/// <see cref="GatewayAlarmMonitor.CurrentAlarms"/> clones the whole active-alarm set under
/// the broadcast lock, so rebuilding it per read stalls every transition and broadcast
/// behind the copy once the dashboard polls a large alarm set. The projection is memoized
/// for as long as the set is unchanged, and every mutation must invalidate it — a stale
/// projection would hide live transitions from the dashboard and the QueryActiveAlarms RPC.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task CurrentAlarmsProjectionIsMemoizedUntilTheAlarmSetChanges()
{
using GatewayMetrics metrics = new();
await using FakeSessionManager sessions = new();
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForSubscribeStartAsync(WaitTimeout);
// Seed through a reconcile (forced by a provider-mode probe) so the cache holds one
// unacked alarm and no further mutation is in flight.
sessions.SetReconcileSnapshot(Snapshot(AlarmConditionState.Active));
sessions.EmitEvent(ProviderModeProbe(1));
await WaitUntilAsync(
() => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference
&& alarm.CurrentState == AlarmConditionState.Active),
WaitTimeout);
IReadOnlyList<ActiveAlarmSnapshot> first = monitor.CurrentAlarms;
Assert.Same(first, monitor.CurrentAlarms);
// A live Acknowledge replaces the cached snapshot, so the next read must rebuild.
sessions.EmitEvent(Transition(2, AlarmTransitionKind.Acknowledge));
await WaitUntilAsync(
() => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference
&& alarm.CurrentState == AlarmConditionState.ActiveAcked),
WaitTimeout);
IReadOnlyList<ActiveAlarmSnapshot> second = monitor.CurrentAlarms;
Assert.NotSame(first, second);
Assert.Same(second, monitor.CurrentAlarms);
// The pre-transition projection is a snapshot of the old generation, not a live view.
Assert.Equal(AlarmConditionState.Active, Assert.Single(first).CurrentState);
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics)
{
AlarmsOptions options = new()
@@ -63,6 +63,49 @@ public sealed class GalaxyRepositoryOptionsValidatorTests
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies an absolute snapshot path inside the application directory fails. Rooted is not the
/// same as safe: the upgrade procedure renames that directory, so a snapshot cached there is
/// discarded on every deploy and the gateway starts cold each time.
/// </summary>
[Fact]
public void Validate_Fails_WhenSnapshotPathIsUnderContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GalaxyRepositoryOptions options = new()
{
PersistSnapshot = true,
SnapshotCachePath = Path.Combine(contentRoot, "galaxy-snapshot.json"),
};
ValidateOptionsResult result =
new GalaxyRepositoryOptionsValidator(contentRoot).Validate(null, options);
Assert.True(result.Failed);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Galaxy:SnapshotCachePath")
&& f.Contains("must not be inside the application directory"));
}
/// <summary>Verifies a snapshot path outside the application directory still passes.</summary>
[Fact]
public void Validate_Succeeds_WhenSnapshotPathIsOutsideContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GalaxyRepositoryOptions options = new()
{
PersistSnapshot = true,
SnapshotCachePath =
Path.Combine(Path.GetTempPath(), $"mxgw-data-{Guid.NewGuid():N}", "galaxy-snapshot.json"),
};
ValidateOptionsResult result =
new GalaxyRepositoryOptionsValidator(contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies the gateway supplies a rooted per-OS default when the shipped config leaves
/// SnapshotCachePath blank, so the removed appsettings literal is not needed and validation
@@ -163,6 +163,121 @@ public sealed class GatewayOptionsValidatorTests
Tls = source.Tls,
};
/// <summary>Verifies the alarm poll cadence and per-fetch cap defaults pass validation.</summary>
[Fact]
public void Validate_Succeeds_WithDefaultAlarmPollCadenceAndFetchCap()
{
AlarmsOptions alarms = new();
Assert.Equal(500, alarms.PollIntervalMilliseconds);
Assert.Equal(1024, alarms.MaxAlarmsPerFetch);
ValidateOptionsResult result = new GatewayOptionsValidator()
.Validate(null, CloneWithAlarms(ValidOptions(), alarms));
Assert.True(result.Succeeded);
}
/// <summary>
/// A poll cadence outside the 100 ms 1 h range must fail validation.
/// Both values are stamped onto every worker launch environment, so
/// they are validated whether or not the central alarm monitor is
/// enabled.
/// </summary>
/// <param name="pollIntervalMilliseconds">Cadence under test.</param>
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
[Theory]
[InlineData(99, false)]
[InlineData(0, false)]
[InlineData(-1, false)]
[InlineData(99, true)]
// Above the one-hour ceiling the cadence stops being a cadence: int.MaxValue
// milliseconds is ~24 days, which silently disables alarm polling.
[InlineData(3_600_001, false)]
[InlineData(int.MaxValue, false)]
[InlineData(int.MaxValue, true)]
public void Validate_Fails_WhenAlarmPollIntervalOutOfRange(
int pollIntervalMilliseconds,
bool alarmsEnabled)
{
GatewayOptions options = CloneWithAlarms(
ValidOptions(),
new AlarmsOptions
{
Enabled = alarmsEnabled,
DefaultArea = "Galaxy",
PollIntervalMilliseconds = pollIntervalMilliseconds,
});
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
Assert.False(result.Succeeded);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Alarms:PollIntervalMilliseconds", StringComparison.Ordinal));
}
/// <summary>
/// A per-fetch cap outside the 64 65,536 range must fail validation.
/// The cap doubles as the truncation-detection threshold in the worker,
/// so a tiny cap would make almost every fetch read as truncated; and
/// the worker is a 32-bit process that materializes each reply as one
/// BSTR plus a full XmlDocument, so an unbounded cap is an
/// out-of-memory fault on the STA rather than a slow poll.
/// </summary>
/// <param name="maxAlarmsPerFetch">Cap under test.</param>
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
[Theory]
[InlineData(63, false)]
[InlineData(0, false)]
[InlineData(-1, false)]
[InlineData(63, true)]
[InlineData(65_537, false)]
[InlineData(int.MaxValue, false)]
[InlineData(int.MaxValue, true)]
public void Validate_Fails_WhenMaxAlarmsPerFetchOutOfRange(
int maxAlarmsPerFetch,
bool alarmsEnabled)
{
GatewayOptions options = CloneWithAlarms(
ValidOptions(),
new AlarmsOptions
{
Enabled = alarmsEnabled,
DefaultArea = "Galaxy",
MaxAlarmsPerFetch = maxAlarmsPerFetch,
});
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
Assert.False(result.Succeeded);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Alarms:MaxAlarmsPerFetch", StringComparison.Ordinal));
}
/// <summary>Verifies the boundary values themselves are accepted at both ends.</summary>
/// <param name="pollIntervalMilliseconds">Cadence under test.</param>
/// <param name="maxAlarmsPerFetch">Cap under test.</param>
[Theory]
[InlineData(100, 64)] // floors
[InlineData(3_600_000, 65_536)] // ceilings
public void Validate_Succeeds_AtAlarmPollCadenceAndFetchCapBoundaries(
int pollIntervalMilliseconds,
int maxAlarmsPerFetch)
{
GatewayOptions options = CloneWithAlarms(
ValidOptions(),
new AlarmsOptions
{
Enabled = true,
DefaultArea = "Galaxy",
PollIntervalMilliseconds = pollIntervalMilliseconds,
MaxAlarmsPerFetch = maxAlarmsPerFetch,
});
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>Verifies an invalid fallback mode is not validated when alarms are disabled.</summary>
[Fact]
public void Validate_Succeeds_WhenAlarmsDisabled_FallbackNotValidated()
@@ -598,6 +713,69 @@ public sealed class GatewayOptionsValidatorTests
f => f.Contains("MxGateway:Authentication:SqlitePath") && f.Contains("rooted"));
}
/// <summary>
/// Verifies an absolute auth DB path <em>inside</em> the application directory fails. This is
/// the gap the rooted check does not close: the path that lost every API key on a production
/// host on 2026-08-09 was absolute and passed rooting cleanly — it simply lived in the directory
/// the upgrade procedure renames away.
/// </summary>
[Fact]
public void Validate_Fails_WhenSqlitePathIsUnderContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions { SqlitePath = Path.Combine(contentRoot, "gateway-auth.db") });
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Failed);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Authentication:SqlitePath")
&& f.Contains("must not be inside the application directory"));
}
/// <summary>
/// Verifies the content-root rule is not a bare string prefix test: a sibling directory whose
/// name merely begins with the content root's must still pass.
/// </summary>
[Fact]
public void Validate_Succeeds_WhenSqlitePathIsSiblingOfContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions { SqlitePath = contentRoot + "-data" + Path.DirectorySeparatorChar + "gateway-auth.db" });
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies a store path outside the application directory passes — the rule must reject only
/// the genuinely unsafe location, not every absolute path.
/// </summary>
[Fact]
public void Validate_Succeeds_WhenSqlitePathIsOutsideContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions
{
SqlitePath = Path.Combine(Path.GetTempPath(), $"mxgw-data-{Guid.NewGuid():N}", "gateway-auth.db"),
});
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies rooting is host-meaningful (SEC-33): a Windows drive-qualified literal fails on a
/// Unix host (where it is not rooted) rather than being blessed and written as a junk-named
@@ -939,4 +1117,59 @@ public sealed class GatewayOptionsValidatorTests
Assert.True(result.Failed);
Assert.Contains(result.Failures!, f => f.Contains("MaxMessageBytes") && f.Contains("reserve"));
}
/// <summary>
/// Verifies the shipped worker event-queue capacity default passes validation and still matches
/// the worker-side <c>MxAccessEventQueue.DefaultCapacity</c> the environment variable falls back
/// to when the launcher value is unusable.
/// </summary>
[Fact]
public void Validate_Succeeds_WithDefaultEventQueueCapacity()
{
Assert.Equal(10000, new WorkerOptions().EventQueueCapacity);
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, ValidOptions());
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies a worker event-queue capacity outside the supported range fails validation. Too
/// small leaves no burst headroom (an overflow faults the whole session); too large commits the
/// 32-bit worker to an outsized pre-allocation.
/// </summary>
/// <param name="eventQueueCapacity">Capacity under test.</param>
[Theory]
[InlineData(0)]
[InlineData(-1)]
[InlineData(999)]
[InlineData(1_000_001)]
public void Validate_Fails_WhenEventQueueCapacityOutOfRange(int eventQueueCapacity)
{
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
null,
WithWorkerAndProtocol(
new WorkerOptions { EventQueueCapacity = eventQueueCapacity },
new ProtocolOptions()));
Assert.True(result.Failed);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Worker:EventQueueCapacity", StringComparison.Ordinal));
}
/// <summary>Verifies the worker event-queue capacity bounds themselves are accepted.</summary>
/// <param name="eventQueueCapacity">Capacity under test.</param>
[Theory]
[InlineData(1000)]
[InlineData(1_000_000)]
public void Validate_Succeeds_AtEventQueueCapacityBounds(int eventQueueCapacity)
{
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
null,
WithWorkerAndProtocol(
new WorkerOptions { EventQueueCapacity = eventQueueCapacity },
new ProtocolOptions()));
Assert.True(result.Succeeded);
}
}
@@ -0,0 +1,128 @@
using Microsoft.AspNetCore.Builder;
using ZB.MOM.WW.MxGateway.Server;
namespace ZB.MOM.WW.MxGateway.Tests.Configuration;
// Mutates the process-global Secrets__SqlitePath that GatewayApplication.CreateBuilder reads;
// serialized against every other collection so a parallel host-building test cannot inherit the
// deliberately-rejected path. See GlobalEnvironmentCollection.
[Collection(TestSupport.GlobalEnvironmentCollection.Name)]
public sealed class SecretsStorePathGuardTests
{
private const string SqlitePathVariable = "Secrets__SqlitePath";
/// <summary>
/// Verifies the store-path guard runs in the <em>pre-host</em> secrets container, which is the
/// only place it matters.
/// </summary>
/// <remarks>
/// <para>
/// <c>CreateBuilder</c> resolves <c>${secret:}</c> references before the host exists, using a
/// throwaway <see cref="Microsoft.Extensions.DependencyInjection.ServiceCollection"/> that
/// contains no <c>IHostEnvironment</c> — and it runs the store migrator, which <b>creates the
/// database</b>. A library that infers the content root from <c>IHostEnvironment</c> alone
/// cannot distinguish "no content root" from "no host registered" and skips the rule here, so
/// the store is created at the rejected path and only then does the real host refuse to start.
/// The leftover empty database with its <c>-wal</c>/<c>-shm</c> siblings is precisely the
/// artifact that made the 2026-08-09 credential loss read as "the database is there, it's just
/// empty". The gateway therefore passes the content root explicitly.
/// </para>
/// <para>
/// The assertion that no file was created is the load-bearing one. A test that merely observed
/// a failed boot would pass even while the store was being written, because the failure arrives
/// afterwards either way — which is exactly how this defect survived its first release.
/// </para>
/// </remarks>
[Fact]
public void CreateBuilder_RejectsSecretsStoreUnderContentRoot_WithoutCreatingIt()
{
string? original = Environment.GetEnvironmentVariable(SqlitePathVariable);
string contentRoot = ResolveContentRoot();
string rejected = Path.Combine(contentRoot, $"probe-secrets-{Guid.NewGuid():N}.db");
try
{
Environment.SetEnvironmentVariable(SqlitePathVariable, rejected);
// Capture rather than Assert.ThrowsAny, so the store-creation assertions below are
// reported first. Ordering matters here: "it threw" is the weaker claim, and asserting
// it first would mask the stronger one — that nothing was written before it threw.
Exception? thrown = Record.Exception(() => GatewayApplication.CreateBuilder([]));
Assert.False(File.Exists(rejected), $"the rejected store was created at {rejected}");
Assert.False(File.Exists(rejected + "-wal"), "a write-ahead log was created for the rejected store");
Assert.False(File.Exists(rejected + "-shm"), "a shared-memory file was created for the rejected store");
Assert.NotNull(thrown);
}
finally
{
Environment.SetEnvironmentVariable(SqlitePathVariable, original);
// Delete defensively: if the guard ever regresses this test writes a database into the
// content root, which on a dev machine is the source tree.
foreach (string leftover in new[] { rejected, rejected + "-wal", rejected + "-shm" })
{
if (File.Exists(leftover))
{
File.Delete(leftover);
}
}
}
}
/// <summary>
/// Verifies a store path outside the content root is accepted <em>and actually used</em>.
/// </summary>
/// <remarks>
/// The assertion is that the database exists afterwards, not merely that nothing threw. A
/// not-null builder is very close to a tautology once no exception escaped, so it would pass
/// even if the pre-host container had stopped opening the store altogether — which would also
/// silently void the negative test above, since that one can only observe a file the migration
/// would otherwise have written. Proving the accepted path gets a real database is what keeps
/// the rejected-path assertion meaningful.
/// </remarks>
[Fact]
public void CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt()
{
string? original = Environment.GetEnvironmentVariable(SqlitePathVariable);
string directory = Directory.CreateTempSubdirectory("mxgw-secrets-ok").FullName;
string accepted = Path.Combine(directory, "secrets.db");
try
{
Environment.SetEnvironmentVariable(SqlitePathVariable, accepted);
WebApplicationBuilder builder = GatewayApplication.CreateBuilder([]);
Assert.NotNull(builder);
Assert.True(File.Exists(accepted), $"the accepted store was not created at {accepted}");
}
finally
{
Environment.SetEnvironmentVariable(SqlitePathVariable, original);
// The store runs in WAL mode with connection pooling, so a pooled handle can outlive the
// migration and keep secrets.db (plus its -wal/-shm sidecars) open. Windows refuses to
// delete a directory holding open files where Unix does not, so clear the pool first;
// the catch is belt-and-braces for a sidecar whose handle outlasts even that.
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
try
{
Directory.Delete(directory, recursive: true);
}
catch (IOException)
{
// Best-effort cleanup of the temp store; a locked file must not fail the test.
}
catch (UnauthorizedAccessException)
{
// Best-effort cleanup of the temp store; a locked file must not fail the test.
}
}
}
// The content root CreateBuilder will use, taken from a builder created with the suite's normal
// (valid) store path rather than assumed from the test's working directory.
private static string ResolveContentRoot() =>
GatewayApplication.CreateBuilder([]).Environment.ContentRootPath;
}
@@ -0,0 +1,104 @@
using System.Security.Claims;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Auth.AspNetCore;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.Secrets.Ui;
namespace ZB.MOM.WW.MxGateway.Tests.Dashboard;
/// <summary>
/// Covers the authorization decision behind the side rail's Secrets link.
/// </summary>
/// <remarks>
/// <para>
/// The rail gates that item with <c>&lt;AuthorizeView Policy="secrets:manage"&gt;</c> — the same
/// policy the mounted <c>/admin/secrets</c> page enforces — rather than a role literal, so nav
/// visibility cannot drift from page access. These tests pin the policy's verdict per principal,
/// which is the behaviour the gate delegates to.
/// </para>
/// <para>
/// This is deliberately not a rendering test: the suite has no component-testing harness, and
/// adding one to assert a single <c>AuthorizeView</c> would be a large dependency for a small
/// claim. What is asserted here is the part that can actually be wrong — which principals the
/// policy admits. The link's presence in <c>MainLayout.razor</c> and the route's existence are
/// covered separately (see <c>GatewayApplicationTests</c>, which asserts <c>/admin/secrets</c> is
/// mapped), so an unmapped route cannot masquerade as a working link.
/// </para>
/// </remarks>
public sealed class SecretsNavGateTests
{
/// <summary>An Administrator sees the Secrets link, because the policy admits that role.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ManagePolicy_AdmitsAdministrator()
{
await using WebApplication app = GatewayApplication.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
PrincipalWithRoles(DashboardRoles.Admin),
resource: null,
SecretsAuthorization.ManagePolicy);
Assert.True(result.Succeeded);
}
/// <summary>
/// A Viewer does not. This is the case the gate exists for: the secrets page denies a Viewer
/// outright, so an ungated link would be a dead end rather than a degraded-but-useful view —
/// which is why the sibling API Keys item is deliberately left ungated (that page does render
/// read-only for Viewers).
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ManagePolicy_RefusesViewer()
{
await using WebApplication app = GatewayApplication.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
PrincipalWithRoles(DashboardRoles.Viewer),
resource: null,
SecretsAuthorization.ManagePolicy);
Assert.False(result.Succeeded);
}
/// <summary>
/// An unauthenticated principal does not. Covers the anonymous-localhost path, which grants a
/// read-only identity without authenticating it.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ManagePolicy_RefusesUnauthenticated()
{
await using WebApplication app = GatewayApplication.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
new ClaimsPrincipal(new ClaimsIdentity()),
resource: null,
SecretsAuthorization.ManagePolicy);
Assert.False(result.Succeeded);
}
// Mirrors what DashboardAuthenticator issues: roles as ZbClaimTypes.Role (== ClaimTypes.Role),
// with the identity told to treat that claim as its role type. Constructing the identity with
// an authentication type is what makes it authenticated — without one, every policy that
// requires an authenticated user fails for the wrong reason and the role assertions above
// would pass vacuously.
private static ClaimsPrincipal PrincipalWithRoles(params string[] roles)
{
var identity = new ClaimsIdentity(
roles.Select(role => new Claim(ZbClaimTypes.Role, role)),
authenticationType: "Test",
nameType: ZbClaimTypes.Name,
roleType: ZbClaimTypes.Role);
return new ClaimsPrincipal(identity);
}
}
@@ -0,0 +1,145 @@
using System.Security.Claims;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.AspNetCore.Components.Web;
using Microsoft.AspNetCore.Components.Web.HtmlRendering;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.Auth.AspNetCore;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Layout;
using ZB.MOM.WW.Secrets.Ui;
namespace ZB.MOM.WW.MxGateway.Tests.Dashboard;
/// <summary>
/// Renders <see cref="MainLayout"/> and asserts whether the side rail emits the Secrets link.
/// </summary>
/// <remarks>
/// <para>
/// The policy-level tests in <c>SecretsNavGateTests</c> are not sufficient on their own, and the
/// reason is worth stating because it is easy to miss: they would stay green if the
/// <c>&lt;AuthorizeView&gt;</c> were deleted outright. They prove the policy decides correctly, not
/// that the rail asks it. The wiring is the part this change actually introduced, so it is the part
/// that needs its own evidence.
/// </para>
/// <para>
/// The load-bearing assertion is the NEGATIVE one. "An Administrator sees the link" is identical to
/// the behaviour before the gate existed, so it cannot distinguish a working gate from an inert one.
/// Only a principal without <c>secrets:manage</c> failing to see the item proves a gate is there at
/// all — which is why <see cref="Rail_OmitsSecretsLink_ForViewer"/> is the test that matters and the
/// Administrator case is its control.
/// </para>
/// <para>
/// Uses the framework's static <see cref="HtmlRenderer"/> rather than a component-testing package:
/// no new dependency, and static rendering is enough because the assertion is about markup the
/// server emits, not about interactivity.
/// </para>
/// </remarks>
public sealed class SecretsNavRenderTests
{
private const string SecretsHref = "/admin/secrets";
/// <summary>
/// The gate's proof. A Viewer holds a real, authenticated identity and still must not be offered
/// the link, because the page would refuse them.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Rail_OmitsSecretsLink_ForViewer()
{
string html = await RenderRailAsync(DashboardRoles.Viewer);
Assert.DoesNotContain(SecretsHref, html, StringComparison.Ordinal);
}
/// <summary>Anonymous callers (the read-only localhost path) are likewise not offered it.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Rail_OmitsSecretsLink_ForAnonymous()
{
string html = await RenderRailAsync();
Assert.DoesNotContain(SecretsHref, html, StringComparison.Ordinal);
}
/// <summary>
/// The control for the two negatives. Without this, a rail that rendered no nav at all — a
/// broken layout, a throwing component swallowed somewhere — would satisfy both absence
/// assertions and the suite would report a working gate over a blank page. The sibling
/// assertions on the always-present items are what make the absence above mean "gated" rather
/// than "nothing rendered".
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Rail_EmitsSecretsLink_ForAdministrator()
{
string html = await RenderRailAsync(DashboardRoles.Admin);
Assert.Contains(SecretsHref, html, StringComparison.Ordinal);
Assert.Contains("/apikeys", html, StringComparison.Ordinal);
}
/// <summary>
/// Pins the deliberate asymmetry: the ungated sibling stays visible to a Viewer. If someone
/// later "fixes the inconsistency" by wrapping the API Keys item in the same AuthorizeView,
/// this fails — that page renders read-only for Viewers, so hiding its link would remove
/// legitimate access.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Rail_StillEmitsApiKeysLink_ForViewer()
{
string html = await RenderRailAsync(DashboardRoles.Viewer);
Assert.Contains("/apikeys", html, StringComparison.Ordinal);
}
private static async Task<string> RenderRailAsync(params string[] roles)
{
var services = new ServiceCollection();
services.AddLogging();
services.AddAuthorization(options => options.AddSecretsAuthorization());
services.AddCascadingAuthenticationState();
services.AddSingleton<AuthenticationStateProvider>(new StubAuthenticationStateProvider(roles));
services.AddSingleton<NavigationManager, StubNavigationManager>();
await using ServiceProvider provider = services.BuildServiceProvider();
await using var renderer = new HtmlRenderer(
provider,
provider.GetRequiredService<ILoggerFactory>());
return await renderer.Dispatcher.InvokeAsync(async () =>
{
HtmlRootComponent output = await renderer.RenderComponentAsync<MainLayout>();
return output.ToHtmlString();
});
}
// Supplies the authentication state the rail's AuthorizeView reads. An empty role list yields an
// unauthenticated principal; otherwise the identity carries an authentication type, without
// which every policy would fail for the wrong reason and the negative assertions would pass
// vacuously.
private sealed class StubAuthenticationStateProvider(string[] roles) : AuthenticationStateProvider
{
public override Task<AuthenticationState> GetAuthenticationStateAsync()
{
ClaimsIdentity identity = roles.Length == 0
? new ClaimsIdentity()
: new ClaimsIdentity(
roles.Select(role => new Claim(ZbClaimTypes.Role, role)),
authenticationType: "Test",
nameType: ZbClaimTypes.Name,
roleType: ZbClaimTypes.Role);
return Task.FromResult(new AuthenticationState(new ClaimsPrincipal(identity)));
}
}
// NavLink resolves hrefs against the current URI, so the rail needs a NavigationManager even
// under static rendering.
private sealed class StubNavigationManager : NavigationManager
{
public StubNavigationManager() => Initialize("https://localhost/", "https://localhost/");
}
}
@@ -24,6 +24,94 @@ public sealed class GatewayLogRedactorTests
Assert.DoesNotContain("super_secret_value", redacted);
}
/// <summary>
/// Verifies that a bearer credential the gateway does not issue is redacted too. A client that
/// pastes a JWT (or any other token) into the authorization header must not have it logged.
/// </summary>
[Fact]
public void RedactClientIdentity_RedactsNonGatewayBearerCredential()
{
const string token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIn0.c2lnbmF0dXJl";
string? redacted = GatewayLogRedactor.RedactClientIdentity($"Bearer {token}");
Assert.Equal("Bearer [redacted]", redacted);
Assert.DoesNotContain(token, redacted, StringComparison.Ordinal);
Assert.DoesNotContain("eyJ", redacted, StringComparison.Ordinal);
}
/// <summary>Verifies that a gateway API key keeps its key-id shape so an operator can still tell keys apart.</summary>
[Fact]
public void RedactClientIdentity_PreservesGatewayKeyIdShape()
{
string? redacted = GatewayLogRedactor.RedactClientIdentity("Bearer mxgw_operator01_super-secret");
Assert.Equal("Bearer mxgw_operator01_[redacted]", redacted);
Assert.DoesNotContain("super-secret", redacted, StringComparison.Ordinal);
}
/// <summary>
/// Pins the 64-character key-id boundary in both directions. Nothing validates key-id length at
/// creation time (neither <c>ApiKeyAdminCommandLineParser.IsValidKeyId</c> nor
/// <c>DashboardApiKeyManagementService.ValidateKeyId</c> caps it), so the cap here is a redaction
/// heuristic that a real key id can cross. This test fixes which way it fails when it does: at
/// exactly 64 the id is still an identifier and survives; at 65 the whole run is treated as secret
/// material and goes. Losing an identifier is the cheap failure; logging a secret is not.
/// </summary>
/// <param name="keyIdLength">Length of the key id presented before the secret separator.</param>
/// <param name="expectsKeyIdPreserved">Whether the key id must survive redaction at that length.</param>
[Theory]
[InlineData(64, true)]
[InlineData(65, false)]
public void RedactClientIdentity_KeyIdLengthBoundary_FailsTowardRedaction(
int keyIdLength,
bool expectsKeyIdPreserved)
{
string keyId = new('a', keyIdLength);
string? redacted = GatewayLogRedactor.RedactClientIdentity($"Bearer mxgw_{keyId}_super-secret");
Assert.Equal(
expectsKeyIdPreserved ? $"Bearer mxgw_{keyId}_[redacted]" : "Bearer mxgw_[redacted]",
redacted);
Assert.DoesNotContain("super-secret", redacted, StringComparison.Ordinal);
}
/// <summary>
/// Verifies that anything not recognized as a gateway API key fails closed: the scheme word
/// survives only when it looks like an auth scheme, and the credential never does.
/// </summary>
/// <param name="clientIdentity">The raw client identity value.</param>
/// <param name="expected">The expected redacted value.</param>
[Theory]
[InlineData("Bearer", "[redacted]")]
[InlineData("Bearer ", "[redacted]")]
[InlineData("Basic dXNlcjpwYXNzd29yZA==", "Basic [redacted]")]
[InlineData("Negotiate YIIJvwYGKwYBBQUCoIIJ", "Negotiate [redacted]")]
[InlineData("mxgw_operator01_super-secret", "[redacted]")]
[InlineData("mxgw_operator01_super-secret trailing", "[redacted]")]
[InlineData("Bearer mxgw_operator01", "Bearer mxgw_[redacted]")]
[InlineData("Bearer mxgw_", "Bearer mxgw_[redacted]")]
[InlineData("Bearer mxgw__super-secret", "Bearer mxgw_[redacted]")]
[InlineData("bearer mxgw_operator01_super-secret", "bearer mxgw_operator01_[redacted]")]
[InlineData("anonymous", "[redacted]")]
[InlineData("some random junk", "[redacted]")]
public void RedactClientIdentity_FailsClosedForUnrecognizedCredentials(string clientIdentity, string expected)
{
Assert.Equal(expected, GatewayLogRedactor.RedactClientIdentity(clientIdentity));
}
/// <summary>Verifies that a blank client identity is passed through — there is nothing to redact.</summary>
/// <param name="clientIdentity">The raw client identity value.</param>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void RedactClientIdentity_PassesThroughBlankValues(string? clientIdentity)
{
Assert.Equal(clientIdentity, GatewayLogRedactor.RedactClientIdentity(clientIdentity));
}
/// <summary>Verifies that IsCredentialBearingCommand identifies credential-bearing MXAccess commands.</summary>
/// <param name="commandMethod">Name of the MXAccess command method.</param>
[Theory]
@@ -0,0 +1,138 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Diagnostics;
using ZB.MOM.WW.MxGateway.Server.Sessions;
namespace ZB.MOM.WW.MxGateway.Tests.Diagnostics;
public sealed class SessionHealthCheckTests
{
/// <summary>
/// An idle gateway is healthy. This is the load-bearing case: a gateway holding no sessions is
/// the normal steady state on a host nothing dials yet, and a probe that reports red there is
/// one operators learn to ignore.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenNoSessionsAreOpen()
{
var check = new SessionHealthCheck(new SessionRegistry());
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(0, result.Data["total"]);
Assert.Equal("No MXAccess sessions are open.", result.Description);
}
/// <summary>Every session ready reports healthy, with the counts carried as entry data.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenAllSessionsReady()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Ready, SessionState.Ready));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(2, result.Data["total"]);
Assert.Equal(2, result.Data["ready"]);
Assert.Equal(0, result.Data["faulted"]);
}
/// <summary>
/// A faulted session alongside a usable one is degraded, not unhealthy — the gateway is still
/// serving the sessions that work.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Degraded_WhenSomeFaultedAndSomeReady()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Ready, SessionState.Faulted));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Degraded, result.Status);
Assert.Equal(1, result.Data["ready"]);
Assert.Equal(1, result.Data["faulted"]);
Assert.Contains("1 faulted", result.Description, StringComparison.Ordinal);
}
/// <summary>
/// A session still starting counts as usable for grading, so a fault beside it is degraded
/// rather than unhealthy — the startup has not failed yet.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Degraded_WhenFaultedBesideAStartingSession()
{
var check = new SessionHealthCheck(
RegistryWith(SessionState.Faulted, SessionState.StartingWorker));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Degraded, result.Status);
Assert.Equal(1, result.Data["starting"]);
}
/// <summary>Every session faulted is the genuinely bad condition, and the only unhealthy one.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Unhealthy_WhenEverySessionIsFaulted()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Faulted, SessionState.Faulted));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Equal(2, result.Data["faulted"]);
Assert.Equal(0, result.Data["ready"]);
}
/// <summary>
/// Closed sessions linger in the registry until they are removed. They are counted separately
/// and excluded from the verdict, so a gateway whose sessions all closed cleanly is healthy —
/// not unhealthy for having zero ready ones.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenOnlyClosedSessionsRemain()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Closed, SessionState.Closed));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(2, result.Data["closing"]);
Assert.Equal(0, result.Data["ready"]);
}
private static SessionRegistry RegistryWith(params SessionState[] states)
{
var registry = new SessionRegistry();
for (int i = 0; i < states.Length; i++)
{
GatewaySession session = CreateSession($"session-{i}");
session.TransitionTo(states[i]);
Assert.True(registry.TryAdd(session));
}
return registry;
}
private static GatewaySession CreateSession(string sessionId)
{
return new GatewaySession(
sessionId,
"mxaccess",
$"mxaccess-gateway-1-{sessionId}",
"nonce",
clientIdentity: null,
clientSessionName: "test-session",
clientCorrelationId: "client-correlation",
TimeSpan.FromSeconds(30),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(5),
DateTimeOffset.UnixEpoch);
}
}
@@ -11,16 +11,64 @@ namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
/// Verifies that <see cref="DashboardEventBroadcaster"/> honours
/// <c>MxGateway:Dashboard:ShowTagValues</c> (SEC-25): tag values are stripped
/// from the mirrored copy when the flag is off, present when it is on, and the
/// shared source event is never mutated.
/// shared source event is never mutated. Also verifies the viewer gate — the
/// mirror does no work at all for a session nobody is watching.
/// </summary>
public sealed class DashboardEventBroadcasterTests
{
/// <summary>An unwatched session costs neither a redaction clone nor a send.</summary>
[Fact]
public void Publish_WithNoRegisteredViewers_DoesNotCloneOrSend()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
MxEvent source = BuildEventWithValue();
broadcaster.Publish("session-1", source);
Assert.Equal(0, hubContext.SendCount);
Assert.Null(hubContext.LastArgument);
}
/// <summary>A viewer on a different session does not open the gate for this one.</summary>
[Fact]
public void Publish_WithViewersOnAnotherSessionOnly_DoesNotSend()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
viewers.AddViewer("conn-1", "session-2");
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(0, hubContext.SendCount);
}
/// <summary>Once the last viewer leaves, the mirror stops sending again.</summary>
[Fact]
public void Publish_AfterLastViewerLeaves_StopsSending()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
viewers.AddViewer("conn-1", "session-1");
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
viewers.RemoveViewer("conn-1", "session-1");
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
}
/// <summary>Values are stripped from the mirror when ShowTagValues is off; metadata survives.</summary>
[Fact]
public void Publish_WhenShowTagValuesFalse_RedactsValuesButKeepsMetadata()
{
CapturingHubContext hubContext = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false);
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, WatchedSession1());
MxEvent source = BuildEventWithValue();
broadcaster.Publish("session-1", source);
@@ -43,7 +91,7 @@ public sealed class DashboardEventBroadcasterTests
public void Publish_WhenShowTagValuesFalse_DoesNotMutateSourceEvent()
{
CapturingHubContext hubContext = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false);
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, WatchedSession1());
MxEvent source = BuildEventWithValue();
broadcaster.Publish("session-1", source);
@@ -61,7 +109,7 @@ public sealed class DashboardEventBroadcasterTests
public void Publish_WhenShowTagValuesTrue_KeepsValues()
{
CapturingHubContext hubContext = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: true);
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: true, WatchedSession1());
MxEvent source = BuildEventWithValue();
broadcaster.Publish("session-1", source);
@@ -73,7 +121,144 @@ public sealed class DashboardEventBroadcasterTests
Assert.NotNull(sent.OnAlarmTransition.LimitValue);
}
private static DashboardEventBroadcaster Create(CapturingHubContext hubContext, bool showTagValues)
/// <summary>
/// An in-process subscriber gets the same redacted clone the hub group gets,
/// and the shared source event is still left untouched.
/// </summary>
[Fact]
public void Subscribe_WhenShowTagValuesFalse_DeliversRedactedCloneWithoutMutatingSource()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
using IDashboardEventSubscription subscription = broadcaster.Subscribe("session-1");
MxEvent source = BuildEventWithValue();
broadcaster.Publish("session-1", source);
MxEvent received = ReadOne(subscription);
Assert.Null(received.Value);
Assert.Null(received.OnAlarmTransition.CurrentValue);
Assert.Null(received.OnAlarmTransition.LimitValue);
Assert.Equal("Tank01.Level.HiHi", received.OnAlarmTransition.AlarmFullReference);
// One clone feeds both audiences — the hub group and the in-process feed.
Assert.Same(hubContext.LastArgument, received);
// The source is shared with the gRPC stream and the replay ring.
Assert.NotSame(source, received);
Assert.NotNull(source.Value);
Assert.Equal(42.5, source.Value.DoubleValue);
Assert.NotNull(source.OnAlarmTransition.CurrentValue);
Assert.NotNull(source.OnAlarmTransition.LimitValue);
}
/// <summary>An in-process subscription opens the viewer gate the same way a hub client does.</summary>
[Fact]
public void Subscribe_OpensTheViewerGate()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(0, hubContext.SendCount);
Assert.Null(hubContext.LastArgument);
using IDashboardEventSubscription subscription = broadcaster.Subscribe("session-1");
Assert.True(viewers.HasViewers("session-1"));
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
Assert.NotNull(hubContext.LastArgument);
}
/// <summary>Disposing the last in-process subscription restores the no-viewers short-circuit.</summary>
[Fact]
public void Dispose_OfLastInProcessSubscription_RestoresTheShortCircuit()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
IDashboardEventSubscription subscription = broadcaster.Subscribe("session-1");
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
Assert.Equal("session-1", ReadOne(subscription).SessionId);
subscription.Dispose();
Assert.False(viewers.HasViewers("session-1"));
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
Assert.False(subscription.Reader.TryRead(out _));
}
/// <summary>Hub viewers and in-process subscribers are audiences of their own session only.</summary>
[Fact]
public void Publish_DeliversOnlyToTheSubscribedSessionsAudience()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
viewers.AddViewer("conn-1", "session-1");
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
using IDashboardEventSubscription subscription = broadcaster.Subscribe("session-2");
// The hub viewer's session must not spill into the in-process feed.
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
Assert.False(subscription.Reader.TryRead(out _));
broadcaster.Publish("session-2", BuildEventWithValue("session-2"));
Assert.Equal("session-2", ReadOne(subscription).SessionId);
// A session with no audience at all still short-circuits.
broadcaster.Publish("session-3", BuildEventWithValue("session-3"));
Assert.Equal(2, hubContext.SendCount);
}
/// <summary>Disposing twice is a no-op and cannot release a sibling subscription's registration.</summary>
[Fact]
public void Dispose_CalledTwice_IsSafeAndLeavesSiblingSubscriptionsAlone()
{
CapturingHubContext hubContext = new();
EventsHubViewerRegistry viewers = new();
DashboardEventBroadcaster broadcaster = Create(hubContext, showTagValues: false, viewers);
IDashboardEventSubscription first = broadcaster.Subscribe("session-1");
using IDashboardEventSubscription second = broadcaster.Subscribe("session-1");
first.Dispose();
first.Dispose();
Assert.True(viewers.HasViewers("session-1"));
broadcaster.Publish("session-1", BuildEventWithValue());
Assert.Equal(1, hubContext.SendCount);
Assert.False(first.Reader.TryRead(out _));
Assert.Equal("session-1", ReadOne(second).SessionId);
}
/// <summary>Reads exactly one event from a subscription, failing the test if none is queued.</summary>
/// <param name="subscription">The subscription to read from.</param>
/// <returns>The event that was read.</returns>
private static MxEvent ReadOne(IDashboardEventSubscription subscription)
{
Assert.True(subscription.Reader.TryRead(out MxEvent? received));
return Assert.IsType<MxEvent>(received);
}
private static DashboardEventBroadcaster Create(
CapturingHubContext hubContext,
bool showTagValues,
EventsHubViewerRegistry viewers)
{
GatewayOptions gatewayOptions = new()
{
@@ -82,16 +267,29 @@ public sealed class DashboardEventBroadcasterTests
return new DashboardEventBroadcaster(
hubContext,
viewers,
Options.Create(gatewayOptions),
NullLogger<DashboardEventBroadcaster>.Instance);
}
private static MxEvent BuildEventWithValue()
/// <summary>A registry with one hub connection watching <c>session-1</c>.</summary>
/// <returns>The populated registry.</returns>
private static EventsHubViewerRegistry WatchedSession1()
{
EventsHubViewerRegistry viewers = new();
viewers.AddViewer("conn-1", "session-1");
return viewers;
}
/// <summary>Builds a value-bearing alarm-transition event for the given session.</summary>
/// <param name="sessionId">Session id stamped on the event.</param>
/// <returns>The event.</returns>
private static MxEvent BuildEventWithValue(string sessionId = "session-1")
{
return new MxEvent
{
Family = MxEventFamily.OnAlarmTransition,
SessionId = "session-1",
SessionId = sessionId,
ServerHandle = 7,
ItemHandle = 11,
Quality = 192,
@@ -117,6 +315,9 @@ public sealed class DashboardEventBroadcasterTests
/// <summary>Gets the first argument of the most recent send call.</summary>
public object? LastArgument => _clients.GroupProxy.LastArgument;
/// <summary>Gets the number of send calls this fake has observed.</summary>
public int SendCount => _clients.GroupProxy.SendCount;
}
private sealed class CapturingHubClients : IHubClients
@@ -148,6 +349,9 @@ public sealed class DashboardEventBroadcasterTests
/// <summary>Gets the first argument of the most recent send call.</summary>
public object? LastArgument { get; private set; }
/// <summary>Gets the number of send calls made through this proxy.</summary>
public int SendCount { get; private set; }
/// <summary>Records the send call arguments and completes synchronously.</summary>
/// <param name="method">The SignalR method name.</param>
/// <param name="args">The method arguments.</param>
@@ -155,6 +359,7 @@ public sealed class DashboardEventBroadcasterTests
/// <returns>A completed task.</returns>
public Task SendCoreAsync(string method, object?[] args, CancellationToken cancellationToken = default)
{
SendCount++;
LastArgument = args.Length > 0 ? args[0] : null;
return Task.CompletedTask;
}
@@ -45,4 +45,23 @@ public sealed class DashboardHubsRegistrationTests
.GetRequiredService<DashboardHubConnectionFactory>();
Assert.NotNull(factory);
}
/// <summary>
/// The publish and in-process subscribe faces of the event mirror must resolve to
/// one instance: two would leave the session-details page reading a mirror the
/// session pipeline never publishes to, and the viewer gate would never open.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Build_WhenDashboardEnabled_ResolvesBothEventMirrorInterfacesToOneInstance()
{
await using WebApplication app = GatewayApplication.Build([]);
IDashboardEventBroadcaster broadcaster = app.Services.GetRequiredService<IDashboardEventBroadcaster>();
IDashboardSessionEventSubscriber subscriber = app.Services
.GetRequiredService<IDashboardSessionEventSubscriber>();
Assert.Same(broadcaster, subscriber);
Assert.Same(app.Services.GetRequiredService<DashboardEventBroadcaster>(), broadcaster);
}
}
@@ -0,0 +1,454 @@
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Sessions;
using ZB.MOM.WW.MxGateway.Server.Workers;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
public sealed class DashboardLiveDataServiceTests
{
// Mirrors DashboardLiveDataService.MaxSubscribedTags — the cap is private, so the
// tests drive it through the public read surface at exactly its documented size.
private const int MaxSubscribedTags = 256;
/// <summary>
/// Verifies a tag already in the advise set is not subscribed again on a later read.
/// </summary>
[Fact]
public async Task ReadAsync_WhenTagAlreadySubscribed_DoesNotResubscribe()
{
RecordingWorkerClient worker = new();
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
await service.ReadAsync(["Tank_001.PV", "Tank_002.PV"], CancellationToken.None);
DashboardLiveReadResult second = await service.ReadAsync(
["Tank_001.PV", "Tank_002.PV"],
CancellationToken.None);
Assert.Null(second.Error);
Assert.Equal(["Tank_001.PV", "Tank_002.PV"], worker.SubscribedTags);
Assert.Empty(worker.UnsubscribedHandles);
}
/// <summary>
/// Verifies the advise set is capped: subscribing past the cap unadvises the
/// least-recently-read tag on the worker and leaves the re-read tag advised.
/// </summary>
[Fact]
public async Task ReadAsync_PastCap_EvictsLeastRecentlyReadTag()
{
RecordingWorkerClient worker = new();
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
// Within one read the last tag counts as most recently read, so filler[0] is
// the tail of the recency list.
string[] filler = CreateTagAddresses(MaxSubscribedTags);
await service.ReadAsync(filler, CancellationToken.None);
// Re-read the tail, making it most recent: the tag read before it is now the
// eviction candidate.
await service.ReadAsync([filler[0]], CancellationToken.None);
Assert.Equal(MaxSubscribedTags, worker.SubscribedTags.Count);
DashboardLiveReadResult overflow = await service.ReadAsync(
["Overflow.PV"],
CancellationToken.None);
Assert.Null(overflow.Error);
Assert.Equal([worker.HandleFor(filler[1])], worker.UnsubscribedHandles);
Assert.Equal("Overflow.PV", worker.SubscribedTags[^1]);
Assert.Equal(MaxSubscribedTags + 1, worker.SubscribedTags.Count);
// The evicted tag is no longer tracked and re-subscribes; the re-read one does not.
await service.ReadAsync([filler[0], filler[1]], CancellationToken.None);
Assert.Equal(filler[1], worker.SubscribedTags[^1]);
Assert.Equal(MaxSubscribedTags + 2, worker.SubscribedTags.Count);
}
/// <summary>
/// Verifies the cap is per-read, not absolute: a single read of more distinct tags
/// than the cap keeps them all (a read never evicts a tag it is about to return),
/// and the next read that subscribes anything squeezes the overshoot back out.
/// </summary>
[Fact]
public async Task ReadAsync_WithMoreDistinctTagsThanCap_KeepsThemAllThenSelfCorrects()
{
RecordingWorkerClient worker = new();
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
string[] oversize = CreateTagAddresses(300);
DashboardLiveReadResult oversizeResult = await service.ReadAsync(oversize, CancellationToken.None);
Assert.Null(oversizeResult.Error);
Assert.Equal(300, oversizeResult.Values.Count);
Assert.Equal(300, worker.SubscribedTags.Count);
Assert.Empty(worker.UnsubscribedHandles);
// 300 + 1 - 256 = 45 evicted in one pass, landing the set back on the cap.
await service.ReadAsync(["Overflow.PV"], CancellationToken.None);
Assert.Equal(oversize[..45].Select(worker.HandleFor), worker.UnsubscribedHandles);
// Exactly at the cap now: one more new tag evicts exactly one.
worker.UnsubscribedHandles.Clear();
await service.ReadAsync(["Overflow2.PV"], CancellationToken.None);
Assert.Equal([worker.HandleFor(oversize[45])], worker.UnsubscribedHandles);
}
/// <summary>
/// Verifies tags read in the same call are never evicted for each other: a read that
/// touches nearly the whole advise set evicts only the untouched remainder, ends over
/// the cap, and the following read trims it back.
/// </summary>
[Fact]
public async Task ReadAsync_WhenTouchedTagsFillTheCap_EvictsOnlyUntouchedTags()
{
RecordingWorkerClient worker = new();
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
string[] filler = CreateTagAddresses(MaxSubscribedTags);
await service.ReadAsync(filler, CancellationToken.None);
// 250 already-advised tags + 10 new ones: only the 6 untouched tags are
// evictable, so the set ends at 260.
string[] fresh = CreateTagAddresses(10, "Fresh");
await service.ReadAsync([.. filler[..250], .. fresh], CancellationToken.None);
Assert.Equal(filler[250..].Select(worker.HandleFor), worker.UnsubscribedHandles);
// 260 + 1 - 256 = 5 evicted on the next read that subscribes anything.
worker.UnsubscribedHandles.Clear();
await service.ReadAsync(["Overflow.PV"], CancellationToken.None);
Assert.Equal(5, worker.UnsubscribedHandles.Count);
}
/// <summary>
/// Verifies a tag the worker failed to advise still occupies a slot but is evicted
/// without any unsubscribe command — there is no item handle to unadvise.
/// </summary>
[Fact]
public async Task ReadAsync_WhenAdviseFailed_EvictsTagWithoutUnsubscribing()
{
RecordingWorkerClient worker = new();
worker.FailSubscribeFor.Add("Bad.PV");
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
// Bad.PV is read first, so it is the least recently read of the batch and the
// first tag evicted.
string[] filler = CreateTagAddresses(MaxSubscribedTags - 1);
await service.ReadAsync(["Bad.PV", .. filler], CancellationToken.None);
DashboardLiveReadResult overflow = await service.ReadAsync(
["Overflow.PV"],
CancellationToken.None);
Assert.Null(overflow.Error);
Assert.Empty(worker.UnsubscribedHandles);
Assert.Equal(0, worker.UnsubscribeCommandCount);
// It was dropped from tracking all the same, so reading it again re-advises it.
await service.ReadAsync(["Bad.PV"], CancellationToken.None);
Assert.Equal("Bad.PV", worker.SubscribedTags[^1]);
}
/// <summary>
/// Verifies a failed unadvise of an evicted tag does not fail the read, and the
/// evicted tag is dropped from tracking anyway.
/// </summary>
[Fact]
public async Task ReadAsync_WhenEvictionUnsubscribeFails_StillCompletesRead()
{
RecordingWorkerClient worker = new() { FailUnsubscribe = true };
await using FakeSessionManager sessionManager = new(worker);
await using DashboardLiveDataService service = CreateService(sessionManager);
string[] filler = CreateTagAddresses(MaxSubscribedTags);
await service.ReadAsync(filler, CancellationToken.None);
DashboardLiveReadResult overflow = await service.ReadAsync(
["Overflow.PV"],
CancellationToken.None);
Assert.Null(overflow.Error);
Assert.Equal("Overflow.PV", Assert.Single(overflow.Values).TagAddress);
Assert.Equal(1, sessionManager.OpenCount);
// The evicted tag was dropped from tracking despite the failed unadvise.
await service.ReadAsync([filler[0]], CancellationToken.None);
Assert.Equal(filler[0], worker.SubscribedTags[^1]);
}
private static DashboardLiveDataService CreateService(ISessionManager sessionManager)
{
return new DashboardLiveDataService(
sessionManager,
new FakeGatewayAlarmService(),
NullLogger<DashboardLiveDataService>.Instance);
}
private static string[] CreateTagAddresses(int count, string prefix = "Tank")
{
string[] addresses = new string[count];
for (int i = 0; i < count; i++)
{
addresses[i] = $"{prefix}_{i:D4}.PV";
}
return addresses;
}
// Serves the dashboard service a single Ready session backed by the recording
// worker, so reads exercise the real GatewaySession bulk command path.
private sealed class FakeSessionManager(RecordingWorkerClient workerClient) : ISessionManager, IAsyncDisposable
{
private readonly List<GatewaySession> _sessions = [];
/// <summary>Gets the number of sessions the dashboard service opened.</summary>
public int OpenCount { get; private set; }
/// <inheritdoc />
public Task<GatewaySession> OpenSessionAsync(
SessionOpenRequest request,
string? clientIdentity,
string? ownerKeyId,
CancellationToken cancellationToken)
{
OpenCount++;
string sessionId = $"dashboard-session-{OpenCount}";
GatewaySession session = new(
sessionId,
"Galaxy",
$"mxgw-1-{sessionId}",
"nonce",
clientIdentity,
request.ClientSessionName,
request.ClientCorrelationId,
TimeSpan.FromSeconds(30),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(5),
DateTimeOffset.UnixEpoch);
session.AttachWorkerClient(workerClient);
session.MarkReady();
_sessions.Add(session);
return Task.FromResult(session);
}
/// <inheritdoc />
public bool TryGetSession(string sessionId, [MaybeNullWhen(false)] out GatewaySession session)
{
session = _sessions.Find(candidate => candidate.SessionId == sessionId);
return session is not null;
}
/// <inheritdoc />
public Task<SessionCloseResult> CloseSessionAsync(string sessionId, CancellationToken cancellationToken)
{
return Task.FromResult(new SessionCloseResult(sessionId, SessionState.Closed, AlreadyClosed: false));
}
/// <inheritdoc />
public Task<WorkerCommandReply> InvokeAsync(
string sessionId,
WorkerCommand command,
CancellationToken cancellationToken) =>
throw new NotSupportedException();
/// <inheritdoc />
public IAsyncEnumerable<WorkerEvent> ReadEventsAsync(string sessionId, CancellationToken cancellationToken) =>
throw new NotSupportedException();
/// <inheritdoc />
public Task<SessionCloseResult> KillWorkerAsync(
string sessionId,
string reason,
CancellationToken cancellationToken) =>
throw new NotSupportedException();
/// <inheritdoc />
public Task<int> CloseExpiredLeasesAsync(DateTimeOffset now, CancellationToken cancellationToken) =>
throw new NotSupportedException();
/// <inheritdoc />
public Task ShutdownAsync(CancellationToken cancellationToken) => throw new NotSupportedException();
/// <summary>Disposes every session handed to the dashboard service.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
foreach (GatewaySession session in _sessions)
{
await session.DisposeAsync().ConfigureAwait(false);
}
}
}
// Answers Register / SubscribeBulk / UnsubscribeBulk / ReadBulk with successful
// replies and records what the dashboard advised and unadvised.
private sealed class RecordingWorkerClient : IWorkerClient
{
private const int RegisteredServerHandle = 77;
private readonly Dictionary<string, int> _itemHandles = new(StringComparer.OrdinalIgnoreCase);
private int _nextItemHandle = 1000;
/// <inheritdoc />
public string SessionId => "dashboard-session-1";
/// <inheritdoc />
public int? ProcessId => 4242;
/// <inheritdoc />
public WorkerClientState State => WorkerClientState.Ready;
/// <inheritdoc />
public DateTimeOffset LastHeartbeatAt => DateTimeOffset.UnixEpoch;
/// <summary>Gets the tag addresses subscribed, in the order the dashboard asked for them.</summary>
public List<string> SubscribedTags { get; } = [];
/// <summary>Gets the item handles the dashboard unsubscribed, in order.</summary>
public List<int> UnsubscribedHandles { get; } = [];
/// <summary>Gets the number of unsubscribe commands the dashboard sent.</summary>
public int UnsubscribeCommandCount { get; private set; }
/// <summary>Gets the tag addresses the worker refuses to advise.</summary>
public HashSet<string> FailSubscribeFor { get; } = new(StringComparer.OrdinalIgnoreCase);
/// <summary>Gets or sets a value indicating whether unsubscribe commands throw.</summary>
public bool FailUnsubscribe { get; set; }
/// <summary>Gets the item handle bound for a previously subscribed tag.</summary>
/// <param name="tagAddress">Tag address to look up.</param>
/// <returns>The bound item handle.</returns>
public int HandleFor(string tagAddress) => _itemHandles[tagAddress];
/// <inheritdoc />
public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public Task<WorkerCommandReply> InvokeAsync(
WorkerCommand command,
TimeSpan timeout,
CancellationToken cancellationToken)
{
MxCommand mxCommand = command.Command
?? throw new InvalidOperationException("The dashboard sent a command with no payload.");
MxCommandReply reply = new()
{
Kind = mxCommand.Kind,
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
};
switch (mxCommand.Kind)
{
case MxCommandKind.Register:
reply.Register = new RegisterReply { ServerHandle = RegisteredServerHandle };
break;
case MxCommandKind.SubscribeBulk:
reply.SubscribeBulk = Subscribe(mxCommand.SubscribeBulk.TagAddresses);
break;
case MxCommandKind.UnsubscribeBulk:
UnsubscribeCommandCount++;
if (FailUnsubscribe)
{
throw new InvalidOperationException("Simulated worker unsubscribe failure.");
}
UnsubscribedHandles.AddRange(mxCommand.UnsubscribeBulk.ItemHandles);
reply.UnsubscribeBulk = new BulkSubscribeReply();
break;
case MxCommandKind.ReadBulk:
reply.ReadBulk = Read(mxCommand.ReadBulk.TagAddresses);
break;
default:
throw new NotSupportedException($"Unexpected dashboard command {mxCommand.Kind}.");
}
return Task.FromResult(new WorkerCommandReply { Reply = reply });
}
/// <inheritdoc />
public async IAsyncEnumerable<WorkerEvent> ReadEventsAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
await Task.CompletedTask.ConfigureAwait(false);
yield break;
}
/// <inheritdoc />
public Task ShutdownAsync(TimeSpan timeout, CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public void Kill(string reason)
{
}
/// <inheritdoc />
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
private BulkSubscribeReply Subscribe(IEnumerable<string> tagAddresses)
{
BulkSubscribeReply subscribeReply = new();
foreach (string tagAddress in tagAddresses)
{
SubscribedTags.Add(tagAddress);
if (FailSubscribeFor.Contains(tagAddress))
{
subscribeReply.Results.Add(new SubscribeResult
{
ServerHandle = RegisteredServerHandle,
TagAddress = tagAddress,
ItemHandle = 0,
WasSuccessful = false,
ErrorMessage = "Simulated advise failure.",
});
continue;
}
if (!_itemHandles.TryGetValue(tagAddress, out int itemHandle))
{
itemHandle = _nextItemHandle++;
_itemHandles[tagAddress] = itemHandle;
}
subscribeReply.Results.Add(new SubscribeResult
{
ServerHandle = RegisteredServerHandle,
TagAddress = tagAddress,
ItemHandle = itemHandle,
WasSuccessful = true,
});
}
return subscribeReply;
}
private BulkReadReply Read(IEnumerable<string> tagAddresses)
{
BulkReadReply readReply = new();
foreach (string tagAddress in tagAddresses)
{
readReply.Results.Add(new BulkReadResult
{
ServerHandle = RegisteredServerHandle,
TagAddress = tagAddress,
ItemHandle = _itemHandles.TryGetValue(tagAddress, out int itemHandle) ? itemHandle : 0,
WasSuccessful = true,
Quality = 192,
});
}
return readReply;
}
}
}
@@ -0,0 +1,522 @@
using System.Runtime.CompilerServices;
using System.Threading.Channels;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
/// <summary>
/// Covers the in-process snapshot fan-out that replaced the dashboard pages'
/// loopback <c>/hubs/snapshot</c> connections. The invariants under test are the
/// ones that make the feed cheaper than the hub hop: exactly one underlying
/// <see cref="IDashboardSnapshotService.WatchSnapshotsAsync"/> enumeration for any
/// number of viewers, nothing at all while nobody is watching, and a slow viewer
/// that can neither buffer without bound nor stall the others.
/// </summary>
public sealed class DashboardSnapshotFeedTests
{
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
/// <summary>
/// With no page subscribed, the feed must not touch the snapshot service at
/// all — no timer, no snapshot build. This is the whole point of the idle
/// gate: an unattended gateway does no dashboard work.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WithNoSubscribers_NeverEnumeratesUnderlyingWatch()
{
FakeSnapshotService service = new();
DashboardSnapshotFeed feed = new(service);
// Obtaining the enumerable without enumerating it must not subscribe
// either: the pump starts on the first MoveNextAsync, not before.
_ = feed.WatchAsync(CancellationToken.None);
await Task.Delay(TimeSpan.FromMilliseconds(100));
Assert.Equal(0, service.EnumerationCount);
}
/// <summary>
/// Two viewers share one underlying enumeration and both see the same
/// pushed snapshot. Before the feed, each page opened its own SignalR
/// connection and the publisher pulled its own snapshot stream.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WithTwoSubscribers_SharesASingleUnderlyingEnumeration()
{
FakeSnapshotService service = new();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource firstCancellation = new();
using CancellationTokenSource secondCancellation = new();
IAsyncEnumerator<DashboardSnapshot> first =
feed.WatchAsync(firstCancellation.Token).GetAsyncEnumerator(firstCancellation.Token);
Task<bool> firstMove = first.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
IAsyncEnumerator<DashboardSnapshot> second =
feed.WatchAsync(secondCancellation.Token).GetAsyncEnumerator(secondCancellation.Token);
Task<bool> secondMove = second.MoveNextAsync().AsTask();
// Push until both have observed a snapshot: a subscriber only becomes
// visible to the pump once its MoveNextAsync has registered the channel,
// so a single push could race the second registration.
await PushUntilAsync(service, Task.WhenAll(firstMove, secondMove));
Assert.True(await firstMove.WaitAsync(TestTimeout));
Assert.True(await secondMove.WaitAsync(TestTimeout));
Assert.StartsWith("push-", first.Current.GatewayVersion, StringComparison.Ordinal);
Assert.StartsWith("push-", second.Current.GatewayVersion, StringComparison.Ordinal);
Assert.Equal(1, service.EnumerationCount);
await firstCancellation.CancelAsync();
await secondCancellation.CancelAsync();
await DrainAsync(first, firstMove);
await DrainAsync(second, secondMove);
}
/// <summary>
/// The last viewer leaving must cancel the underlying enumeration (idle
/// gate re-armed), and the next viewer must restart it — the rapid
/// unsubscribe/resubscribe path a page navigation exercises.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WhenLastSubscriberLeaves_CancelsPumpAndRestartsForTheNext()
{
FakeSnapshotService service = new();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource firstCancellation = new();
IAsyncEnumerator<DashboardSnapshot> first =
feed.WatchAsync(firstCancellation.Token).GetAsyncEnumerator(firstCancellation.Token);
Task<bool> firstMove = first.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
await firstCancellation.CancelAsync();
await DrainAsync(first, firstMove);
await WaitUntilAsync(() => service.CompletedEnumerationCount >= 1);
Assert.True(service.LastEnumerationWasCancelled);
using CancellationTokenSource secondCancellation = new();
IAsyncEnumerator<DashboardSnapshot> second =
feed.WatchAsync(secondCancellation.Token).GetAsyncEnumerator(secondCancellation.Token);
Task<bool> secondMove = second.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 2);
Assert.Equal(2, service.EnumerationCount);
await secondCancellation.CancelAsync();
await DrainAsync(second, secondMove);
}
/// <summary>
/// A viewer that is not reading must not stall the pump or accumulate
/// snapshots: its bounded channel drops the oldest, so its next read is the
/// newest snapshot the pump has broadcast, not a backlog head. The fast
/// reader's progress is what makes the assertion deterministic — once it has
/// seen the third snapshot the pump has provably broadcast all three.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WithASlowSubscriber_KeepsOnlyTheNewestSnapshot()
{
FakeSnapshotService service = new();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource fastCancellation = new();
using CancellationTokenSource slowCancellation = new();
IAsyncEnumerator<DashboardSnapshot> fast =
feed.WatchAsync(fastCancellation.Token).GetAsyncEnumerator(fastCancellation.Token);
Task<bool> fastMove = fast.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
// The slow subscriber registers but never advances until the very end.
IAsyncEnumerator<DashboardSnapshot> slow =
feed.WatchAsync(slowCancellation.Token).GetAsyncEnumerator(slowCancellation.Token);
Task<bool> slowMove = slow.MoveNextAsync().AsTask();
await PushUntilAsync(service, Task.WhenAll(fastMove, slowMove));
Assert.True(await fastMove.WaitAsync(TestTimeout));
Assert.True(await slowMove.WaitAsync(TestTimeout));
service.Push(CreateSnapshot("s1"));
service.Push(CreateSnapshot("s2"));
service.Push(CreateSnapshot("s3"));
// Drain the fast reader until it sees s3; that proves the pump broadcast
// all three to every subscriber, so the slow channel now holds exactly s3.
string fastLatest = fast.Current.GatewayVersion;
while (fastLatest != "s3")
{
Assert.True(await fast.MoveNextAsync().AsTask().WaitAsync(TestTimeout));
fastLatest = fast.Current.GatewayVersion;
}
Assert.True(await slow.MoveNextAsync().AsTask().WaitAsync(TestTimeout));
Assert.Equal("s3", slow.Current.GatewayVersion);
await fastCancellation.CancelAsync();
await slowCancellation.CancelAsync();
await DrainAsync(fast, Task.FromResult(true));
await DrainAsync(slow, Task.FromResult(true));
}
/// <summary>
/// A fault in the underlying watch is surfaced to the current viewers rather
/// than silently hanging them, and it resets the feed so the next viewer
/// starts a fresh pump instead of attaching to a dead one.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WhenUnderlyingWatchFaults_PropagatesAndRestartsForTheNextSubscriber()
{
FakeSnapshotService service = new();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource firstCancellation = new();
IAsyncEnumerator<DashboardSnapshot> first =
feed.WatchAsync(firstCancellation.Token).GetAsyncEnumerator(firstCancellation.Token);
Task<bool> firstMove = first.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
service.Fault(new InvalidOperationException("simulated snapshot source failure"));
InvalidOperationException failure =
await Assert.ThrowsAsync<InvalidOperationException>(() => firstMove.WaitAsync(TestTimeout));
Assert.Equal("simulated snapshot source failure", failure.Message);
await first.DisposeAsync();
using CancellationTokenSource secondCancellation = new();
IAsyncEnumerator<DashboardSnapshot> second =
feed.WatchAsync(secondCancellation.Token).GetAsyncEnumerator(secondCancellation.Token);
Task<bool> secondMove = second.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 2);
await PushUntilAsync(service, secondMove);
Assert.True(await secondMove.WaitAsync(TestTimeout));
await secondCancellation.CancelAsync();
await DrainAsync(second, secondMove);
}
/// <summary>
/// The race the generation tagging exists for: a page subscribes in the window between
/// the source failing and the dying pump detaching its subscribers. Without generations
/// the newcomer joined the doomed pump, was detached with its error, and no pump ever
/// restarted (only a first subscriber started one) — that page froze for good.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WhenASubscriberJoinsWhileAFaultedPumpUnwinds_IsServedByAFreshPump()
{
FakeSnapshotService service = new();
service.HoldDisposal();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource firstCancellation = new();
IAsyncEnumerator<DashboardSnapshot> first =
feed.WatchAsync(firstCancellation.Token).GetAsyncEnumerator(firstCancellation.Token);
Task<bool> firstMove = first.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
service.Fault(new InvalidOperationException("simulated snapshot source failure"));
// The pump has observed the failure and is parked disposing the enumerator — the
// exact window in which a page used to attach itself to a doomed pump.
await service.DisposalReached.WaitAsync(TestTimeout);
using CancellationTokenSource secondCancellation = new();
IAsyncEnumerator<DashboardSnapshot> second =
feed.WatchAsync(secondCancellation.Token).GetAsyncEnumerator(secondCancellation.Token);
// An async iterator body runs synchronously up to its first await, so the
// subscription is registered by the time MoveNextAsync hands back its task.
Task<bool> secondMove = second.MoveNextAsync().AsTask();
service.ReleaseDisposal();
// The subscriber that was there when the source broke still learns about it...
InvalidOperationException failure =
await Assert.ThrowsAsync<InvalidOperationException>(() => firstMove.WaitAsync(TestTimeout));
Assert.Equal("simulated snapshot source failure", failure.Message);
await first.DisposeAsync();
// ...and the one that joined mid-unwind is served by a restarted enumeration
// instead of inheriting the failure.
await WaitUntilAsync(() => service.EnumerationCount >= 2);
await PushUntilAsync(service, secondMove);
Assert.True(await secondMove.WaitAsync(TestTimeout));
Assert.StartsWith("push-", second.Current.GatewayVersion, StringComparison.Ordinal);
await secondCancellation.CancelAsync();
await DrainAsync(second, secondMove);
}
/// <summary>
/// The idle gate is per generation, not per subscriber count. A viewer that joins while a
/// pump unwinds starts a new generation, and the old generation's viewers linger in the
/// list until that pump's reset runs — so a global "is the list empty" check let the new
/// viewer leave without cancelling the generation it had just started, leaving a pump
/// enumerating the snapshot source with nobody watching it.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchAsync_WhenAJoinerLeavesWhileAnOldGenerationLingers_LeavesNoPumpRunning()
{
FakeSnapshotService service = new();
service.HoldDisposal();
DashboardSnapshotFeed feed = new(service);
using CancellationTokenSource oldCancellation = new();
IAsyncEnumerator<DashboardSnapshot> oldSubscriber =
feed.WatchAsync(oldCancellation.Token).GetAsyncEnumerator(oldCancellation.Token);
Task<bool> oldMove = oldSubscriber.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount >= 1);
service.Fault(new InvalidOperationException("simulated snapshot source failure"));
await service.DisposalReached.WaitAsync(TestTimeout);
// Joins mid-unwind (starting a fresh generation) and leaves again before the dying
// pump has detached the subscriber that is still lingering in the list.
using CancellationTokenSource joinerCancellation = new();
IAsyncEnumerator<DashboardSnapshot> joiner =
feed.WatchAsync(joinerCancellation.Token).GetAsyncEnumerator(joinerCancellation.Token);
Task<bool> joinerMove = joiner.MoveNextAsync().AsTask();
await joinerCancellation.CancelAsync();
// The joiner's unwind is a continuation of its cancelled channel read; give it time to
// run its unsubscribe before the dying pump is released. Only the interleaving depends
// on this delay — the assertions below hold either way.
await Task.Delay(TimeSpan.FromMilliseconds(150));
service.ReleaseDisposal();
await Assert.ThrowsAsync<InvalidOperationException>(() => oldMove.WaitAsync(TestTimeout));
await oldSubscriber.DisposeAsync();
// Completes only once the joiner's unsubscribe has awaited its generation's pump.
await DrainAsync(joiner, joinerMove);
// Nobody is watching, so nothing may consume a snapshot: a surviving pump would drain
// this push within its first read.
service.Push(CreateSnapshot("orphan-check"));
await Task.Delay(TimeSpan.FromMilliseconds(150));
Assert.Equal(1, service.PendingPushCount);
// ...and the next viewer still starts cleanly, picking up the queued snapshot.
int enumerationsBefore = service.EnumerationCount;
using CancellationTokenSource nextCancellation = new();
IAsyncEnumerator<DashboardSnapshot> next =
feed.WatchAsync(nextCancellation.Token).GetAsyncEnumerator(nextCancellation.Token);
Task<bool> nextMove = next.MoveNextAsync().AsTask();
await WaitUntilAsync(() => service.EnumerationCount > enumerationsBefore);
Assert.True(await nextMove.WaitAsync(TestTimeout));
Assert.Equal("orphan-check", next.Current.GatewayVersion);
await nextCancellation.CancelAsync();
await DrainAsync(next, nextMove);
}
/// <summary>Builds a snapshot whose version string identifies it in assertions.</summary>
/// <param name="version">Identity marker carried in <c>GatewayVersion</c>.</param>
/// <returns>A snapshot carrying the supplied identity marker.</returns>
private static DashboardSnapshot CreateSnapshot(string version)
{
return new DashboardSnapshot(
GeneratedAt: DateTimeOffset.UnixEpoch,
GatewayStartedAt: DateTimeOffset.UnixEpoch,
GatewayUptime: TimeSpan.Zero,
GatewayStatus: "Healthy",
GatewayVersion: version,
Sessions: Array.Empty<DashboardSessionSummary>(),
Workers: Array.Empty<DashboardWorkerSummary>(),
Metrics: Array.Empty<DashboardMetricSummary>(),
Faults: Array.Empty<DashboardFaultSummary>(),
ApiKeys: Array.Empty<DashboardApiKeySummary>(),
Configuration: null!,
Galaxy: null!);
}
/// <summary>
/// Pushes snapshots until the supplied task completes, so a test never
/// depends on a single push landing after a subscriber has registered.
/// </summary>
/// <param name="service">Fake snapshot source to push through.</param>
/// <param name="until">Task whose completion stops the pushes.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
private static async Task PushUntilAsync(FakeSnapshotService service, Task until)
{
using CancellationTokenSource cancellation = new(TestTimeout);
int sequence = 0;
while (!until.IsCompleted)
{
service.Push(CreateSnapshot($"push-{sequence++}"));
await Task.Delay(TimeSpan.FromMilliseconds(5), cancellation.Token);
}
}
/// <summary>
/// Observes the cancellation of a pending enumeration and disposes the
/// enumerator, mirroring how <c>await foreach</c> unwinds a cancelled watch.
/// </summary>
/// <param name="enumerator">Enumerator to unwind.</param>
/// <param name="pending">The in-flight move, if any.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
private static async Task DrainAsync(IAsyncEnumerator<DashboardSnapshot> enumerator, Task pending)
{
try
{
await pending.WaitAsync(TestTimeout);
}
catch (OperationCanceledException)
{
}
try
{
await enumerator.DisposeAsync();
}
catch (OperationCanceledException)
{
}
}
private static async Task WaitUntilAsync(Func<bool> predicate)
{
using CancellationTokenSource cancellation = new(TestTimeout);
while (!predicate())
{
await Task.Delay(TimeSpan.FromMilliseconds(5), cancellation.Token);
}
}
/// <summary>
/// Snapshot source under the feed's control: counts enumerations, records how
/// each one ended, and lets the test drive snapshots (or a fault) into the
/// live enumeration.
/// </summary>
private sealed class FakeSnapshotService : IDashboardSnapshotService
{
private readonly Channel<object> _pushes = Channel.CreateUnbounded<object>();
private readonly TaskCompletionSource _disposalReached = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _disposalRelease = new(TaskCreationOptions.RunContinuationsAsynchronously);
private int _enumerationCount;
private int _completedEnumerationCount;
private volatile bool _lastEnumerationWasCancelled;
private volatile bool _holdDisposal;
/// <summary>Gets the number of times the feed started enumerating this source.</summary>
public int EnumerationCount => Volatile.Read(ref _enumerationCount);
/// <summary>Gets a task that completes when a held enumerator disposal is reached.</summary>
public Task DisposalReached => _disposalReached.Task;
/// <summary>
/// Gets the number of queued snapshots no enumeration has taken yet. A live pump
/// drains this even with nobody watching, so a stable count proves the feed is idle.
/// </summary>
public int PendingPushCount => _pushes.Reader.Count;
/// <summary>Gets the number of enumerations that have finished (cancelled, faulted, or completed).</summary>
public int CompletedEnumerationCount => Volatile.Read(ref _completedEnumerationCount);
/// <summary>Gets a value indicating whether the most recently finished enumeration ended cancelled.</summary>
public bool LastEnumerationWasCancelled => _lastEnumerationWasCancelled;
/// <summary>Queues a snapshot for the live enumeration to yield.</summary>
/// <param name="snapshot">Snapshot to yield.</param>
public void Push(DashboardSnapshot snapshot) => _pushes.Writer.TryWrite(snapshot);
/// <summary>Queues a failure for the live enumeration to throw.</summary>
/// <param name="error">Exception to throw from the enumeration.</param>
public void Fault(Exception error) => _pushes.Writer.TryWrite(error);
/// <summary>
/// Parks enumerator disposal until <see cref="ReleaseDisposal"/>, which holds the pump
/// in the window between observing the source's failure and detaching its subscribers.
/// </summary>
public void HoldDisposal() => _holdDisposal = true;
/// <summary>Releases a held enumerator disposal.</summary>
public void ReleaseDisposal() => _disposalRelease.TrySetResult();
/// <inheritdoc />
public DashboardSnapshot GetSnapshot() => CreateSnapshot("current");
/// <inheritdoc />
public IAsyncEnumerable<DashboardSnapshot> WatchSnapshotsAsync(CancellationToken cancellationToken)
=> new GatedEnumerable(this);
private async Task OnDisposingAsync()
{
if (!_holdDisposal)
{
return;
}
_disposalReached.TrySetResult();
await _disposalRelease.Task.ConfigureAwait(false);
}
private async IAsyncEnumerable<DashboardSnapshot> EnumerateAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
Interlocked.Increment(ref _enumerationCount);
try
{
await foreach (object item in _pushes.Reader.ReadAllAsync(cancellationToken).ConfigureAwait(false))
{
if (item is Exception error)
{
throw error;
}
yield return (DashboardSnapshot)item;
}
}
finally
{
_lastEnumerationWasCancelled = cancellationToken.IsCancellationRequested;
Interlocked.Increment(ref _completedEnumerationCount);
}
}
/// <summary>
/// Wraps the iterator so disposal is a control point of its own: the feed ends a pump
/// generation when MoveNextAsync fails, which is strictly before this disposal runs.
/// </summary>
/// <param name="owner">The fake whose enumeration is being wrapped.</param>
private sealed class GatedEnumerable(FakeSnapshotService owner) : IAsyncEnumerable<DashboardSnapshot>
{
/// <summary>Creates a gated enumerator over the fake's enumeration.</summary>
/// <param name="cancellationToken">Token to observe for cancellation.</param>
/// <returns>The gated enumerator.</returns>
public IAsyncEnumerator<DashboardSnapshot> GetAsyncEnumerator(
CancellationToken cancellationToken = default)
=> new GatedEnumerator(owner, owner.EnumerateAsync(cancellationToken).GetAsyncEnumerator(cancellationToken));
}
private sealed class GatedEnumerator(
FakeSnapshotService owner,
IAsyncEnumerator<DashboardSnapshot> inner) : IAsyncEnumerator<DashboardSnapshot>
{
/// <summary>Gets the current snapshot.</summary>
public DashboardSnapshot Current => inner.Current;
/// <summary>Advances the wrapped enumeration.</summary>
/// <returns>A task that yields whether another snapshot is available.</returns>
public ValueTask<bool> MoveNextAsync() => inner.MoveNextAsync();
/// <summary>Parks while the fake holds disposal, then disposes the wrapped enumeration.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
await owner.OnDisposingAsync().ConfigureAwait(false);
await inner.DisposeAsync().ConfigureAwait(false);
}
}
}
}
@@ -0,0 +1,154 @@
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
/// <summary>
/// Verifies <see cref="DashboardSnapshotHubConnectionCounter"/>, the seam
/// <see cref="DashboardSnapshotPublisher"/> reads before building a snapshot.
/// An over-count leaves the publisher ticking for nobody; an under-count is
/// worse — it idle-gates a dashboard that is actually open, so the page silently
/// stops updating. The counter is exercised directly rather than through the hub,
/// mirroring the <see cref="EventsHubViewerRegistry"/> precedent: a SignalR
/// <c>Hub</c> instance needs a caller-clients and connection context fake to
/// invoke <c>OnConnectedAsync</c>, and the hub methods themselves are two lines
/// of delegation to this type.
/// </summary>
public sealed class DashboardSnapshotHubConnectionCounterTests
{
/// <summary>A fresh counter reports no viewers, so the publisher starts idle.</summary>
[Fact]
public void Count_WhenNothingConnected_IsZero()
{
DashboardSnapshotHubConnectionCounter counter = new();
Assert.Equal(0, counter.Count);
}
/// <summary>Connect/disconnect pairs move the count and return the post-operation value.</summary>
[Fact]
public void IncrementThenDecrement_TracksLiveConnections()
{
DashboardSnapshotHubConnectionCounter counter = new();
Assert.Equal(1, counter.Increment());
Assert.Equal(2, counter.Increment());
Assert.Equal(2, counter.Count);
Assert.Equal(1, counter.Decrement());
Assert.Equal(0, counter.Decrement());
Assert.Equal(0, counter.Count);
}
/// <summary>
/// SignalR calls <c>OnDisconnectedAsync</c> for a connection whose
/// <c>OnConnectedAsync</c> faulted, so unmatched decrements happen. They must
/// hold the floor at zero rather than driving the count negative.
/// </summary>
[Fact]
public void Decrement_WithoutMatchingIncrement_HoldsAtZero()
{
DashboardSnapshotHubConnectionCounter counter = new();
Assert.Equal(0, counter.Decrement());
Assert.Equal(0, counter.Decrement());
Assert.Equal(0, counter.Count);
// A genuine connection after unmatched disconnects still registers as one.
Assert.Equal(1, counter.Increment());
}
/// <summary>
/// Many concurrent unmatched decrements must not leave the count below zero:
/// a negative floor would swallow the next real connection's increment and
/// keep the publisher idle-gated while a viewer waits. The clamp lives inside
/// the compare-and-swap, so the floor holds however the calls interleave.
/// </summary>
[Fact]
public void Decrement_UnderConcurrencyFromZero_NeverGoesNegative()
{
DashboardSnapshotHubConnectionCounter counter = new();
Parallel.For(0, 256, _ => counter.Decrement());
Assert.Equal(0, counter.Count);
Assert.Equal(1, counter.Increment());
Assert.Equal(1, counter.Count);
}
/// <summary>
/// Stress check on the invariant the idle gate depends on: real connects
/// interleaved with unmatched disconnects leave the count in [0, connects], and a
/// connect after the storm is always visible to the publisher. The failure this
/// guards is the decrement-then-repair race the CAS retry loop replaced — an early
/// decrementer's stale repair either erases a live connection's increment or leaves
/// a negative value behind, and either way an open dashboard freezes behind the
/// gate.
/// </summary>
/// <remarks>
/// This does not deterministically reproduce that race, and it is not claimed to:
/// the bad interleaving needs a specific few-instruction overlap that cannot be
/// forced through the public API, and a merely low count is a legitimate outcome
/// here (a decrement that runs while the count is positive consumes a real
/// connection). Verified by experiment: the previous implementation passes this
/// test. What is asserted are the observable consequences — never negative, the
/// floor holds, a later connect still registers — with the correctness argument
/// resting on the clamped CAS retry loop itself.
/// </remarks>
[Fact]
public void IncrementAndDecrement_InterleavedUnderConcurrency_StayWithinTheRealConnectionCount()
{
const int LiveConnections = 8;
const int UnmatchedDisconnects = 128;
for (int round = 0; round < 50; round++)
{
DashboardSnapshotHubConnectionCounter counter = new();
// A few workers are real connects that must survive; the rest are
// unmatched disconnects hammering the zero floor around them.
Parallel.For(0, LiveConnections + UnmatchedDisconnects, index =>
{
if (index % 16 == 0 && index / 16 < LiveConnections)
{
counter.Increment();
return;
}
counter.Decrement();
});
Assert.InRange(counter.Count, 0, LiveConnections);
// Every unmatched decrement has completed, so the surviving connections
// disconnect cleanly and the counter must land exactly on zero — never
// below it, and a subsequent connect must be visible to the publisher.
int remaining = counter.Count;
for (int i = 0; i < remaining; i++)
{
counter.Decrement();
}
Assert.Equal(0, counter.Count);
Assert.Equal(1, counter.Increment());
}
}
/// <summary>Matched connect/disconnect pairs under concurrency settle back at zero.</summary>
[Fact]
public void IncrementAndDecrement_MatchedPairsUnderConcurrency_SettleAtZero()
{
DashboardSnapshotHubConnectionCounter counter = new();
Parallel.For(0, 64, _ =>
{
for (int pass = 0; pass < 50; pass++)
{
counter.Increment();
counter.Decrement();
}
});
Assert.Equal(0, counter.Count);
}
}
@@ -9,6 +9,7 @@ namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
public sealed class DashboardSnapshotPublisherTests
{
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
private static readonly TimeSpan IdlePollInterval = TimeSpan.FromMilliseconds(10);
/// <summary>
/// A transient failure inside
@@ -28,8 +29,10 @@ public sealed class DashboardSnapshotPublisherTests
DashboardSnapshotPublisher publisher = new(
snapshotService,
hubContext,
ConnectedCounter(),
NullLogger<DashboardSnapshotPublisher>.Instance,
reconnectDelay);
reconnectDelay,
IdlePollInterval);
using CancellationTokenSource cts = new();
Task execute = publisher.StartAsync(cts.Token);
@@ -73,8 +76,10 @@ public sealed class DashboardSnapshotPublisherTests
DashboardSnapshotPublisher publisher = new(
snapshotService,
hubContext,
ConnectedCounter(),
NullLogger<DashboardSnapshotPublisher>.Instance,
reconnectDelay);
reconnectDelay,
IdlePollInterval);
using CancellationTokenSource cts = new();
Task execute = publisher.StartAsync(cts.Token);
@@ -88,6 +93,54 @@ public sealed class DashboardSnapshotPublisherTests
Assert.True(snapshotService.SubscribeCount >= 2);
}
/// <summary>
/// With no dashboard connected there is nobody to broadcast to, so the publisher must
/// not advance the snapshot enumerator at all — every pull costs a registry snapshot and
/// sort, a locked metrics dictionary copy, and periodically a SQLite key-table read.
/// The first viewer to connect resumes the tick.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ExecuteAsync_WhenNoHubConnections_DoesNotPullSnapshots()
{
CountingSnapshotService snapshotService = new();
RecordingHubContext hubContext = new();
DashboardSnapshotHubConnectionCounter connectionCounter = new();
DashboardSnapshotPublisher publisher = new(
snapshotService,
hubContext,
connectionCounter,
NullLogger<DashboardSnapshotPublisher>.Instance,
TimeSpan.FromMilliseconds(50),
IdlePollInterval);
using CancellationTokenSource cts = new();
await publisher.StartAsync(cts.Token).WaitAsync(TestTimeout);
// Long enough for many idle polls at IdlePollInterval.
await Task.Delay(TimeSpan.FromMilliseconds(250));
Assert.Equal(0, snapshotService.PullCount);
Assert.Equal(0, hubContext.SendCount);
connectionCounter.Increment();
await WaitUntilAsync(() => hubContext.SendCount >= 1);
await cts.CancelAsync();
await publisher.StopAsync(CancellationToken.None);
Assert.True(snapshotService.PullCount >= 1);
}
/// <summary>Creates a connection counter that already has one live viewer.</summary>
/// <returns>A counter reporting a single connection.</returns>
private static DashboardSnapshotHubConnectionCounter ConnectedCounter()
{
DashboardSnapshotHubConnectionCounter counter = new();
counter.Increment();
return counter;
}
private static async Task WaitUntilAsync(Func<bool> predicate)
{
using CancellationTokenSource cancellationTokenSource = new(TestTimeout);
@@ -174,6 +227,34 @@ public sealed class DashboardSnapshotPublisherTests
}
}
private sealed class CountingSnapshotService : IDashboardSnapshotService
{
private int _pullCount;
/// <summary>Gets the number of snapshots the publisher pulled from this source.</summary>
public int PullCount => Volatile.Read(ref _pullCount);
/// <inheritdoc />
public DashboardSnapshot GetSnapshot()
{
return null!;
}
/// <inheritdoc />
public async IAsyncEnumerable<DashboardSnapshot> WatchSnapshotsAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
// Short cadence so a listening publisher pulls quickly; the counter only
// moves when the publisher actually advances the enumerator.
await Task.Delay(TimeSpan.FromMilliseconds(5), cancellationToken).ConfigureAwait(false);
Interlocked.Increment(ref _pullCount);
yield return GetSnapshot();
}
}
}
private sealed class RecordingHubContext : IHubContext<DashboardSnapshotHub>
{
private readonly RecordingHubClients _clients = new();
@@ -1,5 +1,6 @@
using System.Globalization;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Time.Testing;
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
using ZB.MOM.WW.GalaxyRepository;
using ZB.MOM.WW.GalaxyRepository.Grpc;
@@ -457,6 +458,7 @@ public sealed class DashboardSnapshotServiceTests
CreatedUtc: DateTimeOffset.Parse("2026-04-28T12:00:00Z", CultureInfo.InvariantCulture),
LastUsedUtc: null,
RevokedUtc: null));
FakeTimeProvider timeProvider = new(DateTimeOffset.Parse("2026-08-15T00:00:00Z", CultureInfo.InvariantCulture));
DashboardSnapshotService service = CreateService(
new SessionRegistry(),
metrics,
@@ -464,11 +466,12 @@ public sealed class DashboardSnapshotServiceTests
{
Dashboard = new DashboardOptions
{
SnapshotIntervalMilliseconds = 1,
SnapshotIntervalMilliseconds = 1000,
},
},
apiKeyAdminStore: apiKeyAdminStore);
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(2));
apiKeyAdminStore: apiKeyAdminStore,
timeProvider: timeProvider);
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(30));
await using IAsyncEnumerator<DashboardSnapshot> enumerator = service
.WatchSnapshotsAsync(cancellation.Token)
.GetAsyncEnumerator(cancellation.Token);
@@ -477,14 +480,86 @@ public sealed class DashboardSnapshotServiceTests
DashboardSnapshot first = enumerator.Current;
apiKeyAdminStore.FailNext = true;
Assert.True(await enumerator.MoveNextAsync());
DashboardSnapshot second = enumerator.Current;
// Advance past the key-summary refresh interval so the second tick really
// does attempt a refresh — that attempt is the one that fails.
DashboardSnapshot second = await NextSnapshotAsync(enumerator, timeProvider, TimeSpan.FromSeconds(20));
Assert.Equal("operator01", Assert.Single(first.ApiKeys).KeyId);
Assert.Equal("operator01", Assert.Single(second.ApiKeys).KeyId);
Assert.Equal(2, apiKeyAdminStore.ListCount);
}
/// <summary>
/// The API key list is a SQLite read; at the default 1s snapshot cadence it would run
/// ~86k times a day against a table that changes by hand. Ticks inside the refresh
/// interval must reuse the cached summaries and not touch the store.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WatchSnapshotsAsync_WhenTicksFallInsideRefreshInterval_ListsApiKeysOnce()
{
using GatewayMetrics metrics = new();
CountingApiKeyAdminStore apiKeyAdminStore = new(
new ApiKeyListItem(
KeyId: "operator01",
KeyPrefix: "mxgw",
DisplayName: "Operator",
Scopes: new HashSet<string>([GatewayScopes.MetadataRead], StringComparer.Ordinal),
ConstraintsJson: null,
CreatedUtc: DateTimeOffset.Parse("2026-04-28T12:00:00Z", CultureInfo.InvariantCulture),
LastUsedUtc: null,
RevokedUtc: null));
FakeTimeProvider timeProvider = new(DateTimeOffset.Parse("2026-08-15T00:00:00Z", CultureInfo.InvariantCulture));
DashboardSnapshotService service = CreateService(
new SessionRegistry(),
metrics,
new GatewayOptions
{
Dashboard = new DashboardOptions
{
SnapshotIntervalMilliseconds = 1000,
},
},
apiKeyAdminStore: apiKeyAdminStore,
timeProvider: timeProvider);
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(30));
await using IAsyncEnumerator<DashboardSnapshot> enumerator = service
.WatchSnapshotsAsync(cancellation.Token)
.GetAsyncEnumerator(cancellation.Token);
Assert.True(await enumerator.MoveNextAsync());
Assert.Equal(1, apiKeyAdminStore.ListCount);
// Two more 1s ticks, both well inside the 15s refresh interval.
DashboardSnapshot second = await NextSnapshotAsync(enumerator, timeProvider, TimeSpan.FromSeconds(1));
await NextSnapshotAsync(enumerator, timeProvider, TimeSpan.FromSeconds(1));
Assert.Equal(1, apiKeyAdminStore.ListCount);
Assert.Equal("operator01", Assert.Single(second.ApiKeys).KeyId);
// A tick past the interval refreshes again, so an added or revoked key still
// reaches the dashboard within the interval.
await NextSnapshotAsync(enumerator, timeProvider, TimeSpan.FromSeconds(20));
Assert.Equal(2, apiKeyAdminStore.ListCount);
}
/// <summary>
/// The effective configuration is startup-static, so the snapshot must hand out the
/// same instance instead of rebuilding the whole option tree on every tick.
/// </summary>
[Fact]
public void GetSnapshot_ReusesTheSameEffectiveConfigurationInstance()
{
using GatewayMetrics metrics = new();
DashboardSnapshotService service = CreateService(new SessionRegistry(), metrics);
DashboardSnapshot first = service.GetSnapshot();
DashboardSnapshot second = service.GetSnapshot();
Assert.Same(first.Configuration, second.Configuration);
}
/// <summary>Verifies that snapshot service disposes cleanly when subscriber cancels.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -513,12 +588,34 @@ public sealed class DashboardSnapshotServiceTests
Assert.False(hasNext);
}
/// <summary>
/// Advances the fake clock past the next snapshot tick and returns the snapshot it
/// produces. <c>MoveNextAsync</c> is started before the advance because the iterator
/// creates its <see cref="PeriodicTimer"/> synchronously on that call — the timer must
/// exist before the clock moves or the tick is missed.
/// </summary>
/// <param name="enumerator">The snapshot enumerator being driven.</param>
/// <param name="timeProvider">The fake clock backing the snapshot timer.</param>
/// <param name="advance">How far to advance the clock.</param>
/// <returns>The snapshot produced by the tick.</returns>
private static async Task<DashboardSnapshot> NextSnapshotAsync(
IAsyncEnumerator<DashboardSnapshot> enumerator,
FakeTimeProvider timeProvider,
TimeSpan advance)
{
ValueTask<bool> pending = enumerator.MoveNextAsync();
timeProvider.Advance(advance);
Assert.True(await pending.AsTask().WaitAsync(TimeSpan.FromSeconds(10)));
return enumerator.Current;
}
private static DashboardSnapshotService CreateService(
SessionRegistry registry,
GatewayMetrics metrics,
GatewayOptions? options = null,
IGalaxyHierarchyCache? galaxyHierarchyCache = null,
IApiKeyAdminStore? apiKeyAdminStore = null)
IApiKeyAdminStore? apiKeyAdminStore = null,
TimeProvider? timeProvider = null)
{
GatewayOptions resolvedOptions = options ?? new GatewayOptions
{
@@ -535,7 +632,8 @@ public sealed class DashboardSnapshotServiceTests
configurationProvider,
galaxyHierarchyCache ?? new StubGalaxyHierarchyCache(GalaxyHierarchyCacheEntry.Empty),
apiKeyAdminStore ?? new FakeApiKeyAdminStore(),
Options.Create(resolvedOptions));
Options.Create(resolvedOptions),
timeProvider);
}
private sealed class StubGalaxyHierarchyCache(GalaxyHierarchyCacheEntry current) : IGalaxyHierarchyCache
@@ -0,0 +1,165 @@
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
/// <summary>
/// Verifies <see cref="EventsHubViewerRegistry"/>, the seam
/// <see cref="DashboardEventBroadcaster"/> consults before cloning and sending
/// an event: a session is "watched" only while at least one hub connection
/// holds a subscription to it, and a dropped connection releases every
/// subscription it held.
/// </summary>
public sealed class EventsHubViewerRegistryTests
{
/// <summary>A session with no subscriber is not watched.</summary>
[Fact]
public void HasViewers_WithNoSubscribers_IsFalse()
{
EventsHubViewerRegistry registry = new();
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>Adding then removing the only viewer flips the session back to unwatched.</summary>
[Fact]
public void AddViewer_ThenRemoveViewer_TogglesWatchedState()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
Assert.True(registry.HasViewers("session-1"));
registry.RemoveViewer("conn-1", "session-1");
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>Each connection counts once: the session stays watched until the last one leaves.</summary>
[Fact]
public void RemoveViewer_WithOtherConnectionsStillSubscribed_KeepsSessionWatched()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
registry.AddViewer("conn-2", "session-1");
registry.RemoveViewer("conn-1", "session-1");
Assert.True(registry.HasViewers("session-1"));
registry.RemoveViewer("conn-2", "session-1");
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>A repeated subscribe from the same connection is idempotent, so one unsubscribe clears it.</summary>
[Fact]
public void AddViewer_CalledTwiceForSameConnection_CountsOnce()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
registry.AddViewer("conn-1", "session-1");
registry.RemoveViewer("conn-1", "session-1");
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>Viewer counts are tracked per session; unrelated sessions stay unwatched.</summary>
[Fact]
public void AddViewer_TracksSessionsIndependently()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
Assert.True(registry.HasViewers("session-1"));
Assert.False(registry.HasViewers("session-2"));
}
/// <summary>A dropped connection releases every session it held.</summary>
[Fact]
public void ReleaseConnection_ReleasesEverySessionTheConnectionHeld()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
registry.AddViewer("conn-1", "session-2");
registry.AddViewer("conn-2", "session-2");
registry.ReleaseConnection("conn-1");
Assert.False(registry.HasViewers("session-1"));
// conn-2 still watches session-2.
Assert.True(registry.HasViewers("session-2"));
}
/// <summary>Releasing a connection twice does not double-decrement another connection's subscription.</summary>
[Fact]
public void ReleaseConnection_CalledTwice_DoesNotDropOtherViewers()
{
EventsHubViewerRegistry registry = new();
registry.AddViewer("conn-1", "session-1");
registry.AddViewer("conn-2", "session-1");
registry.ReleaseConnection("conn-1");
registry.ReleaseConnection("conn-1");
Assert.True(registry.HasViewers("session-1"));
}
/// <summary>Unmatched removals cannot drive the count negative and strand a session as unwatched.</summary>
[Fact]
public void RemoveViewer_WithoutMatchingAdd_LeavesCountAtZero()
{
EventsHubViewerRegistry registry = new();
registry.RemoveViewer("conn-1", "session-1");
registry.RemoveViewer("conn-1", "session-1");
registry.ReleaseConnection("conn-1");
Assert.False(registry.HasViewers("session-1"));
// A subsequent genuine subscribe must still register as exactly one viewer.
registry.AddViewer("conn-1", "session-1");
Assert.True(registry.HasViewers("session-1"));
registry.RemoveViewer("conn-1", "session-1");
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>Blank connection or session ids are ignored rather than tracked.</summary>
[Theory]
[InlineData("", "session-1")]
[InlineData(" ", "session-1")]
[InlineData("conn-1", "")]
[InlineData("conn-1", " ")]
public void AddViewer_WithBlankIdentifiers_IsIgnored(string connectionId, string sessionId)
{
EventsHubViewerRegistry registry = new();
registry.AddViewer(connectionId, sessionId);
Assert.False(registry.HasViewers(sessionId));
Assert.False(registry.HasViewers("session-1"));
}
/// <summary>Concurrent add/remove pairs settle at zero viewers, never at a stuck-on count.</summary>
[Fact]
public void AddAndRemoveViewer_UnderConcurrency_SettlesAtZero()
{
EventsHubViewerRegistry registry = new();
Parallel.For(0, 64, i =>
{
string connectionId = $"conn-{i}";
for (int pass = 0; pass < 50; pass++)
{
registry.AddViewer(connectionId, "session-1");
registry.RemoveViewer(connectionId, "session-1");
}
});
Assert.False(registry.HasViewers("session-1"));
}
}
@@ -205,6 +205,12 @@ public sealed class GatewayApplicationTests
"/galaxy",
"/apikeys",
"/sessions/{SessionId}",
// Mounted from the ZB.MOM.WW.Secrets.Ui RCL rather than declared here, so it is the
// one nav destination that a routing regression could remove without touching this
// repo's own pages. The side rail links to it (role-gated), which makes an unmapped
// route a visible dead link rather than a silent absence.
"/admin/secrets",
];
foreach (string canonical in canonicalRoutes)
{
@@ -120,6 +120,30 @@ public sealed class MxAccessGrpcMapperTests
Assert.Equal(ProtocolStatusCode.ProtocolViolation, publicReply.ProtocolStatus.Code);
}
/// <summary>
/// Verifies MapCommandReply transfers ownership of the inner MxCommandReply the same way
/// MapEvent does: the returned reference is the instance carried by the WorkerCommandReply,
/// not a clone. The WorkerCommandReply is discarded after mapping and the awaiting Invoke
/// call is its single consumer, so moving the inner reply out is safe and avoids a deep copy
/// of a potentially large bulk-read payload.
/// </summary>
[Fact]
public void MapCommandReply_TransfersOwnershipOfInnerReplyWithoutCloning()
{
MxCommandReply innerReply = new()
{
SessionId = "session-1",
Kind = MxCommandKind.Register,
ProtocolStatus = MxAccessGrpcMapper.Ok(),
Register = new RegisterReply { ServerHandle = 50 },
};
WorkerCommandReply workerReply = new() { Reply = innerReply };
MxCommandReply mapped = new MxAccessGrpcMapper().MapCommandReply(workerReply);
Assert.Same(innerReply, mapped);
}
/// <summary>
/// Verifies MapEvent transfers ownership of the inner MxEvent (GWC-07 / IPC-05): the
/// returned reference is the same instance carried by the WorkerEvent, not a clone. The
@@ -1,3 +1,4 @@
using System.Collections.Concurrent;
using System.Threading.Channels;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Time.Testing;
@@ -1057,6 +1058,161 @@ public sealed class SessionEventDistributorTests
Assert.False(lateCts.IsCancellationRequested);
}
/// <summary>
/// Guards the copy-on-write fan-out snapshot: registrations and unregistrations churn on
/// another thread while the pump is actively fanning events, and the stable subscriber
/// must still receive every event exactly once and in order. The pump captures the
/// subscriber array once per event instead of enumerating the dictionary, so a mutation
/// racing the fan-out must never drop, duplicate, or reorder an event for a subscriber
/// registered throughout — nor leave the array and the dictionary disagreeing on the
/// subscriber count once the churn stops.
/// <para>
/// One assumption worth naming: the snapshot is rebuilt from
/// <c>ConcurrentDictionary.Values</c>, whose bucket order happens to keep the long-lived
/// stable subscriber ahead of the churned ones here, so it is written to before a churned
/// subscriber's disposal can interleave. If that ever stops holding the test does not
/// silently pass — it fails as a <see cref="ReadTimeout"/> expiry on the read below, because
/// an event dropped for the stable subscriber never arrives.
/// </para>
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RegistrationChurnDuringFanOut_StableSubscriberStillReceivesEveryEventInOrder()
{
// Below the 64-event per-subscriber queue capacity, so the stable subscriber cannot
// overflow and be disconnected while the writes race the churn — the assertion stays
// deterministic no matter how the threads interleave.
const int EventCount = 50;
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
await using SessionEventDistributor distributor = CreateDistributor(source.Reader);
await distributor.StartAsync(CancellationToken.None);
using IEventSubscriberLease stable = distributor.Register();
using CancellationTokenSource churnCts = new();
Task churn = Task.Run(async () =>
{
while (!churnCts.IsCancellationRequested)
{
// Register then immediately unregister: every iteration rebuilds the fan-out
// snapshot twice, maximizing the chance of landing inside a fan-out pass.
distributor.Register().Dispose();
await Task.Yield();
}
});
for (ulong sequence = 1; sequence <= EventCount; sequence++)
{
source.Writer.TryWrite(Event(sequence));
}
List<ulong> received = [];
for (int i = 0; i < EventCount; i++)
{
received.Add((await ReadOneAsync(stable.Reader)).WorkerSequence);
}
await churnCts.CancelAsync();
await churn.WaitAsync(ReadTimeout);
Assert.Equal(Enumerable.Range(1, EventCount).Select(sequence => (ulong)sequence), received);
// Only the stable subscriber remains: the snapshot the count is read from tracked every
// add and remove the churn performed.
Assert.Equal(1, distributor.SubscriberCount);
}
/// <summary>
/// Regression: a subscriber that unregisters (lease disposed) after the pump captured the
/// fan-out array is still written to, and <c>TryWrite</c> on its now-completed channel
/// returns false — the same signal a full channel gives. Treating that as backpressure
/// emitted a bogus <c>EventQueueOverflow</c> metric and, under the default
/// single-subscriber FailFast policy, faulted the whole session: a stream ending normally
/// during traffic could kill the session. The overflow path must claim the removal first
/// and bail out when the subscriber is already gone.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task GracefulUnregisterDuringFanOut_DoesNotReportOverflow_OrFaultTheSession()
{
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
// Records every overflow-handler invocation. isInternal distinguishes the deliberate
// overflow (the internal subscriber below) from the graceful unregister under test.
ConcurrentQueue<(bool IsOnlySubscriber, bool IsInternal)> invocations = new();
IEventSubscriberLease? gracefulLease = null;
int disposedGracefulLease = 0;
await using SessionEventDistributor distributor = new(
"session-graceful-unregister",
ct => source.Reader.ReadAllAsync(ct),
subscriberQueueCapacity: 1,
replayBufferCapacity: 0,
replayRetentionSeconds: 0,
NullLogger<SessionEventDistributor>.Instance,
TimeProvider.System,
(isOnlySubscriber, isInternal) =>
{
invocations.Enqueue((isOnlySubscriber, isInternal));
// The seam that makes the race deterministic: this handler runs ON the pump
// thread, part-way through fanning one event to the array it already captured.
// Disposing the graceful lease here unregisters and completes that subscriber
// in exactly the window the fix targets — after the capture, before the pump
// reaches its TryWrite. Only on the first invocation, so a genuine repeat
// overflow cannot re-trigger it.
if (Interlocked.Exchange(ref disposedGracefulLease, 1) == 0)
{
gracefulLease!.Dispose();
}
},
singleSubscriberMode: true);
await distributor.StartAsync(CancellationToken.None);
// Registered FIRST so it precedes the graceful subscriber in the captured fan-out array,
// putting the graceful subscriber's TryWrite after this one's overflow handler. Internal
// so its own (expected) overflow reports isOnlySubscriber == false and can never fault
// the session by itself. Never read from, so its capacity-1 channel fills immediately.
using IEventSubscriberLease overflowing = distributor.Register(isInternal: true);
// External subscriber that will unregister gracefully mid-fan-out. Under the old
// behavior its completed-channel TryWrite reported isOnlySubscriber == true, which is
// precisely the legacy FailFast "fault the session" signal.
gracefulLease = distributor.Register();
// Event 1 fills the internal subscriber's channel and is drained from the graceful one,
// so on event 2 the internal subscriber overflows while the graceful one has room —
// whichever order the array happens to hold, only the internal subscriber overflows.
source.Writer.TryWrite(Event(1));
MxEvent first = await ReadOneAsync(gracefulLease.Reader);
Assert.Equal(1ul, first.WorkerSequence);
// Event 2: the internal subscriber overflows, the handler disposes the graceful lease,
// and the pump then writes event 2 to that already-completed channel.
source.Writer.TryWrite(Event(2));
// The graceful subscriber's channel must complete cleanly — no EventQueueOverflow fault.
await AssertCompletedAsync(gracefulLease.Reader);
// The pump survives and keeps serving a freshly-attached subscriber.
using IEventSubscriberLease later = distributor.Register();
source.Writer.TryWrite(Event(3));
Assert.Equal(3ul, (await ReadOneAsync(later.Reader)).WorkerSequence);
// Guards against a vacuous pass: the deliberate internal overflow must actually have
// fired, since that handler call is the seam that disposes the lease mid-fan-out.
Assert.NotEmpty(invocations);
Assert.Equal(1, Volatile.Read(ref disposedGracefulLease));
// The deliberate internal overflow is expected; the graceful unregister must NOT have
// produced an overflow report of its own. An isOnlySubscriber == true invocation is the
// exact signal that would have faulted the session.
Assert.All(invocations, invocation => Assert.True(invocation.IsInternal));
Assert.DoesNotContain(invocations, invocation => invocation.IsOnlySubscriber);
}
private static SessionEventDistributor CreateDistributor(ChannelReader<MxEvent> source)
=> CreateDistributor(source, replayBufferCapacity: 1024, replayRetentionSeconds: 300);
@@ -1087,6 +1087,117 @@ public sealed class SessionManagerTests
Assert.Equal(1, workerClient.ShutdownCount);
}
/// <summary>
/// A sweep pass tears the selected sessions down concurrently rather than one worker
/// shutdown after another: with a mass expiry, a few hung workers would otherwise
/// serialize reaping (each close is bounded by <c>Worker:ShutdownTimeoutSeconds</c>) and
/// starve session slots. Overlap is asserted by counting concurrent entries into the fake
/// worker's shutdown rather than by wall clock, which is sturdier on a loaded box.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task CloseExpiredLeasesAsync_ClosesExpiredSessionsConcurrently()
{
ShutdownConcurrencyProbe probe = new(expectedConcurrency: 2);
FakeWorkerClient firstClient = new() { ShutdownConcurrencyProbe = probe };
FakeWorkerClient secondClient = new() { ShutdownConcurrencyProbe = probe };
SessionManager manager = CreateManager(new QueueingSessionWorkerClientFactory(firstClient, secondClient));
GatewaySession firstSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None);
GatewaySession secondSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-2", ownerKeyId: null, CancellationToken.None);
DateTimeOffset now = DateTimeOffset.UtcNow;
firstSession.ExtendLease(now.AddSeconds(-1));
secondSession.ExtendLease(now.AddSeconds(-1));
int closedCount = await manager.CloseExpiredLeasesAsync(now, CancellationToken.None);
Assert.Equal(2, closedCount);
Assert.Equal(SessionState.Closed, firstSession.State);
Assert.Equal(SessionState.Closed, secondSession.State);
Assert.Equal(2, probe.MaxObservedConcurrency);
}
/// <summary>
/// Host stop drains sessions concurrently: 50 sessions at a worst-case 10 s shutdown each
/// would exceed any host stop-timeout if drained one at a time, leaving the tail to the
/// orphan killer.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ShutdownAsync_ClosesSessionsConcurrently()
{
ShutdownConcurrencyProbe probe = new(expectedConcurrency: 2);
FakeWorkerClient firstClient = new() { ShutdownConcurrencyProbe = probe };
FakeWorkerClient secondClient = new() { ShutdownConcurrencyProbe = probe };
SessionManager manager = CreateManager(new QueueingSessionWorkerClientFactory(firstClient, secondClient));
GatewaySession firstSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None);
GatewaySession secondSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-2", ownerKeyId: null, CancellationToken.None);
await manager.ShutdownAsync(CancellationToken.None);
Assert.Equal(SessionState.Closed, firstSession.State);
Assert.Equal(SessionState.Closed, secondSession.State);
Assert.Equal(2, probe.MaxObservedConcurrency);
}
/// <summary>
/// A close that throws must not abandon the rest of the selected set: the sweep still
/// tears the healthy expired session down, and the failure still surfaces to the lease
/// monitor (which logs it) exactly as the sequential loop did.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task CloseExpiredLeasesAsync_WhenOneCloseFails_StillClosesRemainingSessionsAndRethrows()
{
FakeWorkerClient failingClient = new()
{
ShutdownException = new InvalidOperationException("worker shutdown failed"),
KillException = new InvalidOperationException("worker kill failed"),
};
FakeWorkerClient healthyClient = new();
SessionManager manager = CreateManager(new QueueingSessionWorkerClientFactory(failingClient, healthyClient));
GatewaySession failingSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None);
GatewaySession healthySession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-2", ownerKeyId: null, CancellationToken.None);
DateTimeOffset now = DateTimeOffset.UtcNow;
failingSession.ExtendLease(now.AddSeconds(-1));
healthySession.ExtendLease(now.AddSeconds(-1));
SessionManagerException exception = await Assert.ThrowsAsync<SessionManagerException>(
async () => await manager.CloseExpiredLeasesAsync(now, CancellationToken.None));
Assert.Equal(SessionManagerErrorCode.CloseFailed, exception.ErrorCode);
Assert.Equal(1, healthyClient.ShutdownCount);
Assert.Equal(SessionState.Closed, healthySession.State);
Assert.False(manager.TryGetSession(healthySession.SessionId, out _));
Assert.False(manager.TryGetSession(failingSession.SessionId, out _));
}
/// <summary>
/// A drain whose token is already cancelled (the host stop deadline elapsed) must still
/// kill every worker rather than skip the teardown: an unkilled worker is a leaked x86
/// process, and a restarted gateway terminates orphans instead of reattaching to them. This
/// pins both halves of the fix — the parallel loop is not bound to the caller's token, and
/// the kill fallback does not run on it.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ShutdownAsync_WhenCancelledBeforeDraining_StillKillsEveryWorker()
{
FakeWorkerClient firstClient = new();
FakeWorkerClient secondClient = new();
SessionManager manager = CreateManager(new QueueingSessionWorkerClientFactory(firstClient, secondClient));
GatewaySession firstSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None);
GatewaySession secondSession = await manager.OpenSessionAsync(CreateOpenRequest(), "client-2", ownerKeyId: null, CancellationToken.None);
using CancellationTokenSource cancellation = new();
await cancellation.CancelAsync();
await manager.ShutdownAsync(cancellation.Token);
Assert.Equal(1, firstClient.KillCount);
Assert.Equal(1, secondClient.KillCount);
Assert.False(manager.TryGetSession(firstSession.SessionId, out _));
Assert.False(manager.TryGetSession(secondSession.SessionId, out _));
}
/// <summary>Verifies that shutdown closes all registered sessions.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -1274,6 +1385,13 @@ public sealed class SessionManagerTests
/// <summary>Gets a value indicating whether to block shutdown on the fake worker client.</summary>
public bool BlockShutdown { get; init; }
/// <summary>
/// Gets the rendezvous that records how many shutdowns overlap, shared by the fakes of
/// the sessions a single teardown pass closes. Null when the test does not measure
/// teardown concurrency.
/// </summary>
public ShutdownConcurrencyProbe? ShutdownConcurrencyProbe { get; init; }
/// <summary>Gets the last command invoked on the fake worker client.</summary>
public WorkerCommand? LastCommand { get; private set; }
@@ -1335,6 +1453,11 @@ public sealed class SessionManagerTests
throw ShutdownException;
}
if (ShutdownConcurrencyProbe is not null)
{
await ShutdownConcurrencyProbe.EnterAsync(cancellationToken);
}
if (BlockShutdown)
{
ShutdownStarted.TrySetResult();
@@ -1379,4 +1502,67 @@ public sealed class SessionManagerTests
}
}
/// <summary>
/// Rendezvous that measures how many worker shutdowns a teardown pass runs at once. Each
/// entering shutdown records the in-flight count and waits until <paramref name="expectedConcurrency"/>
/// shutdowns are in flight, so a genuinely parallel teardown releases immediately while a
/// sequential one can only release on the bounded timeout — with a max observed concurrency
/// of one, which is the assertion that fails.
/// </summary>
/// <param name="expectedConcurrency">Number of overlapping shutdowns that releases the rendezvous.</param>
private sealed class ShutdownConcurrencyProbe(int expectedConcurrency)
{
private static readonly TimeSpan RendezvousTimeout = TimeSpan.FromSeconds(5);
private readonly TaskCompletionSource _reached = new(TaskCreationOptions.RunContinuationsAsynchronously);
private int _inFlight;
private int _maxInFlight;
/// <summary>Gets the highest number of shutdowns observed in flight at the same time.</summary>
public int MaxObservedConcurrency => Volatile.Read(ref _maxInFlight);
/// <summary>Enters the rendezvous for one worker shutdown and waits for the expected overlap.</summary>
/// <param name="cancellationToken">Token that abandons the wait.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task EnterAsync(CancellationToken cancellationToken)
{
int inFlight = Interlocked.Increment(ref _inFlight);
RecordMax(inFlight);
if (inFlight >= expectedConcurrency)
{
_reached.TrySetResult();
}
try
{
await _reached.Task.WaitAsync(RendezvousTimeout, cancellationToken);
}
catch (Exception exception) when (exception is TimeoutException or OperationCanceledException)
{
// Sequential teardown: the expected overlap never happens, so let the shutdown
// finish and let MaxObservedConcurrency report the (failing) truth. Cancellation is
// swallowed for the same reason — a cancelled rendezvous must not turn into a
// second, misleading failure on top of the concurrency assertion.
}
finally
{
Interlocked.Decrement(ref _inFlight);
}
}
private void RecordMax(int inFlight)
{
int observed = Volatile.Read(ref _maxInFlight);
while (inFlight > observed)
{
int previous = Interlocked.CompareExchange(ref _maxInFlight, inFlight, observed);
if (previous == observed)
{
return;
}
observed = previous;
}
}
}
}
@@ -159,7 +159,11 @@ public sealed class WorkerClientTests
CreateCommand(MxCommandKind.GetWorkerInfo),
TestTimeout,
CancellationToken.None);
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
// The timeout also emits a WorkerCancel for the abandoned correlation (GWC-31), which sits
// ahead of the second command on the FIFO pipe; skip it rather than mistaking it for the
// command this assertion is about.
WorkerEnvelope secondCommand = await ReadNextCommandAsync(pipePair, timedOutCommand.CorrelationId);
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
@@ -169,6 +173,49 @@ public sealed class WorkerClientTests
Assert.Equal(MxCommandKind.GetWorkerInfo, reply.Reply.Kind);
}
/// <summary>
/// A command timeout abandons the gateway-side wait, but the worker keeps the correlation on
/// its single STA queue and would still run it — so the gateway forwards a <c>WorkerCancel</c>
/// for the abandoned correlation id (GWC-31). Without it, a client that retries after a
/// timeout stacks work the worker still intends to execute. Asserted on the wire because the
/// cancel is protocol behavior the worker depends on, not an internal detail.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task InvokeAsync_WhenCommandTimesOut_SendsWorkerCancelForThatCorrelation()
{
await using PipePair pipePair = await PipePair.CreateAsync();
await using WorkerClient client = CreateClient(pipePair);
await CompleteHandshakeAsync(client, pipePair);
// Advise rather than a control command: control commands bypass the worker's STA queue, so a
// data command is the case the cancel actually exists for.
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
CreateCommand(MxCommandKind.Advise),
TimeSpan.FromMilliseconds(50),
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
async () => await invokeTask.WaitAsync(TestTimeout));
Assert.Equal(WorkerClientErrorCode.CommandTimeout, exception.ErrorCode);
WorkerEnvelope cancelEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCancel, cancelEnvelope.BodyCase);
Assert.Equal(commandEnvelope.CorrelationId, cancelEnvelope.CorrelationId);
Assert.False(string.IsNullOrWhiteSpace(cancelEnvelope.WorkerCancel.Reason));
Assert.True(
cancelEnvelope.Sequence > commandEnvelope.Sequence,
$"The cancel arrived with sequence {cancelEnvelope.Sequence} after {commandEnvelope.Sequence}; "
+ "envelope sequences must be strictly increasing in wire order.");
// The timeout fails one command; it is not a session fault.
Assert.Equal(WorkerClientState.Ready, client.State);
}
/// <summary>
/// The envelope <c>sequence</c> is a monotonic per-sender counter (gateway.md), so the values
/// observed on the pipe must be strictly increasing in wire order. Stamping the sequence when
@@ -1030,6 +1077,32 @@ public sealed class WorkerClientTests
return envelope;
}
/// <summary>
/// Reads gateway envelopes until a <c>WorkerCommand</c> arrives, skipping the <c>WorkerCancel</c>
/// a command timeout emits for <paramref name="canceledCorrelationId"/>. Anything else on the pipe
/// fails the test rather than being skipped: the point of the skip is to tolerate exactly the one
/// known interleaving, not to make the assertion blind to unexpected gateway traffic.
/// </summary>
/// <param name="pipePair">The connected pipe pair whose worker side is read.</param>
/// <param name="canceledCorrelationId">Correlation id of the timed-out command whose cancel is expected.</param>
/// <returns>The next command envelope written by the gateway.</returns>
private static async Task<WorkerEnvelope> ReadNextCommandAsync(
PipePair pipePair,
string canceledCorrelationId)
{
while (true)
{
WorkerEnvelope envelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerCommand)
{
return envelope;
}
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCancel, envelope.BodyCase);
Assert.Equal(canceledCorrelationId, envelope.CorrelationId);
}
}
private static async Task WaitUntilAsync(
Func<bool> predicate,
TimeSpan timeout)
@@ -47,12 +47,56 @@ public sealed class WorkerProcessLauncherTests
"1500",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerWriteCompletionWaitEnvironmentVariableName]);
// The worker sizes its outbound event queue from the launch environment;
// the queue has no drop policy, so this capacity is the session's burst
// headroom rather than a throttle.
Assert.Equal(
"10000",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerEventQueueCapacityEnvironmentVariableName]);
// MxGateway:Alarms defaults reach the worker's alarm poll loop and its
// GetXmlCurrentAlarms2 cap (which is also its truncation threshold)
// through the launch environment, not the command line.
Assert.Equal(
"500",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerAlarmPollIntervalEnvironmentVariableName]);
Assert.Equal(
"1024",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerMaxAlarmsPerFetchEnvironmentVariableName]);
Assert.DoesNotContain(Nonce, handle.CommandLine.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(Nonce, string.Join(" ", handle.CommandLine.Arguments), StringComparison.Ordinal);
Assert.False(pipeReservation.DisposeCalled);
Assert.Equal(0, metrics.GetSnapshot().WorkersRunning);
}
/// <summary>
/// Verifies that a configured <see cref="WorkerOptions.EventQueueCapacity"/> — not the shipped
/// default — is what reaches the worker, so the option is deployable without a worker rebuild.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task LaunchAsync_WithConfiguredEventQueueCapacity_ExportsItToTheWorkerEnvironment()
{
using TestDirectory directory = TestDirectory.Create();
string executablePath = directory.CreateWorkerExecutable(machine: 0x014c);
FakeWorkerProcessFactory processFactory = new(new FakeWorkerProcess(processId: 1234));
WorkerProcessLauncher launcher = CreateLauncher(
executablePath,
processFactory,
new SucceedingStartupProbe(),
eventQueueCapacity: 65536);
using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest());
Assert.NotNull(processFactory.LastStartInfo);
Assert.Equal(
"65536",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerEventQueueCapacityEnvironmentVariableName]);
}
/// <summary>Verifies that a failed startup probe kills and disposes the worker process.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -205,7 +249,8 @@ public sealed class WorkerProcessLauncherTests
GatewayMetrics? metrics = null,
int startupTimeoutSeconds = 30,
int startupProbeRetryAttempts = 3,
int startupProbeRetryDelayMilliseconds = 250)
int startupProbeRetryDelayMilliseconds = 250,
int eventQueueCapacity = 10000)
{
GatewayOptions options = new()
{
@@ -216,6 +261,7 @@ public sealed class WorkerProcessLauncherTests
StartupTimeoutSeconds = startupTimeoutSeconds,
StartupProbeRetryAttempts = startupProbeRetryAttempts,
StartupProbeRetryDelayMilliseconds = startupProbeRetryDelayMilliseconds,
EventQueueCapacity = eventQueueCapacity,
},
};
@@ -19,7 +19,10 @@ public sealed class GatewayMetricsTests
metrics.CommandFailed("WriteSecured", "AuthorizationFailed", TimeSpan.FromMilliseconds(12));
metrics.EventReceived("session-1", "OnDataChange");
metrics.EventReceived("session-1", "OnDataChange");
metrics.SetWorkerEventQueueDepth(7);
// GWC-30: the worker queue-depth gauge sums one live source per worker client, so the two
// registrations below stand in for two concurrent sessions holding 3 and 4 events.
using IDisposable workerDepthSourceA = metrics.RegisterWorkerEventQueueDepthSource(static () => 3);
using IDisposable workerDepthSourceB = metrics.RegisterWorkerEventQueueDepthSource(static () => 4);
// GWC-15: the gRPC stream queue-depth gauge sums live backlog sources at collection time
// rather than tracking a pushed running total. Register a source reporting 3.
using IDisposable backlogSource = metrics.RegisterEventStreamBacklogSource(static () => 3);
@@ -54,16 +57,111 @@ public sealed class GatewayMetricsTests
Assert.Equal(2, snapshot.EventsBySession["session-1"]);
}
/// <summary>Verifies that negative queue depth is rejected.</summary>
/// <summary>
/// GWC-30: the worker queue-depth gauge sums every registered source rather than holding a
/// single pushed scalar, so concurrent sessions add up instead of overwriting one another,
/// and a disposed registration (a worker client going away) drops out of the sum. Disposal
/// is idempotent because a client's dispose path can run twice.
/// </summary>
[Fact]
public void SetEventQueueDepth_RejectsNegativeDepth()
public void WorkerEventQueueDepthSources_SumAcrossRegistrationsAndDropOnDispose()
{
using GatewayMetrics metrics = new();
ArgumentOutOfRangeException exception = Assert.Throws<ArgumentOutOfRangeException>(
() => metrics.SetWorkerEventQueueDepth(-1));
IDisposable firstSource = metrics.RegisterWorkerEventQueueDepthSource(static () => 3);
using IDisposable secondSource = metrics.RegisterWorkerEventQueueDepthSource(static () => 4);
Assert.Equal("depth", exception.ParamName);
Assert.Equal(7, metrics.GetSnapshot().WorkerEventQueueDepth);
firstSource.Dispose();
Assert.Equal(4, metrics.GetSnapshot().WorkerEventQueueDepth);
firstSource.Dispose();
Assert.Equal(4, metrics.GetSnapshot().WorkerEventQueueDepth);
}
/// <summary>
/// A depth source reads a lock-free counter that a racing decrement can momentarily push
/// below zero, so the sum clamps each reading instead of rejecting it — the pull model has
/// no caller to throw back at.
/// </summary>
[Fact]
public void WorkerEventQueueDepthSources_ClampNegativeReadingsToZero()
{
using GatewayMetrics metrics = new();
using IDisposable negativeSource = metrics.RegisterWorkerEventQueueDepthSource(static () => -5);
using IDisposable positiveSource = metrics.RegisterWorkerEventQueueDepthSource(static () => 4);
Assert.Equal(4, metrics.GetSnapshot().WorkerEventQueueDepth);
}
/// <summary>
/// Verifies the exported gauge keeps the name <c>mxgateway.events.worker_queue.depth</c> and
/// reports the summed sources, so the pull-model rework is invisible to exporters.
/// </summary>
[Fact]
public void WorkerEventQueueDepthGauge_ReportsSummedSources()
{
using GatewayMetrics metrics = new();
using MeterListener listener = new();
int? capturedDepth = null;
listener.InstrumentPublished = (instrument, meterListener) =>
{
if (ReferenceEquals(instrument.Meter, metrics.Meter)
&& instrument.Name == "mxgateway.events.worker_queue.depth")
{
meterListener.EnableMeasurementEvents(instrument);
}
};
listener.SetMeasurementEventCallback<int>(
(instrument, measurement, _, _) =>
{
if (ReferenceEquals(instrument.Meter, metrics.Meter)
&& instrument.Name == "mxgateway.events.worker_queue.depth")
{
capturedDepth = measurement;
}
});
listener.Start();
using IDisposable firstSource = metrics.RegisterWorkerEventQueueDepthSource(static () => 5);
using IDisposable secondSource = metrics.RegisterWorkerEventQueueDepthSource(static () => 6);
listener.RecordObservableInstruments();
Assert.Equal(11, capturedDepth);
}
/// <summary>
/// The command counters are incremented with <see cref="Interlocked"/> rather than under the
/// process-wide metrics lock, so this asserts no increment is lost when every gRPC thread
/// records at once.
/// </summary>
[Fact]
public void CommandCounters_CountEveryConcurrentInvocation()
{
const int workers = 8;
const int perWorker = 500;
using GatewayMetrics metrics = new();
Parallel.For(0, workers, _ =>
{
for (int index = 0; index < perWorker; index++)
{
metrics.CommandStarted("Register");
metrics.CommandSucceeded("Register", TimeSpan.FromMilliseconds(1));
metrics.CommandFailed("WriteSecured", "AuthorizationFailed", TimeSpan.FromMilliseconds(1));
}
});
GatewayMetricsSnapshot snapshot = metrics.GetSnapshot();
Assert.Equal(workers * perWorker, snapshot.CommandsStarted);
Assert.Equal(workers * perWorker, snapshot.CommandsSucceeded);
Assert.Equal(workers * perWorker, snapshot.CommandsFailed);
Assert.Equal(workers * perWorker, snapshot.CommandFailuresByMethod["WriteSecured"]);
}
/// <summary>

Some files were not shown because too many files have changed in this diff Show More