perf(worker,gateway): pin MXSTATUS_PROXY record info, dedicated event-drain thread, cap thread-pool spin

At ~240 events/s the worker burned 8-12% system CPU on wonder-app-vd03. A WPR
trace put 72% of it in lmxproxy's per-event GetRecordInfoFromGuids (type
library reload through the registry, amplified by WOW64 and the EDR registry
filter) and 26% in the .NET Framework thread pool spinning after each
per-event work item.

- Worker: MxStatusProxyRecordInfoPin holds the LMXProxy ITypeLib and the
  MXSTATUS_PROXY IRecordInfo for the process lifetime, so oleaut32 serves the
  per-event lookup from cache (~1 ms -> <1 us per call). Best-effort, never
  fatal; status logged at handshake.
- Worker: the event drain loop runs on a dedicated background thread with a
  synchronous wait and a synchronous pipe flush, removing the per-event
  thread-pool hops.
- Gateway: runtimeconfig sets ThreadPool.UnfairSemaphoreSpinLimit=10 to cut
  PortableThreadPool spin-wait CPU.

windev bench at ~242 ev/s: worker 67% -> 6.2% of one core; gateway 8.1% -> 6.9%.
Worker suite 785 passed / 14 skipped (x86, windev).
This commit is contained in:
Joseph Doherty
2026-11-21 08:00:13 -05:00
parent 08a9cfd68a
commit 659bb831a2
15 changed files with 398 additions and 34 deletions
+47 -3
View File
@@ -479,6 +479,32 @@ Creation rules:
- Never marshal the raw COM object to pipe reader/writer threads.
- Capture COM creation HRESULT or exception details.
**Record-info pin.** Before the first `LMXProxyServerClass` is created,
`MxAccessComObjectFactory.Create()` calls `MxStatusProxyRecordInfoPin.EnsurePinned()`. Once per
process, this loads the LMXProxy type library and the `MXSTATUS_PROXY` record info, and holds
both references until exit. The GUIDs and version come from the interop assembly's metadata:
library `{77e896ec-b3e3-4de4-b96f-f32570164211}` v3.2 (`MXAccess32.tlb`), record
`{A162D9E4-9019-42F3-A087-79264007E542}`.
The pin exists because every `OnDataChange` carries a SAFEARRAY of `MXSTATUS_PROXY` records, and
`lmxproxy.dll` resolves that record type with `GetRecordInfoFromGuids` each time it builds one.
With no live reference, oleaut32 re-reads the registry and reloads the type library on every
call:
- about 1 ms per event on the dev box;
- worse under WOW64 registry redirection plus an EDR registry filter, where a production WPR trace
put it at ~70% of the STA's CPU.
While one `IRecordInfo` is held, the same call returns oleaut32's cached instance in under a
microsecond. Measured at ~240 events/s on the dev box, worker CPU fell from 67% to 10% of a core
with the pin alone.
The pin only warms a cache. It changes no MXAccess behaviour, and a failure is never fatal. The
outcome is in `MxStatusProxyRecordInfoPin.Status`, and the worker logs it as `WorkerRecordInfoPin`
just before `WorkerReady`. Worker logs go to stderr, which the gateway does not redirect, so the
line lands wherever the host captures the gateway's stderr (the NSSM stderr log on deployed
boxes).
If COM creation fails, the worker should send a structured fault with:
- fault category,
@@ -564,9 +590,27 @@ everything that arrived — the loop must therefore re-check `DrainFault()` and
re-drain after every wait, never treat a wake as "exactly one event". The 25 ms
`EventDrainInterval` survives as the **fallback ceiling** on an unsignalled wait,
not as a latency floor: an event arriving at an idle worker is framed at signal
latency instead of waiting out a tick, an idle worker parks instead of waking 40
times a second, and the interval only bounds how long the loop may sleep if some
future path mutates the queue without signalling.
latency instead of waiting out a tick, and the interval only bounds how long the
loop may sleep if some future path mutates the queue without signalling. An idle
worker still wakes at that ceiling, but a wake on an empty queue is cheap.
The loop runs on a **dedicated background thread** (`mxgw-event-drain`), not on the
CLR thread pool:
- It parks in the blocking `IWorkerRuntimeSession.WaitForEvents`, which uses the
same coalesced signal and single-waiter contract as `WaitForEventsAsync`.
- It blocks on the frame writer's batch write.
- `RunEventDrainLoopAsync` starts the thread and returns a task that completes,
faults or cancels exactly as the loop does, so the message loop's `Task.WhenAny`
supervision is unchanged.
The reason is .NET Framework's thread pool. As an async loop, every enqueue's
semaphore release queued a continuation to the pool, and each pool worker that ran
one then spun in `SleepEx` before parking. A production WPR trace measured ~150
context switches per event and ~26% of the worker's CPU in that spin. With the
dedicated thread and the frame writer's synchronous flush, the steady-state event
path schedules no managed pool work. The overlapped pipe write still completes on
an I/O completion thread.
A single event whose envelope exceeds the negotiated frame maximum is
**undeliverable end to end** — the pipe maximum sits only the envelope-overhead