perf(worker,gateway): pin MXSTATUS_PROXY record info, dedicated event-drain thread, cap thread-pool spin
At ~240 events/s the worker burned 8-12% system CPU on wonder-app-vd03. A WPR trace put 72% of it in lmxproxy's per-event GetRecordInfoFromGuids (type library reload through the registry, amplified by WOW64 and the EDR registry filter) and 26% in the .NET Framework thread pool spinning after each per-event work item. - Worker: MxStatusProxyRecordInfoPin holds the LMXProxy ITypeLib and the MXSTATUS_PROXY IRecordInfo for the process lifetime, so oleaut32 serves the per-event lookup from cache (~1 ms -> <1 us per call). Best-effort, never fatal; status logged at handshake. - Worker: the event drain loop runs on a dedicated background thread with a synchronous wait and a synchronous pipe flush, removing the per-event thread-pool hops. - Gateway: runtimeconfig sets ThreadPool.UnfairSemaphoreSpinLimit=10 to cut PortableThreadPool spin-wait CPU. windev bench at ~242 ev/s: worker 67% -> 6.2% of one core; gateway 8.1% -> 6.9%. Worker suite 785 passed / 14 skipped (x86, windev).
This commit is contained in:
@@ -807,6 +807,21 @@ Worst-case on-disk log footprint is therefore about `retainedFileCountLimit x fi
|
|||||||
Raise or lower these in the deployed `appsettings.json` (or via `Serilog__WriteTo__1__Args__*` environment
|
Raise or lower these in the deployed `appsettings.json` (or via `Serilog__WriteTo__1__Args__*` environment
|
||||||
variables) to fit the host's disk budget.
|
variables) to fit the host's disk budget.
|
||||||
|
|
||||||
|
## Runtime Settings (runtimeconfig)
|
||||||
|
|
||||||
|
Some settings belong to the .NET runtime rather than to `MxGateway:*` options. They ship in the
|
||||||
|
build through `RuntimeHostConfigurationOption` items in the Server csproj, which write them into
|
||||||
|
`ZB.MOM.WW.MxGateway.Server.runtimeconfig.json`. They therefore travel with every deploy and need
|
||||||
|
no NSSM environment variables.
|
||||||
|
|
||||||
|
| Setting | Shipped value | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| `System.Threading.ThreadPool.UnfairSemaphoreSpinLimit` | `10` (runtime default `70`) | A pool worker that finds no work spins before parking, and past step 10 it calls `Sleep(0)` on every other step. The event relay queues several pool work items per event, so on a busy host those spins became ~93% of the gateway's context switches (WPR trace on a 4-CPU production host, ~240 events/s). With `10`, idle workers spin briefly and then park. Measured on the dev box at ~240 events/s: gateway CPU −15%. |
|
||||||
|
|
||||||
|
To override a setting on one host without rebuilding, use the environment-variable form, which
|
||||||
|
takes precedence over runtimeconfig: `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=<n>`. `0`
|
||||||
|
disables the spin entirely.
|
||||||
|
|
||||||
## Related Documentation
|
## Related Documentation
|
||||||
|
|
||||||
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
||||||
|
|||||||
@@ -479,6 +479,32 @@ Creation rules:
|
|||||||
- Never marshal the raw COM object to pipe reader/writer threads.
|
- Never marshal the raw COM object to pipe reader/writer threads.
|
||||||
- Capture COM creation HRESULT or exception details.
|
- Capture COM creation HRESULT or exception details.
|
||||||
|
|
||||||
|
**Record-info pin.** Before the first `LMXProxyServerClass` is created,
|
||||||
|
`MxAccessComObjectFactory.Create()` calls `MxStatusProxyRecordInfoPin.EnsurePinned()`. Once per
|
||||||
|
process, this loads the LMXProxy type library and the `MXSTATUS_PROXY` record info, and holds
|
||||||
|
both references until exit. The GUIDs and version come from the interop assembly's metadata:
|
||||||
|
library `{77e896ec-b3e3-4de4-b96f-f32570164211}` v3.2 (`MXAccess32.tlb`), record
|
||||||
|
`{A162D9E4-9019-42F3-A087-79264007E542}`.
|
||||||
|
|
||||||
|
The pin exists because every `OnDataChange` carries a SAFEARRAY of `MXSTATUS_PROXY` records, and
|
||||||
|
`lmxproxy.dll` resolves that record type with `GetRecordInfoFromGuids` each time it builds one.
|
||||||
|
With no live reference, oleaut32 re-reads the registry and reloads the type library on every
|
||||||
|
call:
|
||||||
|
|
||||||
|
- about 1 ms per event on the dev box;
|
||||||
|
- worse under WOW64 registry redirection plus an EDR registry filter, where a production WPR trace
|
||||||
|
put it at ~70% of the STA's CPU.
|
||||||
|
|
||||||
|
While one `IRecordInfo` is held, the same call returns oleaut32's cached instance in under a
|
||||||
|
microsecond. Measured at ~240 events/s on the dev box, worker CPU fell from 67% to 10% of a core
|
||||||
|
with the pin alone.
|
||||||
|
|
||||||
|
The pin only warms a cache. It changes no MXAccess behaviour, and a failure is never fatal. The
|
||||||
|
outcome is in `MxStatusProxyRecordInfoPin.Status`, and the worker logs it as `WorkerRecordInfoPin`
|
||||||
|
just before `WorkerReady`. Worker logs go to stderr, which the gateway does not redirect, so the
|
||||||
|
line lands wherever the host captures the gateway's stderr (the NSSM stderr log on deployed
|
||||||
|
boxes).
|
||||||
|
|
||||||
If COM creation fails, the worker should send a structured fault with:
|
If COM creation fails, the worker should send a structured fault with:
|
||||||
|
|
||||||
- fault category,
|
- fault category,
|
||||||
@@ -564,9 +590,27 @@ everything that arrived — the loop must therefore re-check `DrainFault()` and
|
|||||||
re-drain after every wait, never treat a wake as "exactly one event". The 25 ms
|
re-drain after every wait, never treat a wake as "exactly one event". The 25 ms
|
||||||
`EventDrainInterval` survives as the **fallback ceiling** on an unsignalled wait,
|
`EventDrainInterval` survives as the **fallback ceiling** on an unsignalled wait,
|
||||||
not as a latency floor: an event arriving at an idle worker is framed at signal
|
not as a latency floor: an event arriving at an idle worker is framed at signal
|
||||||
latency instead of waiting out a tick, an idle worker parks instead of waking 40
|
latency instead of waiting out a tick, and the interval only bounds how long the
|
||||||
times a second, and the interval only bounds how long the loop may sleep if some
|
loop may sleep if some future path mutates the queue without signalling. An idle
|
||||||
future path mutates the queue without signalling.
|
worker still wakes at that ceiling, but a wake on an empty queue is cheap.
|
||||||
|
|
||||||
|
The loop runs on a **dedicated background thread** (`mxgw-event-drain`), not on the
|
||||||
|
CLR thread pool:
|
||||||
|
|
||||||
|
- It parks in the blocking `IWorkerRuntimeSession.WaitForEvents`, which uses the
|
||||||
|
same coalesced signal and single-waiter contract as `WaitForEventsAsync`.
|
||||||
|
- It blocks on the frame writer's batch write.
|
||||||
|
- `RunEventDrainLoopAsync` starts the thread and returns a task that completes,
|
||||||
|
faults or cancels exactly as the loop does, so the message loop's `Task.WhenAny`
|
||||||
|
supervision is unchanged.
|
||||||
|
|
||||||
|
The reason is .NET Framework's thread pool. As an async loop, every enqueue's
|
||||||
|
semaphore release queued a continuation to the pool, and each pool worker that ran
|
||||||
|
one then spun in `SleepEx` before parking. A production WPR trace measured ~150
|
||||||
|
context switches per event and ~26% of the worker's CPU in that spin. With the
|
||||||
|
dedicated thread and the frame writer's synchronous flush, the steady-state event
|
||||||
|
path schedules no managed pool work. The overlapped pipe write still completes on
|
||||||
|
an I/O completion thread.
|
||||||
|
|
||||||
A single event whose envelope exceeds the negotiated frame maximum is
|
A single event whose envelope exceeds the negotiated frame maximum is
|
||||||
**undeliverable end to end** — the pipe maximum sits only the envelope-overhead
|
**undeliverable end to end** — the pipe maximum sits only the envelope-overhead
|
||||||
|
|||||||
@@ -135,7 +135,7 @@ Two failure shapes are distinguished during a drain pass:
|
|||||||
|
|
||||||
Flushes are coalesced across a *run of same-class frames* inside a drain
|
Flushes are coalesced across a *run of same-class frames* inside a drain
|
||||||
pass: each frame in the run is written to the stream without an individual
|
pass: each frame in the run is written to the stream without an individual
|
||||||
flush, then one `FlushAsync` runs — at the end of the pass, and additionally
|
flush, then one flush runs — at the end of the pass, and additionally
|
||||||
at every control-to-event boundary — and only then does every
|
at every control-to-event boundary — and only then does every
|
||||||
successfully-written frame of that run resolve its completion. A caller's
|
successfully-written frame of that run resolve its completion. A caller's
|
||||||
`WriteAsync` therefore still does not complete until its bytes are both
|
`WriteAsync` therefore still does not complete until its bytes are both
|
||||||
@@ -166,11 +166,18 @@ end-of-pass flush, and additionally fails the event frame the drain had
|
|||||||
already claimed off its queue (nothing else would ever complete it) along
|
already claimed off its queue (nothing else would ever complete it) along
|
||||||
with every frame still queued.
|
with every frame still queued.
|
||||||
|
|
||||||
|
The flush is a synchronous `Stream.Flush()`, not `FlushAsync`. `PipeStream` does not
|
||||||
|
override `FlushAsync`, so on .NET Framework the base implementation runs `Flush()`
|
||||||
|
through `Task.Factory.StartNew`. That is one thread-pool work item per drain pass for
|
||||||
|
a `PipeStream.Flush` that does no I/O, because it never calls `FlushFileBuffers`.
|
||||||
|
Calling `Flush()` inline keeps the same written-and-flushed contract for any stream.
|
||||||
|
|
||||||
Note the ordering all of this implies at the peer: the frames reach the pipe
|
Note the ordering all of this implies at the peer: the frames reach the pipe
|
||||||
before the flush that follows them, so the gateway can read a whole batch
|
before the flush that follows them, so the gateway can read a whole batch
|
||||||
while the writer has not yet flushed it. Anything observing the flush itself
|
while the writer has not yet flushed it. Anything observing the flush itself
|
||||||
(a test counting flushes, for instance) must wait for the flush, not infer it
|
(a test counting flushes, for instance) must wait for the flush, not infer it
|
||||||
from frames arriving. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
|
from frames arriving. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`, which
|
||||||
|
runs on a dedicated thread; see `MxAccessWorkerInstanceDesign.md`)
|
||||||
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
|
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
|
||||||
every frame under one `_gate` acquisition, takes the write lock once, and
|
every frame under one `_gate` acquisition, takes the write lock once, and
|
||||||
drains them together, so a burst of N events costs one flush rather than N —
|
drains them together, so a burst of N events costs one flush rather than N —
|
||||||
|
|||||||
@@ -38,6 +38,17 @@
|
|||||||
<PackageReference Include="Polly.Core" Version="8.6.6" />
|
<PackageReference Include="Polly.Core" Version="8.6.6" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<!--
|
||||||
|
Cap the thread pool's idle spin. The default (70 steps, Sleep(0) on every other step past
|
||||||
|
the tenth) makes a worker that finds no work yield ~30 times before parking; with several
|
||||||
|
pool work items per relayed event, a WPR trace on a busy 4-CPU host put ~93% of the
|
||||||
|
gateway's context switches in those yields. 10 keeps a short busy-spin and then parks.
|
||||||
|
Measured on windev at ~240 events/s: gateway CPU -15%. See docs/GatewayConfiguration.md.
|
||||||
|
-->
|
||||||
|
<RuntimeHostConfigurationOption Include="System.Threading.ThreadPool.UnfairSemaphoreSpinLimit" Value="10" />
|
||||||
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\ZB.MOM.WW.MxGateway.Contracts\ZB.MOM.WW.MxGateway.Contracts.csproj" />
|
<ProjectReference Include="..\ZB.MOM.WW.MxGateway.Contracts\ZB.MOM.WW.MxGateway.Contracts.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|||||||
@@ -375,7 +375,7 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Verifies the writer coalesces the flush across a batch of frames drained together: four frames
|
/// Verifies the writer coalesces the flush across a batch of frames drained together: four frames
|
||||||
/// queued behind an in-progress write drain in a single pass and share one FlushAsync, not four.
|
/// queued behind an in-progress write drain in a single pass and share one Flush, not four.
|
||||||
/// Every frame still reaches the wire intact.
|
/// Every frame still reaches the wire intact.
|
||||||
/// <para>
|
/// <para>
|
||||||
/// The burst is all-event on purpose. The control-frame completion decoupling made the drain flush
|
/// The burst is all-event on purpose. The control-frame completion decoupling made the drain flush
|
||||||
@@ -1386,20 +1386,20 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// A MemoryStream that counts FlushAsync calls without gating any write, so a batch write can be
|
// A MemoryStream that counts Flush calls without gating any write, so a batch write can be
|
||||||
// asserted to flush exactly once.
|
// asserted to flush exactly once.
|
||||||
private sealed class FlushCountingStream : MemoryStream
|
private sealed class FlushCountingStream : MemoryStream
|
||||||
{
|
{
|
||||||
private int _flushCount;
|
private int _flushCount;
|
||||||
|
|
||||||
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
|
/// <summary>Gets the number of <see cref="Flush"/> calls observed so far.</summary>
|
||||||
public int FlushCount => Volatile.Read(ref _flushCount);
|
public int FlushCount => Volatile.Read(ref _flushCount);
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public override Task FlushAsync(CancellationToken cancellationToken)
|
public override void Flush()
|
||||||
{
|
{
|
||||||
Interlocked.Increment(ref _flushCount);
|
Interlocked.Increment(ref _flushCount);
|
||||||
return base.FlushAsync(cancellationToken);
|
base.Flush();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1500,7 +1500,7 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
/// <summary>Gets a task that completes once the second gated <see cref="WriteAsync"/> call has started blocking.</summary>
|
/// <summary>Gets a task that completes once the second gated <see cref="WriteAsync"/> call has started blocking.</summary>
|
||||||
public Task SecondGateWriteStarted => _secondGateWriteStarted.Task;
|
public Task SecondGateWriteStarted => _secondGateWriteStarted.Task;
|
||||||
|
|
||||||
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
|
/// <summary>Gets the number of <see cref="Flush"/> calls observed so far.</summary>
|
||||||
public int FlushCount => Volatile.Read(ref _flushCount);
|
public int FlushCount => Volatile.Read(ref _flushCount);
|
||||||
|
|
||||||
/// <summary>Releases the first blocked write so it can complete.</summary>
|
/// <summary>Releases the first blocked write so it can complete.</summary>
|
||||||
@@ -1528,10 +1528,10 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public override Task FlushAsync(CancellationToken cancellationToken)
|
public override void Flush()
|
||||||
{
|
{
|
||||||
Interlocked.Increment(ref _flushCount);
|
Interlocked.Increment(ref _flushCount);
|
||||||
return base.FlushAsync(cancellationToken);
|
base.Flush();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -1547,7 +1547,7 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A MemoryStream whose first write blocks until released and whose every FlushAsync throws, so a test
|
// A MemoryStream whose first write blocks until released and whose every Flush throws, so a test
|
||||||
// can fault the class-boundary flush with control frames already written and an event frame already
|
// can fault the class-boundary flush with control frames already written and an event frame already
|
||||||
// claimed off its queue.
|
// claimed off its queue.
|
||||||
private sealed class FlushFaultingGatedStream : MemoryStream
|
private sealed class FlushFaultingGatedStream : MemoryStream
|
||||||
@@ -1584,9 +1584,9 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public override Task FlushAsync(CancellationToken cancellationToken)
|
public override void Flush()
|
||||||
{
|
{
|
||||||
return Task.FromException(new IOException(_faultMessage));
|
throw new IOException(_faultMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
|
|||||||
@@ -367,6 +367,45 @@ public sealed class WorkerPipeSessionTests
|
|||||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The event drain loop runs on its own dedicated background thread, not the CLR thread
|
||||||
|
/// pool: on .NET Framework every pool continuation per event also paid the pool's idle
|
||||||
|
/// spin, which a production trace measured at ~150 context switches per event. The parked
|
||||||
|
/// wait is observed from the fake, which records the calling thread.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task RunAsync_EventDrainLoop_RunsOnDedicatedNonPoolThread()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(10));
|
||||||
|
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||||
|
FakeRuntimeSession runtime = new()
|
||||||
|
{
|
||||||
|
WaitForEventsOnSignalOnly = true,
|
||||||
|
};
|
||||||
|
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
while (runtime.LastWaitForEventsTimeout is null)
|
||||||
|
{
|
||||||
|
await Task.Delay(TimeSpan.FromMilliseconds(5), cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Equal("mxgw-event-drain", runtime.LastWaitForEventsThreadName);
|
||||||
|
Assert.False(runtime.LastWaitForEventsOnThreadPool);
|
||||||
|
|
||||||
|
// The dedicated thread still ships events and still stops cleanly on shutdown.
|
||||||
|
runtime.EnqueueEvent(CreateWorkerEvent(sequence: 3));
|
||||||
|
WorkerEnvelope workerEvent = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerEvent,
|
||||||
|
cancellation.Token);
|
||||||
|
Assert.Equal(3UL, workerEvent.WorkerEvent.Event.WorkerSequence);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Verifies that a Ping control command is answered on the worker side
|
/// Verifies that a Ping control command is answered on the worker side
|
||||||
/// (not dispatched to the STA) with an OK reply that echoes the ping
|
/// (not dispatched to the STA) with an OK reply that echoes the ping
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
using ZB.MOM.WW.MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.MxGateway.Worker.Tests.MxAccess;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The record-info pin is a best-effort, process-wide cache warm-up. Whether it succeeds depends
|
||||||
|
/// on the host's MXAccess registration, so these tests assert only what must hold on every host:
|
||||||
|
/// it never throws, it always reports an outcome, and only the first call does work.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class MxStatusProxyRecordInfoPinTests
|
||||||
|
{
|
||||||
|
/// <summary>Repeated calls return the same outcome and leave a status describing it.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void EnsurePinned_CalledTwice_IsIdempotentAndReportsStatus()
|
||||||
|
{
|
||||||
|
bool first = MxStatusProxyRecordInfoPin.EnsurePinned();
|
||||||
|
string? statusAfterFirst = MxStatusProxyRecordInfoPin.Status;
|
||||||
|
bool second = MxStatusProxyRecordInfoPin.EnsurePinned();
|
||||||
|
|
||||||
|
Assert.Equal(first, second);
|
||||||
|
Assert.False(string.IsNullOrWhiteSpace(statusAfterFirst));
|
||||||
|
Assert.Same(statusAfterFirst, MxStatusProxyRecordInfoPin.Status);
|
||||||
|
if (first)
|
||||||
|
{
|
||||||
|
Assert.StartsWith("pinned ", statusAfterFirst);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -387,6 +387,22 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
|||||||
return eventSignal.WaitAsync(timeout, cancellationToken);
|
return eventSignal.WaitAsync(timeout, cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Gets the name of the thread that made the most recent <see cref="WaitForEvents"/> call.</summary>
|
||||||
|
public string? LastWaitForEventsThreadName { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>Gets whether the most recent <see cref="WaitForEvents"/> call ran on a thread-pool thread.</summary>
|
||||||
|
public bool? LastWaitForEventsOnThreadPool { get; private set; }
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public void WaitForEvents(TimeSpan timeout, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
LastWaitForEventsThreadName = Thread.CurrentThread.Name;
|
||||||
|
LastWaitForEventsOnThreadPool = Thread.CurrentThread.IsThreadPoolThread;
|
||||||
|
// Same knobs as the async form (signal-only, recorded timeout, backing queue); the drain
|
||||||
|
// loop now runs on a dedicated thread, so blocking on the task here is the production shape.
|
||||||
|
WaitForEventsAsync(timeout, cancellationToken).GetAwaiter().GetResult();
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Gets or sets a fault the next <see cref="DrainFault"/> call hands back once, standing in
|
/// Gets or sets a fault the next <see cref="DrainFault"/> call hands back once, standing in
|
||||||
/// for the fault a real event queue self-records on overflow.
|
/// for the fault a real event queue self-records on overflow.
|
||||||
|
|||||||
@@ -588,7 +588,7 @@ public sealed class WorkerFrameWriter
|
|||||||
// and now flushed, so completing them early is the same contract, earlier.
|
// and now flushed, so completing them early is the same contract, earlier.
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _stream.FlushAsync(CancellationToken.None).ConfigureAwait(false);
|
FlushStream();
|
||||||
}
|
}
|
||||||
catch (Exception exception)
|
catch (Exception exception)
|
||||||
{
|
{
|
||||||
@@ -640,7 +640,7 @@ public sealed class WorkerFrameWriter
|
|||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _stream.FlushAsync(CancellationToken.None).ConfigureAwait(false);
|
FlushStream();
|
||||||
}
|
}
|
||||||
catch (Exception exception)
|
catch (Exception exception)
|
||||||
{
|
{
|
||||||
@@ -655,6 +655,15 @@ public sealed class WorkerFrameWriter
|
|||||||
CompleteFrames(written);
|
CompleteFrames(written);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Synchronous on purpose. PipeStream does not override FlushAsync, so the base Stream.FlushAsync
|
||||||
|
// runs Flush() through Task.Factory.StartNew: one thread-pool work item per drain pass, for a
|
||||||
|
// PipeStream.Flush that has nothing to do (it never calls FlushFileBuffers). Calling Flush()
|
||||||
|
// inline keeps the contract for any stream and drops the pool hop.
|
||||||
|
private void FlushStream()
|
||||||
|
{
|
||||||
|
_stream.Flush();
|
||||||
|
}
|
||||||
|
|
||||||
private static void CompleteFrames(List<PendingFrame> frames)
|
private static void CompleteFrames(List<PendingFrame> frames)
|
||||||
{
|
{
|
||||||
foreach (PendingFrame frame in frames)
|
foreach (PendingFrame frame in frames)
|
||||||
|
|||||||
@@ -355,6 +355,13 @@ public sealed class WorkerPipeSession
|
|||||||
await WriteWorkerHelloAsync(cancellationToken).ConfigureAwait(false);
|
await WriteWorkerHelloAsync(cancellationToken).ConfigureAwait(false);
|
||||||
_state = WorkerState.InitializingSta;
|
_state = WorkerState.InitializingSta;
|
||||||
WorkerReady ready = await initializeMxAccessAsync(cancellationToken).ConfigureAwait(false);
|
WorkerReady ready = await initializeMxAccessAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (MxStatusProxyRecordInfoPin.Status is { } pinStatus)
|
||||||
|
{
|
||||||
|
_logger?.Information(
|
||||||
|
"WorkerRecordInfoPin",
|
||||||
|
new Dictionary<string, object?> { ["status"] = pinStatus });
|
||||||
|
}
|
||||||
|
|
||||||
await WriteWorkerReadyAsync(ready, cancellationToken).ConfigureAwait(false);
|
await WriteWorkerReadyAsync(ready, cancellationToken).ConfigureAwait(false);
|
||||||
_state = WorkerState.Ready;
|
_state = WorkerState.Ready;
|
||||||
}
|
}
|
||||||
@@ -589,14 +596,59 @@ public sealed class WorkerPipeSession
|
|||||||
TaskScheduler.Default);
|
TaskScheduler.Default);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task RunEventDrainLoopAsync(CancellationToken cancellationToken)
|
/// <summary>
|
||||||
|
/// Starts the event drain loop on a dedicated background thread and returns a task that
|
||||||
|
/// completes, faults, or cancels exactly as the loop does, so the message loop's
|
||||||
|
/// <c>Task.WhenAny</c> supervision is unchanged.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The loop used to be an async method woken by <c>SemaphoreSlim.WaitAsync</c>. On .NET
|
||||||
|
/// Framework every release of that semaphore queues the continuation to the CLR thread pool,
|
||||||
|
/// and each pool worker that runs one then spins in <c>SleepEx</c> before parking: a WPR trace
|
||||||
|
/// on wonder-app-vd03 measured ~150 context switches per event and ~26% of the worker's CPU
|
||||||
|
/// there. A dedicated thread parks on the signal instead, and with a synchronous flush the
|
||||||
|
/// steady-state event path schedules no managed thread-pool work at all. The pipe write itself
|
||||||
|
/// stays overlapped; its completion runs on an I/O completion thread, not a worker thread.
|
||||||
|
/// </remarks>
|
||||||
|
private Task RunEventDrainLoopAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
TaskCompletionSource<bool> completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
Thread thread = new(() =>
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
RunEventDrainLoop(cancellationToken);
|
||||||
|
completion.TrySetResult(true);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
completion.TrySetCanceled(cancellationToken);
|
||||||
|
}
|
||||||
|
catch (Exception exception)
|
||||||
|
{
|
||||||
|
completion.TrySetException(exception);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
{
|
||||||
|
IsBackground = true,
|
||||||
|
Name = "mxgw-event-drain",
|
||||||
|
};
|
||||||
|
thread.Start();
|
||||||
|
return completion.Task;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void RunEventDrainLoop(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
while (!cancellationToken.IsCancellationRequested)
|
while (!cancellationToken.IsCancellationRequested)
|
||||||
{
|
{
|
||||||
IWorkerRuntimeSession? runtimeSession = _runtimeSession;
|
IWorkerRuntimeSession? runtimeSession = _runtimeSession;
|
||||||
if (runtimeSession is null)
|
if (runtimeSession is null)
|
||||||
{
|
{
|
||||||
await Task.Delay(EventDrainInterval, cancellationToken).ConfigureAwait(false);
|
if (cancellationToken.WaitHandle.WaitOne(EventDrainInterval))
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -604,7 +656,7 @@ public sealed class WorkerPipeSession
|
|||||||
if (fault is not null)
|
if (fault is not null)
|
||||||
{
|
{
|
||||||
_state = WorkerState.Faulted;
|
_state = WorkerState.Faulted;
|
||||||
await TryWriteFaultAsync(fault, cancellationToken).ConfigureAwait(false);
|
TryWriteFaultAsync(fault, cancellationToken).GetAwaiter().GetResult();
|
||||||
throw new WorkerEventQueueFaultException(
|
throw new WorkerEventQueueFaultException(
|
||||||
fault.Category,
|
fault.Category,
|
||||||
string.IsNullOrWhiteSpace(fault.DiagnosticMessage)
|
string.IsNullOrWhiteSpace(fault.DiagnosticMessage)
|
||||||
@@ -615,15 +667,13 @@ public sealed class WorkerPipeSession
|
|||||||
IReadOnlyList<WorkerEvent> events = runtimeSession.DrainEvents(EventDrainBatchSize);
|
IReadOnlyList<WorkerEvent> events = runtimeSession.DrainEvents(EventDrainBatchSize);
|
||||||
if (events.Count == 0)
|
if (events.Count == 0)
|
||||||
{
|
{
|
||||||
// Wait on the queue's wake signal rather than sleeping a fixed tick: an event
|
// Park on the queue's wake signal rather than sleeping a fixed tick: an event
|
||||||
// enqueued by the STA completes this immediately, so the first event of a burst is
|
// enqueued by the STA releases this immediately, so the first event of a burst is
|
||||||
// framed at signal latency instead of waiting out EventDrainInterval, and a session
|
// framed at signal latency, and a session with no traffic wakes only at the fallback
|
||||||
// with no traffic stops waking at all. The wait's outcome is intentionally ignored —
|
// ceiling. The wait's outcome is intentionally ignored — whether a signal or the
|
||||||
// whether a signal or the fallback ended it, the next pass re-checks DrainFault()
|
// fallback ended it, the next pass re-checks DrainFault() and re-drains, which is
|
||||||
// and re-drains, which is also why one coalesced wake for many enqueues is safe.
|
// also why one coalesced wake for many enqueues is safe.
|
||||||
await runtimeSession
|
runtimeSession.WaitForEvents(EventDrainInterval, cancellationToken);
|
||||||
.WaitForEventsAsync(EventDrainInterval, cancellationToken)
|
|
||||||
.ConfigureAwait(false);
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -640,9 +690,10 @@ public sealed class WorkerPipeSession
|
|||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _writer
|
_writer
|
||||||
.WriteBatchAsync(envelopes, WorkerFrameWritePriority.Event, cancellationToken)
|
.WriteBatchAsync(envelopes, WorkerFrameWritePriority.Event, cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.GetAwaiter()
|
||||||
|
.GetResult();
|
||||||
}
|
}
|
||||||
catch (WorkerFrameProtocolException exception)
|
catch (WorkerFrameProtocolException exception)
|
||||||
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||||
@@ -651,11 +702,12 @@ public sealed class WorkerPipeSession
|
|||||||
// still the structured, event-naming fault. Map the rejection back to the first
|
// still the structured, event-naming fault. Map the rejection back to the first
|
||||||
// event in batch order whose envelope overshoots the negotiated maximum — the same
|
// event in batch order whose envelope overshoots the negotiated maximum — the same
|
||||||
// frame the writer rejected first.
|
// frame the writer rejected first.
|
||||||
await FaultOnOversizedEventAsync(
|
FaultOnOversizedEventAsync(
|
||||||
FindOversizedEvent(events, envelopes),
|
FindOversizedEvent(events, envelopes),
|
||||||
exception,
|
exception,
|
||||||
cancellationToken)
|
cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.GetAwaiter()
|
||||||
|
.GetResult();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -79,6 +79,16 @@ public interface IWorkerRuntimeSession : IDisposable
|
|||||||
/// <returns>A task that completes when the queue is signalled or the timeout elapses.</returns>
|
/// <returns>A task that completes when the queue is signalled or the timeout elapses.</returns>
|
||||||
Task WaitForEventsAsync(TimeSpan timeout, CancellationToken cancellationToken);
|
Task WaitForEventsAsync(TimeSpan timeout, CancellationToken cancellationToken);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Blocking form of <see cref="WaitForEventsAsync"/> for a drain loop that runs on its own
|
||||||
|
/// dedicated thread. Same contract (coalesced signal, timeout is a ceiling, at most one
|
||||||
|
/// waiter); it parks the calling thread instead of scheduling a thread-pool continuation per
|
||||||
|
/// wake, which on .NET Framework also triggers the pool's spin-wait on every event.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="timeout">Maximum time to wait before the wait completes unsignalled.</param>
|
||||||
|
/// <param name="cancellationToken">Token cancelling the wait.</param>
|
||||||
|
void WaitForEvents(TimeSpan timeout, CancellationToken cancellationToken);
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Drains a pending fault from the queue, if any.
|
/// Drains a pending fault from the queue, if any.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -8,6 +8,9 @@ public sealed class MxAccessComObjectFactory : IMxAccessComObjectFactory
|
|||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public object Create()
|
public object Create()
|
||||||
{
|
{
|
||||||
|
// Warm oleaut32's record-info cache before the first OnDataChange can fire; see
|
||||||
|
// MxStatusProxyRecordInfoPin for why the per-event lookup is otherwise expensive.
|
||||||
|
MxStatusProxyRecordInfoPin.EnsurePinned();
|
||||||
return new LMXProxyServerClass();
|
return new LMXProxyServerClass();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -320,6 +320,17 @@ public sealed class MxAccessEventQueue
|
|||||||
return eventSignal.WaitAsync(timeout, cancellationToken);
|
return eventSignal.WaitAsync(timeout, cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Blocking form of <see cref="WaitForEventsAsync"/>: parks the calling (dedicated drain)
|
||||||
|
/// thread on the same coalesced signal. Same single-waiter precondition.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="timeout">Maximum time to wait before the wait completes unsignalled.</param>
|
||||||
|
/// <param name="cancellationToken">Token cancelling the wait.</param>
|
||||||
|
public void WaitForEvents(TimeSpan timeout, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
_ = eventSignal.Wait(timeout, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Attempts to dequeue the next event without removing it if empty.
|
/// Attempts to dequeue the next event without removing it if empty.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -682,6 +682,12 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
return eventQueue.WaitForEventsAsync(timeout, cancellationToken);
|
return eventQueue.WaitForEventsAsync(timeout, cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public void WaitForEvents(TimeSpan timeout, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
eventQueue.WaitForEvents(timeout, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public WorkerFault? DrainFault()
|
public WorkerFault? DrainFault()
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
using System;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Runtime.InteropServices;
|
||||||
|
using ArchestrA.MxAccess;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Keeps the LMXProxy type library and the <c>MXSTATUS_PROXY</c> record info alive for the
|
||||||
|
/// lifetime of the worker process, so the MXAccess proxy's per-event
|
||||||
|
/// <c>GetRecordInfoFromGuids</c> call is served from oleaut32's cache.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Every <c>OnDataChange</c> carries a SAFEARRAY of <c>MXSTATUS_PROXY</c> records, and
|
||||||
|
/// <c>lmxproxy.dll</c> resolves that record type with <c>GetRecordInfoFromGuids</c> each time it
|
||||||
|
/// builds the array. With no live reference, oleaut32 re-resolves the type library through the
|
||||||
|
/// registry and reloads it on every call: about 1 ms per event on an idle dev box. Under WOW64
|
||||||
|
/// and an EDR registry filter it is worse; a WPR trace on wonder-app-vd03 put it at about 70% of
|
||||||
|
/// the STA's CPU. While one <c>IRecordInfo</c> for the type is held, the same call returns the
|
||||||
|
/// cached instance in well under a microsecond. The pin is a process-wide cache warm-up only: it
|
||||||
|
/// changes no MXAccess behavior, and failure to pin is reported but never fatal.
|
||||||
|
/// </remarks>
|
||||||
|
public static class MxStatusProxyRecordInfoPin
|
||||||
|
{
|
||||||
|
private static readonly object Gate = new();
|
||||||
|
private static IntPtr pinnedTypeLib;
|
||||||
|
private static IntPtr pinnedRecordInfo;
|
||||||
|
private static bool attempted;
|
||||||
|
|
||||||
|
/// <summary>Gets the outcome of the pin attempt, or <see langword="null"/> before the first attempt.</summary>
|
||||||
|
public static string? Status { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Pins the type library and record info once per process. Safe to call repeatedly and from
|
||||||
|
/// any thread; only the first call does work.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns><see langword="true"/> when the record info is pinned.</returns>
|
||||||
|
public static bool EnsurePinned()
|
||||||
|
{
|
||||||
|
lock (Gate)
|
||||||
|
{
|
||||||
|
if (attempted)
|
||||||
|
{
|
||||||
|
return pinnedRecordInfo != IntPtr.Zero;
|
||||||
|
}
|
||||||
|
|
||||||
|
attempted = true;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Guid libraryGuid = typeof(LMXProxyServerClass).Assembly
|
||||||
|
.GetCustomAttributes(typeof(GuidAttribute), inherit: false)
|
||||||
|
.Cast<GuidAttribute>()
|
||||||
|
.Select(attribute => new Guid(attribute.Value))
|
||||||
|
.Single();
|
||||||
|
TypeLibVersionAttribute version = typeof(LMXProxyServerClass).Assembly
|
||||||
|
.GetCustomAttributes(typeof(TypeLibVersionAttribute), inherit: false)
|
||||||
|
.Cast<TypeLibVersionAttribute>()
|
||||||
|
.Single();
|
||||||
|
Guid recordGuid = typeof(MXSTATUS_PROXY).GUID;
|
||||||
|
|
||||||
|
int hr = LoadRegTypeLib(
|
||||||
|
ref libraryGuid,
|
||||||
|
(ushort)version.MajorVersion,
|
||||||
|
(ushort)version.MinorVersion,
|
||||||
|
0,
|
||||||
|
out pinnedTypeLib);
|
||||||
|
if (hr != 0)
|
||||||
|
{
|
||||||
|
Status = $"LoadRegTypeLib failed (HRESULT 0x{hr:X8})";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
hr = GetRecordInfoFromGuids(
|
||||||
|
ref libraryGuid,
|
||||||
|
(uint)version.MajorVersion,
|
||||||
|
(uint)version.MinorVersion,
|
||||||
|
0,
|
||||||
|
ref recordGuid,
|
||||||
|
out pinnedRecordInfo);
|
||||||
|
if (hr != 0)
|
||||||
|
{
|
||||||
|
Status = $"GetRecordInfoFromGuids failed (HRESULT 0x{hr:X8})";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
Status = $"pinned {libraryGuid:B} v{version.MajorVersion}.{version.MinorVersion} record {recordGuid:B}";
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
catch (Exception exception)
|
||||||
|
{
|
||||||
|
Status = $"pin failed: {exception.GetType().Name}: {exception.Message}";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[DllImport("oleaut32.dll")]
|
||||||
|
private static extern int LoadRegTypeLib(
|
||||||
|
ref Guid libraryGuid,
|
||||||
|
ushort majorVersion,
|
||||||
|
ushort minorVersion,
|
||||||
|
uint lcid,
|
||||||
|
out IntPtr typeLib);
|
||||||
|
|
||||||
|
[DllImport("oleaut32.dll")]
|
||||||
|
private static extern int GetRecordInfoFromGuids(
|
||||||
|
ref Guid libraryGuid,
|
||||||
|
uint majorVersion,
|
||||||
|
uint minorVersion,
|
||||||
|
uint lcid,
|
||||||
|
ref Guid typeGuid,
|
||||||
|
out IntPtr recordInfo);
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user