Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 4a3560c7ee | |||
| a3ccd5c80b | |||
| 0fd954d94c | |||
| 91f2d8dc14 | |||
| fb425da009 | |||
| c7e4c4b614 | |||
| 59c710d789 | |||
| 862f119b91 | |||
| 35e4442c7b | |||
| ed1018c3bb |
@@ -0,0 +1,61 @@
|
|||||||
|
# Gateway Testing
|
||||||
|
|
||||||
|
Gateway tests run without installed MXAccess by using fake workers, fake
|
||||||
|
transports, and in-process gRPC service fakes. Live MXAccess verification belongs
|
||||||
|
in opt-in integration tests because it depends on installed COM components and
|
||||||
|
provider state.
|
||||||
|
|
||||||
|
## Fake Worker Harness
|
||||||
|
|
||||||
|
`FakeWorkerHarness` in `src/MxGateway.Tests/Gateway/Workers/Fakes/` provides an
|
||||||
|
in-process worker side for named-pipe IPC tests. It uses the same
|
||||||
|
`WorkerFrameReader`, `WorkerFrameWriter`, and `WorkerEnvelope` contract as the
|
||||||
|
gateway so tests exercise real frame validation and worker-client state changes.
|
||||||
|
|
||||||
|
Use the harness when a gateway or session test needs worker behavior without
|
||||||
|
starting `MxGateway.Worker.exe` or loading MXAccess COM. The harness scripts:
|
||||||
|
|
||||||
|
- `WorkerHello` and `WorkerReady` startup,
|
||||||
|
- command replies with matching correlation ids,
|
||||||
|
- ordered `WorkerEvent` frames,
|
||||||
|
- `WorkerHeartbeat` frames,
|
||||||
|
- `WorkerFault` frames,
|
||||||
|
- shutdown acknowledgements,
|
||||||
|
- malformed protobuf payloads and oversized frame headers,
|
||||||
|
- slow or hung workers by withholding a reply.
|
||||||
|
|
||||||
|
Session-level tests can connect the harness to the pipe created by
|
||||||
|
`SessionWorkerClientFactory` with `ConnectToGatewayPipeAsync`. Lower-level
|
||||||
|
`WorkerClient` tests can use `CreateConnectedPairAsync` to create both pipe ends
|
||||||
|
inside the test.
|
||||||
|
|
||||||
|
`GatewayEndToEndFakeWorkerSmokeTests` composes the real gRPC service,
|
||||||
|
`SessionManager`, `SessionWorkerClientFactory`, `WorkerClient`, and
|
||||||
|
`EventStreamService` with a scripted fake worker launcher. The smoke test covers
|
||||||
|
`OpenSession`, `Register`, `AddItem`, `Advise`, one streamed `OnDataChange`
|
||||||
|
event, and `CloseSession` without loading MXAccess COM.
|
||||||
|
|
||||||
|
## Focused Commands
|
||||||
|
|
||||||
|
Run the fake worker tests after changing gateway worker IPC, session startup, or
|
||||||
|
event streaming behavior:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~FakeWorkerHarnessTests
|
||||||
|
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~SessionWorkerClientFactoryFakeWorkerTests
|
||||||
|
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndFakeWorkerSmokeTests
|
||||||
|
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests
|
||||||
|
dotnet test src/MxGateway.Worker.Tests/MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter FullyQualifiedName~WorkerPipeSessionTests
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the gateway test project after shared gateway test infrastructure changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Gateway Process Design](./gateway-process-design.md)
|
||||||
|
- [Worker Frame Protocol](./WorkerFrameProtocol.md)
|
||||||
|
- [MXAccess Worker Instance Detailed Design](./mxaccess-worker-instance-design.md)
|
||||||
@@ -891,6 +891,11 @@ behavior unless an explicit non-parity backend is designed.
|
|||||||
Gateway tests should be able to run without installed MXAccess by using fake
|
Gateway tests should be able to run without installed MXAccess by using fake
|
||||||
workers and fake transports.
|
workers and fake transports.
|
||||||
|
|
||||||
|
Use `FakeWorkerHarness` for tests that need real gateway-to-worker framing,
|
||||||
|
handshake, command, event, fault, or malformed-protocol behavior without loading
|
||||||
|
MXAccess COM. See [Gateway Testing](./GatewayTesting.md) for the harness scope
|
||||||
|
and focused test commands.
|
||||||
|
|
||||||
Focused tests:
|
Focused tests:
|
||||||
|
|
||||||
- session state transitions,
|
- session state transitions,
|
||||||
|
|||||||
@@ -218,6 +218,8 @@ Live tests:
|
|||||||
|
|
||||||
Labels: `area:worker`, `type:feature`, `priority:p0`
|
Labels: `area:worker`, `type:feature`, `priority:p0`
|
||||||
|
|
||||||
|
Status: implemented.
|
||||||
|
|
||||||
Deliverables:
|
Deliverables:
|
||||||
|
|
||||||
- `AddItem`,
|
- `AddItem`,
|
||||||
|
|||||||
@@ -432,6 +432,25 @@ HRESULT and converts the reply to `ProtocolStatusCode.MxaccessFailure`.
|
|||||||
`MxAccessStaSession.GetRegisteredServerHandlesAsync` returns an STA-read
|
`MxAccessStaSession.GetRegisteredServerHandlesAsync` returns an STA-read
|
||||||
snapshot of tracked server handles for diagnostics and future cleanup logic.
|
snapshot of tracked server handles for diagnostics and future cleanup logic.
|
||||||
|
|
||||||
|
`MxAccessCommandExecutor` also implements the item lifecycle commands:
|
||||||
|
|
||||||
|
- `AddItem` calls `LMXProxyServerClass.AddItem` with the requested server
|
||||||
|
handle and item definition. It preserves the returned item handle in both
|
||||||
|
`ReturnValue` and `AddItemReply.ItemHandle`.
|
||||||
|
- `AddItem2` calls `LMXProxyServerClass.AddItem2` with the requested server
|
||||||
|
handle, item definition, and context string. The context string is passed to
|
||||||
|
MXAccess exactly as received.
|
||||||
|
- `RemoveItem` calls `LMXProxyServerClass.RemoveItem` with the requested server
|
||||||
|
handle and item handle. The reply has no method-specific payload because the
|
||||||
|
public MXAccess method returns `void`.
|
||||||
|
|
||||||
|
The worker records item handles only after `AddItem` or `AddItem2` returns
|
||||||
|
normally, and removes item handles only after `RemoveItem` returns normally.
|
||||||
|
The registry does not prevalidate server or item handles, so invalid and
|
||||||
|
cross-server handle behavior remains owned by MXAccess. COM exceptions continue
|
||||||
|
through `StaCommandDispatcher`, which preserves the HRESULT and leaves
|
||||||
|
diagnostic registry state unchanged for failed cleanup calls.
|
||||||
|
|
||||||
## Handle Registry
|
## Handle Registry
|
||||||
|
|
||||||
The worker should track MXAccess state for diagnostics and cleanup, while still
|
The worker should track MXAccess state for diagnostics and cleanup, while still
|
||||||
@@ -454,6 +473,8 @@ Rules:
|
|||||||
- Do not rewrite handles returned by MXAccess.
|
- Do not rewrite handles returned by MXAccess.
|
||||||
- Record server handles only after `Register` succeeds.
|
- Record server handles only after `Register` succeeds.
|
||||||
- Remove server handles only after `Unregister` succeeds.
|
- Remove server handles only after `Unregister` succeeds.
|
||||||
|
- Record item handles only after `AddItem` or `AddItem2` succeeds.
|
||||||
|
- Remove item handles only after `RemoveItem` succeeds.
|
||||||
- Preserve invalid-handle behavior from MXAccess.
|
- Preserve invalid-handle behavior from MXAccess.
|
||||||
- Preserve cross-server handle behavior from MXAccess.
|
- Preserve cross-server handle behavior from MXAccess.
|
||||||
- Use registry state for cleanup and diagnostics, not semantic correction.
|
- Use registry state for cleanup and diagnostics, not semantic correction.
|
||||||
@@ -555,13 +576,19 @@ Do not drop or coalesce events in v1.
|
|||||||
|
|
||||||
## Heartbeat And Watchdog
|
## Heartbeat And Watchdog
|
||||||
|
|
||||||
The worker heartbeat should prove that:
|
`WorkerPipeSession` starts the heartbeat loop after the gateway validates
|
||||||
|
`WorkerHello` and receives `WorkerReady`. Heartbeats continue until
|
||||||
|
`WorkerShutdown`, cancellation, or a pipe/protocol failure stops the session.
|
||||||
|
The loop uses `WorkerPipeSessionOptions.HeartbeatInterval`; the default matches
|
||||||
|
the gateway worker heartbeat interval.
|
||||||
|
|
||||||
|
The worker heartbeat proves that:
|
||||||
|
|
||||||
- pipe writer is alive,
|
- pipe writer is alive,
|
||||||
- worker host is alive,
|
- worker host is alive,
|
||||||
- STA has recently pumped or completed work.
|
- STA has recently pumped or completed work.
|
||||||
|
|
||||||
Heartbeat payload should include:
|
Heartbeat payload includes:
|
||||||
|
|
||||||
- worker process id,
|
- worker process id,
|
||||||
- session id,
|
- session id,
|
||||||
@@ -572,13 +599,19 @@ Heartbeat payload should include:
|
|||||||
- event sequence,
|
- event sequence,
|
||||||
- current command correlation id if any.
|
- current command correlation id if any.
|
||||||
|
|
||||||
The STA watchdog should warn when:
|
`MxAccessStaSession.CaptureHeartbeat()` reads `StaRuntime.LastActivityUtc` and
|
||||||
|
`StaCommandDispatcher` queue state without touching the raw MXAccess COM object
|
||||||
|
outside the STA. Event queue depth and event sequence are reported as zero until
|
||||||
|
the event queue implementation owns those counters.
|
||||||
|
|
||||||
- one command exceeds its expected duration,
|
The STA watchdog currently emits a `WorkerFault` with
|
||||||
- the STA has not pumped messages within the heartbeat grace period,
|
`WorkerFaultCategory.StaHung` when `LastStaActivityUtc` is older than
|
||||||
- event queue depth remains high.
|
`WorkerPipeSessionOptions.HeartbeatGrace`. The fault includes the current
|
||||||
|
command correlation id when a command is active. Command duration and high event
|
||||||
The worker can report the problem, but the gateway owns the final kill decision.
|
queue depth remain observable through heartbeat fields until dedicated
|
||||||
|
thresholds own those warnings. The worker reports stale STA activity, but the
|
||||||
|
gateway owns the final kill decision through its existing heartbeat and worker
|
||||||
|
lifecycle policy.
|
||||||
|
|
||||||
## Shutdown
|
## Shutdown
|
||||||
|
|
||||||
@@ -697,6 +730,10 @@ Live MXAccess tests:
|
|||||||
|
|
||||||
Live tests should be opt-in and clearly marked because they depend on installed
|
Live tests should be opt-in and clearly marked because they depend on installed
|
||||||
MXAccess COM and provider state.
|
MXAccess COM and provider state.
|
||||||
|
The worker test suite uses `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1` for these
|
||||||
|
tests. `AddItem` uses `TestChildObject.TestInt` by default and accepts an
|
||||||
|
override through `MXGATEWAY_LIVE_MXACCESS_ITEM`; `AddItem2` uses the captured
|
||||||
|
parity fixture shape `AddItem2("TestInt", "TestChildObject")`.
|
||||||
|
|
||||||
## Initial Implementation Slice
|
## Initial Implementation Slice
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,439 @@
|
|||||||
|
using System.Collections.Concurrent;
|
||||||
|
using Google.Protobuf.WellKnownTypes;
|
||||||
|
using Grpc.Core;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using MxGateway.Contracts;
|
||||||
|
using MxGateway.Contracts.Proto;
|
||||||
|
using MxGateway.Server.Configuration;
|
||||||
|
using MxGateway.Server.Grpc;
|
||||||
|
using MxGateway.Server.Metrics;
|
||||||
|
using MxGateway.Server.Security.Authorization;
|
||||||
|
using MxGateway.Server.Sessions;
|
||||||
|
using MxGateway.Server.Workers;
|
||||||
|
using MxGateway.Tests.Gateway.Workers.Fakes;
|
||||||
|
|
||||||
|
namespace MxGateway.Tests.Gateway;
|
||||||
|
|
||||||
|
public sealed class GatewayEndToEndFakeWorkerSmokeTests
|
||||||
|
{
|
||||||
|
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
|
||||||
|
private const int ServerHandle = 1001;
|
||||||
|
private const int ItemHandle = 2002;
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task GatewayService_WithFakeWorker_CompletesSessionCommandEventAndClosePath()
|
||||||
|
{
|
||||||
|
ScriptedFakeWorkerProcessLauncher launcher = new();
|
||||||
|
await using GatewayServiceFixture fixture = new(launcher);
|
||||||
|
|
||||||
|
OpenSessionReply openReply = await fixture.Service.OpenSession(
|
||||||
|
new OpenSessionRequest
|
||||||
|
{
|
||||||
|
ClientSessionName = "fake-worker-e2e",
|
||||||
|
ClientCorrelationId = "open-correlation",
|
||||||
|
CommandTimeout = Duration.FromTimeSpan(TestTimeout),
|
||||||
|
},
|
||||||
|
new TestServerCallContext());
|
||||||
|
|
||||||
|
RecordingServerStreamWriter<MxEvent> eventWriter = new();
|
||||||
|
Task streamTask = fixture.Service.StreamEvents(
|
||||||
|
new StreamEventsRequest { SessionId = openReply.SessionId },
|
||||||
|
eventWriter,
|
||||||
|
new TestServerCallContext());
|
||||||
|
|
||||||
|
MxCommandReply registerReply = await fixture.Service.Invoke(
|
||||||
|
CreateRegisterRequest(openReply.SessionId),
|
||||||
|
new TestServerCallContext());
|
||||||
|
MxCommandReply addItemReply = await fixture.Service.Invoke(
|
||||||
|
CreateAddItemRequest(openReply.SessionId, registerReply.Register.ServerHandle),
|
||||||
|
new TestServerCallContext());
|
||||||
|
MxCommandReply adviseReply = await fixture.Service.Invoke(
|
||||||
|
CreateAdviseRequest(openReply.SessionId, registerReply.Register.ServerHandle, addItemReply.AddItem.ItemHandle),
|
||||||
|
new TestServerCallContext());
|
||||||
|
|
||||||
|
MxEvent dataChange = await eventWriter.WaitForFirstMessageAsync(TestTimeout);
|
||||||
|
|
||||||
|
CloseSessionReply closeReply = await fixture.Service.CloseSession(
|
||||||
|
new CloseSessionRequest
|
||||||
|
{
|
||||||
|
SessionId = openReply.SessionId,
|
||||||
|
ClientCorrelationId = "close-correlation",
|
||||||
|
},
|
||||||
|
new TestServerCallContext());
|
||||||
|
|
||||||
|
await streamTask.WaitAsync(TestTimeout);
|
||||||
|
await launcher.WorkerTask.WaitAsync(TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(GatewayContractInfo.DefaultBackendName, openReply.BackendName);
|
||||||
|
Assert.Equal(ScriptedFakeWorkerProcessLauncher.ProcessId, openReply.WorkerProcessId);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, registerReply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(ServerHandle, registerReply.Register.ServerHandle);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, addItemReply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(ItemHandle, addItemReply.AddItem.ItemHandle);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, adviseReply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(MxEventFamily.OnDataChange, dataChange.Family);
|
||||||
|
Assert.Equal(openReply.SessionId, dataChange.SessionId);
|
||||||
|
Assert.Equal(ServerHandle, dataChange.ServerHandle);
|
||||||
|
Assert.Equal(ItemHandle, dataChange.ItemHandle);
|
||||||
|
Assert.Equal("scripted-value", dataChange.Value.StringValue);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, closeReply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(SessionState.Closed, closeReply.FinalState);
|
||||||
|
Assert.True(launcher.Process.HasExited);
|
||||||
|
Assert.Equal(
|
||||||
|
[MxCommandKind.Register, MxCommandKind.AddItem, MxCommandKind.Advise],
|
||||||
|
launcher.CommandKinds);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static MxCommandRequest CreateRegisterRequest(string sessionId)
|
||||||
|
{
|
||||||
|
return new MxCommandRequest
|
||||||
|
{
|
||||||
|
SessionId = sessionId,
|
||||||
|
ClientCorrelationId = "register-correlation",
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.Register,
|
||||||
|
Register = new RegisterCommand { ClientName = "fake-worker-e2e-client" },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static MxCommandRequest CreateAddItemRequest(
|
||||||
|
string sessionId,
|
||||||
|
int serverHandle)
|
||||||
|
{
|
||||||
|
return new MxCommandRequest
|
||||||
|
{
|
||||||
|
SessionId = sessionId,
|
||||||
|
ClientCorrelationId = "add-item-correlation",
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem,
|
||||||
|
AddItem = new AddItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemDefinition = "Galaxy.Tag.Value",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static MxCommandRequest CreateAdviseRequest(
|
||||||
|
string sessionId,
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
return new MxCommandRequest
|
||||||
|
{
|
||||||
|
SessionId = sessionId,
|
||||||
|
ClientCorrelationId = "advise-correlation",
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.Advise,
|
||||||
|
Advise = new AdviseCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class GatewayServiceFixture : IAsyncDisposable
|
||||||
|
{
|
||||||
|
private readonly GatewayMetrics _metrics = new();
|
||||||
|
private readonly SessionRegistry _registry = new();
|
||||||
|
|
||||||
|
public GatewayServiceFixture(IWorkerProcessLauncher launcher)
|
||||||
|
{
|
||||||
|
IOptions<GatewayOptions> options = Options.Create(CreateOptions());
|
||||||
|
SessionWorkerClientFactory workerClientFactory = new(
|
||||||
|
launcher,
|
||||||
|
options,
|
||||||
|
_metrics,
|
||||||
|
NullLoggerFactory.Instance);
|
||||||
|
SessionManager sessionManager = new(
|
||||||
|
_registry,
|
||||||
|
workerClientFactory,
|
||||||
|
options,
|
||||||
|
_metrics,
|
||||||
|
logger: NullLogger<SessionManager>.Instance);
|
||||||
|
MxAccessGrpcMapper mapper = new();
|
||||||
|
EventStreamService eventStreamService = new(
|
||||||
|
sessionManager,
|
||||||
|
options,
|
||||||
|
mapper,
|
||||||
|
_metrics,
|
||||||
|
NullLogger<EventStreamService>.Instance);
|
||||||
|
|
||||||
|
Service = new MxAccessGatewayService(
|
||||||
|
sessionManager,
|
||||||
|
new GatewayRequestIdentityAccessor(),
|
||||||
|
new MxAccessGrpcRequestValidator(),
|
||||||
|
mapper,
|
||||||
|
eventStreamService,
|
||||||
|
NullLogger<MxAccessGatewayService>.Instance);
|
||||||
|
}
|
||||||
|
|
||||||
|
public MxAccessGatewayService Service { get; }
|
||||||
|
|
||||||
|
public async ValueTask DisposeAsync()
|
||||||
|
{
|
||||||
|
foreach (GatewaySession session in _registry.Snapshot())
|
||||||
|
{
|
||||||
|
await session.DisposeAsync();
|
||||||
|
}
|
||||||
|
|
||||||
|
_metrics.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static GatewayOptions CreateOptions()
|
||||||
|
{
|
||||||
|
return new GatewayOptions
|
||||||
|
{
|
||||||
|
Worker = new WorkerOptions
|
||||||
|
{
|
||||||
|
StartupTimeoutSeconds = 5,
|
||||||
|
ShutdownTimeoutSeconds = 5,
|
||||||
|
HeartbeatIntervalSeconds = 30,
|
||||||
|
HeartbeatGraceSeconds = 30,
|
||||||
|
MaxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
|
||||||
|
},
|
||||||
|
Sessions = new SessionOptions
|
||||||
|
{
|
||||||
|
DefaultCommandTimeoutSeconds = 5,
|
||||||
|
MaxSessions = 4,
|
||||||
|
},
|
||||||
|
Events = new EventOptions
|
||||||
|
{
|
||||||
|
QueueCapacity = 16,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class ScriptedFakeWorkerProcessLauncher : IWorkerProcessLauncher
|
||||||
|
{
|
||||||
|
public const int ProcessId = 4680;
|
||||||
|
private readonly ConcurrentQueue<MxCommandKind> _commandKinds = new();
|
||||||
|
|
||||||
|
public FakeWorkerProcess Process { get; } = new(ProcessId);
|
||||||
|
|
||||||
|
public IReadOnlyCollection<MxCommandKind> CommandKinds => _commandKinds.ToArray();
|
||||||
|
|
||||||
|
public Task WorkerTask { get; private set; } = Task.CompletedTask;
|
||||||
|
|
||||||
|
public Task<WorkerProcessHandle> LaunchAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
WorkerTask = RunWorkerAsync(request, cancellationToken);
|
||||||
|
|
||||||
|
return Task.FromResult(new WorkerProcessHandle(
|
||||||
|
Process,
|
||||||
|
new WorkerProcessCommandLine("fake-worker.exe", []),
|
||||||
|
DateTimeOffset.UtcNow));
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task RunWorkerAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness harness = await FakeWorkerHarness.ConnectToGatewayPipeAsync(
|
||||||
|
request.SessionId,
|
||||||
|
request.Nonce,
|
||||||
|
request.PipeName,
|
||||||
|
request.ProtocolVersion,
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
await harness.CompleteStartupAsync(ProcessId, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
while (!cancellationToken.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = await harness.ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerCommand)
|
||||||
|
{
|
||||||
|
await ReplyToCommandAsync(harness, envelope, cancellationToken).ConfigureAwait(false);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerShutdown)
|
||||||
|
{
|
||||||
|
await harness.SendShutdownAckAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
Process.MarkExited(0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new InvalidOperationException($"Unexpected gateway envelope {envelope.BodyCase}.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task ReplyToCommandAsync(
|
||||||
|
FakeWorkerHarness harness,
|
||||||
|
WorkerEnvelope commandEnvelope,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
MxCommand command = commandEnvelope.WorkerCommand.Command;
|
||||||
|
_commandKinds.Enqueue(command.Kind);
|
||||||
|
|
||||||
|
await harness.ReplyToCommandAsync(
|
||||||
|
commandEnvelope,
|
||||||
|
configureReply: reply => ConfigureReply(reply, command.Kind),
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
if (command.Kind == MxCommandKind.Advise)
|
||||||
|
{
|
||||||
|
await harness.EmitEventAsync(
|
||||||
|
MxEventFamily.OnDataChange,
|
||||||
|
cancellationToken,
|
||||||
|
mxEvent =>
|
||||||
|
{
|
||||||
|
mxEvent.ServerHandle = command.Advise.ServerHandle;
|
||||||
|
mxEvent.ItemHandle = command.Advise.ItemHandle;
|
||||||
|
mxEvent.Quality = 192;
|
||||||
|
mxEvent.Value = new MxValue
|
||||||
|
{
|
||||||
|
DataType = MxDataType.String,
|
||||||
|
StringValue = "scripted-value",
|
||||||
|
};
|
||||||
|
mxEvent.OnDataChange = new OnDataChangeEvent();
|
||||||
|
}).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void ConfigureReply(
|
||||||
|
MxCommandReply reply,
|
||||||
|
MxCommandKind kind)
|
||||||
|
{
|
||||||
|
switch (kind)
|
||||||
|
{
|
||||||
|
case MxCommandKind.Register:
|
||||||
|
reply.Register = new RegisterReply { ServerHandle = ServerHandle };
|
||||||
|
break;
|
||||||
|
case MxCommandKind.AddItem:
|
||||||
|
reply.AddItem = new AddItemReply { ItemHandle = ItemHandle };
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeWorkerProcess(int processId) : IWorkerProcess
|
||||||
|
{
|
||||||
|
public int Id { get; } = processId;
|
||||||
|
|
||||||
|
public bool HasExited { get; private set; }
|
||||||
|
|
||||||
|
public int? ExitCode { get; private set; }
|
||||||
|
|
||||||
|
public ValueTask WaitForExitAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
HasExited = true;
|
||||||
|
ExitCode ??= 0;
|
||||||
|
return ValueTask.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Kill(bool entireProcessTree)
|
||||||
|
{
|
||||||
|
MarkExited(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
public void MarkExited(int exitCode)
|
||||||
|
{
|
||||||
|
HasExited = true;
|
||||||
|
ExitCode = exitCode;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class RecordingServerStreamWriter<T> : IServerStreamWriter<T>
|
||||||
|
{
|
||||||
|
private readonly object _syncRoot = new();
|
||||||
|
private readonly TaskCompletionSource<T> _firstMessage = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
private readonly List<T> _messages = [];
|
||||||
|
|
||||||
|
public IReadOnlyList<T> Messages
|
||||||
|
{
|
||||||
|
get
|
||||||
|
{
|
||||||
|
lock (_syncRoot)
|
||||||
|
{
|
||||||
|
return _messages.ToArray();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public WriteOptions? WriteOptions { get; set; }
|
||||||
|
|
||||||
|
public Task WriteAsync(T message)
|
||||||
|
{
|
||||||
|
lock (_syncRoot)
|
||||||
|
{
|
||||||
|
_messages.Add(message);
|
||||||
|
}
|
||||||
|
|
||||||
|
_firstMessage.TrySetResult(message);
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<T> WaitForFirstMessageAsync(TimeSpan timeout)
|
||||||
|
{
|
||||||
|
return await _firstMessage.Task.WaitAsync(timeout).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class TestServerCallContext(CancellationToken cancellationToken = default) : ServerCallContext
|
||||||
|
{
|
||||||
|
private readonly Metadata _requestHeaders = [];
|
||||||
|
private readonly Metadata _responseTrailers = [];
|
||||||
|
private readonly Dictionary<object, object> _userState = [];
|
||||||
|
private Status _status;
|
||||||
|
private WriteOptions? _writeOptions;
|
||||||
|
|
||||||
|
protected override string MethodCore => "/mxaccess_gateway.v1.MxAccessGateway/Test";
|
||||||
|
|
||||||
|
protected override string HostCore => "localhost";
|
||||||
|
|
||||||
|
protected override string PeerCore => "ipv4:127.0.0.1:5000";
|
||||||
|
|
||||||
|
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
|
||||||
|
|
||||||
|
protected override Metadata RequestHeadersCore => _requestHeaders;
|
||||||
|
|
||||||
|
protected override CancellationToken CancellationTokenCore => cancellationToken;
|
||||||
|
|
||||||
|
protected override Metadata ResponseTrailersCore => _responseTrailers;
|
||||||
|
|
||||||
|
protected override Status StatusCore
|
||||||
|
{
|
||||||
|
get => _status;
|
||||||
|
set => _status = value;
|
||||||
|
}
|
||||||
|
|
||||||
|
protected override WriteOptions? WriteOptionsCore
|
||||||
|
{
|
||||||
|
get => _writeOptions;
|
||||||
|
set => _writeOptions = value;
|
||||||
|
}
|
||||||
|
|
||||||
|
protected override AuthContext AuthContextCore { get; } = new(
|
||||||
|
string.Empty,
|
||||||
|
new Dictionary<string, List<AuthProperty>>(StringComparer.Ordinal));
|
||||||
|
|
||||||
|
protected override IDictionary<object, object> UserStateCore => _userState;
|
||||||
|
|
||||||
|
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||||
|
{
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
protected override ContextPropagationToken CreatePropagationTokenCore(
|
||||||
|
ContextPropagationOptions? options)
|
||||||
|
{
|
||||||
|
throw new NotSupportedException();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,216 @@
|
|||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using MxGateway.Contracts;
|
||||||
|
using MxGateway.Contracts.Proto;
|
||||||
|
using MxGateway.Server.Configuration;
|
||||||
|
using MxGateway.Server.Metrics;
|
||||||
|
using MxGateway.Server.Sessions;
|
||||||
|
using MxGateway.Server.Workers;
|
||||||
|
using MxGateway.Tests.Gateway.Workers.Fakes;
|
||||||
|
|
||||||
|
namespace MxGateway.Tests.Gateway.Sessions;
|
||||||
|
|
||||||
|
public sealed class SessionWorkerClientFactoryFakeWorkerTests
|
||||||
|
{
|
||||||
|
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task CreateAsync_WithScriptedFakeWorker_ReturnsReadyClient()
|
||||||
|
{
|
||||||
|
ScriptedFakeWorkerProcessLauncher launcher = new();
|
||||||
|
using GatewayMetrics metrics = new();
|
||||||
|
SessionWorkerClientFactory factory = new(
|
||||||
|
launcher,
|
||||||
|
Options.Create(CreateOptions()),
|
||||||
|
metrics,
|
||||||
|
NullLoggerFactory.Instance);
|
||||||
|
GatewaySession session = CreateSession();
|
||||||
|
|
||||||
|
await using IWorkerClient workerClient = await factory.CreateAsync(
|
||||||
|
session,
|
||||||
|
CancellationToken.None);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientState.Ready, workerClient.State);
|
||||||
|
Assert.Equal(ScriptedFakeWorkerProcessLauncher.ProcessId, workerClient.ProcessId);
|
||||||
|
Assert.NotNull(launcher.Harness);
|
||||||
|
|
||||||
|
Task<WorkerCommandReply> invokeTask = workerClient.InvokeAsync(
|
||||||
|
CreateCommand(MxCommandKind.Ping),
|
||||||
|
TestTimeout,
|
||||||
|
CancellationToken.None);
|
||||||
|
WorkerEnvelope commandEnvelope = await launcher.Harness.ReadCommandAsync();
|
||||||
|
await launcher.Harness.ReplyToCommandAsync(commandEnvelope);
|
||||||
|
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.Reply.ProtocolStatus.Code);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task CreateAsync_WhenFakeWorkerStartupFails_ThrowsWorkerClientException()
|
||||||
|
{
|
||||||
|
FailingStartupWorkerProcessLauncher launcher = new();
|
||||||
|
using GatewayMetrics metrics = new();
|
||||||
|
SessionWorkerClientFactory factory = new(
|
||||||
|
launcher,
|
||||||
|
Options.Create(CreateOptions()),
|
||||||
|
metrics,
|
||||||
|
NullLoggerFactory.Instance);
|
||||||
|
GatewaySession session = CreateSession();
|
||||||
|
|
||||||
|
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
||||||
|
async () => await factory.CreateAsync(session, CancellationToken.None).WaitAsync(TestTimeout));
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientErrorCode.ProtocolViolation, exception.ErrorCode);
|
||||||
|
Assert.True(launcher.Process.IsDisposed);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static GatewayOptions CreateOptions()
|
||||||
|
{
|
||||||
|
return new GatewayOptions
|
||||||
|
{
|
||||||
|
Worker = new WorkerOptions
|
||||||
|
{
|
||||||
|
StartupTimeoutSeconds = 5,
|
||||||
|
ShutdownTimeoutSeconds = 5,
|
||||||
|
HeartbeatIntervalSeconds = 30,
|
||||||
|
HeartbeatGraceSeconds = 30,
|
||||||
|
MaxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
|
||||||
|
},
|
||||||
|
Events = new EventOptions
|
||||||
|
{
|
||||||
|
QueueCapacity = 16,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static GatewaySession CreateSession()
|
||||||
|
{
|
||||||
|
return new GatewaySession(
|
||||||
|
FakeWorkerHarness.DefaultSessionId,
|
||||||
|
GatewayContractInfo.DefaultBackendName,
|
||||||
|
$"mxaccessgw-session-fake-worker-{Guid.NewGuid():N}",
|
||||||
|
FakeWorkerHarness.DefaultNonce,
|
||||||
|
"test-client",
|
||||||
|
"fake-worker-session-test",
|
||||||
|
"client-correlation-1",
|
||||||
|
TestTimeout,
|
||||||
|
TestTimeout,
|
||||||
|
TestTimeout,
|
||||||
|
DateTimeOffset.UtcNow);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerCommand CreateCommand(MxCommandKind kind)
|
||||||
|
{
|
||||||
|
return new WorkerCommand
|
||||||
|
{
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = kind,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class ScriptedFakeWorkerProcessLauncher : IWorkerProcessLauncher
|
||||||
|
{
|
||||||
|
public const int ProcessId = 2468;
|
||||||
|
private readonly FakeWorkerProcess _process = new(ProcessId);
|
||||||
|
|
||||||
|
public FakeWorkerHarness? Harness { get; private set; }
|
||||||
|
|
||||||
|
public Task<WorkerProcessHandle> LaunchAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
_ = RunWorkerAsync(request, cancellationToken);
|
||||||
|
|
||||||
|
return Task.FromResult(CreateHandle(_process));
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task RunWorkerAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
Harness = await FakeWorkerHarness.ConnectToGatewayPipeAsync(
|
||||||
|
request.SessionId,
|
||||||
|
request.Nonce,
|
||||||
|
request.PipeName,
|
||||||
|
request.ProtocolVersion,
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
await Harness.CompleteStartupAsync(ProcessId, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FailingStartupWorkerProcessLauncher : IWorkerProcessLauncher
|
||||||
|
{
|
||||||
|
public FakeWorkerProcess Process { get; } = new(processId: 3579);
|
||||||
|
|
||||||
|
public Task<WorkerProcessHandle> LaunchAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
_ = RunWorkerAsync(request, cancellationToken);
|
||||||
|
|
||||||
|
return Task.FromResult(CreateHandle(Process));
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task RunWorkerAsync(
|
||||||
|
WorkerProcessLaunchRequest request,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness harness = await FakeWorkerHarness.ConnectToGatewayPipeAsync(
|
||||||
|
request.SessionId,
|
||||||
|
request.Nonce,
|
||||||
|
request.PipeName,
|
||||||
|
request.ProtocolVersion,
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
_ = await harness.ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
await harness.SendWorkerHelloAsync(
|
||||||
|
workerProcessId: Process.Id,
|
||||||
|
workerProtocolVersion: request.ProtocolVersion + 1,
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerProcessHandle CreateHandle(IWorkerProcess process)
|
||||||
|
{
|
||||||
|
return new WorkerProcessHandle(
|
||||||
|
process,
|
||||||
|
new WorkerProcessCommandLine("fake-worker.exe", []),
|
||||||
|
DateTimeOffset.UtcNow);
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeWorkerProcess(int processId) : IWorkerProcess
|
||||||
|
{
|
||||||
|
private bool _disposed;
|
||||||
|
|
||||||
|
public int Id { get; } = processId;
|
||||||
|
|
||||||
|
public bool HasExited { get; private set; }
|
||||||
|
|
||||||
|
public int? ExitCode { get; private set; }
|
||||||
|
|
||||||
|
public int KillCount { get; private set; }
|
||||||
|
|
||||||
|
public ValueTask WaitForExitAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
HasExited = true;
|
||||||
|
ExitCode = 0;
|
||||||
|
return ValueTask.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Kill(bool entireProcessTree)
|
||||||
|
{
|
||||||
|
KillCount++;
|
||||||
|
HasExited = true;
|
||||||
|
ExitCode = -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
_disposed = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
public bool IsDisposed => _disposed;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
using MxGateway.Contracts;
|
||||||
|
using MxGateway.Contracts.Proto;
|
||||||
|
using MxGateway.Server.Workers;
|
||||||
|
using MxGateway.Tests.Gateway.Workers.Fakes;
|
||||||
|
|
||||||
|
namespace MxGateway.Tests.Gateway.Workers;
|
||||||
|
|
||||||
|
public sealed class FakeWorkerHarnessTests
|
||||||
|
{
|
||||||
|
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task CompleteStartupAsync_WithHelloAndReady_TransitionsClientToReady()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
|
||||||
|
Task startTask = client.StartAsync(CancellationToken.None);
|
||||||
|
WorkerEnvelope gatewayHello = await fakeWorker.CompleteStartupAsync();
|
||||||
|
await startTask.WaitAsync(TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, gatewayHello.BodyCase);
|
||||||
|
Assert.Equal(FakeWorkerHarness.DefaultNonce, gatewayHello.GatewayHello.Nonce);
|
||||||
|
Assert.Equal(WorkerClientState.Ready, client.State);
|
||||||
|
Assert.Equal(FakeWorkerHarness.DefaultWorkerProcessId, client.ProcessId);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task StartAsync_WithProtocolMismatch_FailsStartup()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
|
||||||
|
Task startTask = client.StartAsync(CancellationToken.None);
|
||||||
|
WorkerEnvelope gatewayHello = await fakeWorker.ReadGatewayEnvelopeAsync();
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, gatewayHello.BodyCase);
|
||||||
|
await fakeWorker.SendWorkerHelloAsync(
|
||||||
|
workerProtocolVersion: GatewayContractInfo.WorkerProtocolVersion + 1);
|
||||||
|
|
||||||
|
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
||||||
|
async () => await startTask.WaitAsync(TestTimeout));
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientErrorCode.ProtocolViolation, exception.ErrorCode);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task InvokeAsync_WithScriptedReply_CompletesCommand()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
|
||||||
|
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
||||||
|
CreateCommand(MxCommandKind.Ping),
|
||||||
|
TestTimeout,
|
||||||
|
CancellationToken.None);
|
||||||
|
WorkerEnvelope commandEnvelope = await fakeWorker.ReadCommandAsync();
|
||||||
|
await fakeWorker.ReplyToCommandAsync(commandEnvelope);
|
||||||
|
|
||||||
|
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(commandEnvelope.CorrelationId, reply.Reply.CorrelationId);
|
||||||
|
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.Reply.ProtocolStatus.Code);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ReadEventsAsync_WithScriptedEvents_YieldsOrderedEvents()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
using CancellationTokenSource cancellationTokenSource = new(TestTimeout);
|
||||||
|
|
||||||
|
await using IAsyncEnumerator<WorkerEvent> events =
|
||||||
|
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
|
||||||
|
|
||||||
|
await fakeWorker.EmitEventAsync(MxEventFamily.OnDataChange, cancellationTokenSource.Token);
|
||||||
|
await fakeWorker.EmitEventAsync(MxEventFamily.OperationComplete, cancellationTokenSource.Token);
|
||||||
|
|
||||||
|
Assert.True(await events.MoveNextAsync());
|
||||||
|
Assert.Equal((ulong)3, events.Current.Event.WorkerSequence);
|
||||||
|
Assert.Equal(MxEventFamily.OnDataChange, events.Current.Event.Family);
|
||||||
|
|
||||||
|
Assert.True(await events.MoveNextAsync());
|
||||||
|
Assert.Equal((ulong)4, events.Current.Event.WorkerSequence);
|
||||||
|
Assert.Equal(MxEventFamily.OperationComplete, events.Current.Event.Family);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ReadLoop_WithScriptedFault_FaultsClient()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
|
||||||
|
await fakeWorker.EmitFaultAsync(
|
||||||
|
WorkerFaultCategory.MxaccessCommandFailed,
|
||||||
|
"scripted MXAccess command fault");
|
||||||
|
|
||||||
|
await WaitUntilAsync(
|
||||||
|
() => client.State == WorkerClientState.Faulted,
|
||||||
|
TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientState.Faulted, client.State);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task SendHeartbeatAsync_UpdatesClientHeartbeatState()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
|
||||||
|
|
||||||
|
await Task.Delay(TimeSpan.FromMilliseconds(20));
|
||||||
|
await fakeWorker.SendHeartbeatAsync(
|
||||||
|
configureHeartbeat: heartbeat => heartbeat.WorkerProcessId = 2468);
|
||||||
|
|
||||||
|
await WaitUntilAsync(
|
||||||
|
() => client.ProcessId == 2468 && client.LastHeartbeatAt > previousHeartbeat,
|
||||||
|
TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientState.Ready, client.State);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task InvokeAsync_WithHungWorker_TimesOutPendingCommand()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
|
||||||
|
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
||||||
|
CreateCommand(MxCommandKind.Ping),
|
||||||
|
TimeSpan.FromMilliseconds(50),
|
||||||
|
CancellationToken.None);
|
||||||
|
WorkerEnvelope commandEnvelope = await fakeWorker.ReadCommandAsync();
|
||||||
|
|
||||||
|
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
||||||
|
async () => await invokeTask.WaitAsync(TestTimeout));
|
||||||
|
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
||||||
|
Assert.Equal(WorkerClientErrorCode.CommandTimeout, exception.ErrorCode);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ReadLoop_WithMalformedFrame_FaultsClient()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
|
||||||
|
await fakeWorker.WriteMalformedPayloadAsync(new byte[] { 0x08, 0x96, 0x01 });
|
||||||
|
|
||||||
|
await WaitUntilAsync(
|
||||||
|
() => client.State == WorkerClientState.Faulted,
|
||||||
|
TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientState.Faulted, client.State);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ShutdownAsync_WithShutdownAck_ClosesClient()
|
||||||
|
{
|
||||||
|
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||||
|
await using WorkerClient client = fakeWorker.CreateClient();
|
||||||
|
await StartClientAsync(fakeWorker, client);
|
||||||
|
|
||||||
|
Task shutdownTask = client.ShutdownAsync(TestTimeout, CancellationToken.None);
|
||||||
|
WorkerEnvelope shutdownEnvelope = await fakeWorker.ReadShutdownAsync();
|
||||||
|
await fakeWorker.SendShutdownAckAsync();
|
||||||
|
await shutdownTask.WaitAsync(TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerShutdown, shutdownEnvelope.BodyCase);
|
||||||
|
Assert.Equal(WorkerClientState.Closed, client.State);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task StartClientAsync(
|
||||||
|
FakeWorkerHarness fakeWorker,
|
||||||
|
WorkerClient client)
|
||||||
|
{
|
||||||
|
Task startTask = client.StartAsync(CancellationToken.None);
|
||||||
|
await fakeWorker.CompleteStartupAsync().ConfigureAwait(false);
|
||||||
|
await startTask.WaitAsync(TestTimeout).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerCommand CreateCommand(MxCommandKind kind)
|
||||||
|
{
|
||||||
|
return new WorkerCommand
|
||||||
|
{
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = kind,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task WaitUntilAsync(
|
||||||
|
Func<bool> predicate,
|
||||||
|
TimeSpan timeout)
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellationTokenSource = new(timeout);
|
||||||
|
while (!predicate())
|
||||||
|
{
|
||||||
|
await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationTokenSource.Token);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,406 @@
|
|||||||
|
using System.Buffers.Binary;
|
||||||
|
using System.IO.Pipes;
|
||||||
|
using Google.Protobuf.WellKnownTypes;
|
||||||
|
using MxGateway.Contracts;
|
||||||
|
using MxGateway.Contracts.Proto;
|
||||||
|
using MxGateway.Server.Metrics;
|
||||||
|
using MxGateway.Server.Workers;
|
||||||
|
|
||||||
|
namespace MxGateway.Tests.Gateway.Workers.Fakes;
|
||||||
|
|
||||||
|
public sealed class FakeWorkerHarness : IAsyncDisposable
|
||||||
|
{
|
||||||
|
public const string DefaultSessionId = "session-fake-worker";
|
||||||
|
public const string DefaultNonce = "nonce-fake-worker";
|
||||||
|
public const int DefaultWorkerProcessId = 9321;
|
||||||
|
|
||||||
|
private readonly NamedPipeServerStream? _gatewayStream;
|
||||||
|
private readonly NamedPipeClientStream _workerStream;
|
||||||
|
private readonly WorkerFrameProtocolOptions _frameOptions;
|
||||||
|
private readonly WorkerFrameReader _reader;
|
||||||
|
private readonly WorkerFrameWriter _writer;
|
||||||
|
private bool _workerSideDisposed;
|
||||||
|
|
||||||
|
private FakeWorkerHarness(
|
||||||
|
string sessionId,
|
||||||
|
string nonce,
|
||||||
|
NamedPipeServerStream? gatewayStream,
|
||||||
|
NamedPipeClientStream workerStream,
|
||||||
|
WorkerFrameProtocolOptions frameOptions)
|
||||||
|
{
|
||||||
|
SessionId = sessionId;
|
||||||
|
Nonce = nonce;
|
||||||
|
_gatewayStream = gatewayStream;
|
||||||
|
_workerStream = workerStream;
|
||||||
|
_frameOptions = frameOptions;
|
||||||
|
_reader = new WorkerFrameReader(_workerStream, frameOptions);
|
||||||
|
_writer = new WorkerFrameWriter(_workerStream, frameOptions);
|
||||||
|
}
|
||||||
|
|
||||||
|
public string SessionId { get; }
|
||||||
|
|
||||||
|
public string Nonce { get; }
|
||||||
|
|
||||||
|
public ulong NextWorkerSequence { get; private set; }
|
||||||
|
|
||||||
|
public static async Task<FakeWorkerHarness> CreateConnectedPairAsync(
|
||||||
|
string sessionId = DefaultSessionId,
|
||||||
|
string nonce = DefaultNonce,
|
||||||
|
uint protocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
int maxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string pipeName = $"mxaccessgw-fake-worker-{Guid.NewGuid():N}";
|
||||||
|
NamedPipeServerStream gatewayStream = new(
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
maxNumberOfServerInstances: 1,
|
||||||
|
PipeTransmissionMode.Byte,
|
||||||
|
PipeOptions.Asynchronous);
|
||||||
|
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
|
||||||
|
|
||||||
|
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken);
|
||||||
|
await workerStream.ConnectAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
await waitForConnectionTask.ConfigureAwait(false);
|
||||||
|
|
||||||
|
return new FakeWorkerHarness(
|
||||||
|
sessionId,
|
||||||
|
nonce,
|
||||||
|
gatewayStream,
|
||||||
|
workerStream,
|
||||||
|
new WorkerFrameProtocolOptions(sessionId, protocolVersion, maxMessageBytes));
|
||||||
|
}
|
||||||
|
|
||||||
|
public static async Task<FakeWorkerHarness> ConnectToGatewayPipeAsync(
|
||||||
|
string sessionId,
|
||||||
|
string nonce,
|
||||||
|
string pipeName,
|
||||||
|
uint protocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
int maxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
|
||||||
|
await workerStream.ConnectAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
return new FakeWorkerHarness(
|
||||||
|
sessionId,
|
||||||
|
nonce,
|
||||||
|
gatewayStream: null,
|
||||||
|
workerStream,
|
||||||
|
new WorkerFrameProtocolOptions(sessionId, protocolVersion, maxMessageBytes));
|
||||||
|
}
|
||||||
|
|
||||||
|
public WorkerClient CreateClient(
|
||||||
|
WorkerClientOptions? options = null,
|
||||||
|
GatewayMetrics? metrics = null,
|
||||||
|
TimeProvider? timeProvider = null)
|
||||||
|
{
|
||||||
|
if (_gatewayStream is null)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("This fake worker is connected to a gateway-owned pipe.");
|
||||||
|
}
|
||||||
|
|
||||||
|
WorkerClientConnection connection = new(
|
||||||
|
SessionId,
|
||||||
|
Nonce,
|
||||||
|
_gatewayStream,
|
||||||
|
_frameOptions);
|
||||||
|
|
||||||
|
return new WorkerClient(connection, options, metrics, timeProvider);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<WorkerEnvelope> CompleteStartupAsync(
|
||||||
|
int workerProcessId = DefaultWorkerProcessId,
|
||||||
|
string workerVersion = "fake-worker",
|
||||||
|
string mxaccessProgid = "LMXProxy.LMXProxyServer.1",
|
||||||
|
string mxaccessClsid = "{C30B52F5-2CB5-4760-AF0A-3A344A7EB5DC}",
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
WorkerEnvelope gatewayHello = await ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (gatewayHello.BodyCase != WorkerEnvelope.BodyOneofCase.GatewayHello)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException($"Expected GatewayHello but received {gatewayHello.BodyCase}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
await SendWorkerHelloAsync(
|
||||||
|
workerProcessId,
|
||||||
|
workerVersion,
|
||||||
|
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
await SendWorkerReadyAsync(
|
||||||
|
workerProcessId,
|
||||||
|
mxaccessProgid,
|
||||||
|
mxaccessClsid,
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
return gatewayHello;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<WorkerEnvelope> ReadGatewayEnvelopeAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<WorkerEnvelope> ReadCommandAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = await ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (envelope.BodyCase != WorkerEnvelope.BodyOneofCase.WorkerCommand)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException($"Expected WorkerCommand but received {envelope.BodyCase}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<WorkerEnvelope> ReadShutdownAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = await ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (envelope.BodyCase != WorkerEnvelope.BodyOneofCase.WorkerShutdown)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException($"Expected WorkerShutdown but received {envelope.BodyCase}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task SendWorkerHelloAsync(
|
||||||
|
int workerProcessId = DefaultWorkerProcessId,
|
||||||
|
string workerVersion = "fake-worker",
|
||||||
|
uint? workerProtocolVersion = null,
|
||||||
|
string? nonce = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerHello = new WorkerHello
|
||||||
|
{
|
||||||
|
ProtocolVersion = workerProtocolVersion ?? _frameOptions.ProtocolVersion,
|
||||||
|
Nonce = nonce ?? Nonce,
|
||||||
|
WorkerProcessId = workerProcessId,
|
||||||
|
WorkerVersion = workerVersion,
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task SendWorkerReadyAsync(
|
||||||
|
int workerProcessId = DefaultWorkerProcessId,
|
||||||
|
string mxaccessProgid = "LMXProxy.LMXProxyServer.1",
|
||||||
|
string mxaccessClsid = "{C30B52F5-2CB5-4760-AF0A-3A344A7EB5DC}",
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerReady = new WorkerReady
|
||||||
|
{
|
||||||
|
WorkerProcessId = workerProcessId,
|
||||||
|
MxaccessProgid = mxaccessProgid,
|
||||||
|
MxaccessClsid = mxaccessClsid,
|
||||||
|
ReadyTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task ReplyToCommandAsync(
|
||||||
|
WorkerEnvelope commandEnvelope,
|
||||||
|
ProtocolStatusCode statusCode = ProtocolStatusCode.Ok,
|
||||||
|
string statusMessage = "OK",
|
||||||
|
Action<MxCommandReply>? configureReply = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (commandEnvelope.BodyCase != WorkerEnvelope.BodyOneofCase.WorkerCommand)
|
||||||
|
{
|
||||||
|
throw new ArgumentException("Command envelope must contain WorkerCommand.", nameof(commandEnvelope));
|
||||||
|
}
|
||||||
|
|
||||||
|
MxCommandKind kind = commandEnvelope.WorkerCommand.Command?.Kind ?? MxCommandKind.Unspecified;
|
||||||
|
MxCommandReply reply = new()
|
||||||
|
{
|
||||||
|
SessionId = SessionId,
|
||||||
|
CorrelationId = commandEnvelope.CorrelationId,
|
||||||
|
Kind = kind,
|
||||||
|
ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = statusCode,
|
||||||
|
Message = statusMessage,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
configureReply?.Invoke(reply);
|
||||||
|
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
commandEnvelope.CorrelationId,
|
||||||
|
envelope => envelope.WorkerCommandReply = new WorkerCommandReply
|
||||||
|
{
|
||||||
|
Reply = reply,
|
||||||
|
CompletedTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task EmitEventAsync(
|
||||||
|
MxEventFamily family,
|
||||||
|
CancellationToken cancellationToken = default,
|
||||||
|
Action<MxEvent>? configureEvent = null)
|
||||||
|
{
|
||||||
|
ulong sequence = NextWorkerSequence + 1;
|
||||||
|
MxEvent mxEvent = new()
|
||||||
|
{
|
||||||
|
SessionId = SessionId,
|
||||||
|
Family = family,
|
||||||
|
WorkerSequence = sequence,
|
||||||
|
WorkerTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
};
|
||||||
|
configureEvent?.Invoke(mxEvent);
|
||||||
|
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerEvent = new WorkerEvent
|
||||||
|
{
|
||||||
|
Event = mxEvent,
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task EmitFaultAsync(
|
||||||
|
WorkerFaultCategory category,
|
||||||
|
string diagnosticMessage,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerFault = new WorkerFault
|
||||||
|
{
|
||||||
|
Category = category,
|
||||||
|
DiagnosticMessage = diagnosticMessage,
|
||||||
|
ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.WorkerUnavailable,
|
||||||
|
Message = diagnosticMessage,
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task SendHeartbeatAsync(
|
||||||
|
WorkerState state = WorkerState.Ready,
|
||||||
|
CancellationToken cancellationToken = default,
|
||||||
|
Action<WorkerHeartbeat>? configureHeartbeat = null)
|
||||||
|
{
|
||||||
|
WorkerHeartbeat heartbeat = new()
|
||||||
|
{
|
||||||
|
WorkerProcessId = DefaultWorkerProcessId,
|
||||||
|
State = state,
|
||||||
|
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
};
|
||||||
|
configureHeartbeat?.Invoke(heartbeat);
|
||||||
|
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerHeartbeat = heartbeat),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task SendShutdownAckAsync(
|
||||||
|
ProtocolStatusCode statusCode = ProtocolStatusCode.Ok,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await _writer.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
envelope => envelope.WorkerShutdownAck = new WorkerShutdownAck
|
||||||
|
{
|
||||||
|
Status = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = statusCode,
|
||||||
|
Message = statusCode.ToString(),
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task WriteMalformedPayloadAsync(
|
||||||
|
ReadOnlyMemory<byte> payload,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (payload.IsEmpty)
|
||||||
|
{
|
||||||
|
throw new ArgumentException("Malformed payload must include at least one byte.", nameof(payload));
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] lengthPrefix = new byte[sizeof(uint)];
|
||||||
|
BinaryPrimitives.WriteUInt32LittleEndian(lengthPrefix, (uint)payload.Length);
|
||||||
|
await _workerStream.WriteAsync(lengthPrefix, cancellationToken).ConfigureAwait(false);
|
||||||
|
await _workerStream.WriteAsync(payload, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task WriteOversizedFrameHeaderAsync(
|
||||||
|
uint payloadLength,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (payloadLength <= _frameOptions.MaxMessageBytes)
|
||||||
|
{
|
||||||
|
throw new ArgumentOutOfRangeException(
|
||||||
|
nameof(payloadLength),
|
||||||
|
payloadLength,
|
||||||
|
"Payload length must exceed the configured maximum.");
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] lengthPrefix = new byte[sizeof(uint)];
|
||||||
|
BinaryPrimitives.WriteUInt32LittleEndian(lengthPrefix, payloadLength);
|
||||||
|
await _workerStream.WriteAsync(lengthPrefix, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async ValueTask DisposeWorkerSideAsync()
|
||||||
|
{
|
||||||
|
if (_workerSideDisposed)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await _workerStream.DisposeAsync().ConfigureAwait(false);
|
||||||
|
_workerSideDisposed = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async ValueTask DisposeAsync()
|
||||||
|
{
|
||||||
|
await DisposeWorkerSideAsync().ConfigureAwait(false);
|
||||||
|
if (_gatewayStream is not null)
|
||||||
|
{
|
||||||
|
await _gatewayStream.DisposeAsync().ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateEnvelope(
|
||||||
|
string correlationId,
|
||||||
|
Action<WorkerEnvelope> setBody)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = new()
|
||||||
|
{
|
||||||
|
ProtocolVersion = _frameOptions.ProtocolVersion,
|
||||||
|
SessionId = SessionId,
|
||||||
|
Sequence = AdvanceSequence(),
|
||||||
|
CorrelationId = correlationId,
|
||||||
|
};
|
||||||
|
setBody(envelope);
|
||||||
|
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
private ulong AdvanceSequence()
|
||||||
|
{
|
||||||
|
return ++NextWorkerSequence;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static NamedPipeClientStream CreateWorkerStream(string pipeName)
|
||||||
|
{
|
||||||
|
return new NamedPipeClientStream(
|
||||||
|
".",
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
PipeOptions.Asynchronous);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
using System.IO.Pipes;
|
using System.IO.Pipes;
|
||||||
|
using Google.Protobuf.WellKnownTypes;
|
||||||
using MxGateway.Contracts;
|
using MxGateway.Contracts;
|
||||||
using MxGateway.Contracts.Proto;
|
using MxGateway.Contracts.Proto;
|
||||||
using MxGateway.Server.Workers;
|
using MxGateway.Server.Workers;
|
||||||
@@ -151,6 +152,24 @@ public sealed class WorkerClientTests
|
|||||||
Assert.Equal(WorkerClientState.Faulted, client.State);
|
Assert.Equal(WorkerClientState.Faulted, client.State);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ReadLoop_WhenHeartbeatArrives_UpdatesLastHeartbeatAndWorkerProcess()
|
||||||
|
{
|
||||||
|
await using PipePair pipePair = await PipePair.CreateAsync();
|
||||||
|
await using WorkerClient client = CreateClient(pipePair);
|
||||||
|
await CompleteHandshakeAsync(client, pipePair);
|
||||||
|
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
|
||||||
|
|
||||||
|
await Task.Delay(TimeSpan.FromMilliseconds(20));
|
||||||
|
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
|
||||||
|
|
||||||
|
await WaitUntilAsync(
|
||||||
|
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
|
||||||
|
TestTimeout);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerClientState.Ready, client.State);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task HeartbeatMonitor_WhenHeartbeatExpires_FaultsClient()
|
public async Task HeartbeatMonitor_WhenHeartbeatExpires_FaultsClient()
|
||||||
{
|
{
|
||||||
@@ -276,6 +295,21 @@ public sealed class WorkerClientTests
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerEnvelope CreateHeartbeatEnvelope(int workerProcessId)
|
||||||
|
{
|
||||||
|
return CreateWorkerEnvelope(
|
||||||
|
correlationId: string.Empty,
|
||||||
|
sequence: 20,
|
||||||
|
envelope => envelope.WorkerHeartbeat = new WorkerHeartbeat
|
||||||
|
{
|
||||||
|
WorkerProcessId = workerProcessId,
|
||||||
|
State = WorkerState.Ready,
|
||||||
|
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
PendingCommandCount = 0,
|
||||||
|
OutboundEventQueueDepth = 0,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerEnvelope CreateWorkerEnvelope(
|
private static WorkerEnvelope CreateWorkerEnvelope(
|
||||||
string correlationId,
|
string correlationId,
|
||||||
ulong sequence,
|
ulong sequence,
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ public sealed class WorkerApplicationTests
|
|||||||
Assert.Equal("mxaccess-gateway-123-session-1", entry.Fields["pipe_name"]);
|
Assert.Equal("mxaccess-gateway-123-session-1", entry.Fields["pipe_name"]);
|
||||||
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, entry.Fields["protocol_version"]);
|
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, entry.Fields["protocol_version"]);
|
||||||
Assert.Equal("[redacted]", entry.Fields["nonce"]);
|
Assert.Equal("[redacted]", entry.Fields["nonce"]);
|
||||||
Assert.Equal("WorkerPipeHandshakeSucceeded", logger.Entries[1].EventName);
|
Assert.Equal("WorkerPipeSessionCompleted", logger.Entries[1].EventName);
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
|
|||||||
@@ -1,10 +1,15 @@
|
|||||||
using System;
|
using System;
|
||||||
|
using System.IO;
|
||||||
using System.IO.Pipes;
|
using System.IO.Pipes;
|
||||||
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
|
using Google.Protobuf.WellKnownTypes;
|
||||||
using MxGateway.Contracts;
|
using MxGateway.Contracts;
|
||||||
using MxGateway.Contracts.Proto;
|
using MxGateway.Contracts.Proto;
|
||||||
using MxGateway.Worker.Bootstrap;
|
using MxGateway.Worker.Bootstrap;
|
||||||
using MxGateway.Worker.Ipc;
|
using MxGateway.Worker.Ipc;
|
||||||
|
using MxGateway.Worker.MxAccess;
|
||||||
|
using MxGateway.Worker.Sta;
|
||||||
|
|
||||||
namespace MxGateway.Worker.Tests.Ipc;
|
namespace MxGateway.Worker.Tests.Ipc;
|
||||||
|
|
||||||
@@ -28,7 +33,9 @@ public sealed class WorkerPipeClientTests
|
|||||||
PipeTransmissionMode.Byte,
|
PipeTransmissionMode.Byte,
|
||||||
PipeOptions.Asynchronous);
|
PipeOptions.Asynchronous);
|
||||||
|
|
||||||
WorkerPipeClient client = new(connectTimeoutMilliseconds: 5000);
|
WorkerPipeClient client = new(
|
||||||
|
connectTimeoutMilliseconds: 5000,
|
||||||
|
(stream, options) => CreateSession(stream, options));
|
||||||
Task clientTask = client.RunAsync(workerOptions);
|
Task clientTask = client.RunAsync(workerOptions);
|
||||||
|
|
||||||
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
|
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
|
||||||
@@ -56,6 +63,86 @@ public sealed class WorkerPipeClientTests
|
|||||||
WorkerEnvelope ready = await reader.ReadAsync();
|
WorkerEnvelope ready = await reader.ReadAsync();
|
||||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerReady, ready.BodyCase);
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerReady, ready.BodyCase);
|
||||||
|
|
||||||
|
await writer.WriteAsync(new WorkerEnvelope
|
||||||
|
{
|
||||||
|
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
SessionId = "session-1",
|
||||||
|
Sequence = 2,
|
||||||
|
WorkerShutdown = new WorkerShutdown
|
||||||
|
{
|
||||||
|
GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(1)),
|
||||||
|
Reason = "test-complete",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
WorkerEnvelope shutdownAck = await reader.ReadAsync();
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerShutdownAck, shutdownAck.BodyCase);
|
||||||
await clientTask;
|
await clientTask;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerPipeSession CreateSession(
|
||||||
|
Stream stream,
|
||||||
|
WorkerFrameProtocolOptions options)
|
||||||
|
{
|
||||||
|
return new WorkerPipeSession(
|
||||||
|
new WorkerFrameReader(stream, options),
|
||||||
|
new WorkerFrameWriter(stream, options),
|
||||||
|
options,
|
||||||
|
() => 1234,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromSeconds(30),
|
||||||
|
HeartbeatGrace = TimeSpan.FromSeconds(30),
|
||||||
|
},
|
||||||
|
() => new FakeRuntimeSession());
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||||
|
{
|
||||||
|
public Task<WorkerReady> StartAsync(
|
||||||
|
int workerProcessId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return Task.FromResult(new WorkerReady
|
||||||
|
{
|
||||||
|
WorkerProcessId = workerProcessId,
|
||||||
|
MxaccessProgid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.ProgId,
|
||||||
|
MxaccessClsid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.Clsid,
|
||||||
|
ReadyTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<MxCommandReply> DispatchAsync(StaCommand command)
|
||||||
|
{
|
||||||
|
return Task.FromResult(new MxCommandReply
|
||||||
|
{
|
||||||
|
SessionId = command.SessionId,
|
||||||
|
CorrelationId = command.CorrelationId,
|
||||||
|
Kind = command.Kind,
|
||||||
|
ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.Ok,
|
||||||
|
Message = "OK",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
|
||||||
|
{
|
||||||
|
return new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: string.Empty);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RequestShutdown()
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,11 +1,16 @@
|
|||||||
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
|
using System.IO.Pipes;
|
||||||
using System.Runtime.InteropServices;
|
using System.Runtime.InteropServices;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
|
using Google.Protobuf.WellKnownTypes;
|
||||||
using MxGateway.Contracts;
|
using MxGateway.Contracts;
|
||||||
using MxGateway.Contracts.Proto;
|
using MxGateway.Contracts.Proto;
|
||||||
using MxGateway.Worker.Ipc;
|
using MxGateway.Worker.Ipc;
|
||||||
|
using MxGateway.Worker.MxAccess;
|
||||||
|
using MxGateway.Worker.Sta;
|
||||||
|
|
||||||
namespace MxGateway.Worker.Tests.Ipc;
|
namespace MxGateway.Worker.Tests.Ipc;
|
||||||
|
|
||||||
@@ -147,6 +152,127 @@ public sealed class WorkerPipeSessionTests
|
|||||||
Assert.Equal(ProtocolStatusCode.WorkerUnavailable, written[1].WorkerFault.ProtocolStatus.Code);
|
Assert.Equal(ProtocolStatusCode.WorkerUnavailable, written[1].WorkerFault.ProtocolStatus.Code);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task RunAsync_SendsHeartbeatPayloadFromRuntimeSnapshot()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
|
||||||
|
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||||
|
FakeRuntimeSession runtime = new();
|
||||||
|
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
pendingCommandCount: 2,
|
||||||
|
outboundEventQueueDepth: 3,
|
||||||
|
lastEventSequence: 42,
|
||||||
|
currentCommandCorrelationId: "current-command"));
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
pipePair.WorkerStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
|
||||||
|
HeartbeatGrace = TimeSpan.FromSeconds(5),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
await ThrowIfCompletedAsync(runTask);
|
||||||
|
|
||||||
|
WorkerEnvelope heartbeat = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerState.ExecutingCommand, heartbeat.WorkerHeartbeat.State);
|
||||||
|
Assert.Equal(1234, heartbeat.WorkerHeartbeat.WorkerProcessId);
|
||||||
|
Assert.Equal(2u, heartbeat.WorkerHeartbeat.PendingCommandCount);
|
||||||
|
Assert.Equal(3u, heartbeat.WorkerHeartbeat.OutboundEventQueueDepth);
|
||||||
|
Assert.Equal(42UL, heartbeat.WorkerHeartbeat.LastEventSequence);
|
||||||
|
Assert.Equal("current-command", heartbeat.WorkerHeartbeat.CurrentCommandCorrelationId);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task RunAsync_WhenCommandIsExecuting_HeartbeatReportsCurrentCorrelation()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
|
||||||
|
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||||
|
FakeRuntimeSession runtime = new()
|
||||||
|
{
|
||||||
|
BlockDispatch = true,
|
||||||
|
};
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
pipePair.WorkerStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
|
||||||
|
HeartbeatGrace = TimeSpan.FromSeconds(5),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
await pipePair.GatewayWriter.WriteAsync(
|
||||||
|
CreateCommandEnvelope("command-1"),
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.True(runtime.DispatchStarted.Wait(TimeSpan.FromSeconds(2)));
|
||||||
|
WorkerEnvelope heartbeat = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
|
||||||
|
envelope => envelope.WorkerHeartbeat.CurrentCommandCorrelationId == "command-1",
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.Equal("command-1", heartbeat.WorkerHeartbeat.CurrentCommandCorrelationId);
|
||||||
|
Assert.Equal(WorkerState.ExecutingCommand, heartbeat.WorkerHeartbeat.State);
|
||||||
|
|
||||||
|
runtime.ReleaseDispatch();
|
||||||
|
WorkerEnvelope reply = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.Equal("command-1", reply.CorrelationId);
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.WorkerCommandReply.Reply.ProtocolStatus.Code);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task RunAsync_WhenStaActivityIsStale_WritesWatchdogFault()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
|
||||||
|
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||||
|
FakeRuntimeSession runtime = new();
|
||||||
|
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: "stuck-command"));
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
pipePair.WorkerStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
|
||||||
|
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
WorkerEnvelope fault = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerFault,
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerFaultCategory.StaHung, fault.WorkerFault.Category);
|
||||||
|
Assert.Equal("stuck-command", fault.WorkerFault.CommandMethod);
|
||||||
|
Assert.Contains("STA activity is stale", fault.WorkerFault.DiagnosticMessage);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerPipeSession CreateSession(
|
private static WorkerPipeSession CreateSession(
|
||||||
Stream inbound,
|
Stream inbound,
|
||||||
Stream outbound,
|
Stream outbound,
|
||||||
@@ -159,6 +285,21 @@ public sealed class WorkerPipeSessionTests
|
|||||||
() => 1234);
|
() => 1234);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerPipeSession CreatePipeSession(
|
||||||
|
Stream stream,
|
||||||
|
FakeRuntimeSession runtime,
|
||||||
|
WorkerPipeSessionOptions sessionOptions)
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
return new WorkerPipeSession(
|
||||||
|
new WorkerFrameReader(stream, options),
|
||||||
|
new WorkerFrameWriter(stream, options),
|
||||||
|
options,
|
||||||
|
() => 1234,
|
||||||
|
sessionOptions,
|
||||||
|
() => runtime);
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerFrameProtocolOptions CreateOptions()
|
private static WorkerFrameProtocolOptions CreateOptions()
|
||||||
{
|
{
|
||||||
return new WorkerFrameProtocolOptions(
|
return new WorkerFrameProtocolOptions(
|
||||||
@@ -185,6 +326,119 @@ public sealed class WorkerPipeSessionTests
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerEnvelope CreateCommandEnvelope(string correlationId)
|
||||||
|
{
|
||||||
|
return new WorkerEnvelope
|
||||||
|
{
|
||||||
|
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
SessionId = SessionId,
|
||||||
|
Sequence = 2,
|
||||||
|
CorrelationId = correlationId,
|
||||||
|
WorkerCommand = new WorkerCommand
|
||||||
|
{
|
||||||
|
Command = new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.Ping,
|
||||||
|
Ping = new PingCommand
|
||||||
|
{
|
||||||
|
Message = "ping",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
EnqueueTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerEnvelope CreateShutdownEnvelope()
|
||||||
|
{
|
||||||
|
return new WorkerEnvelope
|
||||||
|
{
|
||||||
|
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
SessionId = SessionId,
|
||||||
|
Sequence = 3,
|
||||||
|
WorkerShutdown = new WorkerShutdown
|
||||||
|
{
|
||||||
|
GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(1)),
|
||||||
|
Reason = "test-complete",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task CompleteGatewayHandshakeAsync(
|
||||||
|
PipePair pipePair,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
await pipePair.GatewayWriter
|
||||||
|
.WriteAsync(CreateGatewayHelloEnvelope(), cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
|
WorkerEnvelope hello = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
WorkerEnvelope ready = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerHello, hello.BodyCase);
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerReady, ready.BodyCase);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task SendShutdownAndWaitAsync(
|
||||||
|
PipePair pipePair,
|
||||||
|
Task runTask,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
await pipePair.GatewayWriter
|
||||||
|
.WriteAsync(CreateShutdownEnvelope(), cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
|
WorkerEnvelope shutdownAck = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerShutdownAck,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, shutdownAck.WorkerShutdownAck.Status.Code);
|
||||||
|
Task completedTask = await Task
|
||||||
|
.WhenAny(runTask, Task.Delay(TimeSpan.FromSeconds(2), cancellationToken))
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
|
Assert.Same(runTask, completedTask);
|
||||||
|
await runTask.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task ThrowIfCompletedAsync(Task task)
|
||||||
|
{
|
||||||
|
await Task.Delay(TimeSpan.FromMilliseconds(100));
|
||||||
|
if (task.IsCompleted)
|
||||||
|
{
|
||||||
|
await task;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Task<WorkerEnvelope> ReadUntilAsync(
|
||||||
|
WorkerFrameReader reader,
|
||||||
|
WorkerEnvelope.BodyOneofCase expectedBody,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
return ReadUntilAsync(
|
||||||
|
reader,
|
||||||
|
expectedBody,
|
||||||
|
_ => true,
|
||||||
|
cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<WorkerEnvelope> ReadUntilAsync(
|
||||||
|
WorkerFrameReader reader,
|
||||||
|
WorkerEnvelope.BodyOneofCase expectedBody,
|
||||||
|
Func<WorkerEnvelope, bool> predicate,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
while (true)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = await reader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (envelope.BodyCase == expectedBody && predicate(envelope))
|
||||||
|
{
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerEnvelope[] ReadWrittenFrames(
|
private static WorkerEnvelope[] ReadWrittenFrames(
|
||||||
MemoryStream stream,
|
MemoryStream stream,
|
||||||
WorkerFrameProtocolOptions options)
|
WorkerFrameProtocolOptions options)
|
||||||
@@ -219,4 +473,157 @@ public sealed class WorkerPipeSessionTests
|
|||||||
buffer[2] = (byte)(value >> 16);
|
buffer[2] = (byte)(value >> 16);
|
||||||
buffer[3] = (byte)(value >> 24);
|
buffer[3] = (byte)(value >> 24);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||||
|
{
|
||||||
|
private readonly ManualResetEventSlim releaseDispatch = new(false);
|
||||||
|
private readonly object gate = new();
|
||||||
|
private WorkerRuntimeHeartbeatSnapshot snapshot = new(
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: string.Empty);
|
||||||
|
|
||||||
|
public ManualResetEventSlim DispatchStarted { get; } = new(false);
|
||||||
|
|
||||||
|
public bool BlockDispatch { get; set; }
|
||||||
|
|
||||||
|
public Task<WorkerReady> StartAsync(
|
||||||
|
int workerProcessId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return Task.FromResult(new WorkerReady
|
||||||
|
{
|
||||||
|
WorkerProcessId = workerProcessId,
|
||||||
|
MxaccessProgid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.ProgId,
|
||||||
|
MxaccessClsid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.Clsid,
|
||||||
|
ReadyTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<MxCommandReply> DispatchAsync(StaCommand command)
|
||||||
|
{
|
||||||
|
return Task.Run(
|
||||||
|
() =>
|
||||||
|
{
|
||||||
|
SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
command.CorrelationId));
|
||||||
|
DispatchStarted.Set();
|
||||||
|
|
||||||
|
if (BlockDispatch)
|
||||||
|
{
|
||||||
|
releaseDispatch.Wait(TimeSpan.FromSeconds(5));
|
||||||
|
}
|
||||||
|
|
||||||
|
SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: string.Empty));
|
||||||
|
|
||||||
|
return new MxCommandReply
|
||||||
|
{
|
||||||
|
SessionId = command.SessionId,
|
||||||
|
CorrelationId = command.CorrelationId,
|
||||||
|
Kind = command.Kind,
|
||||||
|
ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.Ok,
|
||||||
|
Message = "OK",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
|
||||||
|
{
|
||||||
|
lock (gate)
|
||||||
|
{
|
||||||
|
return snapshot;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RequestShutdown()
|
||||||
|
{
|
||||||
|
releaseDispatch.Set();
|
||||||
|
}
|
||||||
|
|
||||||
|
public void ReleaseDispatch()
|
||||||
|
{
|
||||||
|
releaseDispatch.Set();
|
||||||
|
}
|
||||||
|
|
||||||
|
public void SetSnapshot(WorkerRuntimeHeartbeatSnapshot value)
|
||||||
|
{
|
||||||
|
lock (gate)
|
||||||
|
{
|
||||||
|
snapshot = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
releaseDispatch.Set();
|
||||||
|
releaseDispatch.Dispose();
|
||||||
|
DispatchStarted.Dispose();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class PipePair : IDisposable
|
||||||
|
{
|
||||||
|
private readonly NamedPipeServerStream gatewayStream;
|
||||||
|
|
||||||
|
private PipePair(
|
||||||
|
NamedPipeServerStream gatewayStream,
|
||||||
|
NamedPipeClientStream workerStream)
|
||||||
|
{
|
||||||
|
this.gatewayStream = gatewayStream;
|
||||||
|
WorkerStream = workerStream;
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
GatewayReader = new WorkerFrameReader(gatewayStream, options);
|
||||||
|
GatewayWriter = new WorkerFrameWriter(gatewayStream, options);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Stream WorkerStream { get; }
|
||||||
|
|
||||||
|
public WorkerFrameReader GatewayReader { get; }
|
||||||
|
|
||||||
|
public WorkerFrameWriter GatewayWriter { get; }
|
||||||
|
|
||||||
|
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
|
||||||
|
NamedPipeServerStream gatewayStream = new(
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
maxNumberOfServerInstances: 1,
|
||||||
|
PipeTransmissionMode.Byte,
|
||||||
|
PipeOptions.Asynchronous);
|
||||||
|
NamedPipeClientStream workerStream = new(
|
||||||
|
".",
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
PipeOptions.Asynchronous);
|
||||||
|
|
||||||
|
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync();
|
||||||
|
await Task
|
||||||
|
.Run(() => workerStream.Connect(5000), cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
await waitForConnectionTask.ConfigureAwait(false);
|
||||||
|
|
||||||
|
return new PipePair(gatewayStream, workerStream);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
WorkerStream.Dispose();
|
||||||
|
gatewayStream.Dispose();
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -78,6 +78,166 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
Assert.Equal(44, registeredServerHandle.ServerHandle);
|
Assert.Equal(44, registeredServerHandle.ServerHandle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_AddItem_CallsMxAccessOnStaAndTracksItemHandle()
|
||||||
|
{
|
||||||
|
FakeMxAccessComObject fakeComObject = new(
|
||||||
|
registerHandle: 46,
|
||||||
|
addItemHandle: 501);
|
||||||
|
FakeMxAccessComObjectFactory factory = new(fakeComObject);
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
await session.DispatchAsync(CreateRegisterCommand("register-before-add", "client-a"));
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(CreateAddItemCommand(
|
||||||
|
"add-item",
|
||||||
|
46,
|
||||||
|
"Galaxy.Tag.Value"));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(reply.HasHresult);
|
||||||
|
Assert.Equal(0, reply.Hresult);
|
||||||
|
Assert.Equal(501, reply.AddItem.ItemHandle);
|
||||||
|
Assert.Equal(MxDataType.Integer, reply.ReturnValue.DataType);
|
||||||
|
Assert.Equal(501, reply.ReturnValue.Int32Value);
|
||||||
|
Assert.Equal(46, fakeComObject.AddItemServerHandle);
|
||||||
|
Assert.Equal("Galaxy.Tag.Value", fakeComObject.AddItemDefinition);
|
||||||
|
Assert.Equal(runtime.StaThreadId, fakeComObject.AddItemThreadId);
|
||||||
|
|
||||||
|
RegisteredItemHandle registeredItemHandle = Assert.Single(
|
||||||
|
await session.GetRegisteredItemHandlesAsync());
|
||||||
|
Assert.Equal(46, registeredItemHandle.ServerHandle);
|
||||||
|
Assert.Equal(501, registeredItemHandle.ItemHandle);
|
||||||
|
Assert.Equal("Galaxy.Tag.Value", registeredItemHandle.ItemDefinition);
|
||||||
|
Assert.Equal(string.Empty, registeredItemHandle.ItemContext);
|
||||||
|
Assert.False(registeredItemHandle.HasItemContext);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_AddItem2_PassesContextExactlyAndTracksItemHandle()
|
||||||
|
{
|
||||||
|
FakeMxAccessComObject fakeComObject = new(
|
||||||
|
registerHandle: 47,
|
||||||
|
addItem2Handle: 502);
|
||||||
|
FakeMxAccessComObjectFactory factory = new(fakeComObject);
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
await session.DispatchAsync(CreateRegisterCommand("register-before-add2", "client-a"));
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(CreateAddItem2Command(
|
||||||
|
"add-item2",
|
||||||
|
47,
|
||||||
|
"TestInt",
|
||||||
|
"TestChildObject"));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
|
||||||
|
Assert.Equal(502, reply.AddItem2.ItemHandle);
|
||||||
|
Assert.Equal(MxDataType.Integer, reply.ReturnValue.DataType);
|
||||||
|
Assert.Equal(502, reply.ReturnValue.Int32Value);
|
||||||
|
Assert.Equal(47, fakeComObject.AddItem2ServerHandle);
|
||||||
|
Assert.Equal("TestInt", fakeComObject.AddItem2Definition);
|
||||||
|
Assert.Equal("TestChildObject", fakeComObject.AddItem2Context);
|
||||||
|
Assert.Equal(runtime.StaThreadId, fakeComObject.AddItem2ThreadId);
|
||||||
|
|
||||||
|
RegisteredItemHandle registeredItemHandle = Assert.Single(
|
||||||
|
await session.GetRegisteredItemHandlesAsync());
|
||||||
|
Assert.Equal(47, registeredItemHandle.ServerHandle);
|
||||||
|
Assert.Equal(502, registeredItemHandle.ItemHandle);
|
||||||
|
Assert.Equal("TestInt", registeredItemHandle.ItemDefinition);
|
||||||
|
Assert.Equal("TestChildObject", registeredItemHandle.ItemContext);
|
||||||
|
Assert.True(registeredItemHandle.HasItemContext);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_RemoveItem_CallsMxAccessOnStaAndRemovesTrackedItemHandle()
|
||||||
|
{
|
||||||
|
FakeMxAccessComObject fakeComObject = new(
|
||||||
|
registerHandle: 48,
|
||||||
|
addItemHandle: 503);
|
||||||
|
FakeMxAccessComObjectFactory factory = new(fakeComObject);
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
await session.DispatchAsync(CreateRegisterCommand("register-before-remove", "client-a"));
|
||||||
|
await session.DispatchAsync(CreateAddItemCommand("add-before-remove", 48, "Galaxy.Tag.Value"));
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(CreateRemoveItemCommand(
|
||||||
|
"remove-item",
|
||||||
|
48,
|
||||||
|
503));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(reply.HasHresult);
|
||||||
|
Assert.Equal(0, reply.Hresult);
|
||||||
|
Assert.Equal(48, fakeComObject.RemoveItemServerHandle);
|
||||||
|
Assert.Equal(503, fakeComObject.RemovedItemHandle);
|
||||||
|
Assert.Equal(runtime.StaThreadId, fakeComObject.RemoveItemThreadId);
|
||||||
|
Assert.Empty(await session.GetRegisteredItemHandlesAsync());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_RemoveItemWithCrossServerHandle_PreservesHResultAndKeepsTrackedItemHandle()
|
||||||
|
{
|
||||||
|
const int hresult = unchecked((int)0x80070057);
|
||||||
|
FakeMxAccessComObject fakeComObject = new(
|
||||||
|
registerHandle: 49,
|
||||||
|
addItemHandle: 504,
|
||||||
|
removeItemException: new COMException("Invalid item handle.", hresult));
|
||||||
|
FakeMxAccessComObjectFactory factory = new(fakeComObject);
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
await session.DispatchAsync(CreateRegisterCommand("register-before-remove-failure", "client-a"));
|
||||||
|
await session.DispatchAsync(CreateAddItemCommand("add-before-remove-failure", 49, "Galaxy.Tag.Value"));
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(CreateRemoveItemCommand(
|
||||||
|
"remove-item-failure",
|
||||||
|
999,
|
||||||
|
504));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.MxaccessFailure, reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(reply.HasHresult);
|
||||||
|
Assert.Equal(hresult, reply.Hresult);
|
||||||
|
Assert.Contains("0x80070057", reply.DiagnosticMessage);
|
||||||
|
Assert.Equal(999, fakeComObject.RemoveItemServerHandle);
|
||||||
|
Assert.Equal(504, fakeComObject.RemovedItemHandle);
|
||||||
|
|
||||||
|
RegisteredItemHandle registeredItemHandle = Assert.Single(
|
||||||
|
await session.GetRegisteredItemHandlesAsync());
|
||||||
|
Assert.Equal(49, registeredItemHandle.ServerHandle);
|
||||||
|
Assert.Equal(504, registeredItemHandle.ItemHandle);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_AddItem2WhenMxAccessThrows_PreservesHResultAndDoesNotTrackItemHandle()
|
||||||
|
{
|
||||||
|
const int hresult = unchecked((int)0x80070057);
|
||||||
|
FakeMxAccessComObject fakeComObject = new(
|
||||||
|
registerHandle: 50,
|
||||||
|
addItem2Exception: new COMException("Invalid server handle.", hresult));
|
||||||
|
FakeMxAccessComObjectFactory factory = new(fakeComObject);
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(CreateAddItem2Command(
|
||||||
|
"add-item2-failure",
|
||||||
|
9001,
|
||||||
|
"TestInt",
|
||||||
|
"TestChildObject"));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.MxaccessFailure, reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(reply.HasHresult);
|
||||||
|
Assert.Equal(hresult, reply.Hresult);
|
||||||
|
Assert.Contains("0x80070057", reply.DiagnosticMessage);
|
||||||
|
Assert.Equal(9001, fakeComObject.AddItem2ServerHandle);
|
||||||
|
Assert.Equal("TestInt", fakeComObject.AddItem2Definition);
|
||||||
|
Assert.Equal("TestChildObject", fakeComObject.AddItem2Context);
|
||||||
|
Assert.Empty(await session.GetRegisteredItemHandlesAsync());
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task DispatchAsync_RegisterWithoutPayload_ReturnsInvalidRequest()
|
public async Task DispatchAsync_RegisterWithoutPayload_ReturnsInvalidRequest()
|
||||||
{
|
{
|
||||||
@@ -98,6 +258,26 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
Assert.Null(factory.FakeComObject.RegisteredClientName);
|
Assert.Null(factory.FakeComObject.RegisteredClientName);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task DispatchAsync_AddItemWithoutPayload_ReturnsInvalidRequest()
|
||||||
|
{
|
||||||
|
FakeMxAccessComObjectFactory factory = new(new FakeMxAccessComObject(registerHandle: 51));
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"missing-add-payload",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem,
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
|
||||||
|
Assert.Null(factory.FakeComObject.AddItemDefinition);
|
||||||
|
}
|
||||||
|
|
||||||
private static StaCommand CreateRegisterCommand(
|
private static StaCommand CreateRegisterCommand(
|
||||||
string correlationId,
|
string correlationId,
|
||||||
string clientName)
|
string clientName)
|
||||||
@@ -132,6 +312,65 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static StaCommand CreateAddItemCommand(
|
||||||
|
string correlationId,
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition)
|
||||||
|
{
|
||||||
|
return new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
correlationId,
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem,
|
||||||
|
AddItem = new AddItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemDefinition = itemDefinition,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static StaCommand CreateAddItem2Command(
|
||||||
|
string correlationId,
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext)
|
||||||
|
{
|
||||||
|
return new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
correlationId,
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem2,
|
||||||
|
AddItem2 = new AddItem2Command
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemDefinition = itemDefinition,
|
||||||
|
ItemContext = itemContext,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static StaCommand CreateRemoveItemCommand(
|
||||||
|
string correlationId,
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
return new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
correlationId,
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.RemoveItem,
|
||||||
|
RemoveItem = new RemoveItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
private static StaRuntime CreateRuntime()
|
private static StaRuntime CreateRuntime()
|
||||||
{
|
{
|
||||||
return new StaRuntime(
|
return new StaRuntime(
|
||||||
@@ -143,14 +382,29 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
private sealed class FakeMxAccessComObject
|
private sealed class FakeMxAccessComObject
|
||||||
{
|
{
|
||||||
private readonly int registerHandle;
|
private readonly int registerHandle;
|
||||||
|
private readonly int addItemHandle;
|
||||||
|
private readonly int addItem2Handle;
|
||||||
private readonly Exception? unregisterException;
|
private readonly Exception? unregisterException;
|
||||||
|
private readonly Exception? addItemException;
|
||||||
|
private readonly Exception? addItem2Exception;
|
||||||
|
private readonly Exception? removeItemException;
|
||||||
|
|
||||||
public FakeMxAccessComObject(
|
public FakeMxAccessComObject(
|
||||||
int registerHandle,
|
int registerHandle,
|
||||||
Exception? unregisterException = null)
|
int addItemHandle = 0,
|
||||||
|
int addItem2Handle = 0,
|
||||||
|
Exception? unregisterException = null,
|
||||||
|
Exception? addItemException = null,
|
||||||
|
Exception? addItem2Exception = null,
|
||||||
|
Exception? removeItemException = null)
|
||||||
{
|
{
|
||||||
this.registerHandle = registerHandle;
|
this.registerHandle = registerHandle;
|
||||||
|
this.addItemHandle = addItemHandle;
|
||||||
|
this.addItem2Handle = addItem2Handle;
|
||||||
this.unregisterException = unregisterException;
|
this.unregisterException = unregisterException;
|
||||||
|
this.addItemException = addItemException;
|
||||||
|
this.addItem2Exception = addItem2Exception;
|
||||||
|
this.removeItemException = removeItemException;
|
||||||
}
|
}
|
||||||
|
|
||||||
public string? RegisteredClientName { get; private set; }
|
public string? RegisteredClientName { get; private set; }
|
||||||
@@ -161,6 +415,26 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
|
|
||||||
public int? UnregisterThreadId { get; private set; }
|
public int? UnregisterThreadId { get; private set; }
|
||||||
|
|
||||||
|
public int? AddItemServerHandle { get; private set; }
|
||||||
|
|
||||||
|
public string? AddItemDefinition { get; private set; }
|
||||||
|
|
||||||
|
public int? AddItemThreadId { get; private set; }
|
||||||
|
|
||||||
|
public int? AddItem2ServerHandle { get; private set; }
|
||||||
|
|
||||||
|
public string? AddItem2Definition { get; private set; }
|
||||||
|
|
||||||
|
public string? AddItem2Context { get; private set; }
|
||||||
|
|
||||||
|
public int? AddItem2ThreadId { get; private set; }
|
||||||
|
|
||||||
|
public int? RemoveItemServerHandle { get; private set; }
|
||||||
|
|
||||||
|
public int? RemovedItemHandle { get; private set; }
|
||||||
|
|
||||||
|
public int? RemoveItemThreadId { get; private set; }
|
||||||
|
|
||||||
public int Register(string clientName)
|
public int Register(string clientName)
|
||||||
{
|
{
|
||||||
RegisteredClientName = clientName;
|
RegisteredClientName = clientName;
|
||||||
@@ -179,6 +453,54 @@ public sealed class MxAccessCommandExecutorTests
|
|||||||
throw unregisterException;
|
throw unregisterException;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public int AddItem(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition)
|
||||||
|
{
|
||||||
|
AddItemServerHandle = serverHandle;
|
||||||
|
AddItemDefinition = itemDefinition;
|
||||||
|
AddItemThreadId = Environment.CurrentManagedThreadId;
|
||||||
|
|
||||||
|
if (addItemException is not null)
|
||||||
|
{
|
||||||
|
throw addItemException;
|
||||||
|
}
|
||||||
|
|
||||||
|
return addItemHandle;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int AddItem2(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext)
|
||||||
|
{
|
||||||
|
AddItem2ServerHandle = serverHandle;
|
||||||
|
AddItem2Definition = itemDefinition;
|
||||||
|
AddItem2Context = itemContext;
|
||||||
|
AddItem2ThreadId = Environment.CurrentManagedThreadId;
|
||||||
|
|
||||||
|
if (addItem2Exception is not null)
|
||||||
|
{
|
||||||
|
throw addItem2Exception;
|
||||||
|
}
|
||||||
|
|
||||||
|
return addItem2Handle;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RemoveItem(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
RemoveItemServerHandle = serverHandle;
|
||||||
|
RemovedItemHandle = itemHandle;
|
||||||
|
RemoveItemThreadId = Environment.CurrentManagedThreadId;
|
||||||
|
|
||||||
|
if (removeItemException is not null)
|
||||||
|
{
|
||||||
|
throw removeItemException;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private sealed class FakeMxAccessComObjectFactory : IMxAccessComObjectFactory
|
private sealed class FakeMxAccessComObjectFactory : IMxAccessComObjectFactory
|
||||||
|
|||||||
@@ -8,6 +8,11 @@ namespace MxGateway.Worker.Tests.MxAccess;
|
|||||||
|
|
||||||
public sealed class MxAccessLiveComCreationTests
|
public sealed class MxAccessLiveComCreationTests
|
||||||
{
|
{
|
||||||
|
private const string LiveClientName = "MxGateway.Worker.Tests";
|
||||||
|
private const string DefaultLiveAddItemReference = "TestChildObject.TestInt";
|
||||||
|
private const string DefaultLiveAddItem2Definition = "TestInt";
|
||||||
|
private const string DefaultLiveAddItem2Context = "TestChildObject";
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task StartAsync_WhenOptedIn_CreatesInstalledMxAccessComObjectOnSta()
|
public async Task StartAsync_WhenOptedIn_CreatesInstalledMxAccessComObjectOnSta()
|
||||||
{
|
{
|
||||||
@@ -43,7 +48,7 @@ public sealed class MxAccessLiveComCreationTests
|
|||||||
Kind = MxCommandKind.Register,
|
Kind = MxCommandKind.Register,
|
||||||
Register = new RegisterCommand
|
Register = new RegisterCommand
|
||||||
{
|
{
|
||||||
ClientName = "MxGateway.Worker.Tests",
|
ClientName = LiveClientName,
|
||||||
},
|
},
|
||||||
}));
|
}));
|
||||||
|
|
||||||
@@ -65,6 +70,151 @@ public sealed class MxAccessLiveComCreationTests
|
|||||||
Assert.Equal(ProtocolStatusCode.Ok, unregisterReply.ProtocolStatus.Code);
|
Assert.Equal(ProtocolStatusCode.Ok, unregisterReply.ProtocolStatus.Code);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task AddItemAndRemoveItem_WhenOptedIn_RoundTripsInstalledMxAccessItemHandle()
|
||||||
|
{
|
||||||
|
if (!RunLiveMxAccessTests())
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
using MxAccessStaSession session = new();
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
|
||||||
|
MxCommandReply registerReply = await RegisterLiveSessionAsync(session, "live-add-register");
|
||||||
|
int serverHandle = registerReply.Register.ServerHandle;
|
||||||
|
int itemHandle = 0;
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
MxCommandReply addItemReply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-add-item",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem,
|
||||||
|
AddItem = new AddItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemDefinition = GetLiveAddItemReference(),
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, addItemReply.ProtocolStatus.Code);
|
||||||
|
Assert.True(addItemReply.AddItem.ItemHandle > 0);
|
||||||
|
itemHandle = addItemReply.AddItem.ItemHandle;
|
||||||
|
|
||||||
|
MxCommandReply removeItemReply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-remove-item",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.RemoveItem,
|
||||||
|
RemoveItem = new RemoveItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, removeItemReply.ProtocolStatus.Code);
|
||||||
|
itemHandle = 0;
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
if (itemHandle > 0)
|
||||||
|
{
|
||||||
|
await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-remove-item-cleanup",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.RemoveItem,
|
||||||
|
RemoveItem = new RemoveItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
await UnregisterLiveSessionAsync(session, serverHandle, "live-add-unregister");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task AddItem2AndRemoveItem_WhenOptedIn_PreservesContextForInstalledMxAccess()
|
||||||
|
{
|
||||||
|
if (!RunLiveMxAccessTests())
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
using MxAccessStaSession session = new();
|
||||||
|
await session.StartAsync(workerProcessId: 1234);
|
||||||
|
|
||||||
|
MxCommandReply registerReply = await RegisterLiveSessionAsync(session, "live-add2-register");
|
||||||
|
int serverHandle = registerReply.Register.ServerHandle;
|
||||||
|
int itemHandle = 0;
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
MxCommandReply addItem2Reply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-add-item2",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.AddItem2,
|
||||||
|
AddItem2 = new AddItem2Command
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemDefinition = DefaultLiveAddItem2Definition,
|
||||||
|
ItemContext = DefaultLiveAddItem2Context,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, addItem2Reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(addItem2Reply.AddItem2.ItemHandle > 0);
|
||||||
|
itemHandle = addItem2Reply.AddItem2.ItemHandle;
|
||||||
|
|
||||||
|
MxCommandReply removeItemReply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-remove-item2",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.RemoveItem,
|
||||||
|
RemoveItem = new RemoveItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, removeItemReply.ProtocolStatus.Code);
|
||||||
|
itemHandle = 0;
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
if (itemHandle > 0)
|
||||||
|
{
|
||||||
|
await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
"live-remove-item2-cleanup",
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.RemoveItem,
|
||||||
|
RemoveItem = new RemoveItemCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
await UnregisterLiveSessionAsync(session, serverHandle, "live-add2-unregister");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private static bool RunLiveMxAccessTests()
|
private static bool RunLiveMxAccessTests()
|
||||||
{
|
{
|
||||||
return string.Equals(
|
return string.Equals(
|
||||||
@@ -72,4 +222,55 @@ public sealed class MxAccessLiveComCreationTests
|
|||||||
"1",
|
"1",
|
||||||
StringComparison.Ordinal);
|
StringComparison.Ordinal);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static string GetLiveAddItemReference()
|
||||||
|
{
|
||||||
|
string itemReference = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_MXACCESS_ITEM");
|
||||||
|
|
||||||
|
return string.IsNullOrWhiteSpace(itemReference)
|
||||||
|
? DefaultLiveAddItemReference
|
||||||
|
: itemReference;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<MxCommandReply> RegisterLiveSessionAsync(
|
||||||
|
MxAccessStaSession session,
|
||||||
|
string correlationId)
|
||||||
|
{
|
||||||
|
MxCommandReply reply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
correlationId,
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.Register,
|
||||||
|
Register = new RegisterCommand
|
||||||
|
{
|
||||||
|
ClientName = LiveClientName,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
|
||||||
|
Assert.True(reply.Register.ServerHandle > 0);
|
||||||
|
|
||||||
|
return reply;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task UnregisterLiveSessionAsync(
|
||||||
|
MxAccessStaSession session,
|
||||||
|
int serverHandle,
|
||||||
|
string correlationId)
|
||||||
|
{
|
||||||
|
MxCommandReply unregisterReply = await session.DispatchAsync(new StaCommand(
|
||||||
|
"session-1",
|
||||||
|
correlationId,
|
||||||
|
new MxCommand
|
||||||
|
{
|
||||||
|
Kind = MxCommandKind.Unregister,
|
||||||
|
Unregister = new UnregisterCommand
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(ProtocolStatusCode.Ok, unregisterReply.ProtocolStatus.Code);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
using System;
|
using System;
|
||||||
|
using System.IO;
|
||||||
using System.IO.Pipes;
|
using System.IO.Pipes;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
@@ -11,6 +12,7 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
|
|||||||
public const int DefaultConnectTimeoutMilliseconds = 30000;
|
public const int DefaultConnectTimeoutMilliseconds = 30000;
|
||||||
|
|
||||||
private readonly int _connectTimeoutMilliseconds;
|
private readonly int _connectTimeoutMilliseconds;
|
||||||
|
private readonly Func<Stream, WorkerFrameProtocolOptions, WorkerPipeSession> _sessionFactory;
|
||||||
|
|
||||||
public WorkerPipeClient()
|
public WorkerPipeClient()
|
||||||
: this(DefaultConnectTimeoutMilliseconds)
|
: this(DefaultConnectTimeoutMilliseconds)
|
||||||
@@ -18,6 +20,15 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
|
|||||||
}
|
}
|
||||||
|
|
||||||
public WorkerPipeClient(int connectTimeoutMilliseconds)
|
public WorkerPipeClient(int connectTimeoutMilliseconds)
|
||||||
|
: this(
|
||||||
|
connectTimeoutMilliseconds,
|
||||||
|
(stream, frameOptions) => new WorkerPipeSession(stream, frameOptions))
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
public WorkerPipeClient(
|
||||||
|
int connectTimeoutMilliseconds,
|
||||||
|
Func<Stream, WorkerFrameProtocolOptions, WorkerPipeSession> sessionFactory)
|
||||||
{
|
{
|
||||||
if (connectTimeoutMilliseconds <= 0)
|
if (connectTimeoutMilliseconds <= 0)
|
||||||
{
|
{
|
||||||
@@ -26,6 +37,7 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
|
|||||||
"Worker pipe connect timeout must be greater than zero.");
|
"Worker pipe connect timeout must be greater than zero.");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
_sessionFactory = sessionFactory ?? throw new ArgumentNullException(nameof(sessionFactory));
|
||||||
_connectTimeoutMilliseconds = connectTimeoutMilliseconds;
|
_connectTimeoutMilliseconds = connectTimeoutMilliseconds;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -48,8 +60,8 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
|
|||||||
|
|
||||||
await ConnectAsync(pipe, cancellationToken).ConfigureAwait(false);
|
await ConnectAsync(pipe, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
WorkerPipeSession session = new(pipe, frameOptions);
|
WorkerPipeSession session = _sessionFactory(pipe, frameOptions);
|
||||||
await session.CompleteStartupHandshakeAsync(cancellationToken).ConfigureAwait(false);
|
await session.RunAsync(cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Task ConnectAsync(
|
private Task ConnectAsync(
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
|
|||||||
using Google.Protobuf.WellKnownTypes;
|
using Google.Protobuf.WellKnownTypes;
|
||||||
using MxGateway.Contracts.Proto;
|
using MxGateway.Contracts.Proto;
|
||||||
using MxGateway.Worker.MxAccess;
|
using MxGateway.Worker.MxAccess;
|
||||||
|
using MxGateway.Worker.Sta;
|
||||||
|
|
||||||
namespace MxGateway.Worker.Ipc;
|
namespace MxGateway.Worker.Ipc;
|
||||||
|
|
||||||
@@ -13,10 +14,14 @@ public sealed class WorkerPipeSession
|
|||||||
{
|
{
|
||||||
private readonly WorkerFrameProtocolOptions _options;
|
private readonly WorkerFrameProtocolOptions _options;
|
||||||
private readonly Func<int> _processIdProvider;
|
private readonly Func<int> _processIdProvider;
|
||||||
|
private readonly Func<IWorkerRuntimeSession> _runtimeSessionFactory;
|
||||||
|
private readonly WorkerPipeSessionOptions _sessionOptions;
|
||||||
private readonly WorkerFrameReader _reader;
|
private readonly WorkerFrameReader _reader;
|
||||||
private readonly WorkerFrameWriter _writer;
|
private readonly WorkerFrameWriter _writer;
|
||||||
private MxAccessStaSession? _mxAccessStaSession;
|
private IWorkerRuntimeSession? _runtimeSession;
|
||||||
private long _nextSequence;
|
private long _nextSequence;
|
||||||
|
private WorkerState _state = WorkerState.Starting;
|
||||||
|
private bool _watchdogFaultSent;
|
||||||
|
|
||||||
public WorkerPipeSession(
|
public WorkerPipeSession(
|
||||||
Stream stream,
|
Stream stream,
|
||||||
@@ -34,11 +39,49 @@ public sealed class WorkerPipeSession
|
|||||||
WorkerFrameWriter writer,
|
WorkerFrameWriter writer,
|
||||||
WorkerFrameProtocolOptions options,
|
WorkerFrameProtocolOptions options,
|
||||||
Func<int> processIdProvider)
|
Func<int> processIdProvider)
|
||||||
|
: this(
|
||||||
|
reader,
|
||||||
|
writer,
|
||||||
|
options,
|
||||||
|
processIdProvider,
|
||||||
|
new WorkerPipeSessionOptions(),
|
||||||
|
() => new MxAccessStaSession())
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
public WorkerPipeSession(
|
||||||
|
WorkerFrameReader reader,
|
||||||
|
WorkerFrameWriter writer,
|
||||||
|
WorkerFrameProtocolOptions options,
|
||||||
|
Func<int> processIdProvider,
|
||||||
|
WorkerPipeSessionOptions sessionOptions,
|
||||||
|
Func<IWorkerRuntimeSession> runtimeSessionFactory)
|
||||||
{
|
{
|
||||||
_reader = reader ?? throw new ArgumentNullException(nameof(reader));
|
_reader = reader ?? throw new ArgumentNullException(nameof(reader));
|
||||||
_writer = writer ?? throw new ArgumentNullException(nameof(writer));
|
_writer = writer ?? throw new ArgumentNullException(nameof(writer));
|
||||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||||
_processIdProvider = processIdProvider ?? throw new ArgumentNullException(nameof(processIdProvider));
|
_processIdProvider = processIdProvider ?? throw new ArgumentNullException(nameof(processIdProvider));
|
||||||
|
_sessionOptions = sessionOptions ?? throw new ArgumentNullException(nameof(sessionOptions));
|
||||||
|
_runtimeSessionFactory = runtimeSessionFactory ?? throw new ArgumentNullException(nameof(runtimeSessionFactory));
|
||||||
|
_sessionOptions.Validate();
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task RunAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
_runtimeSession = _runtimeSessionFactory();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await CompleteStartupHandshakeAsync(
|
||||||
|
token => _runtimeSession.StartAsync(_processIdProvider(), token),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
await RunMessageLoopAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_runtimeSession?.Dispose();
|
||||||
|
_runtimeSession = null;
|
||||||
|
_state = WorkerState.Stopped;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task CompleteStartupHandshakeAsync(CancellationToken cancellationToken = default)
|
public Task CompleteStartupHandshakeAsync(CancellationToken cancellationToken = default)
|
||||||
@@ -76,11 +119,14 @@ public sealed class WorkerPipeSession
|
|||||||
try
|
try
|
||||||
{
|
{
|
||||||
WorkerEnvelope envelope = await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
WorkerEnvelope envelope = await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
_state = WorkerState.Handshaking;
|
||||||
ValidateGatewayHello(envelope);
|
ValidateGatewayHello(envelope);
|
||||||
|
|
||||||
await WriteWorkerHelloAsync(cancellationToken).ConfigureAwait(false);
|
await WriteWorkerHelloAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
_state = WorkerState.InitializingSta;
|
||||||
WorkerReady ready = await initializeMxAccessAsync(cancellationToken).ConfigureAwait(false);
|
WorkerReady ready = await initializeMxAccessAsync(cancellationToken).ConfigureAwait(false);
|
||||||
await WriteWorkerReadyAsync(ready, cancellationToken).ConfigureAwait(false);
|
await WriteWorkerReadyAsync(ready, cancellationToken).ConfigureAwait(false);
|
||||||
|
_state = WorkerState.Ready;
|
||||||
}
|
}
|
||||||
catch (WorkerFrameProtocolException exception)
|
catch (WorkerFrameProtocolException exception)
|
||||||
{
|
{
|
||||||
@@ -140,6 +186,174 @@ public sealed class WorkerPipeSession
|
|||||||
return _writer.WriteAsync(CreateEnvelope(ready), cancellationToken);
|
return _writer.WriteAsync(CreateEnvelope(ready), cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async Task RunMessageLoopAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
using CancellationTokenSource heartbeatCancellation = CancellationTokenSource
|
||||||
|
.CreateLinkedTokenSource(cancellationToken);
|
||||||
|
Task heartbeatTask = RunHeartbeatLoopAsync(heartbeatCancellation.Token);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
while (!cancellationToken.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
Task<WorkerEnvelope> readTask = _reader.ReadAsync(cancellationToken);
|
||||||
|
Task completedTask = await Task.WhenAny(readTask, heartbeatTask).ConfigureAwait(false);
|
||||||
|
if (completedTask == heartbeatTask)
|
||||||
|
{
|
||||||
|
await heartbeatTask.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
WorkerEnvelope envelope = await readTask.ConfigureAwait(false);
|
||||||
|
bool keepReading = await DispatchGatewayEnvelopeAsync(envelope, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (!keepReading)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
heartbeatCancellation.Cancel();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await heartbeatTask.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<bool> DispatchGatewayEnvelopeAsync(
|
||||||
|
WorkerEnvelope envelope,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
switch (envelope.BodyCase)
|
||||||
|
{
|
||||||
|
case WorkerEnvelope.BodyOneofCase.WorkerCommand:
|
||||||
|
_ = ProcessCommandAsync(envelope, cancellationToken);
|
||||||
|
return true;
|
||||||
|
case WorkerEnvelope.BodyOneofCase.WorkerShutdown:
|
||||||
|
await ShutdownAsync(envelope.WorkerShutdown, cancellationToken).ConfigureAwait(false);
|
||||||
|
return false;
|
||||||
|
case WorkerEnvelope.BodyOneofCase.WorkerCancel:
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
throw new WorkerFrameProtocolException(
|
||||||
|
WorkerFrameProtocolErrorCode.UnexpectedEnvelopeBody,
|
||||||
|
$"Worker received unexpected gateway envelope body {envelope.BodyCase}.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task ProcessCommandAsync(
|
||||||
|
WorkerEnvelope envelope,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
IWorkerRuntimeSession runtimeSession = _runtimeSession
|
||||||
|
?? throw new InvalidOperationException("Worker runtime session has not been initialized.");
|
||||||
|
WorkerCommand workerCommand = envelope.WorkerCommand;
|
||||||
|
MxCommand command = workerCommand.Command;
|
||||||
|
StaCommand staCommand = new(
|
||||||
|
_options.SessionId,
|
||||||
|
envelope.CorrelationId,
|
||||||
|
command,
|
||||||
|
workerCommand.EnqueueTimestamp,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
MxCommandReply reply = await runtimeSession.DispatchAsync(staCommand).ConfigureAwait(false);
|
||||||
|
await _writer
|
||||||
|
.WriteAsync(
|
||||||
|
CreateEnvelope(new WorkerCommandReply
|
||||||
|
{
|
||||||
|
Reply = reply,
|
||||||
|
CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
|
||||||
|
}),
|
||||||
|
cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||||
|
{
|
||||||
|
_state = WorkerState.Faulted;
|
||||||
|
await TryWriteFaultAsync(
|
||||||
|
CreateFault(
|
||||||
|
WorkerFaultCategory.MxaccessCommandFailed,
|
||||||
|
staCommand.MethodName,
|
||||||
|
exception),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task ShutdownAsync(
|
||||||
|
WorkerShutdown shutdown,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
_state = WorkerState.ShuttingDown;
|
||||||
|
_runtimeSession?.RequestShutdown();
|
||||||
|
|
||||||
|
await _writer
|
||||||
|
.WriteAsync(
|
||||||
|
CreateEnvelope(
|
||||||
|
new WorkerShutdownAck
|
||||||
|
{
|
||||||
|
Status = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.Ok,
|
||||||
|
Message = string.IsNullOrWhiteSpace(shutdown.Reason)
|
||||||
|
? "Worker shutdown accepted."
|
||||||
|
: $"Worker shutdown accepted: {shutdown.Reason}",
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task RunHeartbeatLoopAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
while (!cancellationToken.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
await Task.Delay(_sessionOptions.HeartbeatInterval, cancellationToken).ConfigureAwait(false);
|
||||||
|
IWorkerRuntimeSession? runtimeSession = _runtimeSession;
|
||||||
|
if (runtimeSession is null)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
WorkerRuntimeHeartbeatSnapshot snapshot = runtimeSession.CaptureHeartbeat();
|
||||||
|
await _writer
|
||||||
|
.WriteAsync(CreateEnvelope(CreateHeartbeat(snapshot)), cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
|
await ReportWatchdogFaultIfNeededAsync(snapshot, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task ReportWatchdogFaultIfNeededAsync(
|
||||||
|
WorkerRuntimeHeartbeatSnapshot snapshot,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
TimeSpan staleFor = DateTimeOffset.UtcNow - snapshot.LastStaActivityUtc;
|
||||||
|
if (staleFor <= _sessionOptions.HeartbeatGrace)
|
||||||
|
{
|
||||||
|
_watchdogFaultSent = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (_watchdogFaultSent)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
_watchdogFaultSent = true;
|
||||||
|
await TryWriteFaultAsync(
|
||||||
|
CreateFault(
|
||||||
|
WorkerFaultCategory.StaHung,
|
||||||
|
snapshot.CurrentCommandCorrelationId,
|
||||||
|
$"STA activity is stale by {staleFor}."),
|
||||||
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
private async Task TryWriteFaultAsync(
|
private async Task TryWriteFaultAsync(
|
||||||
WorkerFrameProtocolException exception,
|
WorkerFrameProtocolException exception,
|
||||||
CancellationToken cancellationToken)
|
CancellationToken cancellationToken)
|
||||||
@@ -178,6 +392,25 @@ public sealed class WorkerPipeSession
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async Task TryWriteFaultAsync(
|
||||||
|
WorkerFault fault,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await _writer
|
||||||
|
.WriteAsync(CreateEnvelope(fault), cancellationToken)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception faultWriteException) when (
|
||||||
|
faultWriteException is IOException
|
||||||
|
|| faultWriteException is ObjectDisposedException
|
||||||
|
|| faultWriteException is WorkerFrameProtocolException)
|
||||||
|
{
|
||||||
|
// The runtime fault remains observable through worker exit or pipe closure.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private WorkerEnvelope CreateEnvelope(WorkerHello hello)
|
private WorkerEnvelope CreateEnvelope(WorkerHello hello)
|
||||||
{
|
{
|
||||||
return CreateBaseEnvelope(hello);
|
return CreateBaseEnvelope(hello);
|
||||||
@@ -193,6 +426,21 @@ public sealed class WorkerPipeSession
|
|||||||
return CreateBaseEnvelope(fault);
|
return CreateBaseEnvelope(fault);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateEnvelope(WorkerCommandReply reply)
|
||||||
|
{
|
||||||
|
return CreateBaseEnvelope(reply);
|
||||||
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateEnvelope(WorkerShutdownAck shutdownAck)
|
||||||
|
{
|
||||||
|
return CreateBaseEnvelope(shutdownAck);
|
||||||
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateEnvelope(WorkerHeartbeat heartbeat)
|
||||||
|
{
|
||||||
|
return CreateBaseEnvelope(heartbeat);
|
||||||
|
}
|
||||||
|
|
||||||
private WorkerEnvelope CreateBaseEnvelope(WorkerHello body)
|
private WorkerEnvelope CreateBaseEnvelope(WorkerHello body)
|
||||||
{
|
{
|
||||||
WorkerEnvelope envelope = CreateBaseEnvelope();
|
WorkerEnvelope envelope = CreateBaseEnvelope();
|
||||||
@@ -214,6 +462,28 @@ public sealed class WorkerPipeSession
|
|||||||
return envelope;
|
return envelope;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateBaseEnvelope(WorkerCommandReply body)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = CreateBaseEnvelope();
|
||||||
|
envelope.CorrelationId = body.Reply?.CorrelationId ?? string.Empty;
|
||||||
|
envelope.WorkerCommandReply = body;
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateBaseEnvelope(WorkerShutdownAck body)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = CreateBaseEnvelope();
|
||||||
|
envelope.WorkerShutdownAck = body;
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
private WorkerEnvelope CreateBaseEnvelope(WorkerHeartbeat body)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = CreateBaseEnvelope();
|
||||||
|
envelope.WorkerHeartbeat = body;
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
private WorkerEnvelope CreateBaseEnvelope()
|
private WorkerEnvelope CreateBaseEnvelope()
|
||||||
{
|
{
|
||||||
return new WorkerEnvelope
|
return new WorkerEnvelope
|
||||||
@@ -231,21 +501,39 @@ public sealed class WorkerPipeSession
|
|||||||
|
|
||||||
private async Task<WorkerReady> InitializeMxAccessAsync(CancellationToken cancellationToken)
|
private async Task<WorkerReady> InitializeMxAccessAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
_mxAccessStaSession = new MxAccessStaSession();
|
_runtimeSession = new MxAccessStaSession();
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
return await _mxAccessStaSession
|
return await _runtimeSession
|
||||||
.StartAsync(_processIdProvider(), cancellationToken)
|
.StartAsync(_processIdProvider(), cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch
|
catch
|
||||||
{
|
{
|
||||||
_mxAccessStaSession.Dispose();
|
_runtimeSession.Dispose();
|
||||||
_mxAccessStaSession = null;
|
_runtimeSession = null;
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private WorkerHeartbeat CreateHeartbeat(WorkerRuntimeHeartbeatSnapshot snapshot)
|
||||||
|
{
|
||||||
|
WorkerState state = string.IsNullOrWhiteSpace(snapshot.CurrentCommandCorrelationId)
|
||||||
|
? _state
|
||||||
|
: WorkerState.ExecutingCommand;
|
||||||
|
|
||||||
|
return new WorkerHeartbeat
|
||||||
|
{
|
||||||
|
WorkerProcessId = _processIdProvider(),
|
||||||
|
State = state,
|
||||||
|
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(snapshot.LastStaActivityUtc),
|
||||||
|
PendingCommandCount = snapshot.PendingCommandCount,
|
||||||
|
OutboundEventQueueDepth = snapshot.OutboundEventQueueDepth,
|
||||||
|
LastEventSequence = snapshot.LastEventSequence,
|
||||||
|
CurrentCommandCorrelationId = snapshot.CurrentCommandCorrelationId,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
private WorkerReady CreateWorkerReady()
|
private WorkerReady CreateWorkerReady()
|
||||||
{
|
{
|
||||||
return new WorkerReady
|
return new WorkerReady
|
||||||
@@ -295,6 +583,42 @@ public sealed class WorkerPipeSession
|
|||||||
return fault;
|
return fault;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerFault CreateFault(
|
||||||
|
WorkerFaultCategory category,
|
||||||
|
string commandMethod,
|
||||||
|
Exception exception)
|
||||||
|
{
|
||||||
|
WorkerFault fault = CreateFault(
|
||||||
|
category,
|
||||||
|
commandMethod,
|
||||||
|
exception.Message);
|
||||||
|
fault.ExceptionType = exception.GetType().FullName ?? string.Empty;
|
||||||
|
fault.ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.WorkerUnavailable,
|
||||||
|
Message = exception.Message,
|
||||||
|
};
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerFault CreateFault(
|
||||||
|
WorkerFaultCategory category,
|
||||||
|
string commandMethod,
|
||||||
|
string diagnosticMessage)
|
||||||
|
{
|
||||||
|
return new WorkerFault
|
||||||
|
{
|
||||||
|
Category = category,
|
||||||
|
CommandMethod = commandMethod ?? string.Empty,
|
||||||
|
DiagnosticMessage = diagnosticMessage,
|
||||||
|
ProtocolStatus = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.WorkerUnavailable,
|
||||||
|
Message = diagnosticMessage,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerFaultCategory MapFaultCategory(WorkerFrameProtocolErrorCode errorCode)
|
private static WorkerFaultCategory MapFaultCategory(WorkerFrameProtocolErrorCode errorCode)
|
||||||
{
|
{
|
||||||
return errorCode switch
|
return errorCode switch
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
using System;
|
||||||
|
|
||||||
|
namespace MxGateway.Worker.Ipc;
|
||||||
|
|
||||||
|
public sealed class WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
public static readonly TimeSpan DefaultHeartbeatInterval = TimeSpan.FromSeconds(5);
|
||||||
|
public static readonly TimeSpan DefaultHeartbeatGrace = TimeSpan.FromSeconds(15);
|
||||||
|
|
||||||
|
public WorkerPipeSessionOptions()
|
||||||
|
{
|
||||||
|
HeartbeatInterval = DefaultHeartbeatInterval;
|
||||||
|
HeartbeatGrace = DefaultHeartbeatGrace;
|
||||||
|
}
|
||||||
|
|
||||||
|
public TimeSpan HeartbeatInterval { get; set; }
|
||||||
|
|
||||||
|
public TimeSpan HeartbeatGrace { get; set; }
|
||||||
|
|
||||||
|
public void Validate()
|
||||||
|
{
|
||||||
|
if (HeartbeatInterval <= TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new ArgumentOutOfRangeException(
|
||||||
|
nameof(HeartbeatInterval),
|
||||||
|
"Worker heartbeat interval must be greater than zero.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (HeartbeatGrace <= TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new ArgumentOutOfRangeException(
|
||||||
|
nameof(HeartbeatGrace),
|
||||||
|
"Worker heartbeat grace must be greater than zero.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,4 +5,17 @@ public interface IMxAccessServer
|
|||||||
int Register(string clientName);
|
int Register(string clientName);
|
||||||
|
|
||||||
void Unregister(int serverHandle);
|
void Unregister(int serverHandle);
|
||||||
|
|
||||||
|
int AddItem(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition);
|
||||||
|
|
||||||
|
int AddItem2(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext);
|
||||||
|
|
||||||
|
void RemoveItem(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
using System;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using MxGateway.Contracts.Proto;
|
||||||
|
using MxGateway.Worker.Sta;
|
||||||
|
|
||||||
|
namespace MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
|
public interface IWorkerRuntimeSession : IDisposable
|
||||||
|
{
|
||||||
|
Task<WorkerReady> StartAsync(
|
||||||
|
int workerProcessId,
|
||||||
|
CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
Task<MxCommandReply> DispatchAsync(StaCommand command);
|
||||||
|
|
||||||
|
WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat();
|
||||||
|
|
||||||
|
void RequestShutdown();
|
||||||
|
}
|
||||||
@@ -35,6 +35,44 @@ public sealed class MxAccessComServer : IMxAccessServer
|
|||||||
Invoke(nameof(Unregister), serverHandle);
|
Invoke(nameof(Unregister), serverHandle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public int AddItem(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition)
|
||||||
|
{
|
||||||
|
if (mxAccessComObject is ILMXProxyServer mxAccessServer)
|
||||||
|
{
|
||||||
|
return mxAccessServer.AddItem(serverHandle, itemDefinition);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (int)Invoke(nameof(AddItem), serverHandle, itemDefinition);
|
||||||
|
}
|
||||||
|
|
||||||
|
public int AddItem2(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext)
|
||||||
|
{
|
||||||
|
if (mxAccessComObject is ILMXProxyServer3 mxAccessServer)
|
||||||
|
{
|
||||||
|
return mxAccessServer.AddItem2(serverHandle, itemDefinition, itemContext);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (int)Invoke(nameof(AddItem2), serverHandle, itemDefinition, itemContext);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RemoveItem(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
if (mxAccessComObject is ILMXProxyServer mxAccessServer)
|
||||||
|
{
|
||||||
|
mxAccessServer.RemoveItem(serverHandle, itemHandle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
Invoke(nameof(RemoveItem), serverHandle, itemHandle);
|
||||||
|
}
|
||||||
|
|
||||||
private object Invoke(
|
private object Invoke(
|
||||||
string methodName,
|
string methodName,
|
||||||
params object[] arguments)
|
params object[] arguments)
|
||||||
|
|||||||
@@ -34,6 +34,9 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
|||||||
{
|
{
|
||||||
MxCommandKind.Register => ExecuteRegister(command),
|
MxCommandKind.Register => ExecuteRegister(command),
|
||||||
MxCommandKind.Unregister => ExecuteUnregister(command),
|
MxCommandKind.Unregister => ExecuteUnregister(command),
|
||||||
|
MxCommandKind.AddItem => ExecuteAddItem(command),
|
||||||
|
MxCommandKind.AddItem2 => ExecuteAddItem2(command),
|
||||||
|
MxCommandKind.RemoveItem => ExecuteRemoveItem(command),
|
||||||
_ => CreateInvalidRequestReply(command, $"Unsupported MXAccess command kind {command.Kind}."),
|
_ => CreateInvalidRequestReply(command, $"Unsupported MXAccess command kind {command.Kind}."),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -67,6 +70,66 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
|||||||
return CreateOkReply(command);
|
return CreateOkReply(command);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private MxCommandReply ExecuteAddItem(StaCommand command)
|
||||||
|
{
|
||||||
|
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.AddItem)
|
||||||
|
{
|
||||||
|
return CreateInvalidRequestReply(command, "AddItem command payload is required.");
|
||||||
|
}
|
||||||
|
|
||||||
|
AddItemCommand addItemCommand = command.Command.AddItem;
|
||||||
|
int itemHandle = session.AddItem(
|
||||||
|
addItemCommand.ServerHandle,
|
||||||
|
addItemCommand.ItemDefinition);
|
||||||
|
|
||||||
|
MxCommandReply reply = CreateOkReply(command);
|
||||||
|
reply.ReturnValue = variantConverter.Convert(itemHandle);
|
||||||
|
reply.AddItem = new AddItemReply
|
||||||
|
{
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
};
|
||||||
|
|
||||||
|
return reply;
|
||||||
|
}
|
||||||
|
|
||||||
|
private MxCommandReply ExecuteAddItem2(StaCommand command)
|
||||||
|
{
|
||||||
|
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.AddItem2)
|
||||||
|
{
|
||||||
|
return CreateInvalidRequestReply(command, "AddItem2 command payload is required.");
|
||||||
|
}
|
||||||
|
|
||||||
|
AddItem2Command addItem2Command = command.Command.AddItem2;
|
||||||
|
int itemHandle = session.AddItem2(
|
||||||
|
addItem2Command.ServerHandle,
|
||||||
|
addItem2Command.ItemDefinition,
|
||||||
|
addItem2Command.ItemContext);
|
||||||
|
|
||||||
|
MxCommandReply reply = CreateOkReply(command);
|
||||||
|
reply.ReturnValue = variantConverter.Convert(itemHandle);
|
||||||
|
reply.AddItem2 = new AddItem2Reply
|
||||||
|
{
|
||||||
|
ItemHandle = itemHandle,
|
||||||
|
};
|
||||||
|
|
||||||
|
return reply;
|
||||||
|
}
|
||||||
|
|
||||||
|
private MxCommandReply ExecuteRemoveItem(StaCommand command)
|
||||||
|
{
|
||||||
|
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.RemoveItem)
|
||||||
|
{
|
||||||
|
return CreateInvalidRequestReply(command, "RemoveItem command payload is required.");
|
||||||
|
}
|
||||||
|
|
||||||
|
RemoveItemCommand removeItemCommand = command.Command.RemoveItem;
|
||||||
|
session.RemoveItem(
|
||||||
|
removeItemCommand.ServerHandle,
|
||||||
|
removeItemCommand.ItemHandle);
|
||||||
|
|
||||||
|
return CreateOkReply(command);
|
||||||
|
}
|
||||||
|
|
||||||
private static MxCommandReply CreateOkReply(StaCommand command)
|
private static MxCommandReply CreateOkReply(StaCommand command)
|
||||||
{
|
{
|
||||||
return new MxCommandReply
|
return new MxCommandReply
|
||||||
|
|||||||
@@ -6,12 +6,19 @@ namespace MxGateway.Worker.MxAccess;
|
|||||||
public sealed class MxAccessHandleRegistry
|
public sealed class MxAccessHandleRegistry
|
||||||
{
|
{
|
||||||
private readonly Dictionary<int, RegisteredServerHandle> serverHandles = new();
|
private readonly Dictionary<int, RegisteredServerHandle> serverHandles = new();
|
||||||
|
private readonly Dictionary<long, RegisteredItemHandle> itemHandles = new();
|
||||||
|
|
||||||
public IReadOnlyList<RegisteredServerHandle> ServerHandles => serverHandles
|
public IReadOnlyList<RegisteredServerHandle> ServerHandles => serverHandles
|
||||||
.Values
|
.Values
|
||||||
.OrderBy(handle => handle.ServerHandle)
|
.OrderBy(handle => handle.ServerHandle)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
|
public IReadOnlyList<RegisteredItemHandle> ItemHandles => itemHandles
|
||||||
|
.Values
|
||||||
|
.OrderBy(handle => handle.ServerHandle)
|
||||||
|
.ThenBy(handle => handle.ItemHandle)
|
||||||
|
.ToArray();
|
||||||
|
|
||||||
public void RegisterServerHandle(
|
public void RegisterServerHandle(
|
||||||
int serverHandle,
|
int serverHandle,
|
||||||
string clientName)
|
string clientName)
|
||||||
@@ -22,10 +29,54 @@ public sealed class MxAccessHandleRegistry
|
|||||||
public void UnregisterServerHandle(int serverHandle)
|
public void UnregisterServerHandle(int serverHandle)
|
||||||
{
|
{
|
||||||
serverHandles.Remove(serverHandle);
|
serverHandles.Remove(serverHandle);
|
||||||
|
|
||||||
|
foreach (long key in itemHandles
|
||||||
|
.Where(pair => pair.Value.ServerHandle == serverHandle)
|
||||||
|
.Select(pair => pair.Key)
|
||||||
|
.ToArray())
|
||||||
|
{
|
||||||
|
itemHandles.Remove(key);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public bool ContainsServerHandle(int serverHandle)
|
public bool ContainsServerHandle(int serverHandle)
|
||||||
{
|
{
|
||||||
return serverHandles.ContainsKey(serverHandle);
|
return serverHandles.ContainsKey(serverHandle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public void RegisterItemHandle(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext,
|
||||||
|
bool hasItemContext)
|
||||||
|
{
|
||||||
|
itemHandles[CreateItemKey(serverHandle, itemHandle)] = new RegisteredItemHandle(
|
||||||
|
serverHandle,
|
||||||
|
itemHandle,
|
||||||
|
itemDefinition,
|
||||||
|
itemContext,
|
||||||
|
hasItemContext);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RemoveItemHandle(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
itemHandles.Remove(CreateItemKey(serverHandle, itemHandle));
|
||||||
|
}
|
||||||
|
|
||||||
|
public bool ContainsItemHandle(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
return itemHandles.ContainsKey(CreateItemKey(serverHandle, itemHandle));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static long CreateItemKey(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
return ((long)serverHandle << 32) | (uint)itemHandle;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -106,6 +106,51 @@ public sealed class MxAccessSession : IDisposable
|
|||||||
handleRegistry.UnregisterServerHandle(serverHandle);
|
handleRegistry.UnregisterServerHandle(serverHandle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public int AddItem(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition)
|
||||||
|
{
|
||||||
|
ThrowIfDisposed();
|
||||||
|
|
||||||
|
int itemHandle = mxAccessServer.AddItem(serverHandle, itemDefinition);
|
||||||
|
handleRegistry.RegisterItemHandle(
|
||||||
|
serverHandle,
|
||||||
|
itemHandle,
|
||||||
|
itemDefinition,
|
||||||
|
string.Empty,
|
||||||
|
hasItemContext: false);
|
||||||
|
|
||||||
|
return itemHandle;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int AddItem2(
|
||||||
|
int serverHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext)
|
||||||
|
{
|
||||||
|
ThrowIfDisposed();
|
||||||
|
|
||||||
|
int itemHandle = mxAccessServer.AddItem2(serverHandle, itemDefinition, itemContext);
|
||||||
|
handleRegistry.RegisterItemHandle(
|
||||||
|
serverHandle,
|
||||||
|
itemHandle,
|
||||||
|
itemDefinition,
|
||||||
|
itemContext,
|
||||||
|
hasItemContext: true);
|
||||||
|
|
||||||
|
return itemHandle;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RemoveItem(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle)
|
||||||
|
{
|
||||||
|
ThrowIfDisposed();
|
||||||
|
|
||||||
|
mxAccessServer.RemoveItem(serverHandle, itemHandle);
|
||||||
|
handleRegistry.RemoveItemHandle(serverHandle, itemHandle);
|
||||||
|
}
|
||||||
|
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
if (disposed)
|
if (disposed)
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ using MxGateway.Worker.Sta;
|
|||||||
|
|
||||||
namespace MxGateway.Worker.MxAccess;
|
namespace MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
public sealed class MxAccessStaSession : IDisposable
|
public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
||||||
{
|
{
|
||||||
private readonly IMxAccessComObjectFactory factory;
|
private readonly IMxAccessComObjectFactory factory;
|
||||||
private readonly IMxAccessEventSink eventSink;
|
private readonly IMxAccessEventSink eventSink;
|
||||||
@@ -68,6 +68,30 @@ public sealed class MxAccessStaSession : IDisposable
|
|||||||
return commandDispatcher.DispatchAsync(command);
|
return commandDispatcher.DispatchAsync(command);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
|
||||||
|
{
|
||||||
|
uint pendingCommandCount = 0;
|
||||||
|
string currentCommandCorrelationId = string.Empty;
|
||||||
|
|
||||||
|
if (commandDispatcher is not null)
|
||||||
|
{
|
||||||
|
pendingCommandCount = (uint)commandDispatcher.PendingCommandCount;
|
||||||
|
currentCommandCorrelationId = commandDispatcher.CurrentCommandCorrelationId;
|
||||||
|
}
|
||||||
|
|
||||||
|
return new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
staRuntime.LastActivityUtc,
|
||||||
|
pendingCommandCount,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void RequestShutdown()
|
||||||
|
{
|
||||||
|
commandDispatcher?.RequestShutdown();
|
||||||
|
}
|
||||||
|
|
||||||
public Task<IReadOnlyList<RegisteredServerHandle>> GetRegisteredServerHandlesAsync(
|
public Task<IReadOnlyList<RegisteredServerHandle>> GetRegisteredServerHandlesAsync(
|
||||||
CancellationToken cancellationToken = default)
|
CancellationToken cancellationToken = default)
|
||||||
{
|
{
|
||||||
@@ -81,6 +105,19 @@ public sealed class MxAccessStaSession : IDisposable
|
|||||||
cancellationToken);
|
cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public Task<IReadOnlyList<RegisteredItemHandle>> GetRegisteredItemHandlesAsync(
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (session is null)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("MXAccess COM session has not been started.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return staRuntime.InvokeAsync(
|
||||||
|
() => session.HandleRegistry.ItemHandles,
|
||||||
|
cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
if (disposed)
|
if (disposed)
|
||||||
@@ -88,7 +125,7 @@ public sealed class MxAccessStaSession : IDisposable
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
commandDispatcher?.RequestShutdown();
|
RequestShutdown();
|
||||||
|
|
||||||
if (session is not null)
|
if (session is not null)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
namespace MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
|
public sealed class RegisteredItemHandle
|
||||||
|
{
|
||||||
|
public RegisteredItemHandle(
|
||||||
|
int serverHandle,
|
||||||
|
int itemHandle,
|
||||||
|
string itemDefinition,
|
||||||
|
string itemContext,
|
||||||
|
bool hasItemContext)
|
||||||
|
{
|
||||||
|
ServerHandle = serverHandle;
|
||||||
|
ItemHandle = itemHandle;
|
||||||
|
ItemDefinition = itemDefinition;
|
||||||
|
ItemContext = itemContext;
|
||||||
|
HasItemContext = hasItemContext;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int ServerHandle { get; }
|
||||||
|
|
||||||
|
public int ItemHandle { get; }
|
||||||
|
|
||||||
|
public string ItemDefinition { get; }
|
||||||
|
|
||||||
|
public string ItemContext { get; }
|
||||||
|
|
||||||
|
public bool HasItemContext { get; }
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
using System;
|
||||||
|
|
||||||
|
namespace MxGateway.Worker.MxAccess;
|
||||||
|
|
||||||
|
public sealed class WorkerRuntimeHeartbeatSnapshot
|
||||||
|
{
|
||||||
|
public WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset lastStaActivityUtc,
|
||||||
|
uint pendingCommandCount,
|
||||||
|
uint outboundEventQueueDepth,
|
||||||
|
ulong lastEventSequence,
|
||||||
|
string currentCommandCorrelationId)
|
||||||
|
{
|
||||||
|
LastStaActivityUtc = lastStaActivityUtc;
|
||||||
|
PendingCommandCount = pendingCommandCount;
|
||||||
|
OutboundEventQueueDepth = outboundEventQueueDepth;
|
||||||
|
LastEventSequence = lastEventSequence;
|
||||||
|
CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
public DateTimeOffset LastStaActivityUtc { get; }
|
||||||
|
|
||||||
|
public uint PendingCommandCount { get; }
|
||||||
|
|
||||||
|
public uint OutboundEventQueueDepth { get; }
|
||||||
|
|
||||||
|
public ulong LastEventSequence { get; }
|
||||||
|
|
||||||
|
public string CurrentCommandCorrelationId { get; }
|
||||||
|
}
|
||||||
@@ -84,7 +84,7 @@ public static class WorkerApplication
|
|||||||
|
|
||||||
pipeClient.RunAsync(options).GetAwaiter().GetResult();
|
pipeClient.RunAsync(options).GetAwaiter().GetResult();
|
||||||
|
|
||||||
logger.Information("WorkerPipeHandshakeSucceeded", new Dictionary<string, object?>
|
logger.Information("WorkerPipeSessionCompleted", new Dictionary<string, object?>
|
||||||
{
|
{
|
||||||
["session_id"] = options.SessionId,
|
["session_id"] = options.SessionId,
|
||||||
["pipe_name"] = options.PipeName,
|
["pipe_name"] = options.PipeName,
|
||||||
|
|||||||
Reference in New Issue
Block a user