Compare commits

..

8 Commits

Author SHA1 Message Date
Joseph Doherty 4a3560c7ee Implement worker heartbeat watchdog 2026-04-26 19:12:06 -04:00
dohertj2 a3ccd5c80b Merge pull request #79 from agent-3/issue-19-gateway-e2e-smoke-with-fake-worker
Issue #19: gateway end-to-end smoke with fake worker
2026-04-26 18:34:54 -04:00
dohertj2 0fd954d94c Merge pull request #78 from agent-2/issue-27-implement-additem-additem2-removeitem
Issue #27: implement AddItem, AddItem2, RemoveItem
2026-04-26 18:32:13 -04:00
Joseph Doherty 91f2d8dc14 Merge remote-tracking branch 'origin/main' into agent-3/issue-19-gateway-e2e-smoke-with-fake-worker 2026-04-26 18:31:49 -04:00
Joseph Doherty fb425da009 Add gateway fake worker end-to-end smoke 2026-04-26 18:30:11 -04:00
Joseph Doherty c7e4c4b614 Merge remote-tracking branch 'origin/main' into agent-2/issue-27-implement-additem-additem2-removeitem 2026-04-26 18:27:00 -04:00
Joseph Doherty 59c710d789 Implement worker AddItem commands 2026-04-26 18:26:44 -04:00
dohertj2 862f119b91 Merge pull request #77 from agent-3/issue-18-build-fake-worker-test-harness
Issue #18: build fake worker test harness
2026-04-26 18:25:00 -04:00
25 changed files with 2324 additions and 41 deletions
+10
View File
@@ -18,6 +18,7 @@ starting `MxGateway.Worker.exe` or loading MXAccess COM. The harness scripts:
- `WorkerHello` and `WorkerReady` startup,
- command replies with matching correlation ids,
- ordered `WorkerEvent` frames,
- `WorkerHeartbeat` frames,
- `WorkerFault` frames,
- shutdown acknowledgements,
- malformed protobuf payloads and oversized frame headers,
@@ -28,6 +29,12 @@ Session-level tests can connect the harness to the pipe created by
`WorkerClient` tests can use `CreateConnectedPairAsync` to create both pipe ends
inside the test.
`GatewayEndToEndFakeWorkerSmokeTests` composes the real gRPC service,
`SessionManager`, `SessionWorkerClientFactory`, `WorkerClient`, and
`EventStreamService` with a scripted fake worker launcher. The smoke test covers
`OpenSession`, `Register`, `AddItem`, `Advise`, one streamed `OnDataChange`
event, and `CloseSession` without loading MXAccess COM.
## Focused Commands
Run the fake worker tests after changing gateway worker IPC, session startup, or
@@ -36,6 +43,9 @@ event streaming behavior:
```bash
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~FakeWorkerHarnessTests
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~SessionWorkerClientFactoryFakeWorkerTests
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndFakeWorkerSmokeTests
dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests
dotnet test src/MxGateway.Worker.Tests/MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter FullyQualifiedName~WorkerPipeSessionTests
```
Run the gateway test project after shared gateway test infrastructure changes:
@@ -218,6 +218,8 @@ Live tests:
Labels: `area:worker`, `type:feature`, `priority:p0`
Status: implemented.
Deliverables:
- `AddItem`,
+45 -8
View File
@@ -432,6 +432,25 @@ HRESULT and converts the reply to `ProtocolStatusCode.MxaccessFailure`.
`MxAccessStaSession.GetRegisteredServerHandlesAsync` returns an STA-read
snapshot of tracked server handles for diagnostics and future cleanup logic.
`MxAccessCommandExecutor` also implements the item lifecycle commands:
- `AddItem` calls `LMXProxyServerClass.AddItem` with the requested server
handle and item definition. It preserves the returned item handle in both
`ReturnValue` and `AddItemReply.ItemHandle`.
- `AddItem2` calls `LMXProxyServerClass.AddItem2` with the requested server
handle, item definition, and context string. The context string is passed to
MXAccess exactly as received.
- `RemoveItem` calls `LMXProxyServerClass.RemoveItem` with the requested server
handle and item handle. The reply has no method-specific payload because the
public MXAccess method returns `void`.
The worker records item handles only after `AddItem` or `AddItem2` returns
normally, and removes item handles only after `RemoveItem` returns normally.
The registry does not prevalidate server or item handles, so invalid and
cross-server handle behavior remains owned by MXAccess. COM exceptions continue
through `StaCommandDispatcher`, which preserves the HRESULT and leaves
diagnostic registry state unchanged for failed cleanup calls.
## Handle Registry
The worker should track MXAccess state for diagnostics and cleanup, while still
@@ -454,6 +473,8 @@ Rules:
- Do not rewrite handles returned by MXAccess.
- Record server handles only after `Register` succeeds.
- Remove server handles only after `Unregister` succeeds.
- Record item handles only after `AddItem` or `AddItem2` succeeds.
- Remove item handles only after `RemoveItem` succeeds.
- Preserve invalid-handle behavior from MXAccess.
- Preserve cross-server handle behavior from MXAccess.
- Use registry state for cleanup and diagnostics, not semantic correction.
@@ -555,13 +576,19 @@ Do not drop or coalesce events in v1.
## Heartbeat And Watchdog
The worker heartbeat should prove that:
`WorkerPipeSession` starts the heartbeat loop after the gateway validates
`WorkerHello` and receives `WorkerReady`. Heartbeats continue until
`WorkerShutdown`, cancellation, or a pipe/protocol failure stops the session.
The loop uses `WorkerPipeSessionOptions.HeartbeatInterval`; the default matches
the gateway worker heartbeat interval.
The worker heartbeat proves that:
- pipe writer is alive,
- worker host is alive,
- STA has recently pumped or completed work.
Heartbeat payload should include:
Heartbeat payload includes:
- worker process id,
- session id,
@@ -572,13 +599,19 @@ Heartbeat payload should include:
- event sequence,
- current command correlation id if any.
The STA watchdog should warn when:
`MxAccessStaSession.CaptureHeartbeat()` reads `StaRuntime.LastActivityUtc` and
`StaCommandDispatcher` queue state without touching the raw MXAccess COM object
outside the STA. Event queue depth and event sequence are reported as zero until
the event queue implementation owns those counters.
- one command exceeds its expected duration,
- the STA has not pumped messages within the heartbeat grace period,
- event queue depth remains high.
The worker can report the problem, but the gateway owns the final kill decision.
The STA watchdog currently emits a `WorkerFault` with
`WorkerFaultCategory.StaHung` when `LastStaActivityUtc` is older than
`WorkerPipeSessionOptions.HeartbeatGrace`. The fault includes the current
command correlation id when a command is active. Command duration and high event
queue depth remain observable through heartbeat fields until dedicated
thresholds own those warnings. The worker reports stale STA activity, but the
gateway owns the final kill decision through its existing heartbeat and worker
lifecycle policy.
## Shutdown
@@ -697,6 +730,10 @@ Live MXAccess tests:
Live tests should be opt-in and clearly marked because they depend on installed
MXAccess COM and provider state.
The worker test suite uses `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1` for these
tests. `AddItem` uses `TestChildObject.TestInt` by default and accepts an
override through `MXGATEWAY_LIVE_MXACCESS_ITEM`; `AddItem2` uses the captured
parity fixture shape `AddItem2("TestInt", "TestChildObject")`.
## Initial Implementation Slice
@@ -0,0 +1,439 @@
using System.Collections.Concurrent;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MxGateway.Contracts;
using MxGateway.Contracts.Proto;
using MxGateway.Server.Configuration;
using MxGateway.Server.Grpc;
using MxGateway.Server.Metrics;
using MxGateway.Server.Security.Authorization;
using MxGateway.Server.Sessions;
using MxGateway.Server.Workers;
using MxGateway.Tests.Gateway.Workers.Fakes;
namespace MxGateway.Tests.Gateway;
public sealed class GatewayEndToEndFakeWorkerSmokeTests
{
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
private const int ServerHandle = 1001;
private const int ItemHandle = 2002;
[Fact]
public async Task GatewayService_WithFakeWorker_CompletesSessionCommandEventAndClosePath()
{
ScriptedFakeWorkerProcessLauncher launcher = new();
await using GatewayServiceFixture fixture = new(launcher);
OpenSessionReply openReply = await fixture.Service.OpenSession(
new OpenSessionRequest
{
ClientSessionName = "fake-worker-e2e",
ClientCorrelationId = "open-correlation",
CommandTimeout = Duration.FromTimeSpan(TestTimeout),
},
new TestServerCallContext());
RecordingServerStreamWriter<MxEvent> eventWriter = new();
Task streamTask = fixture.Service.StreamEvents(
new StreamEventsRequest { SessionId = openReply.SessionId },
eventWriter,
new TestServerCallContext());
MxCommandReply registerReply = await fixture.Service.Invoke(
CreateRegisterRequest(openReply.SessionId),
new TestServerCallContext());
MxCommandReply addItemReply = await fixture.Service.Invoke(
CreateAddItemRequest(openReply.SessionId, registerReply.Register.ServerHandle),
new TestServerCallContext());
MxCommandReply adviseReply = await fixture.Service.Invoke(
CreateAdviseRequest(openReply.SessionId, registerReply.Register.ServerHandle, addItemReply.AddItem.ItemHandle),
new TestServerCallContext());
MxEvent dataChange = await eventWriter.WaitForFirstMessageAsync(TestTimeout);
CloseSessionReply closeReply = await fixture.Service.CloseSession(
new CloseSessionRequest
{
SessionId = openReply.SessionId,
ClientCorrelationId = "close-correlation",
},
new TestServerCallContext());
await streamTask.WaitAsync(TestTimeout);
await launcher.WorkerTask.WaitAsync(TestTimeout);
Assert.Equal(ProtocolStatusCode.Ok, openReply.ProtocolStatus.Code);
Assert.Equal(GatewayContractInfo.DefaultBackendName, openReply.BackendName);
Assert.Equal(ScriptedFakeWorkerProcessLauncher.ProcessId, openReply.WorkerProcessId);
Assert.Equal(ProtocolStatusCode.Ok, registerReply.ProtocolStatus.Code);
Assert.Equal(ServerHandle, registerReply.Register.ServerHandle);
Assert.Equal(ProtocolStatusCode.Ok, addItemReply.ProtocolStatus.Code);
Assert.Equal(ItemHandle, addItemReply.AddItem.ItemHandle);
Assert.Equal(ProtocolStatusCode.Ok, adviseReply.ProtocolStatus.Code);
Assert.Equal(MxEventFamily.OnDataChange, dataChange.Family);
Assert.Equal(openReply.SessionId, dataChange.SessionId);
Assert.Equal(ServerHandle, dataChange.ServerHandle);
Assert.Equal(ItemHandle, dataChange.ItemHandle);
Assert.Equal("scripted-value", dataChange.Value.StringValue);
Assert.Equal(ProtocolStatusCode.Ok, closeReply.ProtocolStatus.Code);
Assert.Equal(SessionState.Closed, closeReply.FinalState);
Assert.True(launcher.Process.HasExited);
Assert.Equal(
[MxCommandKind.Register, MxCommandKind.AddItem, MxCommandKind.Advise],
launcher.CommandKinds);
}
private static MxCommandRequest CreateRegisterRequest(string sessionId)
{
return new MxCommandRequest
{
SessionId = sessionId,
ClientCorrelationId = "register-correlation",
Command = new MxCommand
{
Kind = MxCommandKind.Register,
Register = new RegisterCommand { ClientName = "fake-worker-e2e-client" },
},
};
}
private static MxCommandRequest CreateAddItemRequest(
string sessionId,
int serverHandle)
{
return new MxCommandRequest
{
SessionId = sessionId,
ClientCorrelationId = "add-item-correlation",
Command = new MxCommand
{
Kind = MxCommandKind.AddItem,
AddItem = new AddItemCommand
{
ServerHandle = serverHandle,
ItemDefinition = "Galaxy.Tag.Value",
},
},
};
}
private static MxCommandRequest CreateAdviseRequest(
string sessionId,
int serverHandle,
int itemHandle)
{
return new MxCommandRequest
{
SessionId = sessionId,
ClientCorrelationId = "advise-correlation",
Command = new MxCommand
{
Kind = MxCommandKind.Advise,
Advise = new AdviseCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
},
};
}
private sealed class GatewayServiceFixture : IAsyncDisposable
{
private readonly GatewayMetrics _metrics = new();
private readonly SessionRegistry _registry = new();
public GatewayServiceFixture(IWorkerProcessLauncher launcher)
{
IOptions<GatewayOptions> options = Options.Create(CreateOptions());
SessionWorkerClientFactory workerClientFactory = new(
launcher,
options,
_metrics,
NullLoggerFactory.Instance);
SessionManager sessionManager = new(
_registry,
workerClientFactory,
options,
_metrics,
logger: NullLogger<SessionManager>.Instance);
MxAccessGrpcMapper mapper = new();
EventStreamService eventStreamService = new(
sessionManager,
options,
mapper,
_metrics,
NullLogger<EventStreamService>.Instance);
Service = new MxAccessGatewayService(
sessionManager,
new GatewayRequestIdentityAccessor(),
new MxAccessGrpcRequestValidator(),
mapper,
eventStreamService,
NullLogger<MxAccessGatewayService>.Instance);
}
public MxAccessGatewayService Service { get; }
public async ValueTask DisposeAsync()
{
foreach (GatewaySession session in _registry.Snapshot())
{
await session.DisposeAsync();
}
_metrics.Dispose();
}
private static GatewayOptions CreateOptions()
{
return new GatewayOptions
{
Worker = new WorkerOptions
{
StartupTimeoutSeconds = 5,
ShutdownTimeoutSeconds = 5,
HeartbeatIntervalSeconds = 30,
HeartbeatGraceSeconds = 30,
MaxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
},
Sessions = new SessionOptions
{
DefaultCommandTimeoutSeconds = 5,
MaxSessions = 4,
},
Events = new EventOptions
{
QueueCapacity = 16,
},
};
}
}
private sealed class ScriptedFakeWorkerProcessLauncher : IWorkerProcessLauncher
{
public const int ProcessId = 4680;
private readonly ConcurrentQueue<MxCommandKind> _commandKinds = new();
public FakeWorkerProcess Process { get; } = new(ProcessId);
public IReadOnlyCollection<MxCommandKind> CommandKinds => _commandKinds.ToArray();
public Task WorkerTask { get; private set; } = Task.CompletedTask;
public Task<WorkerProcessHandle> LaunchAsync(
WorkerProcessLaunchRequest request,
CancellationToken cancellationToken = default)
{
WorkerTask = RunWorkerAsync(request, cancellationToken);
return Task.FromResult(new WorkerProcessHandle(
Process,
new WorkerProcessCommandLine("fake-worker.exe", []),
DateTimeOffset.UtcNow));
}
private async Task RunWorkerAsync(
WorkerProcessLaunchRequest request,
CancellationToken cancellationToken)
{
await using FakeWorkerHarness harness = await FakeWorkerHarness.ConnectToGatewayPipeAsync(
request.SessionId,
request.Nonce,
request.PipeName,
request.ProtocolVersion,
cancellationToken: cancellationToken).ConfigureAwait(false);
await harness.CompleteStartupAsync(ProcessId, cancellationToken: cancellationToken).ConfigureAwait(false);
while (!cancellationToken.IsCancellationRequested)
{
WorkerEnvelope envelope = await harness.ReadGatewayEnvelopeAsync(cancellationToken).ConfigureAwait(false);
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerCommand)
{
await ReplyToCommandAsync(harness, envelope, cancellationToken).ConfigureAwait(false);
continue;
}
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerShutdown)
{
await harness.SendShutdownAckAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
Process.MarkExited(0);
return;
}
throw new InvalidOperationException($"Unexpected gateway envelope {envelope.BodyCase}.");
}
}
private async Task ReplyToCommandAsync(
FakeWorkerHarness harness,
WorkerEnvelope commandEnvelope,
CancellationToken cancellationToken)
{
MxCommand command = commandEnvelope.WorkerCommand.Command;
_commandKinds.Enqueue(command.Kind);
await harness.ReplyToCommandAsync(
commandEnvelope,
configureReply: reply => ConfigureReply(reply, command.Kind),
cancellationToken: cancellationToken).ConfigureAwait(false);
if (command.Kind == MxCommandKind.Advise)
{
await harness.EmitEventAsync(
MxEventFamily.OnDataChange,
cancellationToken,
mxEvent =>
{
mxEvent.ServerHandle = command.Advise.ServerHandle;
mxEvent.ItemHandle = command.Advise.ItemHandle;
mxEvent.Quality = 192;
mxEvent.Value = new MxValue
{
DataType = MxDataType.String,
StringValue = "scripted-value",
};
mxEvent.OnDataChange = new OnDataChangeEvent();
}).ConfigureAwait(false);
}
}
private static void ConfigureReply(
MxCommandReply reply,
MxCommandKind kind)
{
switch (kind)
{
case MxCommandKind.Register:
reply.Register = new RegisterReply { ServerHandle = ServerHandle };
break;
case MxCommandKind.AddItem:
reply.AddItem = new AddItemReply { ItemHandle = ItemHandle };
break;
}
}
}
private sealed class FakeWorkerProcess(int processId) : IWorkerProcess
{
public int Id { get; } = processId;
public bool HasExited { get; private set; }
public int? ExitCode { get; private set; }
public ValueTask WaitForExitAsync(CancellationToken cancellationToken)
{
HasExited = true;
ExitCode ??= 0;
return ValueTask.CompletedTask;
}
public void Kill(bool entireProcessTree)
{
MarkExited(-1);
}
public void Dispose()
{
}
public void MarkExited(int exitCode)
{
HasExited = true;
ExitCode = exitCode;
}
}
private sealed class RecordingServerStreamWriter<T> : IServerStreamWriter<T>
{
private readonly object _syncRoot = new();
private readonly TaskCompletionSource<T> _firstMessage = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly List<T> _messages = [];
public IReadOnlyList<T> Messages
{
get
{
lock (_syncRoot)
{
return _messages.ToArray();
}
}
}
public WriteOptions? WriteOptions { get; set; }
public Task WriteAsync(T message)
{
lock (_syncRoot)
{
_messages.Add(message);
}
_firstMessage.TrySetResult(message);
return Task.CompletedTask;
}
public async Task<T> WaitForFirstMessageAsync(TimeSpan timeout)
{
return await _firstMessage.Task.WaitAsync(timeout).ConfigureAwait(false);
}
}
private sealed class TestServerCallContext(CancellationToken cancellationToken = default) : ServerCallContext
{
private readonly Metadata _requestHeaders = [];
private readonly Metadata _responseTrailers = [];
private readonly Dictionary<object, object> _userState = [];
private Status _status;
private WriteOptions? _writeOptions;
protected override string MethodCore => "/mxaccess_gateway.v1.MxAccessGateway/Test";
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:5000";
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
protected override Metadata RequestHeadersCore => _requestHeaders;
protected override CancellationToken CancellationTokenCore => cancellationToken;
protected override Metadata ResponseTrailersCore => _responseTrailers;
protected override Status StatusCore
{
get => _status;
set => _status = value;
}
protected override WriteOptions? WriteOptionsCore
{
get => _writeOptions;
set => _writeOptions = value;
}
protected override AuthContext AuthContextCore { get; } = new(
string.Empty,
new Dictionary<string, List<AuthProperty>>(StringComparer.Ordinal));
protected override IDictionary<object, object> UserStateCore => _userState;
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
{
return Task.CompletedTask;
}
protected override ContextPropagationToken CreatePropagationTokenCore(
ContextPropagationOptions? options)
{
throw new NotSupportedException();
}
}
}
@@ -105,6 +105,25 @@ public sealed class FakeWorkerHarnessTests
Assert.Equal(WorkerClientState.Faulted, client.State);
}
[Fact]
public async Task SendHeartbeatAsync_UpdatesClientHeartbeatState()
{
await using FakeWorkerHarness fakeWorker = await FakeWorkerHarness.CreateConnectedPairAsync();
await using WorkerClient client = fakeWorker.CreateClient();
await StartClientAsync(fakeWorker, client);
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
await Task.Delay(TimeSpan.FromMilliseconds(20));
await fakeWorker.SendHeartbeatAsync(
configureHeartbeat: heartbeat => heartbeat.WorkerProcessId = 2468);
await WaitUntilAsync(
() => client.ProcessId == 2468 && client.LastHeartbeatAt > previousHeartbeat,
TestTimeout);
Assert.Equal(WorkerClientState.Ready, client.State);
}
[Fact]
public async Task InvokeAsync_WithHungWorker_TimesOutPendingCommand()
{
@@ -205,6 +205,7 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
WorkerEnvelope commandEnvelope,
ProtocolStatusCode statusCode = ProtocolStatusCode.Ok,
string statusMessage = "OK",
Action<MxCommandReply>? configureReply = null,
CancellationToken cancellationToken = default)
{
if (commandEnvelope.BodyCase != WorkerEnvelope.BodyOneofCase.WorkerCommand)
@@ -213,12 +214,7 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
}
MxCommandKind kind = commandEnvelope.WorkerCommand.Command?.Kind ?? MxCommandKind.Unspecified;
await _writer.WriteAsync(
CreateEnvelope(
commandEnvelope.CorrelationId,
envelope => envelope.WorkerCommandReply = new WorkerCommandReply
{
Reply = new MxCommandReply
MxCommandReply reply = new()
{
SessionId = SessionId,
CorrelationId = commandEnvelope.CorrelationId,
@@ -228,7 +224,15 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
Code = statusCode,
Message = statusMessage,
},
},
};
configureReply?.Invoke(reply);
await _writer.WriteAsync(
CreateEnvelope(
commandEnvelope.CorrelationId,
envelope => envelope.WorkerCommandReply = new WorkerCommandReply
{
Reply = reply,
CompletedTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
}),
cancellationToken).ConfigureAwait(false);
@@ -236,21 +240,25 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
public async Task EmitEventAsync(
MxEventFamily family,
CancellationToken cancellationToken = default)
CancellationToken cancellationToken = default,
Action<MxEvent>? configureEvent = null)
{
ulong sequence = NextWorkerSequence + 1;
await _writer.WriteAsync(
CreateEnvelope(
correlationId: string.Empty,
envelope => envelope.WorkerEvent = new WorkerEvent
{
Event = new MxEvent
MxEvent mxEvent = new()
{
SessionId = SessionId,
Family = family,
WorkerSequence = sequence,
WorkerTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
};
configureEvent?.Invoke(mxEvent);
await _writer.WriteAsync(
CreateEnvelope(
correlationId: string.Empty,
envelope => envelope.WorkerEvent = new WorkerEvent
{
Event = mxEvent,
}),
cancellationToken).ConfigureAwait(false);
}
@@ -276,6 +284,26 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
cancellationToken).ConfigureAwait(false);
}
public async Task SendHeartbeatAsync(
WorkerState state = WorkerState.Ready,
CancellationToken cancellationToken = default,
Action<WorkerHeartbeat>? configureHeartbeat = null)
{
WorkerHeartbeat heartbeat = new()
{
WorkerProcessId = DefaultWorkerProcessId,
State = state,
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
};
configureHeartbeat?.Invoke(heartbeat);
await _writer.WriteAsync(
CreateEnvelope(
correlationId: string.Empty,
envelope => envelope.WorkerHeartbeat = heartbeat),
cancellationToken).ConfigureAwait(false);
}
public async Task SendShutdownAckAsync(
ProtocolStatusCode statusCode = ProtocolStatusCode.Ok,
CancellationToken cancellationToken = default)
@@ -1,4 +1,5 @@
using System.IO.Pipes;
using Google.Protobuf.WellKnownTypes;
using MxGateway.Contracts;
using MxGateway.Contracts.Proto;
using MxGateway.Server.Workers;
@@ -151,6 +152,24 @@ public sealed class WorkerClientTests
Assert.Equal(WorkerClientState.Faulted, client.State);
}
[Fact]
public async Task ReadLoop_WhenHeartbeatArrives_UpdatesLastHeartbeatAndWorkerProcess()
{
await using PipePair pipePair = await PipePair.CreateAsync();
await using WorkerClient client = CreateClient(pipePair);
await CompleteHandshakeAsync(client, pipePair);
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
await Task.Delay(TimeSpan.FromMilliseconds(20));
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await WaitUntilAsync(
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
TestTimeout);
Assert.Equal(WorkerClientState.Ready, client.State);
}
[Fact]
public async Task HeartbeatMonitor_WhenHeartbeatExpires_FaultsClient()
{
@@ -276,6 +295,21 @@ public sealed class WorkerClientTests
});
}
private static WorkerEnvelope CreateHeartbeatEnvelope(int workerProcessId)
{
return CreateWorkerEnvelope(
correlationId: string.Empty,
sequence: 20,
envelope => envelope.WorkerHeartbeat = new WorkerHeartbeat
{
WorkerProcessId = workerProcessId,
State = WorkerState.Ready,
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
PendingCommandCount = 0,
OutboundEventQueueDepth = 0,
});
}
private static WorkerEnvelope CreateWorkerEnvelope(
string correlationId,
ulong sequence,
@@ -30,7 +30,7 @@ public sealed class WorkerApplicationTests
Assert.Equal("mxaccess-gateway-123-session-1", entry.Fields["pipe_name"]);
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, entry.Fields["protocol_version"]);
Assert.Equal("[redacted]", entry.Fields["nonce"]);
Assert.Equal("WorkerPipeHandshakeSucceeded", logger.Entries[1].EventName);
Assert.Equal("WorkerPipeSessionCompleted", logger.Entries[1].EventName);
}
[Fact]
@@ -1,10 +1,15 @@
using System;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
using Google.Protobuf.WellKnownTypes;
using MxGateway.Contracts;
using MxGateway.Contracts.Proto;
using MxGateway.Worker.Bootstrap;
using MxGateway.Worker.Ipc;
using MxGateway.Worker.MxAccess;
using MxGateway.Worker.Sta;
namespace MxGateway.Worker.Tests.Ipc;
@@ -28,7 +33,9 @@ public sealed class WorkerPipeClientTests
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
WorkerPipeClient client = new(connectTimeoutMilliseconds: 5000);
WorkerPipeClient client = new(
connectTimeoutMilliseconds: 5000,
(stream, options) => CreateSession(stream, options));
Task clientTask = client.RunAsync(workerOptions);
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
@@ -56,6 +63,86 @@ public sealed class WorkerPipeClientTests
WorkerEnvelope ready = await reader.ReadAsync();
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerReady, ready.BodyCase);
await writer.WriteAsync(new WorkerEnvelope
{
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
SessionId = "session-1",
Sequence = 2,
WorkerShutdown = new WorkerShutdown
{
GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(1)),
Reason = "test-complete",
},
});
WorkerEnvelope shutdownAck = await reader.ReadAsync();
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerShutdownAck, shutdownAck.BodyCase);
await clientTask;
}
private static WorkerPipeSession CreateSession(
Stream stream,
WorkerFrameProtocolOptions options)
{
return new WorkerPipeSession(
new WorkerFrameReader(stream, options),
new WorkerFrameWriter(stream, options),
options,
() => 1234,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromSeconds(30),
HeartbeatGrace = TimeSpan.FromSeconds(30),
},
() => new FakeRuntimeSession());
}
private sealed class FakeRuntimeSession : IWorkerRuntimeSession
{
public Task<WorkerReady> StartAsync(
int workerProcessId,
CancellationToken cancellationToken = default)
{
return Task.FromResult(new WorkerReady
{
WorkerProcessId = workerProcessId,
MxaccessProgid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.ProgId,
MxaccessClsid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.Clsid,
ReadyTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
});
}
public Task<MxCommandReply> DispatchAsync(StaCommand command)
{
return Task.FromResult(new MxCommandReply
{
SessionId = command.SessionId,
CorrelationId = command.CorrelationId,
Kind = command.Kind,
ProtocolStatus = new ProtocolStatus
{
Code = ProtocolStatusCode.Ok,
Message = "OK",
},
});
}
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
{
return new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow,
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: string.Empty);
}
public void RequestShutdown()
{
}
public void Dispose()
{
}
}
}
@@ -1,11 +1,16 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Pipes;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;
using Google.Protobuf.WellKnownTypes;
using MxGateway.Contracts;
using MxGateway.Contracts.Proto;
using MxGateway.Worker.Ipc;
using MxGateway.Worker.MxAccess;
using MxGateway.Worker.Sta;
namespace MxGateway.Worker.Tests.Ipc;
@@ -147,6 +152,127 @@ public sealed class WorkerPipeSessionTests
Assert.Equal(ProtocolStatusCode.WorkerUnavailable, written[1].WorkerFault.ProtocolStatus.Code);
}
[Fact]
public async Task RunAsync_SendsHeartbeatPayloadFromRuntimeSnapshot()
{
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new();
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow,
pendingCommandCount: 2,
outboundEventQueueDepth: 3,
lastEventSequence: 42,
currentCommandCorrelationId: "current-command"));
WorkerPipeSession session = CreatePipeSession(
pipePair.WorkerStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
HeartbeatGrace = TimeSpan.FromSeconds(5),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
await ThrowIfCompletedAsync(runTask);
WorkerEnvelope heartbeat = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
cancellation.Token);
Assert.Equal(WorkerState.ExecutingCommand, heartbeat.WorkerHeartbeat.State);
Assert.Equal(1234, heartbeat.WorkerHeartbeat.WorkerProcessId);
Assert.Equal(2u, heartbeat.WorkerHeartbeat.PendingCommandCount);
Assert.Equal(3u, heartbeat.WorkerHeartbeat.OutboundEventQueueDepth);
Assert.Equal(42UL, heartbeat.WorkerHeartbeat.LastEventSequence);
Assert.Equal("current-command", heartbeat.WorkerHeartbeat.CurrentCommandCorrelationId);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
[Fact]
public async Task RunAsync_WhenCommandIsExecuting_HeartbeatReportsCurrentCorrelation()
{
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new()
{
BlockDispatch = true,
};
WorkerPipeSession session = CreatePipeSession(
pipePair.WorkerStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
HeartbeatGrace = TimeSpan.FromSeconds(5),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
await pipePair.GatewayWriter.WriteAsync(
CreateCommandEnvelope("command-1"),
cancellation.Token);
Assert.True(runtime.DispatchStarted.Wait(TimeSpan.FromSeconds(2)));
WorkerEnvelope heartbeat = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
envelope => envelope.WorkerHeartbeat.CurrentCommandCorrelationId == "command-1",
cancellation.Token);
Assert.Equal("command-1", heartbeat.WorkerHeartbeat.CurrentCommandCorrelationId);
Assert.Equal(WorkerState.ExecutingCommand, heartbeat.WorkerHeartbeat.State);
runtime.ReleaseDispatch();
WorkerEnvelope reply = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
cancellation.Token);
Assert.Equal("command-1", reply.CorrelationId);
Assert.Equal(ProtocolStatusCode.Ok, reply.WorkerCommandReply.Reply.ProtocolStatus.Code);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
[Fact]
public async Task RunAsync_WhenStaActivityIsStale_WritesWatchdogFault()
{
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(5));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new();
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: "stuck-command"));
WorkerPipeSession session = CreatePipeSession(
pipePair.WorkerStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
WorkerEnvelope fault = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerFault,
cancellation.Token);
Assert.Equal(WorkerFaultCategory.StaHung, fault.WorkerFault.Category);
Assert.Equal("stuck-command", fault.WorkerFault.CommandMethod);
Assert.Contains("STA activity is stale", fault.WorkerFault.DiagnosticMessage);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
private static WorkerPipeSession CreateSession(
Stream inbound,
Stream outbound,
@@ -159,6 +285,21 @@ public sealed class WorkerPipeSessionTests
() => 1234);
}
private static WorkerPipeSession CreatePipeSession(
Stream stream,
FakeRuntimeSession runtime,
WorkerPipeSessionOptions sessionOptions)
{
WorkerFrameProtocolOptions options = CreateOptions();
return new WorkerPipeSession(
new WorkerFrameReader(stream, options),
new WorkerFrameWriter(stream, options),
options,
() => 1234,
sessionOptions,
() => runtime);
}
private static WorkerFrameProtocolOptions CreateOptions()
{
return new WorkerFrameProtocolOptions(
@@ -185,6 +326,119 @@ public sealed class WorkerPipeSessionTests
};
}
private static WorkerEnvelope CreateCommandEnvelope(string correlationId)
{
return new WorkerEnvelope
{
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
SessionId = SessionId,
Sequence = 2,
CorrelationId = correlationId,
WorkerCommand = new WorkerCommand
{
Command = new MxCommand
{
Kind = MxCommandKind.Ping,
Ping = new PingCommand
{
Message = "ping",
},
},
EnqueueTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
};
}
private static WorkerEnvelope CreateShutdownEnvelope()
{
return new WorkerEnvelope
{
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
SessionId = SessionId,
Sequence = 3,
WorkerShutdown = new WorkerShutdown
{
GracePeriod = Duration.FromTimeSpan(TimeSpan.FromSeconds(1)),
Reason = "test-complete",
},
};
}
private static async Task CompleteGatewayHandshakeAsync(
PipePair pipePair,
CancellationToken cancellationToken)
{
await pipePair.GatewayWriter
.WriteAsync(CreateGatewayHelloEnvelope(), cancellationToken)
.ConfigureAwait(false);
WorkerEnvelope hello = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
WorkerEnvelope ready = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerHello, hello.BodyCase);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerReady, ready.BodyCase);
}
private static async Task SendShutdownAndWaitAsync(
PipePair pipePair,
Task runTask,
CancellationToken cancellationToken)
{
await pipePair.GatewayWriter
.WriteAsync(CreateShutdownEnvelope(), cancellationToken)
.ConfigureAwait(false);
WorkerEnvelope shutdownAck = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerShutdownAck,
cancellationToken);
Assert.Equal(ProtocolStatusCode.Ok, shutdownAck.WorkerShutdownAck.Status.Code);
Task completedTask = await Task
.WhenAny(runTask, Task.Delay(TimeSpan.FromSeconds(2), cancellationToken))
.ConfigureAwait(false);
Assert.Same(runTask, completedTask);
await runTask.ConfigureAwait(false);
}
private static async Task ThrowIfCompletedAsync(Task task)
{
await Task.Delay(TimeSpan.FromMilliseconds(100));
if (task.IsCompleted)
{
await task;
}
}
private static Task<WorkerEnvelope> ReadUntilAsync(
WorkerFrameReader reader,
WorkerEnvelope.BodyOneofCase expectedBody,
CancellationToken cancellationToken)
{
return ReadUntilAsync(
reader,
expectedBody,
_ => true,
cancellationToken);
}
private static async Task<WorkerEnvelope> ReadUntilAsync(
WorkerFrameReader reader,
WorkerEnvelope.BodyOneofCase expectedBody,
Func<WorkerEnvelope, bool> predicate,
CancellationToken cancellationToken)
{
while (true)
{
WorkerEnvelope envelope = await reader.ReadAsync(cancellationToken).ConfigureAwait(false);
if (envelope.BodyCase == expectedBody && predicate(envelope))
{
return envelope;
}
}
}
private static WorkerEnvelope[] ReadWrittenFrames(
MemoryStream stream,
WorkerFrameProtocolOptions options)
@@ -219,4 +473,157 @@ public sealed class WorkerPipeSessionTests
buffer[2] = (byte)(value >> 16);
buffer[3] = (byte)(value >> 24);
}
private sealed class FakeRuntimeSession : IWorkerRuntimeSession
{
private readonly ManualResetEventSlim releaseDispatch = new(false);
private readonly object gate = new();
private WorkerRuntimeHeartbeatSnapshot snapshot = new(
DateTimeOffset.UtcNow,
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: string.Empty);
public ManualResetEventSlim DispatchStarted { get; } = new(false);
public bool BlockDispatch { get; set; }
public Task<WorkerReady> StartAsync(
int workerProcessId,
CancellationToken cancellationToken = default)
{
return Task.FromResult(new WorkerReady
{
WorkerProcessId = workerProcessId,
MxaccessProgid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.ProgId,
MxaccessClsid = MxGateway.Worker.MxAccess.MxAccessInteropInfo.Clsid,
ReadyTimestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
});
}
public Task<MxCommandReply> DispatchAsync(StaCommand command)
{
return Task.Run(
() =>
{
SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow,
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
command.CorrelationId));
DispatchStarted.Set();
if (BlockDispatch)
{
releaseDispatch.Wait(TimeSpan.FromSeconds(5));
}
SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow,
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: string.Empty));
return new MxCommandReply
{
SessionId = command.SessionId,
CorrelationId = command.CorrelationId,
Kind = command.Kind,
ProtocolStatus = new ProtocolStatus
{
Code = ProtocolStatusCode.Ok,
Message = "OK",
},
};
});
}
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
{
lock (gate)
{
return snapshot;
}
}
public void RequestShutdown()
{
releaseDispatch.Set();
}
public void ReleaseDispatch()
{
releaseDispatch.Set();
}
public void SetSnapshot(WorkerRuntimeHeartbeatSnapshot value)
{
lock (gate)
{
snapshot = value;
}
}
public void Dispose()
{
releaseDispatch.Set();
releaseDispatch.Dispose();
DispatchStarted.Dispose();
}
}
private sealed class PipePair : IDisposable
{
private readonly NamedPipeServerStream gatewayStream;
private PipePair(
NamedPipeServerStream gatewayStream,
NamedPipeClientStream workerStream)
{
this.gatewayStream = gatewayStream;
WorkerStream = workerStream;
WorkerFrameProtocolOptions options = CreateOptions();
GatewayReader = new WorkerFrameReader(gatewayStream, options);
GatewayWriter = new WorkerFrameWriter(gatewayStream, options);
}
public Stream WorkerStream { get; }
public WorkerFrameReader GatewayReader { get; }
public WorkerFrameWriter GatewayWriter { get; }
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
{
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeClientStream workerStream = new(
".",
pipeName,
PipeDirection.InOut,
PipeOptions.Asynchronous);
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync();
await Task
.Run(() => workerStream.Connect(5000), cancellationToken)
.ConfigureAwait(false);
await waitForConnectionTask.ConfigureAwait(false);
return new PipePair(gatewayStream, workerStream);
}
public void Dispose()
{
WorkerStream.Dispose();
gatewayStream.Dispose();
}
}
}
@@ -78,6 +78,166 @@ public sealed class MxAccessCommandExecutorTests
Assert.Equal(44, registeredServerHandle.ServerHandle);
}
[Fact]
public async Task DispatchAsync_AddItem_CallsMxAccessOnStaAndTracksItemHandle()
{
FakeMxAccessComObject fakeComObject = new(
registerHandle: 46,
addItemHandle: 501);
FakeMxAccessComObjectFactory factory = new(fakeComObject);
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
await session.DispatchAsync(CreateRegisterCommand("register-before-add", "client-a"));
MxCommandReply reply = await session.DispatchAsync(CreateAddItemCommand(
"add-item",
46,
"Galaxy.Tag.Value"));
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.True(reply.HasHresult);
Assert.Equal(0, reply.Hresult);
Assert.Equal(501, reply.AddItem.ItemHandle);
Assert.Equal(MxDataType.Integer, reply.ReturnValue.DataType);
Assert.Equal(501, reply.ReturnValue.Int32Value);
Assert.Equal(46, fakeComObject.AddItemServerHandle);
Assert.Equal("Galaxy.Tag.Value", fakeComObject.AddItemDefinition);
Assert.Equal(runtime.StaThreadId, fakeComObject.AddItemThreadId);
RegisteredItemHandle registeredItemHandle = Assert.Single(
await session.GetRegisteredItemHandlesAsync());
Assert.Equal(46, registeredItemHandle.ServerHandle);
Assert.Equal(501, registeredItemHandle.ItemHandle);
Assert.Equal("Galaxy.Tag.Value", registeredItemHandle.ItemDefinition);
Assert.Equal(string.Empty, registeredItemHandle.ItemContext);
Assert.False(registeredItemHandle.HasItemContext);
}
[Fact]
public async Task DispatchAsync_AddItem2_PassesContextExactlyAndTracksItemHandle()
{
FakeMxAccessComObject fakeComObject = new(
registerHandle: 47,
addItem2Handle: 502);
FakeMxAccessComObjectFactory factory = new(fakeComObject);
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
await session.DispatchAsync(CreateRegisterCommand("register-before-add2", "client-a"));
MxCommandReply reply = await session.DispatchAsync(CreateAddItem2Command(
"add-item2",
47,
"TestInt",
"TestChildObject"));
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.Equal(502, reply.AddItem2.ItemHandle);
Assert.Equal(MxDataType.Integer, reply.ReturnValue.DataType);
Assert.Equal(502, reply.ReturnValue.Int32Value);
Assert.Equal(47, fakeComObject.AddItem2ServerHandle);
Assert.Equal("TestInt", fakeComObject.AddItem2Definition);
Assert.Equal("TestChildObject", fakeComObject.AddItem2Context);
Assert.Equal(runtime.StaThreadId, fakeComObject.AddItem2ThreadId);
RegisteredItemHandle registeredItemHandle = Assert.Single(
await session.GetRegisteredItemHandlesAsync());
Assert.Equal(47, registeredItemHandle.ServerHandle);
Assert.Equal(502, registeredItemHandle.ItemHandle);
Assert.Equal("TestInt", registeredItemHandle.ItemDefinition);
Assert.Equal("TestChildObject", registeredItemHandle.ItemContext);
Assert.True(registeredItemHandle.HasItemContext);
}
[Fact]
public async Task DispatchAsync_RemoveItem_CallsMxAccessOnStaAndRemovesTrackedItemHandle()
{
FakeMxAccessComObject fakeComObject = new(
registerHandle: 48,
addItemHandle: 503);
FakeMxAccessComObjectFactory factory = new(fakeComObject);
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
await session.DispatchAsync(CreateRegisterCommand("register-before-remove", "client-a"));
await session.DispatchAsync(CreateAddItemCommand("add-before-remove", 48, "Galaxy.Tag.Value"));
MxCommandReply reply = await session.DispatchAsync(CreateRemoveItemCommand(
"remove-item",
48,
503));
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.True(reply.HasHresult);
Assert.Equal(0, reply.Hresult);
Assert.Equal(48, fakeComObject.RemoveItemServerHandle);
Assert.Equal(503, fakeComObject.RemovedItemHandle);
Assert.Equal(runtime.StaThreadId, fakeComObject.RemoveItemThreadId);
Assert.Empty(await session.GetRegisteredItemHandlesAsync());
}
[Fact]
public async Task DispatchAsync_RemoveItemWithCrossServerHandle_PreservesHResultAndKeepsTrackedItemHandle()
{
const int hresult = unchecked((int)0x80070057);
FakeMxAccessComObject fakeComObject = new(
registerHandle: 49,
addItemHandle: 504,
removeItemException: new COMException("Invalid item handle.", hresult));
FakeMxAccessComObjectFactory factory = new(fakeComObject);
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
await session.DispatchAsync(CreateRegisterCommand("register-before-remove-failure", "client-a"));
await session.DispatchAsync(CreateAddItemCommand("add-before-remove-failure", 49, "Galaxy.Tag.Value"));
MxCommandReply reply = await session.DispatchAsync(CreateRemoveItemCommand(
"remove-item-failure",
999,
504));
Assert.Equal(ProtocolStatusCode.MxaccessFailure, reply.ProtocolStatus.Code);
Assert.True(reply.HasHresult);
Assert.Equal(hresult, reply.Hresult);
Assert.Contains("0x80070057", reply.DiagnosticMessage);
Assert.Equal(999, fakeComObject.RemoveItemServerHandle);
Assert.Equal(504, fakeComObject.RemovedItemHandle);
RegisteredItemHandle registeredItemHandle = Assert.Single(
await session.GetRegisteredItemHandlesAsync());
Assert.Equal(49, registeredItemHandle.ServerHandle);
Assert.Equal(504, registeredItemHandle.ItemHandle);
}
[Fact]
public async Task DispatchAsync_AddItem2WhenMxAccessThrows_PreservesHResultAndDoesNotTrackItemHandle()
{
const int hresult = unchecked((int)0x80070057);
FakeMxAccessComObject fakeComObject = new(
registerHandle: 50,
addItem2Exception: new COMException("Invalid server handle.", hresult));
FakeMxAccessComObjectFactory factory = new(fakeComObject);
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
MxCommandReply reply = await session.DispatchAsync(CreateAddItem2Command(
"add-item2-failure",
9001,
"TestInt",
"TestChildObject"));
Assert.Equal(ProtocolStatusCode.MxaccessFailure, reply.ProtocolStatus.Code);
Assert.True(reply.HasHresult);
Assert.Equal(hresult, reply.Hresult);
Assert.Contains("0x80070057", reply.DiagnosticMessage);
Assert.Equal(9001, fakeComObject.AddItem2ServerHandle);
Assert.Equal("TestInt", fakeComObject.AddItem2Definition);
Assert.Equal("TestChildObject", fakeComObject.AddItem2Context);
Assert.Empty(await session.GetRegisteredItemHandlesAsync());
}
[Fact]
public async Task DispatchAsync_RegisterWithoutPayload_ReturnsInvalidRequest()
{
@@ -98,6 +258,26 @@ public sealed class MxAccessCommandExecutorTests
Assert.Null(factory.FakeComObject.RegisteredClientName);
}
[Fact]
public async Task DispatchAsync_AddItemWithoutPayload_ReturnsInvalidRequest()
{
FakeMxAccessComObjectFactory factory = new(new FakeMxAccessComObject(registerHandle: 51));
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(runtime, factory, new NoopEventSink());
await session.StartAsync(workerProcessId: 1234);
MxCommandReply reply = await session.DispatchAsync(new StaCommand(
"session-1",
"missing-add-payload",
new MxCommand
{
Kind = MxCommandKind.AddItem,
}));
Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
Assert.Null(factory.FakeComObject.AddItemDefinition);
}
private static StaCommand CreateRegisterCommand(
string correlationId,
string clientName)
@@ -132,6 +312,65 @@ public sealed class MxAccessCommandExecutorTests
});
}
private static StaCommand CreateAddItemCommand(
string correlationId,
int serverHandle,
string itemDefinition)
{
return new StaCommand(
"session-1",
correlationId,
new MxCommand
{
Kind = MxCommandKind.AddItem,
AddItem = new AddItemCommand
{
ServerHandle = serverHandle,
ItemDefinition = itemDefinition,
},
});
}
private static StaCommand CreateAddItem2Command(
string correlationId,
int serverHandle,
string itemDefinition,
string itemContext)
{
return new StaCommand(
"session-1",
correlationId,
new MxCommand
{
Kind = MxCommandKind.AddItem2,
AddItem2 = new AddItem2Command
{
ServerHandle = serverHandle,
ItemDefinition = itemDefinition,
ItemContext = itemContext,
},
});
}
private static StaCommand CreateRemoveItemCommand(
string correlationId,
int serverHandle,
int itemHandle)
{
return new StaCommand(
"session-1",
correlationId,
new MxCommand
{
Kind = MxCommandKind.RemoveItem,
RemoveItem = new RemoveItemCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
});
}
private static StaRuntime CreateRuntime()
{
return new StaRuntime(
@@ -143,14 +382,29 @@ public sealed class MxAccessCommandExecutorTests
private sealed class FakeMxAccessComObject
{
private readonly int registerHandle;
private readonly int addItemHandle;
private readonly int addItem2Handle;
private readonly Exception? unregisterException;
private readonly Exception? addItemException;
private readonly Exception? addItem2Exception;
private readonly Exception? removeItemException;
public FakeMxAccessComObject(
int registerHandle,
Exception? unregisterException = null)
int addItemHandle = 0,
int addItem2Handle = 0,
Exception? unregisterException = null,
Exception? addItemException = null,
Exception? addItem2Exception = null,
Exception? removeItemException = null)
{
this.registerHandle = registerHandle;
this.addItemHandle = addItemHandle;
this.addItem2Handle = addItem2Handle;
this.unregisterException = unregisterException;
this.addItemException = addItemException;
this.addItem2Exception = addItem2Exception;
this.removeItemException = removeItemException;
}
public string? RegisteredClientName { get; private set; }
@@ -161,6 +415,26 @@ public sealed class MxAccessCommandExecutorTests
public int? UnregisterThreadId { get; private set; }
public int? AddItemServerHandle { get; private set; }
public string? AddItemDefinition { get; private set; }
public int? AddItemThreadId { get; private set; }
public int? AddItem2ServerHandle { get; private set; }
public string? AddItem2Definition { get; private set; }
public string? AddItem2Context { get; private set; }
public int? AddItem2ThreadId { get; private set; }
public int? RemoveItemServerHandle { get; private set; }
public int? RemovedItemHandle { get; private set; }
public int? RemoveItemThreadId { get; private set; }
public int Register(string clientName)
{
RegisteredClientName = clientName;
@@ -179,6 +453,54 @@ public sealed class MxAccessCommandExecutorTests
throw unregisterException;
}
}
public int AddItem(
int serverHandle,
string itemDefinition)
{
AddItemServerHandle = serverHandle;
AddItemDefinition = itemDefinition;
AddItemThreadId = Environment.CurrentManagedThreadId;
if (addItemException is not null)
{
throw addItemException;
}
return addItemHandle;
}
public int AddItem2(
int serverHandle,
string itemDefinition,
string itemContext)
{
AddItem2ServerHandle = serverHandle;
AddItem2Definition = itemDefinition;
AddItem2Context = itemContext;
AddItem2ThreadId = Environment.CurrentManagedThreadId;
if (addItem2Exception is not null)
{
throw addItem2Exception;
}
return addItem2Handle;
}
public void RemoveItem(
int serverHandle,
int itemHandle)
{
RemoveItemServerHandle = serverHandle;
RemovedItemHandle = itemHandle;
RemoveItemThreadId = Environment.CurrentManagedThreadId;
if (removeItemException is not null)
{
throw removeItemException;
}
}
}
private sealed class FakeMxAccessComObjectFactory : IMxAccessComObjectFactory
@@ -8,6 +8,11 @@ namespace MxGateway.Worker.Tests.MxAccess;
public sealed class MxAccessLiveComCreationTests
{
private const string LiveClientName = "MxGateway.Worker.Tests";
private const string DefaultLiveAddItemReference = "TestChildObject.TestInt";
private const string DefaultLiveAddItem2Definition = "TestInt";
private const string DefaultLiveAddItem2Context = "TestChildObject";
[Fact]
public async Task StartAsync_WhenOptedIn_CreatesInstalledMxAccessComObjectOnSta()
{
@@ -43,7 +48,7 @@ public sealed class MxAccessLiveComCreationTests
Kind = MxCommandKind.Register,
Register = new RegisterCommand
{
ClientName = "MxGateway.Worker.Tests",
ClientName = LiveClientName,
},
}));
@@ -65,6 +70,151 @@ public sealed class MxAccessLiveComCreationTests
Assert.Equal(ProtocolStatusCode.Ok, unregisterReply.ProtocolStatus.Code);
}
[Fact]
public async Task AddItemAndRemoveItem_WhenOptedIn_RoundTripsInstalledMxAccessItemHandle()
{
if (!RunLiveMxAccessTests())
{
return;
}
using MxAccessStaSession session = new();
await session.StartAsync(workerProcessId: 1234);
MxCommandReply registerReply = await RegisterLiveSessionAsync(session, "live-add-register");
int serverHandle = registerReply.Register.ServerHandle;
int itemHandle = 0;
try
{
MxCommandReply addItemReply = await session.DispatchAsync(new StaCommand(
"session-1",
"live-add-item",
new MxCommand
{
Kind = MxCommandKind.AddItem,
AddItem = new AddItemCommand
{
ServerHandle = serverHandle,
ItemDefinition = GetLiveAddItemReference(),
},
}));
Assert.Equal(ProtocolStatusCode.Ok, addItemReply.ProtocolStatus.Code);
Assert.True(addItemReply.AddItem.ItemHandle > 0);
itemHandle = addItemReply.AddItem.ItemHandle;
MxCommandReply removeItemReply = await session.DispatchAsync(new StaCommand(
"session-1",
"live-remove-item",
new MxCommand
{
Kind = MxCommandKind.RemoveItem,
RemoveItem = new RemoveItemCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
}));
Assert.Equal(ProtocolStatusCode.Ok, removeItemReply.ProtocolStatus.Code);
itemHandle = 0;
}
finally
{
if (itemHandle > 0)
{
await session.DispatchAsync(new StaCommand(
"session-1",
"live-remove-item-cleanup",
new MxCommand
{
Kind = MxCommandKind.RemoveItem,
RemoveItem = new RemoveItemCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
}));
}
await UnregisterLiveSessionAsync(session, serverHandle, "live-add-unregister");
}
}
[Fact]
public async Task AddItem2AndRemoveItem_WhenOptedIn_PreservesContextForInstalledMxAccess()
{
if (!RunLiveMxAccessTests())
{
return;
}
using MxAccessStaSession session = new();
await session.StartAsync(workerProcessId: 1234);
MxCommandReply registerReply = await RegisterLiveSessionAsync(session, "live-add2-register");
int serverHandle = registerReply.Register.ServerHandle;
int itemHandle = 0;
try
{
MxCommandReply addItem2Reply = await session.DispatchAsync(new StaCommand(
"session-1",
"live-add-item2",
new MxCommand
{
Kind = MxCommandKind.AddItem2,
AddItem2 = new AddItem2Command
{
ServerHandle = serverHandle,
ItemDefinition = DefaultLiveAddItem2Definition,
ItemContext = DefaultLiveAddItem2Context,
},
}));
Assert.Equal(ProtocolStatusCode.Ok, addItem2Reply.ProtocolStatus.Code);
Assert.True(addItem2Reply.AddItem2.ItemHandle > 0);
itemHandle = addItem2Reply.AddItem2.ItemHandle;
MxCommandReply removeItemReply = await session.DispatchAsync(new StaCommand(
"session-1",
"live-remove-item2",
new MxCommand
{
Kind = MxCommandKind.RemoveItem,
RemoveItem = new RemoveItemCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
}));
Assert.Equal(ProtocolStatusCode.Ok, removeItemReply.ProtocolStatus.Code);
itemHandle = 0;
}
finally
{
if (itemHandle > 0)
{
await session.DispatchAsync(new StaCommand(
"session-1",
"live-remove-item2-cleanup",
new MxCommand
{
Kind = MxCommandKind.RemoveItem,
RemoveItem = new RemoveItemCommand
{
ServerHandle = serverHandle,
ItemHandle = itemHandle,
},
}));
}
await UnregisterLiveSessionAsync(session, serverHandle, "live-add2-unregister");
}
}
private static bool RunLiveMxAccessTests()
{
return string.Equals(
@@ -72,4 +222,55 @@ public sealed class MxAccessLiveComCreationTests
"1",
StringComparison.Ordinal);
}
private static string GetLiveAddItemReference()
{
string itemReference = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_MXACCESS_ITEM");
return string.IsNullOrWhiteSpace(itemReference)
? DefaultLiveAddItemReference
: itemReference;
}
private static async Task<MxCommandReply> RegisterLiveSessionAsync(
MxAccessStaSession session,
string correlationId)
{
MxCommandReply reply = await session.DispatchAsync(new StaCommand(
"session-1",
correlationId,
new MxCommand
{
Kind = MxCommandKind.Register,
Register = new RegisterCommand
{
ClientName = LiveClientName,
},
}));
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.True(reply.Register.ServerHandle > 0);
return reply;
}
private static async Task UnregisterLiveSessionAsync(
MxAccessStaSession session,
int serverHandle,
string correlationId)
{
MxCommandReply unregisterReply = await session.DispatchAsync(new StaCommand(
"session-1",
correlationId,
new MxCommand
{
Kind = MxCommandKind.Unregister,
Unregister = new UnregisterCommand
{
ServerHandle = serverHandle,
},
}));
Assert.Equal(ProtocolStatusCode.Ok, unregisterReply.ProtocolStatus.Code);
}
}
+14 -2
View File
@@ -1,4 +1,5 @@
using System;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
@@ -11,6 +12,7 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
public const int DefaultConnectTimeoutMilliseconds = 30000;
private readonly int _connectTimeoutMilliseconds;
private readonly Func<Stream, WorkerFrameProtocolOptions, WorkerPipeSession> _sessionFactory;
public WorkerPipeClient()
: this(DefaultConnectTimeoutMilliseconds)
@@ -18,6 +20,15 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
}
public WorkerPipeClient(int connectTimeoutMilliseconds)
: this(
connectTimeoutMilliseconds,
(stream, frameOptions) => new WorkerPipeSession(stream, frameOptions))
{
}
public WorkerPipeClient(
int connectTimeoutMilliseconds,
Func<Stream, WorkerFrameProtocolOptions, WorkerPipeSession> sessionFactory)
{
if (connectTimeoutMilliseconds <= 0)
{
@@ -26,6 +37,7 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
"Worker pipe connect timeout must be greater than zero.");
}
_sessionFactory = sessionFactory ?? throw new ArgumentNullException(nameof(sessionFactory));
_connectTimeoutMilliseconds = connectTimeoutMilliseconds;
}
@@ -48,8 +60,8 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
await ConnectAsync(pipe, cancellationToken).ConfigureAwait(false);
WorkerPipeSession session = new(pipe, frameOptions);
await session.CompleteStartupHandshakeAsync(cancellationToken).ConfigureAwait(false);
WorkerPipeSession session = _sessionFactory(pipe, frameOptions);
await session.RunAsync(cancellationToken).ConfigureAwait(false);
}
private Task ConnectAsync(
+329 -5
View File
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
using Google.Protobuf.WellKnownTypes;
using MxGateway.Contracts.Proto;
using MxGateway.Worker.MxAccess;
using MxGateway.Worker.Sta;
namespace MxGateway.Worker.Ipc;
@@ -13,10 +14,14 @@ public sealed class WorkerPipeSession
{
private readonly WorkerFrameProtocolOptions _options;
private readonly Func<int> _processIdProvider;
private readonly Func<IWorkerRuntimeSession> _runtimeSessionFactory;
private readonly WorkerPipeSessionOptions _sessionOptions;
private readonly WorkerFrameReader _reader;
private readonly WorkerFrameWriter _writer;
private MxAccessStaSession? _mxAccessStaSession;
private IWorkerRuntimeSession? _runtimeSession;
private long _nextSequence;
private WorkerState _state = WorkerState.Starting;
private bool _watchdogFaultSent;
public WorkerPipeSession(
Stream stream,
@@ -34,11 +39,49 @@ public sealed class WorkerPipeSession
WorkerFrameWriter writer,
WorkerFrameProtocolOptions options,
Func<int> processIdProvider)
: this(
reader,
writer,
options,
processIdProvider,
new WorkerPipeSessionOptions(),
() => new MxAccessStaSession())
{
}
public WorkerPipeSession(
WorkerFrameReader reader,
WorkerFrameWriter writer,
WorkerFrameProtocolOptions options,
Func<int> processIdProvider,
WorkerPipeSessionOptions sessionOptions,
Func<IWorkerRuntimeSession> runtimeSessionFactory)
{
_reader = reader ?? throw new ArgumentNullException(nameof(reader));
_writer = writer ?? throw new ArgumentNullException(nameof(writer));
_options = options ?? throw new ArgumentNullException(nameof(options));
_processIdProvider = processIdProvider ?? throw new ArgumentNullException(nameof(processIdProvider));
_sessionOptions = sessionOptions ?? throw new ArgumentNullException(nameof(sessionOptions));
_runtimeSessionFactory = runtimeSessionFactory ?? throw new ArgumentNullException(nameof(runtimeSessionFactory));
_sessionOptions.Validate();
}
public async Task RunAsync(CancellationToken cancellationToken = default)
{
_runtimeSession = _runtimeSessionFactory();
try
{
await CompleteStartupHandshakeAsync(
token => _runtimeSession.StartAsync(_processIdProvider(), token),
cancellationToken).ConfigureAwait(false);
await RunMessageLoopAsync(cancellationToken).ConfigureAwait(false);
}
finally
{
_runtimeSession?.Dispose();
_runtimeSession = null;
_state = WorkerState.Stopped;
}
}
public Task CompleteStartupHandshakeAsync(CancellationToken cancellationToken = default)
@@ -76,11 +119,14 @@ public sealed class WorkerPipeSession
try
{
WorkerEnvelope envelope = await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
_state = WorkerState.Handshaking;
ValidateGatewayHello(envelope);
await WriteWorkerHelloAsync(cancellationToken).ConfigureAwait(false);
_state = WorkerState.InitializingSta;
WorkerReady ready = await initializeMxAccessAsync(cancellationToken).ConfigureAwait(false);
await WriteWorkerReadyAsync(ready, cancellationToken).ConfigureAwait(false);
_state = WorkerState.Ready;
}
catch (WorkerFrameProtocolException exception)
{
@@ -140,6 +186,174 @@ public sealed class WorkerPipeSession
return _writer.WriteAsync(CreateEnvelope(ready), cancellationToken);
}
private async Task RunMessageLoopAsync(CancellationToken cancellationToken)
{
using CancellationTokenSource heartbeatCancellation = CancellationTokenSource
.CreateLinkedTokenSource(cancellationToken);
Task heartbeatTask = RunHeartbeatLoopAsync(heartbeatCancellation.Token);
try
{
while (!cancellationToken.IsCancellationRequested)
{
Task<WorkerEnvelope> readTask = _reader.ReadAsync(cancellationToken);
Task completedTask = await Task.WhenAny(readTask, heartbeatTask).ConfigureAwait(false);
if (completedTask == heartbeatTask)
{
await heartbeatTask.ConfigureAwait(false);
}
WorkerEnvelope envelope = await readTask.ConfigureAwait(false);
bool keepReading = await DispatchGatewayEnvelopeAsync(envelope, cancellationToken).ConfigureAwait(false);
if (!keepReading)
{
return;
}
}
}
finally
{
heartbeatCancellation.Cancel();
try
{
await heartbeatTask.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
}
}
}
private async Task<bool> DispatchGatewayEnvelopeAsync(
WorkerEnvelope envelope,
CancellationToken cancellationToken)
{
switch (envelope.BodyCase)
{
case WorkerEnvelope.BodyOneofCase.WorkerCommand:
_ = ProcessCommandAsync(envelope, cancellationToken);
return true;
case WorkerEnvelope.BodyOneofCase.WorkerShutdown:
await ShutdownAsync(envelope.WorkerShutdown, cancellationToken).ConfigureAwait(false);
return false;
case WorkerEnvelope.BodyOneofCase.WorkerCancel:
return true;
default:
throw new WorkerFrameProtocolException(
WorkerFrameProtocolErrorCode.UnexpectedEnvelopeBody,
$"Worker received unexpected gateway envelope body {envelope.BodyCase}.");
}
}
private async Task ProcessCommandAsync(
WorkerEnvelope envelope,
CancellationToken cancellationToken)
{
IWorkerRuntimeSession runtimeSession = _runtimeSession
?? throw new InvalidOperationException("Worker runtime session has not been initialized.");
WorkerCommand workerCommand = envelope.WorkerCommand;
MxCommand command = workerCommand.Command;
StaCommand staCommand = new(
_options.SessionId,
envelope.CorrelationId,
command,
workerCommand.EnqueueTimestamp,
cancellationToken);
try
{
MxCommandReply reply = await runtimeSession.DispatchAsync(staCommand).ConfigureAwait(false);
await _writer
.WriteAsync(
CreateEnvelope(new WorkerCommandReply
{
Reply = reply,
CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
}),
cancellationToken)
.ConfigureAwait(false);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
_state = WorkerState.Faulted;
await TryWriteFaultAsync(
CreateFault(
WorkerFaultCategory.MxaccessCommandFailed,
staCommand.MethodName,
exception),
cancellationToken).ConfigureAwait(false);
}
}
private async Task ShutdownAsync(
WorkerShutdown shutdown,
CancellationToken cancellationToken)
{
_state = WorkerState.ShuttingDown;
_runtimeSession?.RequestShutdown();
await _writer
.WriteAsync(
CreateEnvelope(
new WorkerShutdownAck
{
Status = new ProtocolStatus
{
Code = ProtocolStatusCode.Ok,
Message = string.IsNullOrWhiteSpace(shutdown.Reason)
? "Worker shutdown accepted."
: $"Worker shutdown accepted: {shutdown.Reason}",
},
}),
cancellationToken)
.ConfigureAwait(false);
}
private async Task RunHeartbeatLoopAsync(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
await Task.Delay(_sessionOptions.HeartbeatInterval, cancellationToken).ConfigureAwait(false);
IWorkerRuntimeSession? runtimeSession = _runtimeSession;
if (runtimeSession is null)
{
continue;
}
WorkerRuntimeHeartbeatSnapshot snapshot = runtimeSession.CaptureHeartbeat();
await _writer
.WriteAsync(CreateEnvelope(CreateHeartbeat(snapshot)), cancellationToken)
.ConfigureAwait(false);
await ReportWatchdogFaultIfNeededAsync(snapshot, cancellationToken).ConfigureAwait(false);
}
}
private async Task ReportWatchdogFaultIfNeededAsync(
WorkerRuntimeHeartbeatSnapshot snapshot,
CancellationToken cancellationToken)
{
TimeSpan staleFor = DateTimeOffset.UtcNow - snapshot.LastStaActivityUtc;
if (staleFor <= _sessionOptions.HeartbeatGrace)
{
_watchdogFaultSent = false;
return;
}
if (_watchdogFaultSent)
{
return;
}
_watchdogFaultSent = true;
await TryWriteFaultAsync(
CreateFault(
WorkerFaultCategory.StaHung,
snapshot.CurrentCommandCorrelationId,
$"STA activity is stale by {staleFor}."),
cancellationToken).ConfigureAwait(false);
}
private async Task TryWriteFaultAsync(
WorkerFrameProtocolException exception,
CancellationToken cancellationToken)
@@ -178,6 +392,25 @@ public sealed class WorkerPipeSession
}
}
private async Task TryWriteFaultAsync(
WorkerFault fault,
CancellationToken cancellationToken)
{
try
{
await _writer
.WriteAsync(CreateEnvelope(fault), cancellationToken)
.ConfigureAwait(false);
}
catch (Exception faultWriteException) when (
faultWriteException is IOException
|| faultWriteException is ObjectDisposedException
|| faultWriteException is WorkerFrameProtocolException)
{
// The runtime fault remains observable through worker exit or pipe closure.
}
}
private WorkerEnvelope CreateEnvelope(WorkerHello hello)
{
return CreateBaseEnvelope(hello);
@@ -193,6 +426,21 @@ public sealed class WorkerPipeSession
return CreateBaseEnvelope(fault);
}
private WorkerEnvelope CreateEnvelope(WorkerCommandReply reply)
{
return CreateBaseEnvelope(reply);
}
private WorkerEnvelope CreateEnvelope(WorkerShutdownAck shutdownAck)
{
return CreateBaseEnvelope(shutdownAck);
}
private WorkerEnvelope CreateEnvelope(WorkerHeartbeat heartbeat)
{
return CreateBaseEnvelope(heartbeat);
}
private WorkerEnvelope CreateBaseEnvelope(WorkerHello body)
{
WorkerEnvelope envelope = CreateBaseEnvelope();
@@ -214,6 +462,28 @@ public sealed class WorkerPipeSession
return envelope;
}
private WorkerEnvelope CreateBaseEnvelope(WorkerCommandReply body)
{
WorkerEnvelope envelope = CreateBaseEnvelope();
envelope.CorrelationId = body.Reply?.CorrelationId ?? string.Empty;
envelope.WorkerCommandReply = body;
return envelope;
}
private WorkerEnvelope CreateBaseEnvelope(WorkerShutdownAck body)
{
WorkerEnvelope envelope = CreateBaseEnvelope();
envelope.WorkerShutdownAck = body;
return envelope;
}
private WorkerEnvelope CreateBaseEnvelope(WorkerHeartbeat body)
{
WorkerEnvelope envelope = CreateBaseEnvelope();
envelope.WorkerHeartbeat = body;
return envelope;
}
private WorkerEnvelope CreateBaseEnvelope()
{
return new WorkerEnvelope
@@ -231,21 +501,39 @@ public sealed class WorkerPipeSession
private async Task<WorkerReady> InitializeMxAccessAsync(CancellationToken cancellationToken)
{
_mxAccessStaSession = new MxAccessStaSession();
_runtimeSession = new MxAccessStaSession();
try
{
return await _mxAccessStaSession
return await _runtimeSession
.StartAsync(_processIdProvider(), cancellationToken)
.ConfigureAwait(false);
}
catch
{
_mxAccessStaSession.Dispose();
_mxAccessStaSession = null;
_runtimeSession.Dispose();
_runtimeSession = null;
throw;
}
}
private WorkerHeartbeat CreateHeartbeat(WorkerRuntimeHeartbeatSnapshot snapshot)
{
WorkerState state = string.IsNullOrWhiteSpace(snapshot.CurrentCommandCorrelationId)
? _state
: WorkerState.ExecutingCommand;
return new WorkerHeartbeat
{
WorkerProcessId = _processIdProvider(),
State = state,
LastStaActivityTimestamp = Timestamp.FromDateTimeOffset(snapshot.LastStaActivityUtc),
PendingCommandCount = snapshot.PendingCommandCount,
OutboundEventQueueDepth = snapshot.OutboundEventQueueDepth,
LastEventSequence = snapshot.LastEventSequence,
CurrentCommandCorrelationId = snapshot.CurrentCommandCorrelationId,
};
}
private WorkerReady CreateWorkerReady()
{
return new WorkerReady
@@ -295,6 +583,42 @@ public sealed class WorkerPipeSession
return fault;
}
private static WorkerFault CreateFault(
WorkerFaultCategory category,
string commandMethod,
Exception exception)
{
WorkerFault fault = CreateFault(
category,
commandMethod,
exception.Message);
fault.ExceptionType = exception.GetType().FullName ?? string.Empty;
fault.ProtocolStatus = new ProtocolStatus
{
Code = ProtocolStatusCode.WorkerUnavailable,
Message = exception.Message,
};
return fault;
}
private static WorkerFault CreateFault(
WorkerFaultCategory category,
string commandMethod,
string diagnosticMessage)
{
return new WorkerFault
{
Category = category,
CommandMethod = commandMethod ?? string.Empty,
DiagnosticMessage = diagnosticMessage,
ProtocolStatus = new ProtocolStatus
{
Code = ProtocolStatusCode.WorkerUnavailable,
Message = diagnosticMessage,
},
};
}
private static WorkerFaultCategory MapFaultCategory(WorkerFrameProtocolErrorCode errorCode)
{
return errorCode switch
@@ -0,0 +1,36 @@
using System;
namespace MxGateway.Worker.Ipc;
public sealed class WorkerPipeSessionOptions
{
public static readonly TimeSpan DefaultHeartbeatInterval = TimeSpan.FromSeconds(5);
public static readonly TimeSpan DefaultHeartbeatGrace = TimeSpan.FromSeconds(15);
public WorkerPipeSessionOptions()
{
HeartbeatInterval = DefaultHeartbeatInterval;
HeartbeatGrace = DefaultHeartbeatGrace;
}
public TimeSpan HeartbeatInterval { get; set; }
public TimeSpan HeartbeatGrace { get; set; }
public void Validate()
{
if (HeartbeatInterval <= TimeSpan.Zero)
{
throw new ArgumentOutOfRangeException(
nameof(HeartbeatInterval),
"Worker heartbeat interval must be greater than zero.");
}
if (HeartbeatGrace <= TimeSpan.Zero)
{
throw new ArgumentOutOfRangeException(
nameof(HeartbeatGrace),
"Worker heartbeat grace must be greater than zero.");
}
}
}
@@ -5,4 +5,17 @@ public interface IMxAccessServer
int Register(string clientName);
void Unregister(int serverHandle);
int AddItem(
int serverHandle,
string itemDefinition);
int AddItem2(
int serverHandle,
string itemDefinition,
string itemContext);
void RemoveItem(
int serverHandle,
int itemHandle);
}
@@ -0,0 +1,20 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using MxGateway.Contracts.Proto;
using MxGateway.Worker.Sta;
namespace MxGateway.Worker.MxAccess;
public interface IWorkerRuntimeSession : IDisposable
{
Task<WorkerReady> StartAsync(
int workerProcessId,
CancellationToken cancellationToken = default);
Task<MxCommandReply> DispatchAsync(StaCommand command);
WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat();
void RequestShutdown();
}
@@ -35,6 +35,44 @@ public sealed class MxAccessComServer : IMxAccessServer
Invoke(nameof(Unregister), serverHandle);
}
public int AddItem(
int serverHandle,
string itemDefinition)
{
if (mxAccessComObject is ILMXProxyServer mxAccessServer)
{
return mxAccessServer.AddItem(serverHandle, itemDefinition);
}
return (int)Invoke(nameof(AddItem), serverHandle, itemDefinition);
}
public int AddItem2(
int serverHandle,
string itemDefinition,
string itemContext)
{
if (mxAccessComObject is ILMXProxyServer3 mxAccessServer)
{
return mxAccessServer.AddItem2(serverHandle, itemDefinition, itemContext);
}
return (int)Invoke(nameof(AddItem2), serverHandle, itemDefinition, itemContext);
}
public void RemoveItem(
int serverHandle,
int itemHandle)
{
if (mxAccessComObject is ILMXProxyServer mxAccessServer)
{
mxAccessServer.RemoveItem(serverHandle, itemHandle);
return;
}
Invoke(nameof(RemoveItem), serverHandle, itemHandle);
}
private object Invoke(
string methodName,
params object[] arguments)
@@ -34,6 +34,9 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
{
MxCommandKind.Register => ExecuteRegister(command),
MxCommandKind.Unregister => ExecuteUnregister(command),
MxCommandKind.AddItem => ExecuteAddItem(command),
MxCommandKind.AddItem2 => ExecuteAddItem2(command),
MxCommandKind.RemoveItem => ExecuteRemoveItem(command),
_ => CreateInvalidRequestReply(command, $"Unsupported MXAccess command kind {command.Kind}."),
};
}
@@ -67,6 +70,66 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
return CreateOkReply(command);
}
private MxCommandReply ExecuteAddItem(StaCommand command)
{
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.AddItem)
{
return CreateInvalidRequestReply(command, "AddItem command payload is required.");
}
AddItemCommand addItemCommand = command.Command.AddItem;
int itemHandle = session.AddItem(
addItemCommand.ServerHandle,
addItemCommand.ItemDefinition);
MxCommandReply reply = CreateOkReply(command);
reply.ReturnValue = variantConverter.Convert(itemHandle);
reply.AddItem = new AddItemReply
{
ItemHandle = itemHandle,
};
return reply;
}
private MxCommandReply ExecuteAddItem2(StaCommand command)
{
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.AddItem2)
{
return CreateInvalidRequestReply(command, "AddItem2 command payload is required.");
}
AddItem2Command addItem2Command = command.Command.AddItem2;
int itemHandle = session.AddItem2(
addItem2Command.ServerHandle,
addItem2Command.ItemDefinition,
addItem2Command.ItemContext);
MxCommandReply reply = CreateOkReply(command);
reply.ReturnValue = variantConverter.Convert(itemHandle);
reply.AddItem2 = new AddItem2Reply
{
ItemHandle = itemHandle,
};
return reply;
}
private MxCommandReply ExecuteRemoveItem(StaCommand command)
{
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.RemoveItem)
{
return CreateInvalidRequestReply(command, "RemoveItem command payload is required.");
}
RemoveItemCommand removeItemCommand = command.Command.RemoveItem;
session.RemoveItem(
removeItemCommand.ServerHandle,
removeItemCommand.ItemHandle);
return CreateOkReply(command);
}
private static MxCommandReply CreateOkReply(StaCommand command)
{
return new MxCommandReply
@@ -6,12 +6,19 @@ namespace MxGateway.Worker.MxAccess;
public sealed class MxAccessHandleRegistry
{
private readonly Dictionary<int, RegisteredServerHandle> serverHandles = new();
private readonly Dictionary<long, RegisteredItemHandle> itemHandles = new();
public IReadOnlyList<RegisteredServerHandle> ServerHandles => serverHandles
.Values
.OrderBy(handle => handle.ServerHandle)
.ToArray();
public IReadOnlyList<RegisteredItemHandle> ItemHandles => itemHandles
.Values
.OrderBy(handle => handle.ServerHandle)
.ThenBy(handle => handle.ItemHandle)
.ToArray();
public void RegisterServerHandle(
int serverHandle,
string clientName)
@@ -22,10 +29,54 @@ public sealed class MxAccessHandleRegistry
public void UnregisterServerHandle(int serverHandle)
{
serverHandles.Remove(serverHandle);
foreach (long key in itemHandles
.Where(pair => pair.Value.ServerHandle == serverHandle)
.Select(pair => pair.Key)
.ToArray())
{
itemHandles.Remove(key);
}
}
public bool ContainsServerHandle(int serverHandle)
{
return serverHandles.ContainsKey(serverHandle);
}
public void RegisterItemHandle(
int serverHandle,
int itemHandle,
string itemDefinition,
string itemContext,
bool hasItemContext)
{
itemHandles[CreateItemKey(serverHandle, itemHandle)] = new RegisteredItemHandle(
serverHandle,
itemHandle,
itemDefinition,
itemContext,
hasItemContext);
}
public void RemoveItemHandle(
int serverHandle,
int itemHandle)
{
itemHandles.Remove(CreateItemKey(serverHandle, itemHandle));
}
public bool ContainsItemHandle(
int serverHandle,
int itemHandle)
{
return itemHandles.ContainsKey(CreateItemKey(serverHandle, itemHandle));
}
private static long CreateItemKey(
int serverHandle,
int itemHandle)
{
return ((long)serverHandle << 32) | (uint)itemHandle;
}
}
@@ -106,6 +106,51 @@ public sealed class MxAccessSession : IDisposable
handleRegistry.UnregisterServerHandle(serverHandle);
}
public int AddItem(
int serverHandle,
string itemDefinition)
{
ThrowIfDisposed();
int itemHandle = mxAccessServer.AddItem(serverHandle, itemDefinition);
handleRegistry.RegisterItemHandle(
serverHandle,
itemHandle,
itemDefinition,
string.Empty,
hasItemContext: false);
return itemHandle;
}
public int AddItem2(
int serverHandle,
string itemDefinition,
string itemContext)
{
ThrowIfDisposed();
int itemHandle = mxAccessServer.AddItem2(serverHandle, itemDefinition, itemContext);
handleRegistry.RegisterItemHandle(
serverHandle,
itemHandle,
itemDefinition,
itemContext,
hasItemContext: true);
return itemHandle;
}
public void RemoveItem(
int serverHandle,
int itemHandle)
{
ThrowIfDisposed();
mxAccessServer.RemoveItem(serverHandle, itemHandle);
handleRegistry.RemoveItemHandle(serverHandle, itemHandle);
}
public void Dispose()
{
if (disposed)
@@ -7,7 +7,7 @@ using MxGateway.Worker.Sta;
namespace MxGateway.Worker.MxAccess;
public sealed class MxAccessStaSession : IDisposable
public sealed class MxAccessStaSession : IWorkerRuntimeSession
{
private readonly IMxAccessComObjectFactory factory;
private readonly IMxAccessEventSink eventSink;
@@ -68,6 +68,30 @@ public sealed class MxAccessStaSession : IDisposable
return commandDispatcher.DispatchAsync(command);
}
public WorkerRuntimeHeartbeatSnapshot CaptureHeartbeat()
{
uint pendingCommandCount = 0;
string currentCommandCorrelationId = string.Empty;
if (commandDispatcher is not null)
{
pendingCommandCount = (uint)commandDispatcher.PendingCommandCount;
currentCommandCorrelationId = commandDispatcher.CurrentCommandCorrelationId;
}
return new WorkerRuntimeHeartbeatSnapshot(
staRuntime.LastActivityUtc,
pendingCommandCount,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId);
}
public void RequestShutdown()
{
commandDispatcher?.RequestShutdown();
}
public Task<IReadOnlyList<RegisteredServerHandle>> GetRegisteredServerHandlesAsync(
CancellationToken cancellationToken = default)
{
@@ -81,6 +105,19 @@ public sealed class MxAccessStaSession : IDisposable
cancellationToken);
}
public Task<IReadOnlyList<RegisteredItemHandle>> GetRegisteredItemHandlesAsync(
CancellationToken cancellationToken = default)
{
if (session is null)
{
throw new InvalidOperationException("MXAccess COM session has not been started.");
}
return staRuntime.InvokeAsync(
() => session.HandleRegistry.ItemHandles,
cancellationToken);
}
public void Dispose()
{
if (disposed)
@@ -88,7 +125,7 @@ public sealed class MxAccessStaSession : IDisposable
return;
}
commandDispatcher?.RequestShutdown();
RequestShutdown();
if (session is not null)
{
@@ -0,0 +1,28 @@
namespace MxGateway.Worker.MxAccess;
public sealed class RegisteredItemHandle
{
public RegisteredItemHandle(
int serverHandle,
int itemHandle,
string itemDefinition,
string itemContext,
bool hasItemContext)
{
ServerHandle = serverHandle;
ItemHandle = itemHandle;
ItemDefinition = itemDefinition;
ItemContext = itemContext;
HasItemContext = hasItemContext;
}
public int ServerHandle { get; }
public int ItemHandle { get; }
public string ItemDefinition { get; }
public string ItemContext { get; }
public bool HasItemContext { get; }
}
@@ -0,0 +1,30 @@
using System;
namespace MxGateway.Worker.MxAccess;
public sealed class WorkerRuntimeHeartbeatSnapshot
{
public WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset lastStaActivityUtc,
uint pendingCommandCount,
uint outboundEventQueueDepth,
ulong lastEventSequence,
string currentCommandCorrelationId)
{
LastStaActivityUtc = lastStaActivityUtc;
PendingCommandCount = pendingCommandCount;
OutboundEventQueueDepth = outboundEventQueueDepth;
LastEventSequence = lastEventSequence;
CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty;
}
public DateTimeOffset LastStaActivityUtc { get; }
public uint PendingCommandCount { get; }
public uint OutboundEventQueueDepth { get; }
public ulong LastEventSequence { get; }
public string CurrentCommandCorrelationId { get; }
}
+1 -1
View File
@@ -84,7 +84,7 @@ public static class WorkerApplication
pipeClient.RunAsync(options).GetAwaiter().GetResult();
logger.Information("WorkerPipeHandshakeSucceeded", new Dictionary<string, object?>
logger.Information("WorkerPipeSessionCompleted", new Dictionary<string, object?>
{
["session_id"] = options.SessionId,
["pipe_name"] = options.PipeName,