Compare commits
6 Commits
focas-tier
...
focas-tier
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8d88ffa14d | ||
| 446a5c022c | |||
|
|
5033609944 | ||
| 9034294b77 | |||
|
|
3892555631 | ||
| 3609a5c676 |
@@ -69,14 +69,32 @@ covers the common address shapes; per-model quirks are not stressed.
|
||||
- Parameter range enforcement (CNC rejects out-of-range writes)
|
||||
- MTB (machine tool builder) custom screens that expose non-standard data
|
||||
|
||||
### 5. Tier-C process isolation behavior
|
||||
### 5. Tier-C process isolation — architecture shipped, Fwlib32 integration hardware-gated
|
||||
|
||||
Per driver-stability.md, FOCAS should run process-isolated because
|
||||
`Fwlib32.dll` has documented crash modes. The test suite runs in-process +
|
||||
only exercises the happy path + mapped error codes — a native access
|
||||
violation from the DLL would take the test host down. The process-isolation
|
||||
path (similar to Galaxy's out-of-process Host) has been scoped but not
|
||||
implemented.
|
||||
The Tier-C architecture is now in place as of PRs #169–#173 (FOCAS
|
||||
PR A–E, task #220):
|
||||
|
||||
- `Driver.FOCAS.Shared` carries MessagePack IPC contracts
|
||||
- `Driver.FOCAS.Host` (.NET 4.8 x86 Windows service via NSSM) accepts
|
||||
a connection on a strictly-ACL'd named pipe + dispatches frames to
|
||||
an `IFocasBackend`
|
||||
- `Driver.FOCAS.Ipc.IpcFocasClient` implements the `IFocasClient` DI
|
||||
seam by forwarding over IPC — swap the DI registration and the
|
||||
driver runs Tier-C with zero other changes
|
||||
- `Driver.FOCAS.Supervisor.FocasHostSupervisor` owns the spawn +
|
||||
heartbeat + respawn + 3-in-5min crash-loop breaker + sticky alert
|
||||
- `Driver.FOCAS.Host.Stability.PostMortemMmf` ↔
|
||||
`Driver.FOCAS.Supervisor.PostMortemReader` — ring-buffer of the
|
||||
last ~1000 IPC operations survives a Host crash
|
||||
|
||||
The one remaining gap is the production `FwlibHostedBackend`: an
|
||||
`IFocasBackend` implementation that wraps the licensed
|
||||
`Fwlib32.dll` P/Invoke. That's hardware-gated on task #222 — we
|
||||
need a CNC on the bench (or the licensed FANUC developer kit DLL
|
||||
with a test harness) to validate it. Until then, the Host ships
|
||||
`FakeFocasBackend` + `UnconfiguredFocasBackend`. Setting
|
||||
`OTOPCUA_FOCAS_BACKEND=fake` lets operators smoke-test the whole
|
||||
Tier-C pipeline end-to-end without any CNC.
|
||||
|
||||
## When to trust FOCAS tests, when to reach for a rig
|
||||
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
# FOCAS Tier-C isolation — plan for task #220
|
||||
|
||||
> **Status**: DRAFT — not yet started. Tracks the multi-PR work to
|
||||
> move `Fwlib32.dll` behind an out-of-process host, mirroring the
|
||||
> Galaxy Tier-C split in [`phase-2-galaxy-out-of-process.md`](phase-2-galaxy-out-of-process.md).
|
||||
> **Status**: PRs A–E shipped. Architecture is in place; the only
|
||||
> remaining FOCAS work is the hardware-dependent production
|
||||
> integration of `Fwlib32.dll` into a real `IFocasBackend`
|
||||
> (`FwlibHostedBackend`), which needs an actual CNC on the bench
|
||||
> and is tracked as a follow-up on #220.
|
||||
>
|
||||
> **Pre-reqs shipped** (this PR): version matrix + pre-flight
|
||||
> validation + unit tests. Those close the cheap half of the
|
||||
> hardware-free stability gap. Tier-C closes the expensive half.
|
||||
> **Pre-reqs shipped**: version matrix + pre-flight validation
|
||||
> (PR #168 — the cheap half of the hardware-free stability gap).
|
||||
|
||||
## Why isolate
|
||||
|
||||
@@ -79,32 +80,41 @@ its own timer + pushes change notifications so the Proxy doesn't
|
||||
round-trip per poll. Matches `Driver.Galaxy.Host` subscription
|
||||
forwarding.
|
||||
|
||||
## PR sequence (proposed)
|
||||
## PR sequence — shipped
|
||||
|
||||
1. **PR A — shared contracts**
|
||||
Create `Driver.FOCAS.Shared` with the MessagePack DTOs. No
|
||||
behaviour change. ~200 LOC + round-trip tests for each DTO.
|
||||
2. **PR B — Host project skeleton**
|
||||
Create `Driver.FOCAS.Host` .NET 4.8 x86 project, NSSM wrapper,
|
||||
pipe server scaffold with the same ACL + caller-SID + shared
|
||||
secret plumbing as Galaxy.Host. No Fwlib32 wiring yet — returns
|
||||
`NotImplemented` for everything. ~400 LOC.
|
||||
3. **PR C — Move Fwlib32 calls into Host**
|
||||
Move `FocasNativeSession`, `FocasTagReader`, `FocasTagWriter`,
|
||||
`FocasPmcBitRmw` + the STA thread into the Host. Proxy forwards
|
||||
over IPC. This is the biggest PR — probably 800-1500 LOC of
|
||||
move-with-translation. Existing unit tests keep passing because
|
||||
`IFocasTagFactory` is the DI seam the tests inject against.
|
||||
4. **PR D — Supervisor + respawn**
|
||||
Proxy-side heartbeat + respawn + crash-loop circuit breaker +
|
||||
BackPressure fan-out on Host death. ~500 LOC + chaos tests.
|
||||
5. **PR E — Post-mortem MMF + operational glue**
|
||||
MMF writer in Host, reader in Proxy. Install scripts for the
|
||||
new `OtOpcUaFocasHost` Windows service. Docs. ~300 LOC.
|
||||
1. **PR A (#169) — shared contracts** ✅
|
||||
`Driver.FOCAS.Shared` netstandard2.0 with MessagePack DTOs for every
|
||||
IPC surface (Hello/Heartbeat/OpenSession/Read/Write/PmcBitWrite/
|
||||
Subscribe/Probe/RuntimeStatus/Recycle/ErrorResponse) + FrameReader/
|
||||
FrameWriter + 24 round-trip tests.
|
||||
2. **PR B (#170) — Host project skeleton** ✅
|
||||
`Driver.FOCAS.Host` net48 x86 Windows Service entry point,
|
||||
`PipeAcl` + `PipeServer` + `IFrameHandler` + `StubFrameHandler`.
|
||||
ACL denies LocalSystem/Administrators; Hello verifies
|
||||
shared-secret + protocol major. 3 handshake tests.
|
||||
3. **PR C (#171) — IPC path end-to-end** ✅
|
||||
Proxy `Ipc/FocasIpcClient` + `Ipc/IpcFocasClient` (implements
|
||||
IFocasClient via IPC). Host `Backend/IFocasBackend` +
|
||||
`FakeFocasBackend` + `UnconfiguredFocasBackend` +
|
||||
`Ipc/FwlibFrameHandler` replacing the stub. 13 new round-trip
|
||||
tests via in-memory loopback.
|
||||
4. **PR D (#172) — Supervisor + respawn** ✅
|
||||
`Supervisor/Backoff` (5s→15s→60s) + `CircuitBreaker` (3-in-5min →
|
||||
1h→4h→manual) + `HeartbeatMonitor` + `IHostProcessLauncher` +
|
||||
`FocasHostSupervisor`. 14 tests.
|
||||
5. **PR E — Ops glue** ✅ (this PR)
|
||||
`ProcessHostLauncher` (real Process.Start + FocasIpcClient
|
||||
connect), `Host/Stability/PostMortemMmf` (magic 'OFPC') +
|
||||
Proxy `Supervisor/PostMortemReader`, `scripts/install/
|
||||
Install-FocasHost.ps1` + `Uninstall-FocasHost.ps1` NSSM wrappers.
|
||||
7 tests (4 MMF round-trip + 3 reader format compatibility).
|
||||
|
||||
Total estimate: 2200-3200 LOC across 5 PRs. Consistent with Galaxy
|
||||
Tier-C but narrower since FOCAS has no Historian + no alarm
|
||||
history.
|
||||
**Post-shipment totals: 189 FOCAS driver tests + 24 Shared tests + 13 Host tests = 226 FOCAS-family tests green.**
|
||||
|
||||
What remains is hardware-dependent: wiring `Fwlib32.dll` P/Invoke
|
||||
into a real `FwlibHostedBackend` implementation of `IFocasBackend`
|
||||
+ validating against a live CNC. The architecture is all the
|
||||
plumbing that work needs.
|
||||
|
||||
## Testing without hardware
|
||||
|
||||
|
||||
108
scripts/install/Install-FocasHost.ps1
Normal file
108
scripts/install/Install-FocasHost.ps1
Normal file
@@ -0,0 +1,108 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Registers the OtOpcUaFocasHost Windows service. Optional companion to
|
||||
Install-Services.ps1 — only run this on nodes where FOCAS driver instances will run
|
||||
with Tier-C process isolation enabled.
|
||||
|
||||
.DESCRIPTION
|
||||
FOCAS PR #220 / Tier-C isolation plan. Wraps OtOpcUa.Driver.FOCAS.Host.exe (net48 x86)
|
||||
as a Windows service using NSSM, running under the same service account as the main
|
||||
OtOpcUa service so the named-pipe ACL works. Passes the per-process shared secret via
|
||||
environment variable at service-start time so it never hits disk.
|
||||
|
||||
.PARAMETER InstallRoot
|
||||
Where the FOCAS Host binaries live (typically
|
||||
C:\Program Files\OtOpcUa\Driver.FOCAS.Host).
|
||||
|
||||
.PARAMETER ServiceAccount
|
||||
Service account SID or DOMAIN\name. Must match the main OtOpcUa server account so the
|
||||
PipeAcl match succeeds.
|
||||
|
||||
.PARAMETER FocasSharedSecret
|
||||
Per-process secret passed via env var. Generated freshly per install if not supplied.
|
||||
|
||||
.PARAMETER FocasBackend
|
||||
Backend selector for the Host process. One of:
|
||||
fwlib32 (default — real Fanuc Fwlib32.dll integration; requires licensed DLL on PATH)
|
||||
fake (in-memory; smoke-test mode)
|
||||
unconfigured (safe default returning structured errors; use until hardware is wired)
|
||||
|
||||
.PARAMETER FocasPipeName
|
||||
Pipe name the Host listens on. Default: OtOpcUaFocas.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-FocasHost.ps1 -InstallRoot 'C:\Program Files\OtOpcUa\Driver.FOCAS.Host' `
|
||||
-ServiceAccount 'OTOPCUA\svc-otopcua' -FocasBackend fwlib32
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$InstallRoot,
|
||||
[Parameter(Mandatory)] [string]$ServiceAccount,
|
||||
[string]$FocasSharedSecret,
|
||||
[ValidateSet('fwlib32','fake','unconfigured')] [string]$FocasBackend = 'unconfigured',
|
||||
[string]$FocasPipeName = 'OtOpcUaFocas',
|
||||
[string]$ServiceName = 'OtOpcUaFocasHost',
|
||||
[string]$NssmPath = 'C:\Program Files\nssm\nssm.exe'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Resolve-Sid {
|
||||
param([string]$Account)
|
||||
if ($Account -match '^S-\d-\d+') { return $Account }
|
||||
try {
|
||||
$nt = New-Object System.Security.Principal.NTAccount($Account)
|
||||
return $nt.Translate([System.Security.Principal.SecurityIdentifier]).Value
|
||||
} catch {
|
||||
throw "Could not resolve '$Account' to a SID. Pass an explicit SID or check the account name."
|
||||
}
|
||||
}
|
||||
|
||||
if (-not (Test-Path $NssmPath)) {
|
||||
throw "nssm.exe not found at '$NssmPath'. Install NSSM or pass -NssmPath."
|
||||
}
|
||||
|
||||
$hostExe = Join-Path $InstallRoot 'OtOpcUa.Driver.FOCAS.Host.exe'
|
||||
if (-not (Test-Path $hostExe)) {
|
||||
throw "FOCAS Host binary not found at '$hostExe'. Publish the Driver.FOCAS.Host project first."
|
||||
}
|
||||
|
||||
if (-not $FocasSharedSecret) {
|
||||
$FocasSharedSecret = [System.Guid]::NewGuid().ToString('N')
|
||||
Write-Host "Generated FocasSharedSecret — store it alongside the OtOpcUa service config."
|
||||
}
|
||||
|
||||
$allowedSid = Resolve-Sid $ServiceAccount
|
||||
|
||||
# Idempotent install — remove + re-create if present.
|
||||
$existing = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
|
||||
if ($existing) {
|
||||
Write-Host "Removing existing '$ServiceName' service..."
|
||||
& $NssmPath stop $ServiceName confirm | Out-Null
|
||||
& $NssmPath remove $ServiceName confirm | Out-Null
|
||||
}
|
||||
|
||||
& $NssmPath install $ServiceName $hostExe | Out-Null
|
||||
& $NssmPath set $ServiceName DisplayName 'OT-OPC-UA FOCAS Host (Tier-C isolated Fwlib32)' | Out-Null
|
||||
& $NssmPath set $ServiceName Description 'Out-of-process Fwlib32.dll host for OtOpcUa FOCAS driver. Crash-isolated from the main OPC UA server.' | Out-Null
|
||||
& $NssmPath set $ServiceName ObjectName $ServiceAccount | Out-Null
|
||||
& $NssmPath set $ServiceName Start SERVICE_AUTO_START | Out-Null
|
||||
& $NssmPath set $ServiceName AppStdout (Join-Path $env:ProgramData 'OtOpcUa\focas-host-stdout.log') | Out-Null
|
||||
& $NssmPath set $ServiceName AppStderr (Join-Path $env:ProgramData 'OtOpcUa\focas-host-stderr.log') | Out-Null
|
||||
& $NssmPath set $ServiceName AppRotateFiles 1 | Out-Null
|
||||
& $NssmPath set $ServiceName AppRotateBytes 10485760 | Out-Null
|
||||
|
||||
& $NssmPath set $ServiceName AppEnvironmentExtra `
|
||||
"OTOPCUA_FOCAS_PIPE=$FocasPipeName" `
|
||||
"OTOPCUA_ALLOWED_SID=$allowedSid" `
|
||||
"OTOPCUA_FOCAS_SECRET=$FocasSharedSecret" `
|
||||
"OTOPCUA_FOCAS_BACKEND=$FocasBackend" | Out-Null
|
||||
|
||||
& $NssmPath set $ServiceName DependOnService OtOpcUa | Out-Null
|
||||
|
||||
Write-Host "Installed '$ServiceName' under '$ServiceAccount' (SID=$allowedSid)."
|
||||
Write-Host "Pipe: \\.\pipe\$FocasPipeName Backend: $FocasBackend"
|
||||
Write-Host "Start the service with: Start-Service $ServiceName"
|
||||
Write-Host ""
|
||||
Write-Host "NOTE: the Fwlib32 backend requires the licensed Fwlib32.dll on PATH"
|
||||
Write-Host "alongside the Host exe. See docs/v2/focas-deployment.md."
|
||||
27
scripts/install/Uninstall-FocasHost.ps1
Normal file
27
scripts/install/Uninstall-FocasHost.ps1
Normal file
@@ -0,0 +1,27 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Removes the OtOpcUaFocasHost Windows service.
|
||||
|
||||
.DESCRIPTION
|
||||
Companion to Install-FocasHost.ps1. Stops + unregisters the service via NSSM.
|
||||
Idempotent — succeeds silently if the service doesn't exist.
|
||||
|
||||
.EXAMPLE
|
||||
.\Uninstall-FocasHost.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ServiceName = 'OtOpcUaFocasHost',
|
||||
[string]$NssmPath = 'C:\Program Files\nssm\nssm.exe'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$svc = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
|
||||
if (-not $svc) { Write-Host "Service '$ServiceName' not present — nothing to do."; return }
|
||||
|
||||
if (-not (Test-Path $NssmPath)) { throw "nssm.exe not found at '$NssmPath'." }
|
||||
|
||||
& $NssmPath stop $ServiceName confirm | Out-Null
|
||||
& $NssmPath remove $ServiceName confirm | Out-Null
|
||||
Write-Host "Removed '$ServiceName'."
|
||||
@@ -0,0 +1,122 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MessagePack;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
|
||||
/// <summary>
|
||||
/// In-memory <see cref="IFocasBackend"/> for tests + an operational stub mode when
|
||||
/// <c>OTOPCUA_FOCAS_BACKEND=fake</c>. Keeps per-address values keyed by a canonical
|
||||
/// string; RMW semantics honor PMC bit-writes against the containing byte so the
|
||||
/// <c>PmcBitWriteRequest</c> path can be exercised end-to-end without hardware.
|
||||
/// </summary>
|
||||
public sealed class FakeFocasBackend : IFocasBackend
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
private long _nextSessionId;
|
||||
private readonly HashSet<long> _openSessions = [];
|
||||
private readonly Dictionary<string, byte[]> _pmcValues = [];
|
||||
private readonly Dictionary<string, byte[]> _paramValues = [];
|
||||
private readonly Dictionary<string, byte[]> _macroValues = [];
|
||||
|
||||
public Task<OpenSessionResponse> OpenSessionAsync(OpenSessionRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
var id = ++_nextSessionId;
|
||||
_openSessions.Add(id);
|
||||
return Task.FromResult(new OpenSessionResponse { Success = true, SessionId = id });
|
||||
}
|
||||
}
|
||||
|
||||
public Task CloseSessionAsync(CloseSessionRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate) { _openSessions.Remove(request.SessionId); }
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task<ReadResponse> ReadAsync(ReadRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_openSessions.Contains(request.SessionId))
|
||||
return Task.FromResult(new ReadResponse { Success = false, StatusCode = 0x80020000u, Error = "session-not-open" });
|
||||
|
||||
var store = StoreFor(request.Address.Kind);
|
||||
var key = CanonicalKey(request.Address);
|
||||
store.TryGetValue(key, out var value);
|
||||
return Task.FromResult(new ReadResponse
|
||||
{
|
||||
Success = true,
|
||||
StatusCode = 0,
|
||||
ValueBytes = value ?? MessagePackSerializer.Serialize((int)0),
|
||||
ValueTypeCode = request.DataType,
|
||||
SourceTimestampUtcUnixMs = System.DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
public Task<WriteResponse> WriteAsync(WriteRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_openSessions.Contains(request.SessionId))
|
||||
return Task.FromResult(new WriteResponse { Success = false, StatusCode = 0x80020000u, Error = "session-not-open" });
|
||||
|
||||
var store = StoreFor(request.Address.Kind);
|
||||
store[CanonicalKey(request.Address)] = request.ValueBytes ?? [];
|
||||
return Task.FromResult(new WriteResponse { Success = true, StatusCode = 0 });
|
||||
}
|
||||
}
|
||||
|
||||
public Task<PmcBitWriteResponse> PmcBitWriteAsync(PmcBitWriteRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_openSessions.Contains(request.SessionId))
|
||||
return Task.FromResult(new PmcBitWriteResponse { Success = false, StatusCode = 0x80020000u, Error = "session-not-open" });
|
||||
if (request.BitIndex is < 0 or > 7)
|
||||
return Task.FromResult(new PmcBitWriteResponse { Success = false, StatusCode = 0x803C0000u, Error = "bit-out-of-range" });
|
||||
|
||||
var key = CanonicalKey(request.Address);
|
||||
_pmcValues.TryGetValue(key, out var current);
|
||||
current ??= MessagePackSerializer.Serialize((byte)0);
|
||||
var b = MessagePackSerializer.Deserialize<byte>(current);
|
||||
var mask = (byte)(1 << request.BitIndex);
|
||||
b = request.Value ? (byte)(b | mask) : (byte)(b & ~mask);
|
||||
_pmcValues[key] = MessagePackSerializer.Serialize(b);
|
||||
return Task.FromResult(new PmcBitWriteResponse { Success = true, StatusCode = 0 });
|
||||
}
|
||||
}
|
||||
|
||||
public Task<ProbeResponse> ProbeAsync(ProbeRequest request, CancellationToken ct)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
return Task.FromResult(new ProbeResponse
|
||||
{
|
||||
Healthy = _openSessions.Contains(request.SessionId),
|
||||
ObservedAtUtcUnixMs = System.DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private Dictionary<string, byte[]> StoreFor(int kind) => kind switch
|
||||
{
|
||||
0 => _pmcValues,
|
||||
1 => _paramValues,
|
||||
2 => _macroValues,
|
||||
_ => _pmcValues,
|
||||
};
|
||||
|
||||
private static string CanonicalKey(FocasAddressDto addr) =>
|
||||
addr.Kind switch
|
||||
{
|
||||
0 => $"{addr.PmcLetter}{addr.Number}",
|
||||
1 => $"P{addr.Number}",
|
||||
2 => $"M{addr.Number}",
|
||||
_ => $"?{addr.Number}",
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
|
||||
/// <summary>
|
||||
/// The Host's view of a FOCAS session. One implementation wraps the real
|
||||
/// <c>Fwlib32.dll</c> via P/Invoke (lands with the real Fwlib32 integration follow-up,
|
||||
/// since no hardware is available today); a second implementation —
|
||||
/// <see cref="FakeFocasBackend"/> — is used by tests.
|
||||
/// Both live on .NET 4.8 x86 so the Host can be deployed in either mode without
|
||||
/// changing the pipe server.
|
||||
/// Invoked via <c>FwlibFrameHandler</c> in the Ipc namespace.
|
||||
/// </summary>
|
||||
public interface IFocasBackend
|
||||
{
|
||||
Task<OpenSessionResponse> OpenSessionAsync(OpenSessionRequest request, CancellationToken ct);
|
||||
Task CloseSessionAsync(CloseSessionRequest request, CancellationToken ct);
|
||||
Task<ReadResponse> ReadAsync(ReadRequest request, CancellationToken ct);
|
||||
Task<WriteResponse> WriteAsync(WriteRequest request, CancellationToken ct);
|
||||
Task<PmcBitWriteResponse> PmcBitWriteAsync(PmcBitWriteRequest request, CancellationToken ct);
|
||||
Task<ProbeResponse> ProbeAsync(ProbeRequest request, CancellationToken ct);
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
|
||||
/// <summary>
|
||||
/// Safe default when the deployment hasn't configured a real Fwlib32 backend.
|
||||
/// Returns structured failure responses instead of throwing so the Proxy can map the
|
||||
/// error to <c>BadDeviceFailure</c> and surface a clear operator message pointing at
|
||||
/// <c>docs/v2/focas-deployment.md</c>. Used when <c>OTOPCUA_FOCAS_BACKEND</c> is unset
|
||||
/// or set to <c>unconfigured</c>.
|
||||
/// </summary>
|
||||
public sealed class UnconfiguredFocasBackend : IFocasBackend
|
||||
{
|
||||
private const uint BadDeviceFailure = 0x80550000u;
|
||||
private const string Reason =
|
||||
"FOCAS Host is running without a real Fwlib32 backend. Set OTOPCUA_FOCAS_BACKEND=fwlib32 " +
|
||||
"and ensure Fwlib32.dll is on PATH — see docs/v2/focas-deployment.md.";
|
||||
|
||||
public Task<OpenSessionResponse> OpenSessionAsync(OpenSessionRequest request, CancellationToken ct) =>
|
||||
Task.FromResult(new OpenSessionResponse { Success = false, Error = Reason, ErrorCode = "NoFwlibBackend" });
|
||||
|
||||
public Task CloseSessionAsync(CloseSessionRequest request, CancellationToken ct) => Task.CompletedTask;
|
||||
|
||||
public Task<ReadResponse> ReadAsync(ReadRequest request, CancellationToken ct) =>
|
||||
Task.FromResult(new ReadResponse { Success = false, StatusCode = BadDeviceFailure, Error = Reason });
|
||||
|
||||
public Task<WriteResponse> WriteAsync(WriteRequest request, CancellationToken ct) =>
|
||||
Task.FromResult(new WriteResponse { Success = false, StatusCode = BadDeviceFailure, Error = Reason });
|
||||
|
||||
public Task<PmcBitWriteResponse> PmcBitWriteAsync(PmcBitWriteRequest request, CancellationToken ct) =>
|
||||
Task.FromResult(new PmcBitWriteResponse { Success = false, StatusCode = BadDeviceFailure, Error = Reason });
|
||||
|
||||
public Task<ProbeResponse> ProbeAsync(ProbeRequest request, CancellationToken ct) =>
|
||||
Task.FromResult(new ProbeResponse { Healthy = false, Error = Reason, ObservedAtUtcUnixMs = System.DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() });
|
||||
}
|
||||
111
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host/Ipc/FwlibFrameHandler.cs
Normal file
111
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host/Ipc/FwlibFrameHandler.cs
Normal file
@@ -0,0 +1,111 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MessagePack;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Ipc;
|
||||
|
||||
/// <summary>
|
||||
/// Real FOCAS frame handler. Deserializes each request DTO, delegates to
|
||||
/// <see cref="IFocasBackend"/>, re-serializes the response. The backend owns the
|
||||
/// Fwlib32 handle + STA thread — the handler is pure dispatch.
|
||||
/// </summary>
|
||||
public sealed class FwlibFrameHandler : IFrameHandler
|
||||
{
|
||||
private readonly IFocasBackend _backend;
|
||||
private readonly ILogger _logger;
|
||||
|
||||
public FwlibFrameHandler(IFocasBackend backend, ILogger logger)
|
||||
{
|
||||
_backend = backend ?? throw new ArgumentNullException(nameof(backend));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public async Task HandleAsync(FocasMessageKind kind, byte[] body, FrameWriter writer, CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
switch (kind)
|
||||
{
|
||||
case FocasMessageKind.Heartbeat:
|
||||
{
|
||||
var hb = MessagePackSerializer.Deserialize<Heartbeat>(body);
|
||||
await writer.WriteAsync(FocasMessageKind.HeartbeatAck,
|
||||
new HeartbeatAck
|
||||
{
|
||||
MonotonicTicks = hb.MonotonicTicks,
|
||||
HostUtcUnixMs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
|
||||
}, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.OpenSessionRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<OpenSessionRequest>(body);
|
||||
var resp = await _backend.OpenSessionAsync(req, ct).ConfigureAwait(false);
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse, resp, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.CloseSessionRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<CloseSessionRequest>(body);
|
||||
await _backend.CloseSessionAsync(req, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.ReadRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<ReadRequest>(body);
|
||||
var resp = await _backend.ReadAsync(req, ct).ConfigureAwait(false);
|
||||
await writer.WriteAsync(FocasMessageKind.ReadResponse, resp, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.WriteRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<WriteRequest>(body);
|
||||
var resp = await _backend.WriteAsync(req, ct).ConfigureAwait(false);
|
||||
await writer.WriteAsync(FocasMessageKind.WriteResponse, resp, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.PmcBitWriteRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<PmcBitWriteRequest>(body);
|
||||
var resp = await _backend.PmcBitWriteAsync(req, ct).ConfigureAwait(false);
|
||||
await writer.WriteAsync(FocasMessageKind.PmcBitWriteResponse, resp, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
case FocasMessageKind.ProbeRequest:
|
||||
{
|
||||
var req = MessagePackSerializer.Deserialize<ProbeRequest>(body);
|
||||
var resp = await _backend.ProbeAsync(req, ct).ConfigureAwait(false);
|
||||
await writer.WriteAsync(FocasMessageKind.ProbeResponse, resp, ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
default:
|
||||
await writer.WriteAsync(FocasMessageKind.ErrorResponse,
|
||||
new ErrorResponse { Code = "unknown-kind", Message = $"Kind {kind} is not handled by the Host" },
|
||||
ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { throw; }
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Error(ex, "FwlibFrameHandler error processing {Kind}", kind);
|
||||
await writer.WriteAsync(FocasMessageKind.ErrorResponse,
|
||||
new ErrorResponse { Code = "backend-exception", Message = ex.Message },
|
||||
ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public IDisposable AttachConnection(FrameWriter writer) => IFrameHandler.NoopAttachment.Instance;
|
||||
}
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Security.Principal;
|
||||
using System.Threading;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Ipc;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host;
|
||||
@@ -44,9 +45,18 @@ public static class Program
|
||||
Log.Information("OtOpcUaFocasHost starting — pipe={Pipe} allowedSid={Sid}",
|
||||
pipeName, allowedSidValue);
|
||||
|
||||
var handler = new StubFrameHandler();
|
||||
Log.Warning("OtOpcUaFocasHost backend=stub — Fwlib32 lift lands in PR C");
|
||||
var backendKind = (Environment.GetEnvironmentVariable("OTOPCUA_FOCAS_BACKEND") ?? "unconfigured")
|
||||
.ToLowerInvariant();
|
||||
IFocasBackend backend = backendKind switch
|
||||
{
|
||||
"fake" => new FakeFocasBackend(),
|
||||
"unconfigured" => new UnconfiguredFocasBackend(),
|
||||
"fwlib32" => new UnconfiguredFocasBackend(), // real Fwlib32 backend lands with hardware integration follow-up
|
||||
_ => new UnconfiguredFocasBackend(),
|
||||
};
|
||||
Log.Information("OtOpcUaFocasHost backend={Backend}", backendKind);
|
||||
|
||||
var handler = new FwlibFrameHandler(backend, Log.Logger);
|
||||
server.RunAsync(handler, cts.Token).GetAwaiter().GetResult();
|
||||
|
||||
Log.Information("OtOpcUaFocasHost stopped cleanly");
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Stability;
|
||||
|
||||
/// <summary>
|
||||
/// Ring-buffer of the last N IPC operations, written into a memory-mapped file. On a
|
||||
/// hard crash the Proxy-side supervisor reads the MMF after the corpse is gone to see
|
||||
/// what was in flight at the moment the Host died. Single-writer (the Host), multi-reader
|
||||
/// (the supervisor) — the file format is identical to the Galaxy Tier-C
|
||||
/// <c>PostMortemMmf</c> so a single reader tool can work both.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// File layout:
|
||||
/// <code>
|
||||
/// [16-byte header: magic(4) | version(4) | capacity(4) | writeIndex(4)]
|
||||
/// [capacity × 256-byte entries: each is [8-byte utcUnixMs | 8-byte opKind | 240-byte UTF-8 message]]
|
||||
/// </code>
|
||||
/// Magic is 'OFPC' (0x4F46_5043) to distinguish a FOCAS file from the Galaxy MMF.
|
||||
/// </remarks>
|
||||
public sealed class PostMortemMmf : IDisposable
|
||||
{
|
||||
private const int Magic = 0x4F465043; // 'OFPC'
|
||||
private const int Version = 1;
|
||||
private const int HeaderBytes = 16;
|
||||
public const int EntryBytes = 256;
|
||||
private const int MessageOffset = 16;
|
||||
private const int MessageCapacity = EntryBytes - MessageOffset;
|
||||
|
||||
public int Capacity { get; }
|
||||
public string Path { get; }
|
||||
|
||||
private readonly MemoryMappedFile _mmf;
|
||||
private readonly MemoryMappedViewAccessor _accessor;
|
||||
private readonly object _writeGate = new();
|
||||
|
||||
public PostMortemMmf(string path, int capacity = 1000)
|
||||
{
|
||||
if (capacity <= 0) throw new ArgumentOutOfRangeException(nameof(capacity));
|
||||
Capacity = capacity;
|
||||
Path = path;
|
||||
|
||||
var fileBytes = HeaderBytes + capacity * EntryBytes;
|
||||
Directory.CreateDirectory(System.IO.Path.GetDirectoryName(path)!);
|
||||
|
||||
var fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read);
|
||||
fs.SetLength(fileBytes);
|
||||
_mmf = MemoryMappedFile.CreateFromFile(fs, null, fileBytes,
|
||||
MemoryMappedFileAccess.ReadWrite, HandleInheritability.None, leaveOpen: false);
|
||||
_accessor = _mmf.CreateViewAccessor(0, fileBytes, MemoryMappedFileAccess.ReadWrite);
|
||||
|
||||
if (_accessor.ReadInt32(0) != Magic)
|
||||
{
|
||||
_accessor.Write(0, Magic);
|
||||
_accessor.Write(4, Version);
|
||||
_accessor.Write(8, capacity);
|
||||
_accessor.Write(12, 0);
|
||||
}
|
||||
}
|
||||
|
||||
public void Write(long opKind, string message)
|
||||
{
|
||||
lock (_writeGate)
|
||||
{
|
||||
var idx = _accessor.ReadInt32(12);
|
||||
var offset = HeaderBytes + idx * EntryBytes;
|
||||
|
||||
_accessor.Write(offset + 0, DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
|
||||
_accessor.Write(offset + 8, opKind);
|
||||
|
||||
var msgBytes = Encoding.UTF8.GetBytes(message ?? string.Empty);
|
||||
var copy = Math.Min(msgBytes.Length, MessageCapacity - 1);
|
||||
_accessor.WriteArray(offset + MessageOffset, msgBytes, 0, copy);
|
||||
_accessor.Write(offset + MessageOffset + copy, (byte)0);
|
||||
|
||||
var next = (idx + 1) % Capacity;
|
||||
_accessor.Write(12, next);
|
||||
}
|
||||
}
|
||||
|
||||
public PostMortemEntry[] ReadAll()
|
||||
{
|
||||
var magic = _accessor.ReadInt32(0);
|
||||
if (magic != Magic) return new PostMortemEntry[0];
|
||||
|
||||
var capacity = _accessor.ReadInt32(8);
|
||||
var writeIndex = _accessor.ReadInt32(12);
|
||||
|
||||
var entries = new PostMortemEntry[capacity];
|
||||
var count = 0;
|
||||
for (var i = 0; i < capacity; i++)
|
||||
{
|
||||
var slot = (writeIndex + i) % capacity;
|
||||
var offset = HeaderBytes + slot * EntryBytes;
|
||||
|
||||
var ts = _accessor.ReadInt64(offset + 0);
|
||||
if (ts == 0) continue;
|
||||
|
||||
var op = _accessor.ReadInt64(offset + 8);
|
||||
var msgBuf = new byte[MessageCapacity];
|
||||
_accessor.ReadArray(offset + MessageOffset, msgBuf, 0, MessageCapacity);
|
||||
var nulTerm = Array.IndexOf<byte>(msgBuf, 0);
|
||||
var msg = Encoding.UTF8.GetString(msgBuf, 0, nulTerm < 0 ? MessageCapacity : nulTerm);
|
||||
|
||||
entries[count++] = new PostMortemEntry(ts, op, msg);
|
||||
}
|
||||
|
||||
Array.Resize(ref entries, count);
|
||||
return entries;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_accessor.Dispose();
|
||||
_mmf.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
public readonly struct PostMortemEntry
|
||||
{
|
||||
public long UtcUnixMs { get; }
|
||||
public long OpKind { get; }
|
||||
public string Message { get; }
|
||||
|
||||
public PostMortemEntry(long utcUnixMs, long opKind, string message)
|
||||
{
|
||||
UtcUnixMs = utcUnixMs;
|
||||
OpKind = opKind;
|
||||
Message = message;
|
||||
}
|
||||
}
|
||||
120
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Ipc/FocasIpcClient.cs
Normal file
120
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Ipc/FocasIpcClient.cs
Normal file
@@ -0,0 +1,120 @@
|
||||
using System.IO;
|
||||
using System.IO.Pipes;
|
||||
using MessagePack;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Ipc;
|
||||
|
||||
/// <summary>
|
||||
/// Proxy-side IPC channel to a running <c>Driver.FOCAS.Host</c>. Owns the pipe connection
|
||||
/// and serializes request/response round-trips through a single call gate so
|
||||
/// concurrent callers don't interleave frames. One instance per FOCAS Host session.
|
||||
/// </summary>
|
||||
public sealed class FocasIpcClient : IAsyncDisposable
|
||||
{
|
||||
private readonly Stream _stream;
|
||||
private readonly FrameReader _reader;
|
||||
private readonly FrameWriter _writer;
|
||||
private readonly SemaphoreSlim _callGate = new(1, 1);
|
||||
|
||||
private FocasIpcClient(Stream stream)
|
||||
{
|
||||
_stream = stream;
|
||||
_reader = new FrameReader(stream, leaveOpen: true);
|
||||
_writer = new FrameWriter(stream, leaveOpen: true);
|
||||
}
|
||||
|
||||
/// <summary>Named-pipe factory: connects, sends Hello, awaits HelloAck.</summary>
|
||||
public static async Task<FocasIpcClient> ConnectAsync(
|
||||
string pipeName, string sharedSecret, TimeSpan connectTimeout, CancellationToken ct)
|
||||
{
|
||||
var stream = new NamedPipeClientStream(
|
||||
serverName: ".",
|
||||
pipeName: pipeName,
|
||||
direction: PipeDirection.InOut,
|
||||
options: PipeOptions.Asynchronous);
|
||||
|
||||
await stream.ConnectAsync((int)connectTimeout.TotalMilliseconds, ct);
|
||||
return await HandshakeAsync(stream, sharedSecret, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Stream factory — used by tests that wire the Proxy against an in-memory stream
|
||||
/// pair instead of a real pipe. <paramref name="stream"/> is owned by the caller
|
||||
/// until <see cref="DisposeAsync"/>.
|
||||
/// </summary>
|
||||
public static Task<FocasIpcClient> ConnectAsync(Stream stream, string sharedSecret, CancellationToken ct)
|
||||
=> HandshakeAsync(stream, sharedSecret, ct);
|
||||
|
||||
private static async Task<FocasIpcClient> HandshakeAsync(Stream stream, string sharedSecret, CancellationToken ct)
|
||||
{
|
||||
var client = new FocasIpcClient(stream);
|
||||
try
|
||||
{
|
||||
await client._writer.WriteAsync(FocasMessageKind.Hello,
|
||||
new Hello { PeerName = "FOCAS.Proxy", SharedSecret = sharedSecret }, ct).ConfigureAwait(false);
|
||||
|
||||
var ack = await client._reader.ReadFrameAsync(ct).ConfigureAwait(false);
|
||||
if (ack is null || ack.Value.Kind != FocasMessageKind.HelloAck)
|
||||
throw new InvalidOperationException("Did not receive HelloAck from FOCAS.Host");
|
||||
|
||||
var ackMsg = FrameReader.Deserialize<HelloAck>(ack.Value.Body);
|
||||
if (!ackMsg.Accepted)
|
||||
throw new UnauthorizedAccessException($"FOCAS.Host rejected Hello: {ackMsg.RejectReason}");
|
||||
|
||||
return client;
|
||||
}
|
||||
catch
|
||||
{
|
||||
await client.DisposeAsync().ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<TResp> CallAsync<TReq, TResp>(
|
||||
FocasMessageKind requestKind, TReq request, FocasMessageKind expectedResponseKind, CancellationToken ct)
|
||||
{
|
||||
await _callGate.WaitAsync(ct).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
await _writer.WriteAsync(requestKind, request, ct).ConfigureAwait(false);
|
||||
|
||||
var frame = await _reader.ReadFrameAsync(ct).ConfigureAwait(false);
|
||||
if (frame is null) throw new EndOfStreamException("FOCAS IPC peer closed before response");
|
||||
|
||||
if (frame.Value.Kind == FocasMessageKind.ErrorResponse)
|
||||
{
|
||||
var err = MessagePackSerializer.Deserialize<ErrorResponse>(frame.Value.Body);
|
||||
throw new FocasIpcException(err.Code, err.Message);
|
||||
}
|
||||
|
||||
if (frame.Value.Kind != expectedResponseKind)
|
||||
throw new InvalidOperationException(
|
||||
$"Expected {expectedResponseKind}, got {frame.Value.Kind}");
|
||||
|
||||
return MessagePackSerializer.Deserialize<TResp>(frame.Value.Body);
|
||||
}
|
||||
finally { _callGate.Release(); }
|
||||
}
|
||||
|
||||
public async Task SendOneWayAsync<TReq>(FocasMessageKind requestKind, TReq request, CancellationToken ct)
|
||||
{
|
||||
await _callGate.WaitAsync(ct).ConfigureAwait(false);
|
||||
try { await _writer.WriteAsync(requestKind, request, ct).ConfigureAwait(false); }
|
||||
finally { _callGate.Release(); }
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
_callGate.Dispose();
|
||||
_reader.Dispose();
|
||||
_writer.Dispose();
|
||||
await _stream.DisposeAsync().ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class FocasIpcException(string code, string message) : Exception($"[{code}] {message}")
|
||||
{
|
||||
public string Code { get; } = code;
|
||||
}
|
||||
199
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Ipc/IpcFocasClient.cs
Normal file
199
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Ipc/IpcFocasClient.cs
Normal file
@@ -0,0 +1,199 @@
|
||||
using MessagePack;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Ipc;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="IFocasClient"/> implementation that forwards every operation over a
|
||||
/// <see cref="FocasIpcClient"/> to a <c>Driver.FOCAS.Host</c> process. Keeps the
|
||||
/// <c>Fwlib32.dll</c> P/Invoke out of the main server process so a native crash
|
||||
/// blast-radius stops at the Host boundary.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Session lifecycle: <see cref="ConnectAsync"/> sends <c>OpenSessionRequest</c> and
|
||||
/// caches the returned <c>SessionId</c>. Subsequent <see cref="ReadAsync"/> /
|
||||
/// <see cref="WriteAsync"/> / <see cref="ProbeAsync"/> calls thread that session id
|
||||
/// onto each request DTO. <see cref="Dispose"/> sends <c>CloseSessionRequest</c> +
|
||||
/// disposes the underlying pipe.
|
||||
/// </remarks>
|
||||
public sealed class IpcFocasClient : IFocasClient
|
||||
{
|
||||
private readonly FocasIpcClient _ipc;
|
||||
private readonly FocasCncSeries _series;
|
||||
private long _sessionId;
|
||||
private bool _connected;
|
||||
|
||||
public IpcFocasClient(FocasIpcClient ipc, FocasCncSeries series = FocasCncSeries.Unknown)
|
||||
{
|
||||
_ipc = ipc ?? throw new ArgumentNullException(nameof(ipc));
|
||||
_series = series;
|
||||
}
|
||||
|
||||
public bool IsConnected => _connected;
|
||||
|
||||
public async Task ConnectAsync(FocasHostAddress address, TimeSpan timeout, CancellationToken cancellationToken)
|
||||
{
|
||||
if (_connected) return;
|
||||
|
||||
var resp = await _ipc.CallAsync<OpenSessionRequest, OpenSessionResponse>(
|
||||
FocasMessageKind.OpenSessionRequest,
|
||||
new OpenSessionRequest
|
||||
{
|
||||
HostAddress = $"{address.Host}:{address.Port}",
|
||||
TimeoutMs = (int)Math.Max(1, timeout.TotalMilliseconds),
|
||||
CncSeries = (int)_series,
|
||||
},
|
||||
FocasMessageKind.OpenSessionResponse,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (!resp.Success)
|
||||
throw new InvalidOperationException(
|
||||
$"FOCAS Host rejected OpenSession for {address}: {resp.ErrorCode ?? "?"} — {resp.Error}");
|
||||
|
||||
_sessionId = resp.SessionId;
|
||||
_connected = true;
|
||||
}
|
||||
|
||||
public async Task<(object? value, uint status)> ReadAsync(
|
||||
FocasAddress address, FocasDataType type, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!_connected) return (null, FocasStatusMapper.BadCommunicationError);
|
||||
|
||||
var resp = await _ipc.CallAsync<ReadRequest, ReadResponse>(
|
||||
FocasMessageKind.ReadRequest,
|
||||
new ReadRequest
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
Address = ToDto(address),
|
||||
DataType = (int)type,
|
||||
},
|
||||
FocasMessageKind.ReadResponse,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (!resp.Success) return (null, resp.StatusCode);
|
||||
|
||||
var value = DecodeValue(resp.ValueBytes, resp.ValueTypeCode);
|
||||
return (value, resp.StatusCode);
|
||||
}
|
||||
|
||||
public async Task<uint> WriteAsync(
|
||||
FocasAddress address, FocasDataType type, object? value, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!_connected) return FocasStatusMapper.BadCommunicationError;
|
||||
|
||||
// PMC bit writes get the first-class RMW frame so the critical section stays on the Host.
|
||||
if (address.Kind == FocasAreaKind.Pmc && type == FocasDataType.Bit && address.BitIndex is int bit)
|
||||
{
|
||||
var bitResp = await _ipc.CallAsync<PmcBitWriteRequest, PmcBitWriteResponse>(
|
||||
FocasMessageKind.PmcBitWriteRequest,
|
||||
new PmcBitWriteRequest
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
Address = ToDto(address),
|
||||
BitIndex = bit,
|
||||
Value = Convert.ToBoolean(value),
|
||||
},
|
||||
FocasMessageKind.PmcBitWriteResponse,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
return bitResp.StatusCode;
|
||||
}
|
||||
|
||||
var resp = await _ipc.CallAsync<WriteRequest, WriteResponse>(
|
||||
FocasMessageKind.WriteRequest,
|
||||
new WriteRequest
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
Address = ToDto(address),
|
||||
DataType = (int)type,
|
||||
ValueTypeCode = (int)type,
|
||||
ValueBytes = EncodeValue(value, type),
|
||||
},
|
||||
FocasMessageKind.WriteResponse,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return resp.StatusCode;
|
||||
}
|
||||
|
||||
public async Task<bool> ProbeAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (!_connected) return false;
|
||||
try
|
||||
{
|
||||
var resp = await _ipc.CallAsync<ProbeRequest, ProbeResponse>(
|
||||
FocasMessageKind.ProbeRequest,
|
||||
new ProbeRequest { SessionId = _sessionId },
|
||||
FocasMessageKind.ProbeResponse,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
return resp.Healthy;
|
||||
}
|
||||
catch { return false; }
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_connected)
|
||||
{
|
||||
try
|
||||
{
|
||||
_ipc.SendOneWayAsync(FocasMessageKind.CloseSessionRequest,
|
||||
new CloseSessionRequest { SessionId = _sessionId }, CancellationToken.None)
|
||||
.GetAwaiter().GetResult();
|
||||
}
|
||||
catch { /* best effort */ }
|
||||
_connected = false;
|
||||
}
|
||||
_ipc.DisposeAsync().AsTask().GetAwaiter().GetResult();
|
||||
}
|
||||
|
||||
private static FocasAddressDto ToDto(FocasAddress addr) => new()
|
||||
{
|
||||
Kind = (int)addr.Kind,
|
||||
PmcLetter = addr.PmcLetter,
|
||||
Number = addr.Number,
|
||||
BitIndex = addr.BitIndex,
|
||||
};
|
||||
|
||||
private static byte[]? EncodeValue(object? value, FocasDataType type)
|
||||
{
|
||||
if (value is null) return null;
|
||||
return type switch
|
||||
{
|
||||
FocasDataType.Bit => MessagePackSerializer.Serialize(Convert.ToBoolean(value)),
|
||||
FocasDataType.Byte => MessagePackSerializer.Serialize(Convert.ToByte(value)),
|
||||
FocasDataType.Int16 => MessagePackSerializer.Serialize(Convert.ToInt16(value)),
|
||||
FocasDataType.Int32 => MessagePackSerializer.Serialize(Convert.ToInt32(value)),
|
||||
FocasDataType.Float32 => MessagePackSerializer.Serialize(Convert.ToSingle(value)),
|
||||
FocasDataType.Float64 => MessagePackSerializer.Serialize(Convert.ToDouble(value)),
|
||||
FocasDataType.String => MessagePackSerializer.Serialize(Convert.ToString(value) ?? string.Empty),
|
||||
_ => MessagePackSerializer.Serialize(Convert.ToInt32(value)),
|
||||
};
|
||||
}
|
||||
|
||||
private static object? DecodeValue(byte[]? bytes, int typeCode)
|
||||
{
|
||||
if (bytes is null) return null;
|
||||
return typeCode switch
|
||||
{
|
||||
FocasDataTypeCode.Bit => MessagePackSerializer.Deserialize<bool>(bytes),
|
||||
FocasDataTypeCode.Byte => MessagePackSerializer.Deserialize<byte>(bytes),
|
||||
FocasDataTypeCode.Int16 => MessagePackSerializer.Deserialize<short>(bytes),
|
||||
FocasDataTypeCode.Int32 => MessagePackSerializer.Deserialize<int>(bytes),
|
||||
FocasDataTypeCode.Float32 => MessagePackSerializer.Deserialize<float>(bytes),
|
||||
FocasDataTypeCode.Float64 => MessagePackSerializer.Deserialize<double>(bytes),
|
||||
FocasDataTypeCode.String => MessagePackSerializer.Deserialize<string>(bytes),
|
||||
_ => MessagePackSerializer.Deserialize<int>(bytes),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Factory producing <see cref="IpcFocasClient"/>s. One pipe connection per
|
||||
/// <c>IFocasClient</c> — matches the driver's one-client-per-device invariant. The
|
||||
/// deployment wires this into the DI container in place of
|
||||
/// <see cref="UnimplementedFocasClientFactory"/>.
|
||||
/// </summary>
|
||||
public sealed class IpcFocasClientFactory(Func<FocasIpcClient> ipcClientFactory, FocasCncSeries series = FocasCncSeries.Unknown)
|
||||
: IFocasClientFactory
|
||||
{
|
||||
public IFocasClient Create() => new IpcFocasClient(ipcClientFactory(), series);
|
||||
}
|
||||
30
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Supervisor/Backoff.cs
Normal file
30
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Supervisor/Backoff.cs
Normal file
@@ -0,0 +1,30 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Respawn-with-backoff schedule for the FOCAS Host process. Matches Galaxy Tier-C:
|
||||
/// 5s → 15s → 60s cap. A sustained stable run (default 2 min) resets the index so a
|
||||
/// one-off crash after hours of steady-state doesn't start from the top of the ladder.
|
||||
/// </summary>
|
||||
public sealed class Backoff
|
||||
{
|
||||
public static TimeSpan[] DefaultSequence { get; } =
|
||||
[TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(60)];
|
||||
|
||||
public TimeSpan StableRunThreshold { get; init; } = TimeSpan.FromMinutes(2);
|
||||
|
||||
private readonly TimeSpan[] _sequence;
|
||||
private int _index;
|
||||
|
||||
public Backoff(TimeSpan[]? sequence = null) => _sequence = sequence ?? DefaultSequence;
|
||||
|
||||
public TimeSpan Next()
|
||||
{
|
||||
var delay = _sequence[Math.Min(_index, _sequence.Length - 1)];
|
||||
_index++;
|
||||
return delay;
|
||||
}
|
||||
|
||||
public void RecordStableRun() => _index = 0;
|
||||
|
||||
public int AttemptIndex => _index;
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Crash-loop circuit breaker for the FOCAS Host. Matches Galaxy Tier-C defaults:
|
||||
/// 3 crashes within 5 minutes opens the breaker; cooldown escalates 1h → 4h → manual
|
||||
/// reset. A sticky alert stays live until the operator explicitly clears it so
|
||||
/// recurring crashes can't silently burn through the cooldown ladder overnight.
|
||||
/// </summary>
|
||||
public sealed class CircuitBreaker
|
||||
{
|
||||
public int CrashesAllowedPerWindow { get; init; } = 3;
|
||||
public TimeSpan Window { get; init; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
public TimeSpan[] CooldownEscalation { get; init; } =
|
||||
[TimeSpan.FromHours(1), TimeSpan.FromHours(4), TimeSpan.MaxValue];
|
||||
|
||||
private readonly List<DateTime> _crashesUtc = [];
|
||||
private DateTime? _openSinceUtc;
|
||||
private int _escalationLevel;
|
||||
public bool StickyAlertActive { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Records a crash + returns <c>true</c> if the supervisor may respawn. On
|
||||
/// <c>false</c>, <paramref name="cooldownRemaining"/> is how long to wait before
|
||||
/// trying again (<c>TimeSpan.MaxValue</c> means manual reset required).
|
||||
/// </summary>
|
||||
public bool TryRecordCrash(DateTime utcNow, out TimeSpan cooldownRemaining)
|
||||
{
|
||||
if (_openSinceUtc is { } openedAt)
|
||||
{
|
||||
var cooldown = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
|
||||
if (cooldown == TimeSpan.MaxValue)
|
||||
{
|
||||
cooldownRemaining = TimeSpan.MaxValue;
|
||||
return false;
|
||||
}
|
||||
if (utcNow - openedAt < cooldown)
|
||||
{
|
||||
cooldownRemaining = cooldown - (utcNow - openedAt);
|
||||
return false;
|
||||
}
|
||||
|
||||
_openSinceUtc = null;
|
||||
_escalationLevel++;
|
||||
}
|
||||
|
||||
_crashesUtc.RemoveAll(t => utcNow - t > Window);
|
||||
_crashesUtc.Add(utcNow);
|
||||
|
||||
if (_crashesUtc.Count > CrashesAllowedPerWindow)
|
||||
{
|
||||
_openSinceUtc = utcNow;
|
||||
StickyAlertActive = true;
|
||||
cooldownRemaining = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
|
||||
return false;
|
||||
}
|
||||
|
||||
cooldownRemaining = TimeSpan.Zero;
|
||||
return true;
|
||||
}
|
||||
|
||||
public void ManualReset()
|
||||
{
|
||||
_crashesUtc.Clear();
|
||||
_openSinceUtc = null;
|
||||
_escalationLevel = 0;
|
||||
StickyAlertActive = false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Ties <see cref="IHostProcessLauncher"/> + <see cref="Backoff"/> +
|
||||
/// <see cref="CircuitBreaker"/> + <see cref="HeartbeatMonitor"/> into one object the
|
||||
/// driver asks for <c>IFocasClient</c>s. On a detected crash (process exit or
|
||||
/// heartbeat loss) the supervisor fans out <c>BadCommunicationError</c> to all
|
||||
/// subscribers via the <see cref="OnUnavailable"/> callback, then respawns with
|
||||
/// backoff unless the breaker is open.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The supervisor itself is I/O-free — it doesn't know how to spawn processes, probe
|
||||
/// pipes, or send heartbeats. Production wires the concrete
|
||||
/// <see cref="IHostProcessLauncher"/> over <c>FocasIpcClient</c> + <c>Process</c>;
|
||||
/// tests drive the same state machine with a deterministic launcher stub.
|
||||
/// </remarks>
|
||||
public sealed class FocasHostSupervisor : IDisposable
|
||||
{
|
||||
private readonly IHostProcessLauncher _launcher;
|
||||
private readonly Backoff _backoff;
|
||||
private readonly CircuitBreaker _breaker;
|
||||
private readonly Func<DateTime> _clock;
|
||||
private IFocasClient? _current;
|
||||
private DateTime _currentStartedUtc;
|
||||
private bool _disposed;
|
||||
|
||||
public FocasHostSupervisor(
|
||||
IHostProcessLauncher launcher,
|
||||
Backoff? backoff = null,
|
||||
CircuitBreaker? breaker = null,
|
||||
Func<DateTime>? clock = null)
|
||||
{
|
||||
_launcher = launcher ?? throw new ArgumentNullException(nameof(launcher));
|
||||
_backoff = backoff ?? new Backoff();
|
||||
_breaker = breaker ?? new CircuitBreaker();
|
||||
_clock = clock ?? (() => DateTime.UtcNow);
|
||||
}
|
||||
|
||||
/// <summary>Raised with a short reason string whenever the Host goes unavailable (crash / heartbeat loss / breaker-open).</summary>
|
||||
public event Action<string>? OnUnavailable;
|
||||
|
||||
/// <summary>Crash count observed in the current process lifetime. Exposed for /hosts Admin telemetry.</summary>
|
||||
public int ObservedCrashes { get; private set; }
|
||||
|
||||
/// <summary><c>true</c> if the crash-loop breaker has latched a sticky alert that needs operator reset.</summary>
|
||||
public bool StickyAlertActive => _breaker.StickyAlertActive;
|
||||
|
||||
public int BackoffAttempt => _backoff.AttemptIndex;
|
||||
|
||||
/// <summary>
|
||||
/// Returns the current live client. If none, tries to launch — applying the
|
||||
/// backoff schedule between attempts and stopping once the breaker opens.
|
||||
/// </summary>
|
||||
public async Task<IFocasClient> GetOrLaunchAsync(CancellationToken ct)
|
||||
{
|
||||
ThrowIfDisposed();
|
||||
if (_current is not null && _launcher.IsProcessAlive) return _current;
|
||||
|
||||
return await LaunchWithBackoffAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Called by the heartbeat task each time a miss threshold is crossed.
|
||||
/// Treated as a crash: fan out Bad status + attempt respawn.
|
||||
/// </summary>
|
||||
public async Task NotifyHostDeadAsync(string reason, CancellationToken ct)
|
||||
{
|
||||
ThrowIfDisposed();
|
||||
OnUnavailable?.Invoke(reason);
|
||||
ObservedCrashes++;
|
||||
try { await _launcher.TerminateAsync(ct).ConfigureAwait(false); }
|
||||
catch { /* best effort */ }
|
||||
_current?.Dispose();
|
||||
_current = null;
|
||||
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown))
|
||||
{
|
||||
OnUnavailable?.Invoke(cooldown == TimeSpan.MaxValue
|
||||
? "circuit-breaker-open-manual-reset-required"
|
||||
: $"circuit-breaker-open-cooldown-{cooldown:g}");
|
||||
return;
|
||||
}
|
||||
// Successful crash recording — do not respawn synchronously; GetOrLaunchAsync will
|
||||
// pick up the attempt on the next call. Keeps the fan-out fast.
|
||||
}
|
||||
|
||||
/// <summary>Operator action — clear the sticky alert + reset the breaker.</summary>
|
||||
public void AcknowledgeAndReset()
|
||||
{
|
||||
_breaker.ManualReset();
|
||||
_backoff.RecordStableRun();
|
||||
}
|
||||
|
||||
private async Task<IFocasClient> LaunchWithBackoffAsync(CancellationToken ct)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
if (_breaker.StickyAlertActive)
|
||||
{
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown) && cooldown == TimeSpan.MaxValue)
|
||||
throw new InvalidOperationException(
|
||||
"FOCAS Host circuit breaker is open and awaiting manual reset. " +
|
||||
"See Admin /hosts; call AcknowledgeAndReset after investigating the Host log.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
_current = await _launcher.LaunchAsync(ct).ConfigureAwait(false);
|
||||
_currentStartedUtc = _clock();
|
||||
|
||||
// If the launch sequence itself takes long enough to count as a stable run,
|
||||
// reset the backoff ladder immediately.
|
||||
if (_clock() - _currentStartedUtc >= _backoff.StableRunThreshold)
|
||||
_backoff.RecordStableRun();
|
||||
|
||||
return _current;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
OnUnavailable?.Invoke($"launch-failed: {ex.Message}");
|
||||
ObservedCrashes++;
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown))
|
||||
{
|
||||
var hint = cooldown == TimeSpan.MaxValue
|
||||
? "manual reset required"
|
||||
: $"cooldown {cooldown:g}";
|
||||
throw new InvalidOperationException(
|
||||
$"FOCAS Host circuit breaker opened after {ObservedCrashes} crashes — {hint}.", ex);
|
||||
}
|
||||
|
||||
var delay = _backoff.Next();
|
||||
await Task.Delay(delay, ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Called from the heartbeat loop after a successful ack run — relaxes the backoff ladder.</summary>
|
||||
public void NotifyStableRun()
|
||||
{
|
||||
if (_current is null) return;
|
||||
if (_clock() - _currentStartedUtc >= _backoff.StableRunThreshold)
|
||||
_backoff.RecordStableRun();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
try { _launcher.TerminateAsync(CancellationToken.None).GetAwaiter().GetResult(); }
|
||||
catch { /* best effort */ }
|
||||
_current?.Dispose();
|
||||
_current = null;
|
||||
}
|
||||
|
||||
private void ThrowIfDisposed()
|
||||
{
|
||||
if (_disposed) throw new ObjectDisposedException(nameof(FocasHostSupervisor));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Tracks missed heartbeats from the FOCAS Host. 2s cadence + 3 consecutive misses =
|
||||
/// host declared dead (~6s detection). Same defaults as Galaxy Tier-C so operators
|
||||
/// see the same cadence across hosts on the /hosts Admin page.
|
||||
/// </summary>
|
||||
public sealed class HeartbeatMonitor
|
||||
{
|
||||
public int MissesUntilDead { get; init; } = 3;
|
||||
|
||||
public TimeSpan Cadence { get; init; } = TimeSpan.FromSeconds(2);
|
||||
|
||||
public int ConsecutiveMisses { get; private set; }
|
||||
public DateTime? LastAckUtc { get; private set; }
|
||||
|
||||
public void RecordAck(DateTime utcNow)
|
||||
{
|
||||
ConsecutiveMisses = 0;
|
||||
LastAckUtc = utcNow;
|
||||
}
|
||||
|
||||
/// <summary>Records a missed heartbeat; returns <c>true</c> when the death threshold is crossed.</summary>
|
||||
public bool RecordMiss()
|
||||
{
|
||||
ConsecutiveMisses++;
|
||||
return ConsecutiveMisses >= MissesUntilDead;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Abstraction over the act of spawning a FOCAS Host process and obtaining an
|
||||
/// <see cref="IFocasClient"/> connected to it. Production wires this to a real
|
||||
/// <c>Process.Start</c> + <c>FocasIpcClient.ConnectAsync</c>; tests use a fake that
|
||||
/// exposes deterministic failure modes so the supervisor logic can be stressed
|
||||
/// without spawning actual exes.
|
||||
/// </summary>
|
||||
public interface IHostProcessLauncher
|
||||
{
|
||||
/// <summary>
|
||||
/// Spawn a new Host process (if one isn't already running) and return a live
|
||||
/// client session. Throws on unrecoverable errors; transient errors (e.g. Host
|
||||
/// not ready yet) should throw <see cref="TimeoutException"/> so the supervisor
|
||||
/// applies the backoff ladder.
|
||||
/// </summary>
|
||||
Task<IFocasClient> LaunchAsync(CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// Terminate the Host process if one is running. Called on Dispose and after a
|
||||
/// heartbeat loss is detected.
|
||||
/// </summary>
|
||||
Task TerminateAsync(CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// <c>true</c> when the most recently spawned Host process is still alive.
|
||||
/// Supervisor polls this at heartbeat cadence; going <c>false</c> without a
|
||||
/// clean shutdown counts as a crash.
|
||||
/// </summary>
|
||||
bool IsProcessAlive { get; }
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Proxy-side reader for the Host's post-mortem MMF. After a Host crash the supervisor
|
||||
/// opens the file (which persists beyond the process lifetime) and enumerates the last
|
||||
/// few thousand IPC operations that were in flight. Format matches
|
||||
/// <c>Driver.FOCAS.Host.Stability.PostMortemMmf</c> — magic 'OFPC' / 256-byte entries.
|
||||
/// </summary>
|
||||
public sealed class PostMortemReader
|
||||
{
|
||||
private const int Magic = 0x4F465043; // 'OFPC'
|
||||
private const int HeaderBytes = 16;
|
||||
private const int EntryBytes = 256;
|
||||
private const int MessageOffset = 16;
|
||||
private const int MessageCapacity = EntryBytes - MessageOffset;
|
||||
|
||||
public string Path { get; }
|
||||
|
||||
public PostMortemReader(string path) => Path = path;
|
||||
|
||||
public PostMortemEntry[] ReadAll()
|
||||
{
|
||||
if (!File.Exists(Path)) return [];
|
||||
|
||||
using var mmf = MemoryMappedFile.CreateFromFile(Path, FileMode.Open, null, 0, MemoryMappedFileAccess.Read);
|
||||
using var accessor = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read);
|
||||
|
||||
if (accessor.ReadInt32(0) != Magic) return [];
|
||||
|
||||
var capacity = accessor.ReadInt32(8);
|
||||
var writeIndex = accessor.ReadInt32(12);
|
||||
var entries = new PostMortemEntry[capacity];
|
||||
var count = 0;
|
||||
|
||||
for (var i = 0; i < capacity; i++)
|
||||
{
|
||||
var slot = (writeIndex + i) % capacity;
|
||||
var offset = HeaderBytes + slot * EntryBytes;
|
||||
var ts = accessor.ReadInt64(offset + 0);
|
||||
if (ts == 0) continue;
|
||||
var op = accessor.ReadInt64(offset + 8);
|
||||
var msgBuf = new byte[MessageCapacity];
|
||||
accessor.ReadArray(offset + MessageOffset, msgBuf, 0, MessageCapacity);
|
||||
var nulTerm = Array.IndexOf<byte>(msgBuf, 0);
|
||||
var msg = Encoding.UTF8.GetString(msgBuf, 0, nulTerm < 0 ? MessageCapacity : nulTerm);
|
||||
entries[count++] = new PostMortemEntry(ts, op, msg);
|
||||
}
|
||||
|
||||
Array.Resize(ref entries, count);
|
||||
return entries;
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct PostMortemEntry(long UtcUnixMs, long OpKind, string Message);
|
||||
@@ -0,0 +1,113 @@
|
||||
using System.Diagnostics;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Ipc;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Production <see cref="IHostProcessLauncher"/>. Spawns <c>OtOpcUa.Driver.FOCAS.Host.exe</c>
|
||||
/// with the pipe name / allowed-SID / per-spawn shared secret in the environment, waits for
|
||||
/// the pipe to come up, then connects a <see cref="FocasIpcClient"/> and wraps it in an
|
||||
/// <see cref="IpcFocasClient"/>. On <see cref="TerminateAsync"/> best-effort kills the
|
||||
/// process and closes the IPC stream.
|
||||
/// </summary>
|
||||
public sealed class ProcessHostLauncher : IHostProcessLauncher
|
||||
{
|
||||
private readonly ProcessHostLauncherOptions _options;
|
||||
private Process? _process;
|
||||
private FocasIpcClient? _ipc;
|
||||
|
||||
public ProcessHostLauncher(ProcessHostLauncherOptions options)
|
||||
{
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
}
|
||||
|
||||
public bool IsProcessAlive => _process is { HasExited: false };
|
||||
|
||||
public async Task<IFocasClient> LaunchAsync(CancellationToken ct)
|
||||
{
|
||||
await TerminateAsync(ct).ConfigureAwait(false);
|
||||
|
||||
var secret = _options.SharedSecret ?? Guid.NewGuid().ToString("N");
|
||||
|
||||
var psi = new ProcessStartInfo
|
||||
{
|
||||
FileName = _options.HostExePath,
|
||||
Arguments = _options.Arguments ?? string.Empty,
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
};
|
||||
psi.Environment["OTOPCUA_FOCAS_PIPE"] = _options.PipeName;
|
||||
psi.Environment["OTOPCUA_ALLOWED_SID"] = _options.AllowedSid;
|
||||
psi.Environment["OTOPCUA_FOCAS_SECRET"] = secret;
|
||||
psi.Environment["OTOPCUA_FOCAS_BACKEND"] = _options.Backend;
|
||||
|
||||
_process = Process.Start(psi)
|
||||
?? throw new InvalidOperationException($"Failed to start {_options.HostExePath}");
|
||||
|
||||
// Poll for pipe readiness up to the configured connect timeout.
|
||||
var deadline = DateTime.UtcNow + _options.ConnectTimeout;
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
if (_process.HasExited)
|
||||
throw new InvalidOperationException(
|
||||
$"FOCAS Host exited before pipe was ready (ExitCode={_process.ExitCode}).");
|
||||
|
||||
try
|
||||
{
|
||||
_ipc = await FocasIpcClient.ConnectAsync(
|
||||
_options.PipeName, secret, TimeSpan.FromSeconds(1), ct).ConfigureAwait(false);
|
||||
break;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
if (DateTime.UtcNow >= deadline)
|
||||
throw new TimeoutException(
|
||||
$"FOCAS Host pipe {_options.PipeName} did not come up within {_options.ConnectTimeout:g}.");
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(250), ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
return new IpcFocasClient(_ipc, _options.Series);
|
||||
}
|
||||
|
||||
public async Task TerminateAsync(CancellationToken ct)
|
||||
{
|
||||
if (_ipc is not null)
|
||||
{
|
||||
try { await _ipc.DisposeAsync().ConfigureAwait(false); }
|
||||
catch { /* best effort */ }
|
||||
_ipc = null;
|
||||
}
|
||||
|
||||
if (_process is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!_process.HasExited)
|
||||
{
|
||||
_process.Kill(entireProcessTree: true);
|
||||
await _process.WaitForExitAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch { /* best effort */ }
|
||||
finally
|
||||
{
|
||||
_process.Dispose();
|
||||
_process = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record ProcessHostLauncherOptions(
|
||||
string HostExePath,
|
||||
string PipeName,
|
||||
string AllowedSid)
|
||||
{
|
||||
public string? SharedSecret { get; init; }
|
||||
public string? Arguments { get; init; }
|
||||
public string Backend { get; init; } = "fwlib32";
|
||||
public TimeSpan ConnectTimeout { get; init; } = TimeSpan.FromSeconds(15);
|
||||
public FocasCncSeries Series { get; init; } = FocasCncSeries.Unknown;
|
||||
}
|
||||
@@ -14,6 +14,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared\ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MessagePack;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Backend;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Ipc;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Tests
|
||||
{
|
||||
/// <summary>
|
||||
/// Validates that <see cref="FwlibFrameHandler"/> correctly dispatches each
|
||||
/// <see cref="FocasMessageKind"/> to the corresponding <see cref="IFocasBackend"/>
|
||||
/// method and serializes the response into the expected response kind. Uses
|
||||
/// <see cref="FakeFocasBackend"/> so no hardware is needed.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class FwlibFrameHandlerTests
|
||||
{
|
||||
private static async Task RoundTripAsync<TReq, TResp>(
|
||||
IFrameHandler handler, FocasMessageKind reqKind, TReq req, FocasMessageKind expectedRespKind,
|
||||
Action<TResp> assertResponse)
|
||||
{
|
||||
using var buffer = new MemoryStream();
|
||||
using var writer = new FrameWriter(buffer, leaveOpen: true);
|
||||
await handler.HandleAsync(reqKind, MessagePackSerializer.Serialize(req), writer, CancellationToken.None);
|
||||
|
||||
buffer.Position = 0;
|
||||
using var reader = new FrameReader(buffer, leaveOpen: true);
|
||||
var frame = await reader.ReadFrameAsync(CancellationToken.None);
|
||||
frame.HasValue.ShouldBeTrue();
|
||||
frame!.Value.Kind.ShouldBe(expectedRespKind);
|
||||
assertResponse(MessagePackSerializer.Deserialize<TResp>(frame.Value.Body));
|
||||
}
|
||||
|
||||
private static FwlibFrameHandler BuildHandler() =>
|
||||
new(new FakeFocasBackend(), new LoggerConfiguration().CreateLogger());
|
||||
|
||||
[Fact]
|
||||
public async Task OpenSession_returns_a_new_session_id()
|
||||
{
|
||||
long sessionId = 0;
|
||||
await RoundTripAsync<OpenSessionRequest, OpenSessionResponse>(
|
||||
BuildHandler(),
|
||||
FocasMessageKind.OpenSessionRequest,
|
||||
new OpenSessionRequest { HostAddress = "h:8193" },
|
||||
FocasMessageKind.OpenSessionResponse,
|
||||
resp => { resp.Success.ShouldBeTrue(); resp.SessionId.ShouldBeGreaterThan(0L); sessionId = resp.SessionId; });
|
||||
sessionId.ShouldBeGreaterThan(0L);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Read_without_open_session_returns_internal_error()
|
||||
{
|
||||
await RoundTripAsync<ReadRequest, ReadResponse>(
|
||||
BuildHandler(),
|
||||
FocasMessageKind.ReadRequest,
|
||||
new ReadRequest
|
||||
{
|
||||
SessionId = 999,
|
||||
Address = new FocasAddressDto { Kind = 0, PmcLetter = "R", Number = 100 },
|
||||
DataType = FocasDataTypeCode.Int32,
|
||||
},
|
||||
FocasMessageKind.ReadResponse,
|
||||
resp => { resp.Success.ShouldBeFalse(); resp.Error.ShouldContain("session-not-open"); });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Full_open_write_read_round_trip_preserves_value()
|
||||
{
|
||||
var handler = BuildHandler();
|
||||
|
||||
// Open.
|
||||
using var buffer = new MemoryStream();
|
||||
using var writer = new FrameWriter(buffer, leaveOpen: true);
|
||||
await handler.HandleAsync(FocasMessageKind.OpenSessionRequest,
|
||||
MessagePackSerializer.Serialize(new OpenSessionRequest { HostAddress = "h:8193" }), writer, CancellationToken.None);
|
||||
|
||||
buffer.Position = 0;
|
||||
using var reader = new FrameReader(buffer, leaveOpen: true);
|
||||
var openFrame = await reader.ReadFrameAsync(CancellationToken.None);
|
||||
var openResp = MessagePackSerializer.Deserialize<OpenSessionResponse>(openFrame!.Value.Body);
|
||||
var sessionId = openResp.SessionId;
|
||||
|
||||
// Write 42 at MACRO:500 as Int32.
|
||||
buffer.Position = 0;
|
||||
buffer.SetLength(0);
|
||||
await handler.HandleAsync(FocasMessageKind.WriteRequest,
|
||||
MessagePackSerializer.Serialize(new WriteRequest
|
||||
{
|
||||
SessionId = sessionId,
|
||||
Address = new FocasAddressDto { Kind = 2, Number = 500 },
|
||||
DataType = FocasDataTypeCode.Int32,
|
||||
ValueTypeCode = FocasDataTypeCode.Int32,
|
||||
ValueBytes = MessagePackSerializer.Serialize((int)42),
|
||||
}), writer, CancellationToken.None);
|
||||
|
||||
// Read back.
|
||||
buffer.Position = 0;
|
||||
buffer.SetLength(0);
|
||||
await handler.HandleAsync(FocasMessageKind.ReadRequest,
|
||||
MessagePackSerializer.Serialize(new ReadRequest
|
||||
{
|
||||
SessionId = sessionId,
|
||||
Address = new FocasAddressDto { Kind = 2, Number = 500 },
|
||||
DataType = FocasDataTypeCode.Int32,
|
||||
}), writer, CancellationToken.None);
|
||||
|
||||
buffer.Position = 0;
|
||||
var readFrame = await reader.ReadFrameAsync(CancellationToken.None);
|
||||
readFrame.HasValue.ShouldBeTrue();
|
||||
readFrame!.Value.Kind.ShouldBe(FocasMessageKind.ReadResponse);
|
||||
// With buffer reuse there may be multiple queued frames; we want the last one.
|
||||
var lastResp = MessagePackSerializer.Deserialize<ReadResponse>(readFrame.Value.Body);
|
||||
// If the Write frame is first, drain it.
|
||||
if (lastResp.ValueBytes is null)
|
||||
{
|
||||
var next = await reader.ReadFrameAsync(CancellationToken.None);
|
||||
lastResp = MessagePackSerializer.Deserialize<ReadResponse>(next!.Value.Body);
|
||||
}
|
||||
lastResp.Success.ShouldBeTrue();
|
||||
MessagePackSerializer.Deserialize<int>(lastResp.ValueBytes!).ShouldBe(42);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PmcBitWrite_sets_specified_bit()
|
||||
{
|
||||
var handler = BuildHandler();
|
||||
using var buffer = new MemoryStream();
|
||||
using var writer = new FrameWriter(buffer, leaveOpen: true);
|
||||
|
||||
await handler.HandleAsync(FocasMessageKind.OpenSessionRequest,
|
||||
MessagePackSerializer.Serialize(new OpenSessionRequest { HostAddress = "h:8193" }), writer, CancellationToken.None);
|
||||
buffer.Position = 0;
|
||||
using var reader = new FrameReader(buffer, leaveOpen: true);
|
||||
var openFrame = await reader.ReadFrameAsync(CancellationToken.None);
|
||||
var sessionId = MessagePackSerializer.Deserialize<OpenSessionResponse>(openFrame!.Value.Body).SessionId;
|
||||
|
||||
buffer.Position = 0; buffer.SetLength(0);
|
||||
await handler.HandleAsync(FocasMessageKind.PmcBitWriteRequest,
|
||||
MessagePackSerializer.Serialize(new PmcBitWriteRequest
|
||||
{
|
||||
SessionId = sessionId,
|
||||
Address = new FocasAddressDto { Kind = 0, PmcLetter = "R", Number = 100 },
|
||||
BitIndex = 3,
|
||||
Value = true,
|
||||
}), writer, CancellationToken.None);
|
||||
|
||||
buffer.Position = 0;
|
||||
var resp = MessagePackSerializer.Deserialize<PmcBitWriteResponse>(
|
||||
(await reader.ReadFrameAsync(CancellationToken.None))!.Value.Body);
|
||||
resp.Success.ShouldBeTrue();
|
||||
resp.StatusCode.ShouldBe(0u);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_reports_healthy_when_session_open()
|
||||
{
|
||||
var handler = BuildHandler();
|
||||
using var buffer = new MemoryStream();
|
||||
using var writer = new FrameWriter(buffer, leaveOpen: true);
|
||||
await handler.HandleAsync(FocasMessageKind.OpenSessionRequest,
|
||||
MessagePackSerializer.Serialize(new OpenSessionRequest { HostAddress = "h:8193" }), writer, CancellationToken.None);
|
||||
buffer.Position = 0;
|
||||
using var reader = new FrameReader(buffer, leaveOpen: true);
|
||||
var sessionId = MessagePackSerializer.Deserialize<OpenSessionResponse>(
|
||||
(await reader.ReadFrameAsync(CancellationToken.None))!.Value.Body).SessionId;
|
||||
|
||||
buffer.Position = 0; buffer.SetLength(0);
|
||||
await handler.HandleAsync(FocasMessageKind.ProbeRequest,
|
||||
MessagePackSerializer.Serialize(new ProbeRequest { SessionId = sessionId }), writer, CancellationToken.None);
|
||||
buffer.Position = 0;
|
||||
var resp = MessagePackSerializer.Deserialize<ProbeResponse>(
|
||||
(await reader.ReadFrameAsync(CancellationToken.None))!.Value.Body);
|
||||
resp.Healthy.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Unconfigured_backend_returns_pointed_error_message()
|
||||
{
|
||||
var handler = new FwlibFrameHandler(new UnconfiguredFocasBackend(), new LoggerConfiguration().CreateLogger());
|
||||
await RoundTripAsync<OpenSessionRequest, OpenSessionResponse>(
|
||||
handler,
|
||||
FocasMessageKind.OpenSessionRequest,
|
||||
new OpenSessionRequest { HostAddress = "h:8193" },
|
||||
FocasMessageKind.OpenSessionResponse,
|
||||
resp =>
|
||||
{
|
||||
resp.Success.ShouldBeFalse();
|
||||
resp.Error.ShouldContain("Fwlib32");
|
||||
resp.ErrorCode.ShouldBe("NoFwlibBackend");
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Stability;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Tests
|
||||
{
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class PostMortemMmfTests : IDisposable
|
||||
{
|
||||
private readonly string _tempPath;
|
||||
|
||||
public PostMortemMmfTests()
|
||||
{
|
||||
_tempPath = Path.Combine(Path.GetTempPath(), $"focas-mmf-{Guid.NewGuid():N}.bin");
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (File.Exists(_tempPath)) File.Delete(_tempPath);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Write_and_read_preserve_order_and_content()
|
||||
{
|
||||
using (var mmf = new PostMortemMmf(_tempPath, capacity: 10))
|
||||
{
|
||||
mmf.Write(opKind: 1, "read R100");
|
||||
mmf.Write(opKind: 2, "write MACRO:500 = 3.14");
|
||||
mmf.Write(opKind: 3, "probe ok");
|
||||
}
|
||||
|
||||
// Reopen (simulating a reader after the writer crashed).
|
||||
using var reader = new PostMortemMmf(_tempPath, capacity: 10);
|
||||
var entries = reader.ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
entries[0].OpKind.ShouldBe(1L);
|
||||
entries[0].Message.ShouldBe("read R100");
|
||||
entries[1].OpKind.ShouldBe(2L);
|
||||
entries[2].Message.ShouldBe("probe ok");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ring_buffer_wraps_at_capacity()
|
||||
{
|
||||
using var mmf = new PostMortemMmf(_tempPath, capacity: 3);
|
||||
for (var i = 0; i < 10; i++) mmf.Write(i, $"op-{i}");
|
||||
|
||||
var entries = mmf.ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
// Oldest surviving entry is op-7 (entries 7,8,9 survive in FIFO order).
|
||||
entries[0].Message.ShouldBe("op-7");
|
||||
entries[1].Message.ShouldBe("op-8");
|
||||
entries[2].Message.ShouldBe("op-9");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Truncated_message_is_null_terminated_and_does_not_overflow()
|
||||
{
|
||||
using var mmf = new PostMortemMmf(_tempPath, capacity: 4);
|
||||
var big = new string('x', 500); // longer than the 240-byte message capacity
|
||||
mmf.Write(42, big);
|
||||
|
||||
var entries = mmf.ReadAll();
|
||||
entries.Length.ShouldBe(1);
|
||||
entries[0].Message.Length.ShouldBeLessThanOrEqualTo(240);
|
||||
entries[0].OpKind.ShouldBe(42L);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reopening_with_existing_data_preserves_entries()
|
||||
{
|
||||
using (var first = new PostMortemMmf(_tempPath, capacity: 5))
|
||||
{
|
||||
first.Write(1, "first-run-1");
|
||||
first.Write(2, "first-run-2");
|
||||
}
|
||||
|
||||
using var second = new PostMortemMmf(_tempPath, capacity: 5);
|
||||
var entries = second.ReadAll();
|
||||
entries.Length.ShouldBe(2);
|
||||
entries[0].Message.ShouldBe("first-run-1");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
using MessagePack;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Ipc;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Shared.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end IPC round-trips over an in-memory loopback: <c>IpcFocasClient</c> talks
|
||||
/// to a test fake that plays the Host's role by reading frames, dispatching on kind,
|
||||
/// and responding with canned DTOs. Validates that every <see cref="IFocasClient"/>
|
||||
/// method translates to the right wire frame + decodes the response correctly.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class IpcFocasClientTests
|
||||
{
|
||||
private const string Secret = "test-secret";
|
||||
|
||||
private static async Task ServerLoopAsync(Stream serverSide, Func<FocasMessageKind, byte[], FrameWriter, Task> dispatch, CancellationToken ct)
|
||||
{
|
||||
using var reader = new FrameReader(serverSide, leaveOpen: true);
|
||||
using var writer = new FrameWriter(serverSide, leaveOpen: true);
|
||||
|
||||
// Hello handshake.
|
||||
var first = await reader.ReadFrameAsync(ct);
|
||||
if (first is null) return;
|
||||
var hello = MessagePackSerializer.Deserialize<Hello>(first.Value.Body);
|
||||
var accepted = hello.SharedSecret == Secret;
|
||||
await writer.WriteAsync(FocasMessageKind.HelloAck,
|
||||
new HelloAck { Accepted = accepted, RejectReason = accepted ? null : "wrong-secret" }, ct);
|
||||
if (!accepted) return;
|
||||
|
||||
while (!ct.IsCancellationRequested)
|
||||
{
|
||||
var frame = await reader.ReadFrameAsync(ct);
|
||||
if (frame is null) return;
|
||||
await dispatch(frame.Value.Kind, frame.Value.Body, writer);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Connect_sends_OpenSessionRequest_and_caches_session_id()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
OpenSessionRequest? received = null;
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
if (kind == FocasMessageKind.OpenSessionRequest)
|
||||
{
|
||||
received = MessagePackSerializer.Deserialize<OpenSessionRequest>(body);
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = true, SessionId = 42 }, cts.Token);
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc, FocasCncSeries.Thirty_i);
|
||||
await client.ConnectAsync(new FocasHostAddress("192.168.1.50", 8193), TimeSpan.FromSeconds(2), cts.Token);
|
||||
|
||||
client.IsConnected.ShouldBeTrue();
|
||||
received.ShouldNotBeNull();
|
||||
received!.HostAddress.ShouldBe("192.168.1.50:8193");
|
||||
received.CncSeries.ShouldBe((int)FocasCncSeries.Thirty_i);
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Connect_throws_when_host_rejects()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
if (kind == FocasMessageKind.OpenSessionRequest)
|
||||
{
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = false, Error = "unreachable", ErrorCode = "EW_SOCKET" }, cts.Token);
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
await Should.ThrowAsync<InvalidOperationException>(async () =>
|
||||
await client.ConnectAsync(new FocasHostAddress("10.0.0.1", 8193), TimeSpan.FromSeconds(1), cts.Token));
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Read_sends_ReadRequest_and_decodes_response()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
ReadRequest? received = null;
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
switch (kind)
|
||||
{
|
||||
case FocasMessageKind.OpenSessionRequest:
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = true, SessionId = 1 }, cts.Token);
|
||||
break;
|
||||
case FocasMessageKind.ReadRequest:
|
||||
received = MessagePackSerializer.Deserialize<ReadRequest>(body);
|
||||
await writer.WriteAsync(FocasMessageKind.ReadResponse,
|
||||
new ReadResponse
|
||||
{
|
||||
Success = true,
|
||||
StatusCode = 0,
|
||||
ValueBytes = MessagePackSerializer.Serialize((int)12345),
|
||||
ValueTypeCode = FocasDataTypeCode.Int32,
|
||||
}, cts.Token);
|
||||
break;
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
await client.ConnectAsync(new FocasHostAddress("h", 8193), TimeSpan.FromSeconds(1), cts.Token);
|
||||
|
||||
var addr = new FocasAddress(FocasAreaKind.Parameter, null, 1815, null);
|
||||
var (value, status) = await client.ReadAsync(addr, FocasDataType.Int32, cts.Token);
|
||||
status.ShouldBe(0u);
|
||||
value.ShouldBe(12345);
|
||||
received!.Address.Number.ShouldBe(1815);
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Write_sends_WriteRequest_and_returns_status()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
switch (kind)
|
||||
{
|
||||
case FocasMessageKind.OpenSessionRequest:
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = true, SessionId = 1 }, cts.Token);
|
||||
break;
|
||||
case FocasMessageKind.WriteRequest:
|
||||
var req = MessagePackSerializer.Deserialize<WriteRequest>(body);
|
||||
MessagePackSerializer.Deserialize<double>(req.ValueBytes!).ShouldBe(3.14);
|
||||
await writer.WriteAsync(FocasMessageKind.WriteResponse,
|
||||
new WriteResponse { Success = true, StatusCode = 0 }, cts.Token);
|
||||
break;
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
await client.ConnectAsync(new FocasHostAddress("h", 8193), TimeSpan.FromSeconds(1), cts.Token);
|
||||
|
||||
var status = await client.WriteAsync(new FocasAddress(FocasAreaKind.Macro, null, 500, null),
|
||||
FocasDataType.Float64, 3.14, cts.Token);
|
||||
status.ShouldBe(0u);
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Write_pmc_bit_sends_first_class_RMW_frame()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
PmcBitWriteRequest? received = null;
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
switch (kind)
|
||||
{
|
||||
case FocasMessageKind.OpenSessionRequest:
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = true, SessionId = 1 }, cts.Token);
|
||||
break;
|
||||
case FocasMessageKind.PmcBitWriteRequest:
|
||||
received = MessagePackSerializer.Deserialize<PmcBitWriteRequest>(body);
|
||||
await writer.WriteAsync(FocasMessageKind.PmcBitWriteResponse,
|
||||
new PmcBitWriteResponse { Success = true, StatusCode = 0 }, cts.Token);
|
||||
break;
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
await client.ConnectAsync(new FocasHostAddress("h", 8193), TimeSpan.FromSeconds(1), cts.Token);
|
||||
|
||||
var addr = new FocasAddress(FocasAreaKind.Pmc, "R", 100, BitIndex: 5);
|
||||
var status = await client.WriteAsync(addr, FocasDataType.Bit, true, cts.Token);
|
||||
status.ShouldBe(0u);
|
||||
received.ShouldNotBeNull();
|
||||
received!.BitIndex.ShouldBe(5);
|
||||
received.Value.ShouldBeTrue();
|
||||
received.Address.PmcLetter.ShouldBe("R");
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_round_trips_health_from_host()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
switch (kind)
|
||||
{
|
||||
case FocasMessageKind.OpenSessionRequest:
|
||||
await writer.WriteAsync(FocasMessageKind.OpenSessionResponse,
|
||||
new OpenSessionResponse { Success = true, SessionId = 1 }, cts.Token);
|
||||
break;
|
||||
case FocasMessageKind.ProbeRequest:
|
||||
await writer.WriteAsync(FocasMessageKind.ProbeResponse,
|
||||
new ProbeResponse { Healthy = true, ObservedAtUtcUnixMs = 1_700_000_000_000 }, cts.Token);
|
||||
break;
|
||||
}
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
await client.ConnectAsync(new FocasHostAddress("h", 8193), TimeSpan.FromSeconds(1), cts.Token);
|
||||
(await client.ProbeAsync(cts.Token)).ShouldBeTrue();
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Error_response_from_host_surfaces_as_FocasIpcException()
|
||||
{
|
||||
await using var loop = new IpcLoopback();
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
|
||||
|
||||
var server = Task.Run(() => ServerLoopAsync(loop.ServerSide, async (kind, body, writer) =>
|
||||
{
|
||||
await writer.WriteAsync(FocasMessageKind.ErrorResponse,
|
||||
new ErrorResponse { Code = "backend-exception", Message = "simulated" }, cts.Token);
|
||||
}, cts.Token));
|
||||
|
||||
var ipc = await FocasIpcClient.ConnectAsync(loop.ClientSide, Secret, cts.Token);
|
||||
var client = new IpcFocasClient(ipc);
|
||||
var ex = await Should.ThrowAsync<FocasIpcException>(async () =>
|
||||
await client.ConnectAsync(new FocasHostAddress("h", 8193), TimeSpan.FromSeconds(1), cts.Token));
|
||||
ex.Code.ShouldBe("backend-exception");
|
||||
|
||||
cts.Cancel();
|
||||
try { await server; } catch { }
|
||||
}
|
||||
}
|
||||
72
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/IpcLoopback.cs
Normal file
72
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/IpcLoopback.cs
Normal file
@@ -0,0 +1,72 @@
|
||||
using System.IO.Pipelines;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Bidirectional in-memory stream pair for IPC tests. Two <c>System.IO.Pipelines.Pipe</c>
|
||||
/// instances — one per direction — exposed as <see cref="System.IO.Stream"/> endpoints
|
||||
/// via <c>PipeReader.AsStream</c> / <c>PipeWriter.AsStream</c>. Lets the test set up a
|
||||
/// <c>FocasIpcClient</c> on one end and a minimal fake server loop on the other without
|
||||
/// standing up a real named pipe.
|
||||
/// </summary>
|
||||
internal sealed class IpcLoopback : IAsyncDisposable
|
||||
{
|
||||
public Stream ClientSide { get; }
|
||||
public Stream ServerSide { get; }
|
||||
|
||||
public IpcLoopback()
|
||||
{
|
||||
var clientToServer = new Pipe();
|
||||
var serverToClient = new Pipe();
|
||||
|
||||
ClientSide = new DuplexPipeStream(serverToClient.Reader.AsStream(), clientToServer.Writer.AsStream());
|
||||
ServerSide = new DuplexPipeStream(clientToServer.Reader.AsStream(), serverToClient.Writer.AsStream());
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await ClientSide.DisposeAsync();
|
||||
await ServerSide.DisposeAsync();
|
||||
}
|
||||
|
||||
private sealed class DuplexPipeStream(Stream read, Stream write) : Stream
|
||||
{
|
||||
public override bool CanRead => true;
|
||||
public override bool CanWrite => true;
|
||||
public override bool CanSeek => false;
|
||||
public override long Length => throw new NotSupportedException();
|
||||
public override long Position
|
||||
{
|
||||
get => throw new NotSupportedException();
|
||||
set => throw new NotSupportedException();
|
||||
}
|
||||
|
||||
public override int Read(byte[] buffer, int offset, int count) => read.Read(buffer, offset, count);
|
||||
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken ct) =>
|
||||
read.ReadAsync(buffer, offset, count, ct);
|
||||
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken ct = default) =>
|
||||
read.ReadAsync(buffer, ct);
|
||||
|
||||
public override void Write(byte[] buffer, int offset, int count) => write.Write(buffer, offset, count);
|
||||
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken ct) =>
|
||||
write.WriteAsync(buffer, offset, count, ct);
|
||||
public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken ct = default) =>
|
||||
write.WriteAsync(buffer, ct);
|
||||
|
||||
public override void Flush() => write.Flush();
|
||||
public override Task FlushAsync(CancellationToken ct) => write.FlushAsync(ct);
|
||||
|
||||
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
|
||||
public override void SetLength(long value) => throw new NotSupportedException();
|
||||
|
||||
protected override void Dispose(bool disposing)
|
||||
{
|
||||
if (disposing)
|
||||
{
|
||||
read.Dispose();
|
||||
write.Dispose();
|
||||
}
|
||||
base.Dispose(disposing);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The Proxy-side <see cref="PostMortemReader"/> must read the Host's MMF format
|
||||
/// (magic 'OFPC', 256-byte entries). This test writes a hand-crafted file that mimics
|
||||
/// the Host's layout exactly + asserts the reader decodes it correctly. Keeps the two
|
||||
/// codebases in lockstep on the wire format without needing to reference the net48
|
||||
/// Host assembly from the net10 test project.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class PostMortemReaderCompatibilityTests : IDisposable
|
||||
{
|
||||
private readonly string _tempPath = Path.Combine(Path.GetTempPath(), $"focas-mmf-compat-{Guid.NewGuid():N}.bin");
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (File.Exists(_tempPath)) File.Delete(_tempPath);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_parses_host_format_and_returns_entries_in_oldest_first_order()
|
||||
{
|
||||
const int magic = 0x4F465043;
|
||||
const int capacity = 5;
|
||||
const int headerBytes = 16;
|
||||
const int entryBytes = 256;
|
||||
const int messageOffset = 16;
|
||||
var fileBytes = headerBytes + capacity * entryBytes;
|
||||
|
||||
using (var fs = new FileStream(_tempPath, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.Read))
|
||||
{
|
||||
fs.SetLength(fileBytes);
|
||||
using var mmf = MemoryMappedFile.CreateFromFile(fs, null, fileBytes,
|
||||
MemoryMappedFileAccess.ReadWrite, HandleInheritability.None, leaveOpen: false);
|
||||
using var acc = mmf.CreateViewAccessor(0, fileBytes, MemoryMappedFileAccess.ReadWrite);
|
||||
acc.Write(0, magic);
|
||||
acc.Write(4, 1);
|
||||
acc.Write(8, capacity);
|
||||
acc.Write(12, 2); // writeIndex — next write would land at slot 2
|
||||
|
||||
void WriteEntry(int slot, long ts, long op, string msg)
|
||||
{
|
||||
var offset = headerBytes + slot * entryBytes;
|
||||
acc.Write(offset + 0, ts);
|
||||
acc.Write(offset + 8, op);
|
||||
var bytes = Encoding.UTF8.GetBytes(msg);
|
||||
acc.WriteArray(offset + messageOffset, bytes, 0, bytes.Length);
|
||||
acc.Write(offset + messageOffset + bytes.Length, (byte)0);
|
||||
}
|
||||
|
||||
WriteEntry(0, 100, 1, "op-a");
|
||||
WriteEntry(1, 200, 2, "op-b");
|
||||
// Slots 2,3 unwritten (ts=0) — reader must skip.
|
||||
WriteEntry(4, 50, 9, "old-wrapped");
|
||||
}
|
||||
|
||||
var entries = new PostMortemReader(_tempPath).ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
// writeIndex=2 means the ring walk starts at slot 2, so iteration order is 2→3→4→0→1.
|
||||
// Slots 2 and 3 are empty; 4 yields "old-wrapped"; then 0="op-a", 1="op-b".
|
||||
entries[0].Message.ShouldBe("old-wrapped");
|
||||
entries[1].Message.ShouldBe("op-a");
|
||||
entries[2].Message.ShouldBe("op-b");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_returns_empty_when_file_missing()
|
||||
{
|
||||
new PostMortemReader(_tempPath + "-does-not-exist").ReadAll().ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_returns_empty_when_magic_mismatches()
|
||||
{
|
||||
File.WriteAllBytes(_tempPath, new byte[1024]);
|
||||
new PostMortemReader(_tempPath).ReadAll().ShouldBeEmpty();
|
||||
}
|
||||
}
|
||||
249
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/SupervisorTests.cs
Normal file
249
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/SupervisorTests.cs
Normal file
@@ -0,0 +1,249 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class BackoffTests
|
||||
{
|
||||
[Fact]
|
||||
public void Default_sequence_is_5s_15s_60s_then_clamped()
|
||||
{
|
||||
var b = new Backoff();
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(5));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(15));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RecordStableRun_resets_the_ladder_to_the_start()
|
||||
{
|
||||
var b = new Backoff();
|
||||
b.Next(); b.Next();
|
||||
b.AttemptIndex.ShouldBe(2);
|
||||
b.RecordStableRun();
|
||||
b.AttemptIndex.ShouldBe(0);
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class CircuitBreakerTests
|
||||
{
|
||||
[Fact]
|
||||
public void Allows_crashes_below_threshold()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
b.TryRecordCrash(now, out _).ShouldBeTrue();
|
||||
b.TryRecordCrash(now.AddSeconds(1), out _).ShouldBeTrue();
|
||||
b.TryRecordCrash(now.AddSeconds(2), out _).ShouldBeTrue();
|
||||
b.StickyAlertActive.ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Opens_when_exceeding_threshold_in_window()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
b.TryRecordCrash(now, out _);
|
||||
b.TryRecordCrash(now.AddSeconds(1), out _);
|
||||
b.TryRecordCrash(now.AddSeconds(2), out _);
|
||||
b.TryRecordCrash(now.AddSeconds(3), out var cooldown).ShouldBeFalse();
|
||||
cooldown.ShouldBe(TimeSpan.FromHours(1));
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Escalates_cooldown_after_second_open()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var t0 = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
// First burst — 4 crashes opens breaker with 1h cooldown.
|
||||
for (var i = 0; i < 4; i++) b.TryRecordCrash(t0.AddSeconds(i), out _);
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
|
||||
// Wait past cooldown. The first crash after cooldown-elapsed resets _openSinceUtc and
|
||||
// bumps escalation level; the next 3 crashes then re-open with the escalated 4h cooldown.
|
||||
b.TryRecordCrash(t0.AddHours(1).AddMinutes(1), out _);
|
||||
var t1 = t0.AddHours(1).AddMinutes(1).AddSeconds(1);
|
||||
b.TryRecordCrash(t1, out _);
|
||||
b.TryRecordCrash(t1.AddSeconds(1), out _);
|
||||
b.TryRecordCrash(t1.AddSeconds(2), out var cooldown).ShouldBeFalse();
|
||||
cooldown.ShouldBe(TimeSpan.FromHours(4));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManualReset_clears_everything()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
for (var i = 0; i < 5; i++) b.TryRecordCrash(now.AddSeconds(i), out _);
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
b.ManualReset();
|
||||
b.StickyAlertActive.ShouldBeFalse();
|
||||
b.TryRecordCrash(now.AddSeconds(10), out _).ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class HeartbeatMonitorTests
|
||||
{
|
||||
[Fact]
|
||||
public void Three_consecutive_misses_declares_dead()
|
||||
{
|
||||
var m = new HeartbeatMonitor();
|
||||
m.RecordMiss().ShouldBeFalse();
|
||||
m.RecordMiss().ShouldBeFalse();
|
||||
m.RecordMiss().ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ack_resets_the_miss_counter()
|
||||
{
|
||||
var m = new HeartbeatMonitor();
|
||||
m.RecordMiss(); m.RecordMiss();
|
||||
m.ConsecutiveMisses.ShouldBe(2);
|
||||
m.RecordAck(DateTime.UtcNow);
|
||||
m.ConsecutiveMisses.ShouldBe(0);
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class FocasHostSupervisorTests
|
||||
{
|
||||
private sealed class FakeLauncher : IHostProcessLauncher
|
||||
{
|
||||
public int LaunchAttempts { get; private set; }
|
||||
public int Terminations { get; private set; }
|
||||
public Queue<Func<IFocasClient>> Plan { get; } = new();
|
||||
public bool IsProcessAlive { get; set; }
|
||||
|
||||
public Task<IFocasClient> LaunchAsync(CancellationToken ct)
|
||||
{
|
||||
LaunchAttempts++;
|
||||
if (Plan.Count == 0) throw new InvalidOperationException("FakeLauncher plan exhausted");
|
||||
var next = Plan.Dequeue()();
|
||||
IsProcessAlive = true;
|
||||
return Task.FromResult(next);
|
||||
}
|
||||
|
||||
public Task TerminateAsync(CancellationToken ct)
|
||||
{
|
||||
Terminations++;
|
||||
IsProcessAlive = false;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class StubFocasClient : IFocasClient
|
||||
{
|
||||
public bool IsConnected => true;
|
||||
public Task ConnectAsync(FocasHostAddress address, TimeSpan timeout, CancellationToken ct) => Task.CompletedTask;
|
||||
public Task<(object? value, uint status)> ReadAsync(FocasAddress a, FocasDataType t, CancellationToken ct) =>
|
||||
Task.FromResult<(object?, uint)>((0, 0));
|
||||
public Task<uint> WriteAsync(FocasAddress a, FocasDataType t, object? v, CancellationToken ct) => Task.FromResult(0u);
|
||||
public Task<bool> ProbeAsync(CancellationToken ct) => Task.FromResult(true);
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetOrLaunch_returns_client_on_first_success()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
var client = await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
client.ShouldNotBeNull();
|
||||
launcher.LaunchAttempts.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetOrLaunch_retries_after_transient_failure_with_backoff()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => throw new TimeoutException("pipe not ready"));
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
|
||||
var backoff = new Backoff([TimeSpan.FromMilliseconds(10), TimeSpan.FromMilliseconds(20)]);
|
||||
var supervisor = new FocasHostSupervisor(launcher, backoff);
|
||||
|
||||
var unavailableMessages = new List<string>();
|
||||
supervisor.OnUnavailable += m => unavailableMessages.Add(m);
|
||||
|
||||
var client = await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
client.ShouldNotBeNull();
|
||||
launcher.LaunchAttempts.ShouldBe(2);
|
||||
unavailableMessages.Count.ShouldBe(1);
|
||||
unavailableMessages[0].ShouldContain("launch-failed");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Repeated_launch_failures_open_breaker_and_surface_InvalidOperation()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
for (var i = 0; i < 10; i++)
|
||||
launcher.Plan.Enqueue(() => throw new InvalidOperationException("simulated host refused"));
|
||||
|
||||
var supervisor = new FocasHostSupervisor(
|
||||
launcher,
|
||||
backoff: new Backoff([TimeSpan.FromMilliseconds(1)]),
|
||||
breaker: new CircuitBreaker { CrashesAllowedPerWindow = 2, Window = TimeSpan.FromMinutes(5) });
|
||||
|
||||
var ex = await Should.ThrowAsync<InvalidOperationException>(async () =>
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken));
|
||||
ex.Message.ShouldContain("circuit breaker");
|
||||
supervisor.StickyAlertActive.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NotifyHostDeadAsync_terminates_current_and_fans_out_unavailable()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
|
||||
var messages = new List<string>();
|
||||
supervisor.OnUnavailable += m => messages.Add(m);
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
await supervisor.NotifyHostDeadAsync("heartbeat-loss", TestContext.Current.CancellationToken);
|
||||
|
||||
launcher.Terminations.ShouldBe(1);
|
||||
messages.ShouldContain("heartbeat-loss");
|
||||
supervisor.ObservedCrashes.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AcknowledgeAndReset_clears_sticky_alert()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
for (var i = 0; i < 10; i++)
|
||||
launcher.Plan.Enqueue(() => throw new InvalidOperationException("refused"));
|
||||
var supervisor = new FocasHostSupervisor(
|
||||
launcher,
|
||||
backoff: new Backoff([TimeSpan.FromMilliseconds(1)]),
|
||||
breaker: new CircuitBreaker { CrashesAllowedPerWindow = 1 });
|
||||
|
||||
try { await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken); } catch { }
|
||||
supervisor.StickyAlertActive.ShouldBeTrue();
|
||||
|
||||
supervisor.AcknowledgeAndReset();
|
||||
supervisor.StickyAlertActive.ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Dispose_terminates_host_process()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
supervisor.Dispose();
|
||||
launcher.Terminations.ShouldBe(1);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user