Compare commits
4 Commits
focas-tier
...
focas-tier
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8d88ffa14d | ||
| 446a5c022c | |||
|
|
5033609944 | ||
| 9034294b77 |
@@ -69,14 +69,32 @@ covers the common address shapes; per-model quirks are not stressed.
|
||||
- Parameter range enforcement (CNC rejects out-of-range writes)
|
||||
- MTB (machine tool builder) custom screens that expose non-standard data
|
||||
|
||||
### 5. Tier-C process isolation behavior
|
||||
### 5. Tier-C process isolation — architecture shipped, Fwlib32 integration hardware-gated
|
||||
|
||||
Per driver-stability.md, FOCAS should run process-isolated because
|
||||
`Fwlib32.dll` has documented crash modes. The test suite runs in-process +
|
||||
only exercises the happy path + mapped error codes — a native access
|
||||
violation from the DLL would take the test host down. The process-isolation
|
||||
path (similar to Galaxy's out-of-process Host) has been scoped but not
|
||||
implemented.
|
||||
The Tier-C architecture is now in place as of PRs #169–#173 (FOCAS
|
||||
PR A–E, task #220):
|
||||
|
||||
- `Driver.FOCAS.Shared` carries MessagePack IPC contracts
|
||||
- `Driver.FOCAS.Host` (.NET 4.8 x86 Windows service via NSSM) accepts
|
||||
a connection on a strictly-ACL'd named pipe + dispatches frames to
|
||||
an `IFocasBackend`
|
||||
- `Driver.FOCAS.Ipc.IpcFocasClient` implements the `IFocasClient` DI
|
||||
seam by forwarding over IPC — swap the DI registration and the
|
||||
driver runs Tier-C with zero other changes
|
||||
- `Driver.FOCAS.Supervisor.FocasHostSupervisor` owns the spawn +
|
||||
heartbeat + respawn + 3-in-5min crash-loop breaker + sticky alert
|
||||
- `Driver.FOCAS.Host.Stability.PostMortemMmf` ↔
|
||||
`Driver.FOCAS.Supervisor.PostMortemReader` — ring-buffer of the
|
||||
last ~1000 IPC operations survives a Host crash
|
||||
|
||||
The one remaining gap is the production `FwlibHostedBackend`: an
|
||||
`IFocasBackend` implementation that wraps the licensed
|
||||
`Fwlib32.dll` P/Invoke. That's hardware-gated on task #222 — we
|
||||
need a CNC on the bench (or the licensed FANUC developer kit DLL
|
||||
with a test harness) to validate it. Until then, the Host ships
|
||||
`FakeFocasBackend` + `UnconfiguredFocasBackend`. Setting
|
||||
`OTOPCUA_FOCAS_BACKEND=fake` lets operators smoke-test the whole
|
||||
Tier-C pipeline end-to-end without any CNC.
|
||||
|
||||
## When to trust FOCAS tests, when to reach for a rig
|
||||
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
# FOCAS Tier-C isolation — plan for task #220
|
||||
|
||||
> **Status**: DRAFT — not yet started. Tracks the multi-PR work to
|
||||
> move `Fwlib32.dll` behind an out-of-process host, mirroring the
|
||||
> Galaxy Tier-C split in [`phase-2-galaxy-out-of-process.md`](phase-2-galaxy-out-of-process.md).
|
||||
> **Status**: PRs A–E shipped. Architecture is in place; the only
|
||||
> remaining FOCAS work is the hardware-dependent production
|
||||
> integration of `Fwlib32.dll` into a real `IFocasBackend`
|
||||
> (`FwlibHostedBackend`), which needs an actual CNC on the bench
|
||||
> and is tracked as a follow-up on #220.
|
||||
>
|
||||
> **Pre-reqs shipped** (this PR): version matrix + pre-flight
|
||||
> validation + unit tests. Those close the cheap half of the
|
||||
> hardware-free stability gap. Tier-C closes the expensive half.
|
||||
> **Pre-reqs shipped**: version matrix + pre-flight validation
|
||||
> (PR #168 — the cheap half of the hardware-free stability gap).
|
||||
|
||||
## Why isolate
|
||||
|
||||
@@ -79,32 +80,41 @@ its own timer + pushes change notifications so the Proxy doesn't
|
||||
round-trip per poll. Matches `Driver.Galaxy.Host` subscription
|
||||
forwarding.
|
||||
|
||||
## PR sequence (proposed)
|
||||
## PR sequence — shipped
|
||||
|
||||
1. **PR A — shared contracts**
|
||||
Create `Driver.FOCAS.Shared` with the MessagePack DTOs. No
|
||||
behaviour change. ~200 LOC + round-trip tests for each DTO.
|
||||
2. **PR B — Host project skeleton**
|
||||
Create `Driver.FOCAS.Host` .NET 4.8 x86 project, NSSM wrapper,
|
||||
pipe server scaffold with the same ACL + caller-SID + shared
|
||||
secret plumbing as Galaxy.Host. No Fwlib32 wiring yet — returns
|
||||
`NotImplemented` for everything. ~400 LOC.
|
||||
3. **PR C — Move Fwlib32 calls into Host**
|
||||
Move `FocasNativeSession`, `FocasTagReader`, `FocasTagWriter`,
|
||||
`FocasPmcBitRmw` + the STA thread into the Host. Proxy forwards
|
||||
over IPC. This is the biggest PR — probably 800-1500 LOC of
|
||||
move-with-translation. Existing unit tests keep passing because
|
||||
`IFocasTagFactory` is the DI seam the tests inject against.
|
||||
4. **PR D — Supervisor + respawn**
|
||||
Proxy-side heartbeat + respawn + crash-loop circuit breaker +
|
||||
BackPressure fan-out on Host death. ~500 LOC + chaos tests.
|
||||
5. **PR E — Post-mortem MMF + operational glue**
|
||||
MMF writer in Host, reader in Proxy. Install scripts for the
|
||||
new `OtOpcUaFocasHost` Windows service. Docs. ~300 LOC.
|
||||
1. **PR A (#169) — shared contracts** ✅
|
||||
`Driver.FOCAS.Shared` netstandard2.0 with MessagePack DTOs for every
|
||||
IPC surface (Hello/Heartbeat/OpenSession/Read/Write/PmcBitWrite/
|
||||
Subscribe/Probe/RuntimeStatus/Recycle/ErrorResponse) + FrameReader/
|
||||
FrameWriter + 24 round-trip tests.
|
||||
2. **PR B (#170) — Host project skeleton** ✅
|
||||
`Driver.FOCAS.Host` net48 x86 Windows Service entry point,
|
||||
`PipeAcl` + `PipeServer` + `IFrameHandler` + `StubFrameHandler`.
|
||||
ACL denies LocalSystem/Administrators; Hello verifies
|
||||
shared-secret + protocol major. 3 handshake tests.
|
||||
3. **PR C (#171) — IPC path end-to-end** ✅
|
||||
Proxy `Ipc/FocasIpcClient` + `Ipc/IpcFocasClient` (implements
|
||||
IFocasClient via IPC). Host `Backend/IFocasBackend` +
|
||||
`FakeFocasBackend` + `UnconfiguredFocasBackend` +
|
||||
`Ipc/FwlibFrameHandler` replacing the stub. 13 new round-trip
|
||||
tests via in-memory loopback.
|
||||
4. **PR D (#172) — Supervisor + respawn** ✅
|
||||
`Supervisor/Backoff` (5s→15s→60s) + `CircuitBreaker` (3-in-5min →
|
||||
1h→4h→manual) + `HeartbeatMonitor` + `IHostProcessLauncher` +
|
||||
`FocasHostSupervisor`. 14 tests.
|
||||
5. **PR E — Ops glue** ✅ (this PR)
|
||||
`ProcessHostLauncher` (real Process.Start + FocasIpcClient
|
||||
connect), `Host/Stability/PostMortemMmf` (magic 'OFPC') +
|
||||
Proxy `Supervisor/PostMortemReader`, `scripts/install/
|
||||
Install-FocasHost.ps1` + `Uninstall-FocasHost.ps1` NSSM wrappers.
|
||||
7 tests (4 MMF round-trip + 3 reader format compatibility).
|
||||
|
||||
Total estimate: 2200-3200 LOC across 5 PRs. Consistent with Galaxy
|
||||
Tier-C but narrower since FOCAS has no Historian + no alarm
|
||||
history.
|
||||
**Post-shipment totals: 189 FOCAS driver tests + 24 Shared tests + 13 Host tests = 226 FOCAS-family tests green.**
|
||||
|
||||
What remains is hardware-dependent: wiring `Fwlib32.dll` P/Invoke
|
||||
into a real `FwlibHostedBackend` implementation of `IFocasBackend`
|
||||
+ validating against a live CNC. The architecture is all the
|
||||
plumbing that work needs.
|
||||
|
||||
## Testing without hardware
|
||||
|
||||
|
||||
108
scripts/install/Install-FocasHost.ps1
Normal file
108
scripts/install/Install-FocasHost.ps1
Normal file
@@ -0,0 +1,108 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Registers the OtOpcUaFocasHost Windows service. Optional companion to
|
||||
Install-Services.ps1 — only run this on nodes where FOCAS driver instances will run
|
||||
with Tier-C process isolation enabled.
|
||||
|
||||
.DESCRIPTION
|
||||
FOCAS PR #220 / Tier-C isolation plan. Wraps OtOpcUa.Driver.FOCAS.Host.exe (net48 x86)
|
||||
as a Windows service using NSSM, running under the same service account as the main
|
||||
OtOpcUa service so the named-pipe ACL works. Passes the per-process shared secret via
|
||||
environment variable at service-start time so it never hits disk.
|
||||
|
||||
.PARAMETER InstallRoot
|
||||
Where the FOCAS Host binaries live (typically
|
||||
C:\Program Files\OtOpcUa\Driver.FOCAS.Host).
|
||||
|
||||
.PARAMETER ServiceAccount
|
||||
Service account SID or DOMAIN\name. Must match the main OtOpcUa server account so the
|
||||
PipeAcl match succeeds.
|
||||
|
||||
.PARAMETER FocasSharedSecret
|
||||
Per-process secret passed via env var. Generated freshly per install if not supplied.
|
||||
|
||||
.PARAMETER FocasBackend
|
||||
Backend selector for the Host process. One of:
|
||||
fwlib32 (default — real Fanuc Fwlib32.dll integration; requires licensed DLL on PATH)
|
||||
fake (in-memory; smoke-test mode)
|
||||
unconfigured (safe default returning structured errors; use until hardware is wired)
|
||||
|
||||
.PARAMETER FocasPipeName
|
||||
Pipe name the Host listens on. Default: OtOpcUaFocas.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-FocasHost.ps1 -InstallRoot 'C:\Program Files\OtOpcUa\Driver.FOCAS.Host' `
|
||||
-ServiceAccount 'OTOPCUA\svc-otopcua' -FocasBackend fwlib32
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$InstallRoot,
|
||||
[Parameter(Mandatory)] [string]$ServiceAccount,
|
||||
[string]$FocasSharedSecret,
|
||||
[ValidateSet('fwlib32','fake','unconfigured')] [string]$FocasBackend = 'unconfigured',
|
||||
[string]$FocasPipeName = 'OtOpcUaFocas',
|
||||
[string]$ServiceName = 'OtOpcUaFocasHost',
|
||||
[string]$NssmPath = 'C:\Program Files\nssm\nssm.exe'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Resolve-Sid {
|
||||
param([string]$Account)
|
||||
if ($Account -match '^S-\d-\d+') { return $Account }
|
||||
try {
|
||||
$nt = New-Object System.Security.Principal.NTAccount($Account)
|
||||
return $nt.Translate([System.Security.Principal.SecurityIdentifier]).Value
|
||||
} catch {
|
||||
throw "Could not resolve '$Account' to a SID. Pass an explicit SID or check the account name."
|
||||
}
|
||||
}
|
||||
|
||||
if (-not (Test-Path $NssmPath)) {
|
||||
throw "nssm.exe not found at '$NssmPath'. Install NSSM or pass -NssmPath."
|
||||
}
|
||||
|
||||
$hostExe = Join-Path $InstallRoot 'OtOpcUa.Driver.FOCAS.Host.exe'
|
||||
if (-not (Test-Path $hostExe)) {
|
||||
throw "FOCAS Host binary not found at '$hostExe'. Publish the Driver.FOCAS.Host project first."
|
||||
}
|
||||
|
||||
if (-not $FocasSharedSecret) {
|
||||
$FocasSharedSecret = [System.Guid]::NewGuid().ToString('N')
|
||||
Write-Host "Generated FocasSharedSecret — store it alongside the OtOpcUa service config."
|
||||
}
|
||||
|
||||
$allowedSid = Resolve-Sid $ServiceAccount
|
||||
|
||||
# Idempotent install — remove + re-create if present.
|
||||
$existing = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
|
||||
if ($existing) {
|
||||
Write-Host "Removing existing '$ServiceName' service..."
|
||||
& $NssmPath stop $ServiceName confirm | Out-Null
|
||||
& $NssmPath remove $ServiceName confirm | Out-Null
|
||||
}
|
||||
|
||||
& $NssmPath install $ServiceName $hostExe | Out-Null
|
||||
& $NssmPath set $ServiceName DisplayName 'OT-OPC-UA FOCAS Host (Tier-C isolated Fwlib32)' | Out-Null
|
||||
& $NssmPath set $ServiceName Description 'Out-of-process Fwlib32.dll host for OtOpcUa FOCAS driver. Crash-isolated from the main OPC UA server.' | Out-Null
|
||||
& $NssmPath set $ServiceName ObjectName $ServiceAccount | Out-Null
|
||||
& $NssmPath set $ServiceName Start SERVICE_AUTO_START | Out-Null
|
||||
& $NssmPath set $ServiceName AppStdout (Join-Path $env:ProgramData 'OtOpcUa\focas-host-stdout.log') | Out-Null
|
||||
& $NssmPath set $ServiceName AppStderr (Join-Path $env:ProgramData 'OtOpcUa\focas-host-stderr.log') | Out-Null
|
||||
& $NssmPath set $ServiceName AppRotateFiles 1 | Out-Null
|
||||
& $NssmPath set $ServiceName AppRotateBytes 10485760 | Out-Null
|
||||
|
||||
& $NssmPath set $ServiceName AppEnvironmentExtra `
|
||||
"OTOPCUA_FOCAS_PIPE=$FocasPipeName" `
|
||||
"OTOPCUA_ALLOWED_SID=$allowedSid" `
|
||||
"OTOPCUA_FOCAS_SECRET=$FocasSharedSecret" `
|
||||
"OTOPCUA_FOCAS_BACKEND=$FocasBackend" | Out-Null
|
||||
|
||||
& $NssmPath set $ServiceName DependOnService OtOpcUa | Out-Null
|
||||
|
||||
Write-Host "Installed '$ServiceName' under '$ServiceAccount' (SID=$allowedSid)."
|
||||
Write-Host "Pipe: \\.\pipe\$FocasPipeName Backend: $FocasBackend"
|
||||
Write-Host "Start the service with: Start-Service $ServiceName"
|
||||
Write-Host ""
|
||||
Write-Host "NOTE: the Fwlib32 backend requires the licensed Fwlib32.dll on PATH"
|
||||
Write-Host "alongside the Host exe. See docs/v2/focas-deployment.md."
|
||||
27
scripts/install/Uninstall-FocasHost.ps1
Normal file
27
scripts/install/Uninstall-FocasHost.ps1
Normal file
@@ -0,0 +1,27 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Removes the OtOpcUaFocasHost Windows service.
|
||||
|
||||
.DESCRIPTION
|
||||
Companion to Install-FocasHost.ps1. Stops + unregisters the service via NSSM.
|
||||
Idempotent — succeeds silently if the service doesn't exist.
|
||||
|
||||
.EXAMPLE
|
||||
.\Uninstall-FocasHost.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ServiceName = 'OtOpcUaFocasHost',
|
||||
[string]$NssmPath = 'C:\Program Files\nssm\nssm.exe'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$svc = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
|
||||
if (-not $svc) { Write-Host "Service '$ServiceName' not present — nothing to do."; return }
|
||||
|
||||
if (-not (Test-Path $NssmPath)) { throw "nssm.exe not found at '$NssmPath'." }
|
||||
|
||||
& $NssmPath stop $ServiceName confirm | Out-Null
|
||||
& $NssmPath remove $ServiceName confirm | Out-Null
|
||||
Write-Host "Removed '$ServiceName'."
|
||||
@@ -0,0 +1,133 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Stability;
|
||||
|
||||
/// <summary>
|
||||
/// Ring-buffer of the last N IPC operations, written into a memory-mapped file. On a
|
||||
/// hard crash the Proxy-side supervisor reads the MMF after the corpse is gone to see
|
||||
/// what was in flight at the moment the Host died. Single-writer (the Host), multi-reader
|
||||
/// (the supervisor) — the file format is identical to the Galaxy Tier-C
|
||||
/// <c>PostMortemMmf</c> so a single reader tool can work both.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// File layout:
|
||||
/// <code>
|
||||
/// [16-byte header: magic(4) | version(4) | capacity(4) | writeIndex(4)]
|
||||
/// [capacity × 256-byte entries: each is [8-byte utcUnixMs | 8-byte opKind | 240-byte UTF-8 message]]
|
||||
/// </code>
|
||||
/// Magic is 'OFPC' (0x4F46_5043) to distinguish a FOCAS file from the Galaxy MMF.
|
||||
/// </remarks>
|
||||
public sealed class PostMortemMmf : IDisposable
|
||||
{
|
||||
private const int Magic = 0x4F465043; // 'OFPC'
|
||||
private const int Version = 1;
|
||||
private const int HeaderBytes = 16;
|
||||
public const int EntryBytes = 256;
|
||||
private const int MessageOffset = 16;
|
||||
private const int MessageCapacity = EntryBytes - MessageOffset;
|
||||
|
||||
public int Capacity { get; }
|
||||
public string Path { get; }
|
||||
|
||||
private readonly MemoryMappedFile _mmf;
|
||||
private readonly MemoryMappedViewAccessor _accessor;
|
||||
private readonly object _writeGate = new();
|
||||
|
||||
public PostMortemMmf(string path, int capacity = 1000)
|
||||
{
|
||||
if (capacity <= 0) throw new ArgumentOutOfRangeException(nameof(capacity));
|
||||
Capacity = capacity;
|
||||
Path = path;
|
||||
|
||||
var fileBytes = HeaderBytes + capacity * EntryBytes;
|
||||
Directory.CreateDirectory(System.IO.Path.GetDirectoryName(path)!);
|
||||
|
||||
var fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read);
|
||||
fs.SetLength(fileBytes);
|
||||
_mmf = MemoryMappedFile.CreateFromFile(fs, null, fileBytes,
|
||||
MemoryMappedFileAccess.ReadWrite, HandleInheritability.None, leaveOpen: false);
|
||||
_accessor = _mmf.CreateViewAccessor(0, fileBytes, MemoryMappedFileAccess.ReadWrite);
|
||||
|
||||
if (_accessor.ReadInt32(0) != Magic)
|
||||
{
|
||||
_accessor.Write(0, Magic);
|
||||
_accessor.Write(4, Version);
|
||||
_accessor.Write(8, capacity);
|
||||
_accessor.Write(12, 0);
|
||||
}
|
||||
}
|
||||
|
||||
public void Write(long opKind, string message)
|
||||
{
|
||||
lock (_writeGate)
|
||||
{
|
||||
var idx = _accessor.ReadInt32(12);
|
||||
var offset = HeaderBytes + idx * EntryBytes;
|
||||
|
||||
_accessor.Write(offset + 0, DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
|
||||
_accessor.Write(offset + 8, opKind);
|
||||
|
||||
var msgBytes = Encoding.UTF8.GetBytes(message ?? string.Empty);
|
||||
var copy = Math.Min(msgBytes.Length, MessageCapacity - 1);
|
||||
_accessor.WriteArray(offset + MessageOffset, msgBytes, 0, copy);
|
||||
_accessor.Write(offset + MessageOffset + copy, (byte)0);
|
||||
|
||||
var next = (idx + 1) % Capacity;
|
||||
_accessor.Write(12, next);
|
||||
}
|
||||
}
|
||||
|
||||
public PostMortemEntry[] ReadAll()
|
||||
{
|
||||
var magic = _accessor.ReadInt32(0);
|
||||
if (magic != Magic) return new PostMortemEntry[0];
|
||||
|
||||
var capacity = _accessor.ReadInt32(8);
|
||||
var writeIndex = _accessor.ReadInt32(12);
|
||||
|
||||
var entries = new PostMortemEntry[capacity];
|
||||
var count = 0;
|
||||
for (var i = 0; i < capacity; i++)
|
||||
{
|
||||
var slot = (writeIndex + i) % capacity;
|
||||
var offset = HeaderBytes + slot * EntryBytes;
|
||||
|
||||
var ts = _accessor.ReadInt64(offset + 0);
|
||||
if (ts == 0) continue;
|
||||
|
||||
var op = _accessor.ReadInt64(offset + 8);
|
||||
var msgBuf = new byte[MessageCapacity];
|
||||
_accessor.ReadArray(offset + MessageOffset, msgBuf, 0, MessageCapacity);
|
||||
var nulTerm = Array.IndexOf<byte>(msgBuf, 0);
|
||||
var msg = Encoding.UTF8.GetString(msgBuf, 0, nulTerm < 0 ? MessageCapacity : nulTerm);
|
||||
|
||||
entries[count++] = new PostMortemEntry(ts, op, msg);
|
||||
}
|
||||
|
||||
Array.Resize(ref entries, count);
|
||||
return entries;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_accessor.Dispose();
|
||||
_mmf.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
public readonly struct PostMortemEntry
|
||||
{
|
||||
public long UtcUnixMs { get; }
|
||||
public long OpKind { get; }
|
||||
public string Message { get; }
|
||||
|
||||
public PostMortemEntry(long utcUnixMs, long opKind, string message)
|
||||
{
|
||||
UtcUnixMs = utcUnixMs;
|
||||
OpKind = opKind;
|
||||
Message = message;
|
||||
}
|
||||
}
|
||||
30
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Supervisor/Backoff.cs
Normal file
30
src/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Supervisor/Backoff.cs
Normal file
@@ -0,0 +1,30 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Respawn-with-backoff schedule for the FOCAS Host process. Matches Galaxy Tier-C:
|
||||
/// 5s → 15s → 60s cap. A sustained stable run (default 2 min) resets the index so a
|
||||
/// one-off crash after hours of steady-state doesn't start from the top of the ladder.
|
||||
/// </summary>
|
||||
public sealed class Backoff
|
||||
{
|
||||
public static TimeSpan[] DefaultSequence { get; } =
|
||||
[TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(60)];
|
||||
|
||||
public TimeSpan StableRunThreshold { get; init; } = TimeSpan.FromMinutes(2);
|
||||
|
||||
private readonly TimeSpan[] _sequence;
|
||||
private int _index;
|
||||
|
||||
public Backoff(TimeSpan[]? sequence = null) => _sequence = sequence ?? DefaultSequence;
|
||||
|
||||
public TimeSpan Next()
|
||||
{
|
||||
var delay = _sequence[Math.Min(_index, _sequence.Length - 1)];
|
||||
_index++;
|
||||
return delay;
|
||||
}
|
||||
|
||||
public void RecordStableRun() => _index = 0;
|
||||
|
||||
public int AttemptIndex => _index;
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Crash-loop circuit breaker for the FOCAS Host. Matches Galaxy Tier-C defaults:
|
||||
/// 3 crashes within 5 minutes opens the breaker; cooldown escalates 1h → 4h → manual
|
||||
/// reset. A sticky alert stays live until the operator explicitly clears it so
|
||||
/// recurring crashes can't silently burn through the cooldown ladder overnight.
|
||||
/// </summary>
|
||||
public sealed class CircuitBreaker
|
||||
{
|
||||
public int CrashesAllowedPerWindow { get; init; } = 3;
|
||||
public TimeSpan Window { get; init; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
public TimeSpan[] CooldownEscalation { get; init; } =
|
||||
[TimeSpan.FromHours(1), TimeSpan.FromHours(4), TimeSpan.MaxValue];
|
||||
|
||||
private readonly List<DateTime> _crashesUtc = [];
|
||||
private DateTime? _openSinceUtc;
|
||||
private int _escalationLevel;
|
||||
public bool StickyAlertActive { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Records a crash + returns <c>true</c> if the supervisor may respawn. On
|
||||
/// <c>false</c>, <paramref name="cooldownRemaining"/> is how long to wait before
|
||||
/// trying again (<c>TimeSpan.MaxValue</c> means manual reset required).
|
||||
/// </summary>
|
||||
public bool TryRecordCrash(DateTime utcNow, out TimeSpan cooldownRemaining)
|
||||
{
|
||||
if (_openSinceUtc is { } openedAt)
|
||||
{
|
||||
var cooldown = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
|
||||
if (cooldown == TimeSpan.MaxValue)
|
||||
{
|
||||
cooldownRemaining = TimeSpan.MaxValue;
|
||||
return false;
|
||||
}
|
||||
if (utcNow - openedAt < cooldown)
|
||||
{
|
||||
cooldownRemaining = cooldown - (utcNow - openedAt);
|
||||
return false;
|
||||
}
|
||||
|
||||
_openSinceUtc = null;
|
||||
_escalationLevel++;
|
||||
}
|
||||
|
||||
_crashesUtc.RemoveAll(t => utcNow - t > Window);
|
||||
_crashesUtc.Add(utcNow);
|
||||
|
||||
if (_crashesUtc.Count > CrashesAllowedPerWindow)
|
||||
{
|
||||
_openSinceUtc = utcNow;
|
||||
StickyAlertActive = true;
|
||||
cooldownRemaining = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
|
||||
return false;
|
||||
}
|
||||
|
||||
cooldownRemaining = TimeSpan.Zero;
|
||||
return true;
|
||||
}
|
||||
|
||||
public void ManualReset()
|
||||
{
|
||||
_crashesUtc.Clear();
|
||||
_openSinceUtc = null;
|
||||
_escalationLevel = 0;
|
||||
StickyAlertActive = false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Ties <see cref="IHostProcessLauncher"/> + <see cref="Backoff"/> +
|
||||
/// <see cref="CircuitBreaker"/> + <see cref="HeartbeatMonitor"/> into one object the
|
||||
/// driver asks for <c>IFocasClient</c>s. On a detected crash (process exit or
|
||||
/// heartbeat loss) the supervisor fans out <c>BadCommunicationError</c> to all
|
||||
/// subscribers via the <see cref="OnUnavailable"/> callback, then respawns with
|
||||
/// backoff unless the breaker is open.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The supervisor itself is I/O-free — it doesn't know how to spawn processes, probe
|
||||
/// pipes, or send heartbeats. Production wires the concrete
|
||||
/// <see cref="IHostProcessLauncher"/> over <c>FocasIpcClient</c> + <c>Process</c>;
|
||||
/// tests drive the same state machine with a deterministic launcher stub.
|
||||
/// </remarks>
|
||||
public sealed class FocasHostSupervisor : IDisposable
|
||||
{
|
||||
private readonly IHostProcessLauncher _launcher;
|
||||
private readonly Backoff _backoff;
|
||||
private readonly CircuitBreaker _breaker;
|
||||
private readonly Func<DateTime> _clock;
|
||||
private IFocasClient? _current;
|
||||
private DateTime _currentStartedUtc;
|
||||
private bool _disposed;
|
||||
|
||||
public FocasHostSupervisor(
|
||||
IHostProcessLauncher launcher,
|
||||
Backoff? backoff = null,
|
||||
CircuitBreaker? breaker = null,
|
||||
Func<DateTime>? clock = null)
|
||||
{
|
||||
_launcher = launcher ?? throw new ArgumentNullException(nameof(launcher));
|
||||
_backoff = backoff ?? new Backoff();
|
||||
_breaker = breaker ?? new CircuitBreaker();
|
||||
_clock = clock ?? (() => DateTime.UtcNow);
|
||||
}
|
||||
|
||||
/// <summary>Raised with a short reason string whenever the Host goes unavailable (crash / heartbeat loss / breaker-open).</summary>
|
||||
public event Action<string>? OnUnavailable;
|
||||
|
||||
/// <summary>Crash count observed in the current process lifetime. Exposed for /hosts Admin telemetry.</summary>
|
||||
public int ObservedCrashes { get; private set; }
|
||||
|
||||
/// <summary><c>true</c> if the crash-loop breaker has latched a sticky alert that needs operator reset.</summary>
|
||||
public bool StickyAlertActive => _breaker.StickyAlertActive;
|
||||
|
||||
public int BackoffAttempt => _backoff.AttemptIndex;
|
||||
|
||||
/// <summary>
|
||||
/// Returns the current live client. If none, tries to launch — applying the
|
||||
/// backoff schedule between attempts and stopping once the breaker opens.
|
||||
/// </summary>
|
||||
public async Task<IFocasClient> GetOrLaunchAsync(CancellationToken ct)
|
||||
{
|
||||
ThrowIfDisposed();
|
||||
if (_current is not null && _launcher.IsProcessAlive) return _current;
|
||||
|
||||
return await LaunchWithBackoffAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Called by the heartbeat task each time a miss threshold is crossed.
|
||||
/// Treated as a crash: fan out Bad status + attempt respawn.
|
||||
/// </summary>
|
||||
public async Task NotifyHostDeadAsync(string reason, CancellationToken ct)
|
||||
{
|
||||
ThrowIfDisposed();
|
||||
OnUnavailable?.Invoke(reason);
|
||||
ObservedCrashes++;
|
||||
try { await _launcher.TerminateAsync(ct).ConfigureAwait(false); }
|
||||
catch { /* best effort */ }
|
||||
_current?.Dispose();
|
||||
_current = null;
|
||||
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown))
|
||||
{
|
||||
OnUnavailable?.Invoke(cooldown == TimeSpan.MaxValue
|
||||
? "circuit-breaker-open-manual-reset-required"
|
||||
: $"circuit-breaker-open-cooldown-{cooldown:g}");
|
||||
return;
|
||||
}
|
||||
// Successful crash recording — do not respawn synchronously; GetOrLaunchAsync will
|
||||
// pick up the attempt on the next call. Keeps the fan-out fast.
|
||||
}
|
||||
|
||||
/// <summary>Operator action — clear the sticky alert + reset the breaker.</summary>
|
||||
public void AcknowledgeAndReset()
|
||||
{
|
||||
_breaker.ManualReset();
|
||||
_backoff.RecordStableRun();
|
||||
}
|
||||
|
||||
private async Task<IFocasClient> LaunchWithBackoffAsync(CancellationToken ct)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
if (_breaker.StickyAlertActive)
|
||||
{
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown) && cooldown == TimeSpan.MaxValue)
|
||||
throw new InvalidOperationException(
|
||||
"FOCAS Host circuit breaker is open and awaiting manual reset. " +
|
||||
"See Admin /hosts; call AcknowledgeAndReset after investigating the Host log.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
_current = await _launcher.LaunchAsync(ct).ConfigureAwait(false);
|
||||
_currentStartedUtc = _clock();
|
||||
|
||||
// If the launch sequence itself takes long enough to count as a stable run,
|
||||
// reset the backoff ladder immediately.
|
||||
if (_clock() - _currentStartedUtc >= _backoff.StableRunThreshold)
|
||||
_backoff.RecordStableRun();
|
||||
|
||||
return _current;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
OnUnavailable?.Invoke($"launch-failed: {ex.Message}");
|
||||
ObservedCrashes++;
|
||||
if (!_breaker.TryRecordCrash(_clock(), out var cooldown))
|
||||
{
|
||||
var hint = cooldown == TimeSpan.MaxValue
|
||||
? "manual reset required"
|
||||
: $"cooldown {cooldown:g}";
|
||||
throw new InvalidOperationException(
|
||||
$"FOCAS Host circuit breaker opened after {ObservedCrashes} crashes — {hint}.", ex);
|
||||
}
|
||||
|
||||
var delay = _backoff.Next();
|
||||
await Task.Delay(delay, ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Called from the heartbeat loop after a successful ack run — relaxes the backoff ladder.</summary>
|
||||
public void NotifyStableRun()
|
||||
{
|
||||
if (_current is null) return;
|
||||
if (_clock() - _currentStartedUtc >= _backoff.StableRunThreshold)
|
||||
_backoff.RecordStableRun();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
try { _launcher.TerminateAsync(CancellationToken.None).GetAwaiter().GetResult(); }
|
||||
catch { /* best effort */ }
|
||||
_current?.Dispose();
|
||||
_current = null;
|
||||
}
|
||||
|
||||
private void ThrowIfDisposed()
|
||||
{
|
||||
if (_disposed) throw new ObjectDisposedException(nameof(FocasHostSupervisor));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Tracks missed heartbeats from the FOCAS Host. 2s cadence + 3 consecutive misses =
|
||||
/// host declared dead (~6s detection). Same defaults as Galaxy Tier-C so operators
|
||||
/// see the same cadence across hosts on the /hosts Admin page.
|
||||
/// </summary>
|
||||
public sealed class HeartbeatMonitor
|
||||
{
|
||||
public int MissesUntilDead { get; init; } = 3;
|
||||
|
||||
public TimeSpan Cadence { get; init; } = TimeSpan.FromSeconds(2);
|
||||
|
||||
public int ConsecutiveMisses { get; private set; }
|
||||
public DateTime? LastAckUtc { get; private set; }
|
||||
|
||||
public void RecordAck(DateTime utcNow)
|
||||
{
|
||||
ConsecutiveMisses = 0;
|
||||
LastAckUtc = utcNow;
|
||||
}
|
||||
|
||||
/// <summary>Records a missed heartbeat; returns <c>true</c> when the death threshold is crossed.</summary>
|
||||
public bool RecordMiss()
|
||||
{
|
||||
ConsecutiveMisses++;
|
||||
return ConsecutiveMisses >= MissesUntilDead;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Abstraction over the act of spawning a FOCAS Host process and obtaining an
|
||||
/// <see cref="IFocasClient"/> connected to it. Production wires this to a real
|
||||
/// <c>Process.Start</c> + <c>FocasIpcClient.ConnectAsync</c>; tests use a fake that
|
||||
/// exposes deterministic failure modes so the supervisor logic can be stressed
|
||||
/// without spawning actual exes.
|
||||
/// </summary>
|
||||
public interface IHostProcessLauncher
|
||||
{
|
||||
/// <summary>
|
||||
/// Spawn a new Host process (if one isn't already running) and return a live
|
||||
/// client session. Throws on unrecoverable errors; transient errors (e.g. Host
|
||||
/// not ready yet) should throw <see cref="TimeoutException"/> so the supervisor
|
||||
/// applies the backoff ladder.
|
||||
/// </summary>
|
||||
Task<IFocasClient> LaunchAsync(CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// Terminate the Host process if one is running. Called on Dispose and after a
|
||||
/// heartbeat loss is detected.
|
||||
/// </summary>
|
||||
Task TerminateAsync(CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// <c>true</c> when the most recently spawned Host process is still alive.
|
||||
/// Supervisor polls this at heartbeat cadence; going <c>false</c> without a
|
||||
/// clean shutdown counts as a crash.
|
||||
/// </summary>
|
||||
bool IsProcessAlive { get; }
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Proxy-side reader for the Host's post-mortem MMF. After a Host crash the supervisor
|
||||
/// opens the file (which persists beyond the process lifetime) and enumerates the last
|
||||
/// few thousand IPC operations that were in flight. Format matches
|
||||
/// <c>Driver.FOCAS.Host.Stability.PostMortemMmf</c> — magic 'OFPC' / 256-byte entries.
|
||||
/// </summary>
|
||||
public sealed class PostMortemReader
|
||||
{
|
||||
private const int Magic = 0x4F465043; // 'OFPC'
|
||||
private const int HeaderBytes = 16;
|
||||
private const int EntryBytes = 256;
|
||||
private const int MessageOffset = 16;
|
||||
private const int MessageCapacity = EntryBytes - MessageOffset;
|
||||
|
||||
public string Path { get; }
|
||||
|
||||
public PostMortemReader(string path) => Path = path;
|
||||
|
||||
public PostMortemEntry[] ReadAll()
|
||||
{
|
||||
if (!File.Exists(Path)) return [];
|
||||
|
||||
using var mmf = MemoryMappedFile.CreateFromFile(Path, FileMode.Open, null, 0, MemoryMappedFileAccess.Read);
|
||||
using var accessor = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read);
|
||||
|
||||
if (accessor.ReadInt32(0) != Magic) return [];
|
||||
|
||||
var capacity = accessor.ReadInt32(8);
|
||||
var writeIndex = accessor.ReadInt32(12);
|
||||
var entries = new PostMortemEntry[capacity];
|
||||
var count = 0;
|
||||
|
||||
for (var i = 0; i < capacity; i++)
|
||||
{
|
||||
var slot = (writeIndex + i) % capacity;
|
||||
var offset = HeaderBytes + slot * EntryBytes;
|
||||
var ts = accessor.ReadInt64(offset + 0);
|
||||
if (ts == 0) continue;
|
||||
var op = accessor.ReadInt64(offset + 8);
|
||||
var msgBuf = new byte[MessageCapacity];
|
||||
accessor.ReadArray(offset + MessageOffset, msgBuf, 0, MessageCapacity);
|
||||
var nulTerm = Array.IndexOf<byte>(msgBuf, 0);
|
||||
var msg = Encoding.UTF8.GetString(msgBuf, 0, nulTerm < 0 ? MessageCapacity : nulTerm);
|
||||
entries[count++] = new PostMortemEntry(ts, op, msg);
|
||||
}
|
||||
|
||||
Array.Resize(ref entries, count);
|
||||
return entries;
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct PostMortemEntry(long UtcUnixMs, long OpKind, string Message);
|
||||
@@ -0,0 +1,113 @@
|
||||
using System.Diagnostics;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Ipc;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
/// <summary>
|
||||
/// Production <see cref="IHostProcessLauncher"/>. Spawns <c>OtOpcUa.Driver.FOCAS.Host.exe</c>
|
||||
/// with the pipe name / allowed-SID / per-spawn shared secret in the environment, waits for
|
||||
/// the pipe to come up, then connects a <see cref="FocasIpcClient"/> and wraps it in an
|
||||
/// <see cref="IpcFocasClient"/>. On <see cref="TerminateAsync"/> best-effort kills the
|
||||
/// process and closes the IPC stream.
|
||||
/// </summary>
|
||||
public sealed class ProcessHostLauncher : IHostProcessLauncher
|
||||
{
|
||||
private readonly ProcessHostLauncherOptions _options;
|
||||
private Process? _process;
|
||||
private FocasIpcClient? _ipc;
|
||||
|
||||
public ProcessHostLauncher(ProcessHostLauncherOptions options)
|
||||
{
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
}
|
||||
|
||||
public bool IsProcessAlive => _process is { HasExited: false };
|
||||
|
||||
public async Task<IFocasClient> LaunchAsync(CancellationToken ct)
|
||||
{
|
||||
await TerminateAsync(ct).ConfigureAwait(false);
|
||||
|
||||
var secret = _options.SharedSecret ?? Guid.NewGuid().ToString("N");
|
||||
|
||||
var psi = new ProcessStartInfo
|
||||
{
|
||||
FileName = _options.HostExePath,
|
||||
Arguments = _options.Arguments ?? string.Empty,
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
};
|
||||
psi.Environment["OTOPCUA_FOCAS_PIPE"] = _options.PipeName;
|
||||
psi.Environment["OTOPCUA_ALLOWED_SID"] = _options.AllowedSid;
|
||||
psi.Environment["OTOPCUA_FOCAS_SECRET"] = secret;
|
||||
psi.Environment["OTOPCUA_FOCAS_BACKEND"] = _options.Backend;
|
||||
|
||||
_process = Process.Start(psi)
|
||||
?? throw new InvalidOperationException($"Failed to start {_options.HostExePath}");
|
||||
|
||||
// Poll for pipe readiness up to the configured connect timeout.
|
||||
var deadline = DateTime.UtcNow + _options.ConnectTimeout;
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
if (_process.HasExited)
|
||||
throw new InvalidOperationException(
|
||||
$"FOCAS Host exited before pipe was ready (ExitCode={_process.ExitCode}).");
|
||||
|
||||
try
|
||||
{
|
||||
_ipc = await FocasIpcClient.ConnectAsync(
|
||||
_options.PipeName, secret, TimeSpan.FromSeconds(1), ct).ConfigureAwait(false);
|
||||
break;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
if (DateTime.UtcNow >= deadline)
|
||||
throw new TimeoutException(
|
||||
$"FOCAS Host pipe {_options.PipeName} did not come up within {_options.ConnectTimeout:g}.");
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(250), ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
return new IpcFocasClient(_ipc, _options.Series);
|
||||
}
|
||||
|
||||
public async Task TerminateAsync(CancellationToken ct)
|
||||
{
|
||||
if (_ipc is not null)
|
||||
{
|
||||
try { await _ipc.DisposeAsync().ConfigureAwait(false); }
|
||||
catch { /* best effort */ }
|
||||
_ipc = null;
|
||||
}
|
||||
|
||||
if (_process is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!_process.HasExited)
|
||||
{
|
||||
_process.Kill(entireProcessTree: true);
|
||||
await _process.WaitForExitAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch { /* best effort */ }
|
||||
finally
|
||||
{
|
||||
_process.Dispose();
|
||||
_process = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record ProcessHostLauncherOptions(
|
||||
string HostExePath,
|
||||
string PipeName,
|
||||
string AllowedSid)
|
||||
{
|
||||
public string? SharedSecret { get; init; }
|
||||
public string? Arguments { get; init; }
|
||||
public string Backend { get; init; } = "fwlib32";
|
||||
public TimeSpan ConnectTimeout { get; init; } = TimeSpan.FromSeconds(15);
|
||||
public FocasCncSeries Series { get; init; } = FocasCncSeries.Unknown;
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Stability;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Host.Tests
|
||||
{
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class PostMortemMmfTests : IDisposable
|
||||
{
|
||||
private readonly string _tempPath;
|
||||
|
||||
public PostMortemMmfTests()
|
||||
{
|
||||
_tempPath = Path.Combine(Path.GetTempPath(), $"focas-mmf-{Guid.NewGuid():N}.bin");
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (File.Exists(_tempPath)) File.Delete(_tempPath);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Write_and_read_preserve_order_and_content()
|
||||
{
|
||||
using (var mmf = new PostMortemMmf(_tempPath, capacity: 10))
|
||||
{
|
||||
mmf.Write(opKind: 1, "read R100");
|
||||
mmf.Write(opKind: 2, "write MACRO:500 = 3.14");
|
||||
mmf.Write(opKind: 3, "probe ok");
|
||||
}
|
||||
|
||||
// Reopen (simulating a reader after the writer crashed).
|
||||
using var reader = new PostMortemMmf(_tempPath, capacity: 10);
|
||||
var entries = reader.ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
entries[0].OpKind.ShouldBe(1L);
|
||||
entries[0].Message.ShouldBe("read R100");
|
||||
entries[1].OpKind.ShouldBe(2L);
|
||||
entries[2].Message.ShouldBe("probe ok");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ring_buffer_wraps_at_capacity()
|
||||
{
|
||||
using var mmf = new PostMortemMmf(_tempPath, capacity: 3);
|
||||
for (var i = 0; i < 10; i++) mmf.Write(i, $"op-{i}");
|
||||
|
||||
var entries = mmf.ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
// Oldest surviving entry is op-7 (entries 7,8,9 survive in FIFO order).
|
||||
entries[0].Message.ShouldBe("op-7");
|
||||
entries[1].Message.ShouldBe("op-8");
|
||||
entries[2].Message.ShouldBe("op-9");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Truncated_message_is_null_terminated_and_does_not_overflow()
|
||||
{
|
||||
using var mmf = new PostMortemMmf(_tempPath, capacity: 4);
|
||||
var big = new string('x', 500); // longer than the 240-byte message capacity
|
||||
mmf.Write(42, big);
|
||||
|
||||
var entries = mmf.ReadAll();
|
||||
entries.Length.ShouldBe(1);
|
||||
entries[0].Message.Length.ShouldBeLessThanOrEqualTo(240);
|
||||
entries[0].OpKind.ShouldBe(42L);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reopening_with_existing_data_preserves_entries()
|
||||
{
|
||||
using (var first = new PostMortemMmf(_tempPath, capacity: 5))
|
||||
{
|
||||
first.Write(1, "first-run-1");
|
||||
first.Write(2, "first-run-2");
|
||||
}
|
||||
|
||||
using var second = new PostMortemMmf(_tempPath, capacity: 5);
|
||||
var entries = second.ReadAll();
|
||||
entries.Length.ShouldBe(2);
|
||||
entries[0].Message.ShouldBe("first-run-1");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
using System.IO.MemoryMappedFiles;
|
||||
using System.Text;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The Proxy-side <see cref="PostMortemReader"/> must read the Host's MMF format
|
||||
/// (magic 'OFPC', 256-byte entries). This test writes a hand-crafted file that mimics
|
||||
/// the Host's layout exactly + asserts the reader decodes it correctly. Keeps the two
|
||||
/// codebases in lockstep on the wire format without needing to reference the net48
|
||||
/// Host assembly from the net10 test project.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class PostMortemReaderCompatibilityTests : IDisposable
|
||||
{
|
||||
private readonly string _tempPath = Path.Combine(Path.GetTempPath(), $"focas-mmf-compat-{Guid.NewGuid():N}.bin");
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (File.Exists(_tempPath)) File.Delete(_tempPath);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_parses_host_format_and_returns_entries_in_oldest_first_order()
|
||||
{
|
||||
const int magic = 0x4F465043;
|
||||
const int capacity = 5;
|
||||
const int headerBytes = 16;
|
||||
const int entryBytes = 256;
|
||||
const int messageOffset = 16;
|
||||
var fileBytes = headerBytes + capacity * entryBytes;
|
||||
|
||||
using (var fs = new FileStream(_tempPath, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.Read))
|
||||
{
|
||||
fs.SetLength(fileBytes);
|
||||
using var mmf = MemoryMappedFile.CreateFromFile(fs, null, fileBytes,
|
||||
MemoryMappedFileAccess.ReadWrite, HandleInheritability.None, leaveOpen: false);
|
||||
using var acc = mmf.CreateViewAccessor(0, fileBytes, MemoryMappedFileAccess.ReadWrite);
|
||||
acc.Write(0, magic);
|
||||
acc.Write(4, 1);
|
||||
acc.Write(8, capacity);
|
||||
acc.Write(12, 2); // writeIndex — next write would land at slot 2
|
||||
|
||||
void WriteEntry(int slot, long ts, long op, string msg)
|
||||
{
|
||||
var offset = headerBytes + slot * entryBytes;
|
||||
acc.Write(offset + 0, ts);
|
||||
acc.Write(offset + 8, op);
|
||||
var bytes = Encoding.UTF8.GetBytes(msg);
|
||||
acc.WriteArray(offset + messageOffset, bytes, 0, bytes.Length);
|
||||
acc.Write(offset + messageOffset + bytes.Length, (byte)0);
|
||||
}
|
||||
|
||||
WriteEntry(0, 100, 1, "op-a");
|
||||
WriteEntry(1, 200, 2, "op-b");
|
||||
// Slots 2,3 unwritten (ts=0) — reader must skip.
|
||||
WriteEntry(4, 50, 9, "old-wrapped");
|
||||
}
|
||||
|
||||
var entries = new PostMortemReader(_tempPath).ReadAll();
|
||||
entries.Length.ShouldBe(3);
|
||||
// writeIndex=2 means the ring walk starts at slot 2, so iteration order is 2→3→4→0→1.
|
||||
// Slots 2 and 3 are empty; 4 yields "old-wrapped"; then 0="op-a", 1="op-b".
|
||||
entries[0].Message.ShouldBe("old-wrapped");
|
||||
entries[1].Message.ShouldBe("op-a");
|
||||
entries[2].Message.ShouldBe("op-b");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_returns_empty_when_file_missing()
|
||||
{
|
||||
new PostMortemReader(_tempPath + "-does-not-exist").ReadAll().ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reader_returns_empty_when_magic_mismatches()
|
||||
{
|
||||
File.WriteAllBytes(_tempPath, new byte[1024]);
|
||||
new PostMortemReader(_tempPath).ReadAll().ShouldBeEmpty();
|
||||
}
|
||||
}
|
||||
249
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/SupervisorTests.cs
Normal file
249
tests/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/SupervisorTests.cs
Normal file
@@ -0,0 +1,249 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Supervisor;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class BackoffTests
|
||||
{
|
||||
[Fact]
|
||||
public void Default_sequence_is_5s_15s_60s_then_clamped()
|
||||
{
|
||||
var b = new Backoff();
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(5));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(15));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(60));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RecordStableRun_resets_the_ladder_to_the_start()
|
||||
{
|
||||
var b = new Backoff();
|
||||
b.Next(); b.Next();
|
||||
b.AttemptIndex.ShouldBe(2);
|
||||
b.RecordStableRun();
|
||||
b.AttemptIndex.ShouldBe(0);
|
||||
b.Next().ShouldBe(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class CircuitBreakerTests
|
||||
{
|
||||
[Fact]
|
||||
public void Allows_crashes_below_threshold()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
b.TryRecordCrash(now, out _).ShouldBeTrue();
|
||||
b.TryRecordCrash(now.AddSeconds(1), out _).ShouldBeTrue();
|
||||
b.TryRecordCrash(now.AddSeconds(2), out _).ShouldBeTrue();
|
||||
b.StickyAlertActive.ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Opens_when_exceeding_threshold_in_window()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
b.TryRecordCrash(now, out _);
|
||||
b.TryRecordCrash(now.AddSeconds(1), out _);
|
||||
b.TryRecordCrash(now.AddSeconds(2), out _);
|
||||
b.TryRecordCrash(now.AddSeconds(3), out var cooldown).ShouldBeFalse();
|
||||
cooldown.ShouldBe(TimeSpan.FromHours(1));
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Escalates_cooldown_after_second_open()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var t0 = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
// First burst — 4 crashes opens breaker with 1h cooldown.
|
||||
for (var i = 0; i < 4; i++) b.TryRecordCrash(t0.AddSeconds(i), out _);
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
|
||||
// Wait past cooldown. The first crash after cooldown-elapsed resets _openSinceUtc and
|
||||
// bumps escalation level; the next 3 crashes then re-open with the escalated 4h cooldown.
|
||||
b.TryRecordCrash(t0.AddHours(1).AddMinutes(1), out _);
|
||||
var t1 = t0.AddHours(1).AddMinutes(1).AddSeconds(1);
|
||||
b.TryRecordCrash(t1, out _);
|
||||
b.TryRecordCrash(t1.AddSeconds(1), out _);
|
||||
b.TryRecordCrash(t1.AddSeconds(2), out var cooldown).ShouldBeFalse();
|
||||
cooldown.ShouldBe(TimeSpan.FromHours(4));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManualReset_clears_everything()
|
||||
{
|
||||
var b = new CircuitBreaker();
|
||||
var now = DateTime.UtcNow;
|
||||
for (var i = 0; i < 5; i++) b.TryRecordCrash(now.AddSeconds(i), out _);
|
||||
b.StickyAlertActive.ShouldBeTrue();
|
||||
b.ManualReset();
|
||||
b.StickyAlertActive.ShouldBeFalse();
|
||||
b.TryRecordCrash(now.AddSeconds(10), out _).ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class HeartbeatMonitorTests
|
||||
{
|
||||
[Fact]
|
||||
public void Three_consecutive_misses_declares_dead()
|
||||
{
|
||||
var m = new HeartbeatMonitor();
|
||||
m.RecordMiss().ShouldBeFalse();
|
||||
m.RecordMiss().ShouldBeFalse();
|
||||
m.RecordMiss().ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ack_resets_the_miss_counter()
|
||||
{
|
||||
var m = new HeartbeatMonitor();
|
||||
m.RecordMiss(); m.RecordMiss();
|
||||
m.ConsecutiveMisses.ShouldBe(2);
|
||||
m.RecordAck(DateTime.UtcNow);
|
||||
m.ConsecutiveMisses.ShouldBe(0);
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class FocasHostSupervisorTests
|
||||
{
|
||||
private sealed class FakeLauncher : IHostProcessLauncher
|
||||
{
|
||||
public int LaunchAttempts { get; private set; }
|
||||
public int Terminations { get; private set; }
|
||||
public Queue<Func<IFocasClient>> Plan { get; } = new();
|
||||
public bool IsProcessAlive { get; set; }
|
||||
|
||||
public Task<IFocasClient> LaunchAsync(CancellationToken ct)
|
||||
{
|
||||
LaunchAttempts++;
|
||||
if (Plan.Count == 0) throw new InvalidOperationException("FakeLauncher plan exhausted");
|
||||
var next = Plan.Dequeue()();
|
||||
IsProcessAlive = true;
|
||||
return Task.FromResult(next);
|
||||
}
|
||||
|
||||
public Task TerminateAsync(CancellationToken ct)
|
||||
{
|
||||
Terminations++;
|
||||
IsProcessAlive = false;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class StubFocasClient : IFocasClient
|
||||
{
|
||||
public bool IsConnected => true;
|
||||
public Task ConnectAsync(FocasHostAddress address, TimeSpan timeout, CancellationToken ct) => Task.CompletedTask;
|
||||
public Task<(object? value, uint status)> ReadAsync(FocasAddress a, FocasDataType t, CancellationToken ct) =>
|
||||
Task.FromResult<(object?, uint)>((0, 0));
|
||||
public Task<uint> WriteAsync(FocasAddress a, FocasDataType t, object? v, CancellationToken ct) => Task.FromResult(0u);
|
||||
public Task<bool> ProbeAsync(CancellationToken ct) => Task.FromResult(true);
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetOrLaunch_returns_client_on_first_success()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
var client = await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
client.ShouldNotBeNull();
|
||||
launcher.LaunchAttempts.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetOrLaunch_retries_after_transient_failure_with_backoff()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => throw new TimeoutException("pipe not ready"));
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
|
||||
var backoff = new Backoff([TimeSpan.FromMilliseconds(10), TimeSpan.FromMilliseconds(20)]);
|
||||
var supervisor = new FocasHostSupervisor(launcher, backoff);
|
||||
|
||||
var unavailableMessages = new List<string>();
|
||||
supervisor.OnUnavailable += m => unavailableMessages.Add(m);
|
||||
|
||||
var client = await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
client.ShouldNotBeNull();
|
||||
launcher.LaunchAttempts.ShouldBe(2);
|
||||
unavailableMessages.Count.ShouldBe(1);
|
||||
unavailableMessages[0].ShouldContain("launch-failed");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Repeated_launch_failures_open_breaker_and_surface_InvalidOperation()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
for (var i = 0; i < 10; i++)
|
||||
launcher.Plan.Enqueue(() => throw new InvalidOperationException("simulated host refused"));
|
||||
|
||||
var supervisor = new FocasHostSupervisor(
|
||||
launcher,
|
||||
backoff: new Backoff([TimeSpan.FromMilliseconds(1)]),
|
||||
breaker: new CircuitBreaker { CrashesAllowedPerWindow = 2, Window = TimeSpan.FromMinutes(5) });
|
||||
|
||||
var ex = await Should.ThrowAsync<InvalidOperationException>(async () =>
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken));
|
||||
ex.Message.ShouldContain("circuit breaker");
|
||||
supervisor.StickyAlertActive.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NotifyHostDeadAsync_terminates_current_and_fans_out_unavailable()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
|
||||
var messages = new List<string>();
|
||||
supervisor.OnUnavailable += m => messages.Add(m);
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
await supervisor.NotifyHostDeadAsync("heartbeat-loss", TestContext.Current.CancellationToken);
|
||||
|
||||
launcher.Terminations.ShouldBe(1);
|
||||
messages.ShouldContain("heartbeat-loss");
|
||||
supervisor.ObservedCrashes.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AcknowledgeAndReset_clears_sticky_alert()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
for (var i = 0; i < 10; i++)
|
||||
launcher.Plan.Enqueue(() => throw new InvalidOperationException("refused"));
|
||||
var supervisor = new FocasHostSupervisor(
|
||||
launcher,
|
||||
backoff: new Backoff([TimeSpan.FromMilliseconds(1)]),
|
||||
breaker: new CircuitBreaker { CrashesAllowedPerWindow = 1 });
|
||||
|
||||
try { await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken); } catch { }
|
||||
supervisor.StickyAlertActive.ShouldBeTrue();
|
||||
|
||||
supervisor.AcknowledgeAndReset();
|
||||
supervisor.StickyAlertActive.ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Dispose_terminates_host_process()
|
||||
{
|
||||
var launcher = new FakeLauncher();
|
||||
launcher.Plan.Enqueue(() => new StubFocasClient());
|
||||
var supervisor = new FocasHostSupervisor(launcher);
|
||||
await supervisor.GetOrLaunchAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
supervisor.Dispose();
|
||||
launcher.Terminations.ShouldBe(1);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user