Phase 1 Streams B–E scaffold + Phase 2 Streams A–C scaffold — 8 new projects with ~70 new tests, all green alongside the 494 v1 IntegrationTests baseline (parity preserved: no v1 tests broken; legacy OtOpcUa.Host untouched). Phase 1 finish: Configuration project (16 entities + 10 enums + DbContext + DesignTimeDbContextFactory + InitialSchema/StoredProcedures/AuthorizationGrants migrations — 8 procs including sp_PublishGeneration with MERGE on ExternalIdReservation per decision #124, sp_RollbackToGeneration cloning rows into a new published generation, sp_ValidateDraft with cross-cluster-namespace + EquipmentUuid-immutability + ZTag/SAPID reservation pre-flight, sp_ComputeGenerationDiff with CHECKSUM-based row signature — plus OtOpcUaNode/OtOpcUaAdmin SQL roles with EXECUTE grants scoped to per-principal-class proc sets and DENY UPDATE/DELETE/INSERT/SELECT on dbo schema); managed DraftValidator covering UNS segment regex, path length, EquipmentUuid immutability across generations, same-cluster namespace binding (decision #122), reservation pre-flight, EquipmentId derivation (decision #125), driver↔namespace compatibility — returning every failing rule in one pass; LiteDB local cache with round-trip + ring pruning + corruption-fast-fail; GenerationApplier with per-entity Added/Removed/Modified diff and dependency-ordered callbacks (namespace → driver → device → equipment → poll-group → tag, Removed before Added); Core project with GenericDriverNodeManager (scaffold for the Phase 2 Galaxy port) and DriverHost lifecycle registry; Server project using Microsoft.Extensions.Hosting BackgroundService replacing TopShelf, with NodeBootstrap that falls back to LiteDB cache when the central DB is unreachable (decision #79); Admin project scaffolded as Blazor Server with Bootstrap 5 sidebar layout, cookie auth, three admin roles (ConfigViewer/ConfigEditor/FleetAdmin), Cluster + Generation services fronting the stored procs. Phase 2 scaffold: Driver.Galaxy.Shared (netstandard2.0) with full MessagePack IPC contract surface — Hello version negotiation, Open/CloseSession, Heartbeat, DiscoverHierarchy + GalaxyObjectInfo/GalaxyAttributeInfo, Read/WriteValues, Subscribe/Unsubscribe/OnDataChange, AlarmSubscribe/Event/Ack, HistoryRead, HostConnectivityStatus, Recycle — plus length-prefixed framing (decision #28) with a 16 MiB cap and thread-safe FrameWriter/FrameReader; Driver.Galaxy.Host (net48) implementing the Tier C cross-cutting protections from driver-stability.md — strict PipeAcl (allow configured server SID only, explicit deny on LocalSystem + Administrators), PipeServer with caller-SID verification via pipe.RunAsClient + WindowsIdentity.GetCurrent and per-process shared-secret Hello, Galaxy-specific MemoryWatchdog (warn at max(1.5×baseline, +200 MB), soft-recycle at max(2×baseline, +200 MB), hard ceiling 1.5 GB, slope ≥5 MB/min over 30-min rolling window), RecyclePolicy (1 soft recycle per hour cap + 03:00 local daily scheduled), PostMortemMmf (1000-entry ring buffer in %ProgramData%\OtOpcUa\driver-postmortem\galaxy.mmf, survives hard crash, readable cross-process), MxAccessHandle : SafeHandle (ReleaseHandle loops Marshal.ReleaseComObject until refcount=0 then calls optional unregister callback), StaPump with responsiveness probe (BlockingCollection dispatcher for Phase 1 — real Win32 GetMessage/DispatchMessage pump slots in with the same semantics when the Galaxy code lift happens), IsExternalInit shim for init setters on .NET 4.8; Driver.Galaxy.Proxy (net10) implementing IDriver + ITagDiscovery forwarding over the IPC channel with MX data-type and security-classification mapping, plus Supervisor pieces — Backoff (5s → 15s → 60s capped, reset-on-stable-run), CircuitBreaker (3 crashes per 5 min opens; 1h → 4h → manual cooldown escalation; sticky alert doesn't auto-clear), HeartbeatMonitor (2s cadence, 3 consecutive misses = host dead per driver-stability.md). Infrastructure: docker SQL Server remapped to host port 14330 to coexist with the native MSSQL14 Galaxy ZB DB instance on 1433; NuGetAuditSuppress applied per-project for two System.Security.Cryptography.Xml advisories that only reach via EF Core Design with PrivateAssets=all (fix ships in 11.0.0-preview); .slnx gains 14 project registrations. Deferred with explicit TODOs in docs/v2/implementation/phase-2-partial-exit-evidence.md: Phase 1 Stream E Admin UI pages (Generations listing + draft-diff-publish, Equipment CRUD with OPC 40010 fields, UNS Areas/Lines tabs, ACLs + permission simulator, Generic JSON config editor, SignalR real-time, Release-Reservation + Merge-Equipment workflows, LDAP login page, AppServer smoke test per decision #142), Phase 2 Stream D (Galaxy MXAccess code lift out of legacy OtOpcUa.Host, dual-service installer, appsettings → DriverConfig migration script, legacy Host deletion — blocked by parity), Phase 2 Stream E (v1 IntegrationTests against v2 topology, Client.CLI walkthrough diff, four 2026-04-13 stability findings regression tests, adversarial review — requires live MXAccess runtime).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 21:35:25 -04:00
parent fc0ce36308
commit 01fd90c178
128 changed files with 12352 additions and 4 deletions
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/GalaxyProxyDriver.cs
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/GalaxyProxyDriver.cs
@@ -0,0 +1,144 @@
+using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.Ipc;
+using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Shared.Contracts;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy;
+
+/// <summary>
+///     <see cref="IDriver"/> implementation that forwards every capability over the Galaxy IPC
+///     channel to the out-of-process Host. Implements <see cref="ITagDiscovery"/> as the
+///     Phase 2 minimum; other capability interfaces (<see cref="IReadable"/>, etc.) will be wired
+///     in once the Host's MXAccess code lift is complete and end-to-end parity tests run.
+/// </summary>
+public sealed class GalaxyProxyDriver(GalaxyProxyOptions options)
+    : IDriver, ITagDiscovery, IDisposable
+{
+    private GalaxyIpcClient? _client;
+    private long _sessionId;
+    private DriverHealth _health = new(DriverState.Unknown, null, null);
+
+    public string DriverInstanceId => options.DriverInstanceId;
+    public string DriverType => "Galaxy";
+
+    public async Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
+    {
+        _health = new DriverHealth(DriverState.Initializing, null, null);
+        try
+        {
+            _client = await GalaxyIpcClient.ConnectAsync(
+                options.PipeName, options.SharedSecret, options.ConnectTimeout, cancellationToken);
+
+            var resp = await _client.CallAsync<OpenSessionRequest, OpenSessionResponse>(
+                MessageKind.OpenSessionRequest,
+                new OpenSessionRequest { DriverInstanceId = DriverInstanceId, DriverConfigJson = driverConfigJson },
+                MessageKind.OpenSessionResponse,
+                cancellationToken);
+
+            if (!resp.Success)
+                throw new InvalidOperationException($"Galaxy.Host OpenSession failed: {resp.Error}");
+
+            _sessionId = resp.SessionId;
+            _health = new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null);
+        }
+        catch (Exception ex)
+        {
+            _health = new DriverHealth(DriverState.Faulted, null, ex.Message);
+            throw;
+        }
+    }
+
+    public async Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
+    {
+        await ShutdownAsync(cancellationToken);
+        await InitializeAsync(driverConfigJson, cancellationToken);
+    }
+
+    public async Task ShutdownAsync(CancellationToken cancellationToken)
+    {
+        if (_client is null) return;
+
+        try
+        {
+            await _client.CallAsync<CloseSessionRequest, ErrorResponse>(
+                MessageKind.CloseSessionRequest, new CloseSessionRequest { SessionId = _sessionId },
+                MessageKind.ErrorResponse, cancellationToken);
+        }
+        catch { /* shutdown is best effort */ }
+
+        await _client.DisposeAsync();
+        _client = null;
+        _health = new DriverHealth(DriverState.Unknown, _health.LastSuccessfulRead, null);
+    }
+
+    public DriverHealth GetHealth() => _health;
+
+    public long GetMemoryFootprint() => 0; // Tier C footprint is reported by the Host over IPC
+
+    public Task FlushOptionalCachesAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+
+    public async Task DiscoverAsync(IAddressSpaceBuilder builder, CancellationToken cancellationToken)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        if (_client is null) throw new InvalidOperationException("Driver not initialized");
+
+        var resp = await _client.CallAsync<DiscoverHierarchyRequest, DiscoverHierarchyResponse>(
+            MessageKind.DiscoverHierarchyRequest,
+            new DiscoverHierarchyRequest { SessionId = _sessionId },
+            MessageKind.DiscoverHierarchyResponse,
+            cancellationToken);
+
+        if (!resp.Success)
+            throw new InvalidOperationException($"Galaxy.Host DiscoverHierarchy failed: {resp.Error}");
+
+        foreach (var obj in resp.Objects)
+        {
+            var folder = builder.Folder(obj.ContainedName, obj.ContainedName);
+            foreach (var attr in obj.Attributes)
+            {
+                folder.Variable(
+                    attr.AttributeName,
+                    attr.AttributeName,
+                    new DriverAttributeInfo(
+                        FullName: $"{obj.TagName}.{attr.AttributeName}",
+                        DriverDataType: MapDataType(attr.MxDataType),
+                        IsArray: attr.IsArray,
+                        ArrayDim: attr.ArrayDim,
+                        SecurityClass: MapSecurity(attr.SecurityClassification),
+                        IsHistorized: attr.IsHistorized));
+            }
+        }
+    }
+
+    private static DriverDataType MapDataType(int mxDataType) => mxDataType switch
+    {
+        0 => DriverDataType.Boolean,
+        1 => DriverDataType.Int32,
+        2 => DriverDataType.Float32,
+        3 => DriverDataType.Float64,
+        4 => DriverDataType.String,
+        5 => DriverDataType.DateTime,
+        _ => DriverDataType.String,
+    };
+
+    private static SecurityClassification MapSecurity(int mxSec) => mxSec switch
+    {
+        0 => SecurityClassification.FreeAccess,
+        1 => SecurityClassification.Operate,
+        2 => SecurityClassification.SecuredWrite,
+        3 => SecurityClassification.VerifiedWrite,
+        4 => SecurityClassification.Tune,
+        5 => SecurityClassification.Configure,
+        6 => SecurityClassification.ViewOnly,
+        _ => SecurityClassification.FreeAccess,
+    };
+
+    public void Dispose() => _client?.DisposeAsync().AsTask().GetAwaiter().GetResult();
+}
+
+public sealed class GalaxyProxyOptions
+{
+    public required string DriverInstanceId { get; init; }
+    public required string PipeName { get; init; }
+    public required string SharedSecret { get; init; }
+    public TimeSpan ConnectTimeout { get; init; } = TimeSpan.FromSeconds(10);
+}
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Ipc/GalaxyIpcClient.cs
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Ipc/GalaxyIpcClient.cs
@@ -0,0 +1,101 @@
+using System.IO.Pipes;
+using MessagePack;
+using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Shared;
+using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Shared.Contracts;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.Ipc;
+
+/// <summary>
+///     Client-side IPC channel to a running <c>Driver.Galaxy.Host</c>. Owns the data-plane pipe
+///     connection and serializes request/response round-trips. One instance per session.
+/// </summary>
+public sealed class GalaxyIpcClient : IAsyncDisposable
+{
+    private readonly NamedPipeClientStream _stream;
+    private readonly FrameReader _reader;
+    private readonly FrameWriter _writer;
+    private readonly SemaphoreSlim _callGate = new(1, 1);
+
+    private GalaxyIpcClient(NamedPipeClientStream stream)
+    {
+        _stream = stream;
+        _reader = new FrameReader(stream, leaveOpen: true);
+        _writer = new FrameWriter(stream, leaveOpen: true);
+    }
+
+    /// <summary>Connects, sends Hello with the shared secret, and awaits HelloAck. Throws on rejection.</summary>
+    public static async Task<GalaxyIpcClient> ConnectAsync(
+        string pipeName, string sharedSecret, TimeSpan connectTimeout, CancellationToken ct)
+    {
+        var stream = new NamedPipeClientStream(
+            serverName: ".",
+            pipeName: pipeName,
+            direction: PipeDirection.InOut,
+            options: PipeOptions.Asynchronous);
+
+        await stream.ConnectAsync((int)connectTimeout.TotalMilliseconds, ct);
+
+        var client = new GalaxyIpcClient(stream);
+        try
+        {
+            await client._writer.WriteAsync(MessageKind.Hello,
+                new Hello { PeerName = "Galaxy.Proxy", SharedSecret = sharedSecret }, ct);
+
+            var ack = await client._reader.ReadFrameAsync(ct);
+            if (ack is null || ack.Value.Kind != MessageKind.HelloAck)
+                throw new InvalidOperationException("Did not receive HelloAck from Galaxy.Host");
+
+            var ackMsg = FrameReader.Deserialize<HelloAck>(ack.Value.Body);
+            if (!ackMsg.Accepted)
+                throw new UnauthorizedAccessException($"Galaxy.Host rejected Hello: {ackMsg.RejectReason}");
+
+            return client;
+        }
+        catch
+        {
+            await client.DisposeAsync();
+            throw;
+        }
+    }
+
+    /// <summary>Round-trips a request and returns the first frame of the response.</summary>
+    public async Task<TResp> CallAsync<TReq, TResp>(
+        MessageKind requestKind, TReq request, MessageKind expectedResponseKind, CancellationToken ct)
+    {
+        await _callGate.WaitAsync(ct);
+        try
+        {
+            await _writer.WriteAsync(requestKind, request, ct);
+
+            var frame = await _reader.ReadFrameAsync(ct);
+            if (frame is null) throw new EndOfStreamException("IPC peer closed before response");
+
+            if (frame.Value.Kind == MessageKind.ErrorResponse)
+            {
+                var err = MessagePackSerializer.Deserialize<ErrorResponse>(frame.Value.Body);
+                throw new GalaxyIpcException(err.Code, err.Message);
+            }
+
+            if (frame.Value.Kind != expectedResponseKind)
+                throw new InvalidOperationException(
+                    $"Expected {expectedResponseKind}, got {frame.Value.Kind}");
+
+            return MessagePackSerializer.Deserialize<TResp>(frame.Value.Body);
+        }
+        finally { _callGate.Release(); }
+    }
+
+    public async ValueTask DisposeAsync()
+    {
+        _callGate.Dispose();
+        _reader.Dispose();
+        _writer.Dispose();
+        await _stream.DisposeAsync();
+    }
+}
+
+public sealed class GalaxyIpcException(string code, string message)
+    : Exception($"[{code}] {message}")
+{
+    public string Code { get; } = code;
+}
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/Backoff.cs
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/Backoff.cs
@@ -0,0 +1,29 @@
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.Supervisor;
+
+/// <summary>
+///     Respawn-with-backoff schedule per <c>driver-stability.md §"Crash-loop circuit breaker"</c>:
+///     5s → 15s → 60s, capped. Reset on a successful (&gt; <see cref="StableRunThreshold"/>)
+///     run.
+/// </summary>
+public sealed class Backoff
+{
+    public static TimeSpan[] DefaultSequence { get; } =
+        [TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(60)];
+
+    public TimeSpan StableRunThreshold { get; init; } = TimeSpan.FromMinutes(2);
+
+    private readonly TimeSpan[] _sequence;
+    private int _index;
+
+    public Backoff(TimeSpan[]? sequence = null) => _sequence = sequence ?? DefaultSequence;
+
+    public TimeSpan Next()
+    {
+        var delay = _sequence[Math.Min(_index, _sequence.Length - 1)];
+        _index++;
+        return delay;
+    }
+
+    /// <summary>Called when the spawned process has stayed up past the stable threshold.</summary>
+    public void RecordStableRun() => _index = 0;
+}
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/CircuitBreaker.cs
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/CircuitBreaker.cs
@@ -0,0 +1,68 @@
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.Supervisor;
+
+/// <summary>
+///     Crash-loop circuit breaker per <c>driver-stability.md</c>:
+///     3 crashes within 5 min → open with escalating cooldown 1h → 4h → 24h manual. A sticky
+///     alert stays until the operator explicitly resets.
+/// </summary>
+public sealed class CircuitBreaker
+{
+    public int CrashesAllowedPerWindow { get; init; } = 3;
+    public TimeSpan Window { get; init; } = TimeSpan.FromMinutes(5);
+
+    public TimeSpan[] CooldownEscalation { get; init; } =
+        [TimeSpan.FromHours(1), TimeSpan.FromHours(4), TimeSpan.MaxValue];
+
+    private readonly List<DateTime> _crashesUtc = [];
+    private DateTime? _openSinceUtc;
+    private int _escalationLevel;
+    public bool StickyAlertActive { get; private set; }
+
+    /// <summary>
+    ///     Called by the supervisor each time the host process exits unexpectedly. Returns
+    ///     <c>false</c> when the breaker is open — supervisor must not respawn.
+    /// </summary>
+    public bool TryRecordCrash(DateTime utcNow, out TimeSpan cooldownRemaining)
+    {
+        if (_openSinceUtc is { } openedAt)
+        {
+            var cooldown = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
+            if (cooldown == TimeSpan.MaxValue)
+            {
+                cooldownRemaining = TimeSpan.MaxValue;
+                return false; // manual reset required
+            }
+            if (utcNow - openedAt < cooldown)
+            {
+                cooldownRemaining = cooldown - (utcNow - openedAt);
+                return false;
+            }
+
+            // Cooldown elapsed — close the breaker but keep the sticky alert per spec.
+            _openSinceUtc = null;
+            _escalationLevel++;
+        }
+
+        _crashesUtc.RemoveAll(t => utcNow - t > Window);
+        _crashesUtc.Add(utcNow);
+
+        if (_crashesUtc.Count > CrashesAllowedPerWindow)
+        {
+            _openSinceUtc = utcNow;
+            StickyAlertActive = true;
+            cooldownRemaining = CooldownEscalation[Math.Min(_escalationLevel, CooldownEscalation.Length - 1)];
+            return false;
+        }
+
+        cooldownRemaining = TimeSpan.Zero;
+        return true;
+    }
+
+    public void ManualReset()
+    {
+        _crashesUtc.Clear();
+        _openSinceUtc = null;
+        _escalationLevel = 0;
+        StickyAlertActive = false;
+    }
+}
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/HeartbeatMonitor.cs
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/HeartbeatMonitor.cs
@@ -0,0 +1,28 @@
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.Supervisor;
+
+/// <summary>
+///     Tracks missed heartbeats on the dedicated heartbeat pipe per
+///     <c>driver-stability.md §"Heartbeat between proxy and host"</c>: 2s cadence, 3 consecutive
+///     misses = host declared dead (~6s detection).
+/// </summary>
+public sealed class HeartbeatMonitor
+{
+    public int MissesUntilDead { get; init; } = 3;
+
+    public TimeSpan Cadence { get; init; } = TimeSpan.FromSeconds(2);
+
+    public int ConsecutiveMisses { get; private set; }
+    public DateTime? LastAckUtc { get; private set; }
+
+    public void RecordAck(DateTime utcNow)
+    {
+        ConsecutiveMisses = 0;
+        LastAckUtc = utcNow;
+    }
+
+    public bool RecordMiss()
+    {
+        ConsecutiveMisses++;
+        return ConsecutiveMisses >= MissesUntilDead;
+    }
+}
--- a/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.csproj
+++ b/src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy.csproj
@@ -0,0 +1,24 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+    <PropertyGroup>
+        <TargetFramework>net10.0</TargetFramework>
+        <Nullable>enable</Nullable>
+        <ImplicitUsings>enable</ImplicitUsings>
+        <LangVersion>latest</LangVersion>
+        <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
+        <GenerateDocumentationFile>true</GenerateDocumentationFile>
+        <NoWarn>$(NoWarn);CS1591</NoWarn>
+        <RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy</RootNamespace>
+    </PropertyGroup>
+
+    <ItemGroup>
+        <ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
+        <ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Shared\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Shared.csproj"/>
+    </ItemGroup>
+
+    <ItemGroup>
+        <NuGetAuditSuppress Include="https://github.com/advisories/GHSA-37gx-xxp4-5rgx"/>
+        <NuGetAuditSuppress Include="https://github.com/advisories/GHSA-w3x6-4m5h-cxqf"/>
+    </ItemGroup>
+
+</Project>