dohertj2/lmxopcua
1
0
Fork 0
Code Issues 17 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(drivers): delete the inert driver-tier recycle machinery (#522)

Browse Source 
The tier system was documented, operator-authorable, and inert. Deleted rather than activated,
because its premise is gone rather than merely unused.

"Tier C" meant a driver running out-of-process behind an IDriverSupervisor that could restart its
Host without tearing down the OPC UA session. No such process exists anywhere: Galaxy reaches
MXAccess over gRPC to the external mxaccessgw sidecar (PR 7.2 retired the in-process
Galaxy.Host/Proxy/Shared projects) and FOCAS has run in-process since its managed wire client
landed 2026-04-24. Consistently, IDriverSupervisor had ZERO implementations and there was nothing
for one to implement against.

The issue understated the inertness. It says the Tier-C-only protections never engaged, implying
the Tier A/B parts did. They did not: nothing constructs MemoryTracking, MemoryRecycle or
ScheduledRecycleScheduler outside their own unit tests, so the whole Core/Stability recycle layer
was dead — meaning option (a), "pass real tiers", was never a flag flip. It would have meant
writing the wiring that never existed AND arming it.

Deleted: MemoryTracking, MemoryRecycle, ScheduledRecycleScheduler, IDriverSupervisor, the
vestigial DriverTypeRegistry (referenced only by its own tests), and the RecycleIntervalSeconds
knob — which the AdminUI let an operator author and the parser validated while it configured
nothing.

DriverTier itself SURVIVES and is load-bearing: DriverResilienceOptions.GetTierDefaults supplies
the real per-capability timeout/retry/breaker policies via DriverFactoryRegistry.GetTier and
DriverCapabilityInvokerFactory. Only the isolation-and-recycle layer above it is gone.

Deliberately NOT deleted:

- WedgeDetector came along in the same directory and is equally dead in production, but it is
  tier-agnostic and is not recycle machinery — it only shares the folder. Restored rather than
  swept up in a decision that was not about it.
- IDriver.GetMemoryFootprint() and FlushOptionalCachesAsync() lose their only consumer here.
  Removing them touches all 12 drivers and every test stub, so they are documented as
  consumerless and filed as #525 instead of buried in this diff.

Compatibility: a deployed ResilienceConfig blob still carrying "recycleIntervalSeconds" parses
cleanly (unknown keys are ignored — guarded by a new test, because a blob that suddenly failed to
parse would fall back to tier defaults and silently discard the operator's real overrides), and
the AdminUI's preserve-unknown-keys bag keeps the key rather than rewriting stored config on an
unrelated edit.

The 01/U-6 knob-inertness guard carried an explicit carve-out admitting RecycleIntervalSeconds was
dormant and out of scope; that carve-out is now gone, so the expected set is literally what the
test's name claims.

Note: Host.IntegrationTests has 2 failures (DriverProbeRegistrationTests.is_idempotent,
PrimaryGateFailoverTests) — verified pre-existing by reproducing both on clean master dc9d947b.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-30 04:44:04 -04:00
parent 30d0697c28
commit cbb882293b
23 changed files with 153 additions and 1145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceOptions.cs
+11 -9
View File
@@ -19,15 +19,17 @@ public sealed record DriverResilienceOptions
public IReadOnlyDictionary<DriverCapability, CapabilityPolicy> CapabilityPolicies { get; init; }
= new Dictionary<DriverCapability, CapabilityPolicy>();
/// <summary>
/// Periodic scheduled recycle interval for Tier C out-of-process hosts, in seconds.
/// Null (the default) = no scheduled recycle; the driver's Host process keeps running
/// indefinitely unless a memory breach or operator action triggers a recycle. Only
/// respected for <see cref="DriverTier.C"/>; Tier A/B recycle would tear down every
/// OPC UA session, so the loader ignores non-null values for those tiers + logs a
/// warning.
/// </summary>
public int? RecycleIntervalSeconds { get; init; }
// RecycleIntervalSeconds is GONE (Gitea #522). It configured a scheduled recycle of a Tier C
// driver's out-of-process Host — a process that no longer exists anywhere: Galaxy reaches MXAccess
// over gRPC to the external mxaccessgw sidecar (PR 7.2 retired the in-process Host/Proxy/Shared
// projects) and FOCAS has run in-process since its managed wire client landed 2026-04-24. The
// scheduler it fed was constructed only in tests, and IDriverSupervisor — the thing that would
// have performed the recycle — had zero implementations. The parser nevertheless validated the
// knob, so an operator could author a recycle interval through the AdminUI and get nothing.
//
// The JSON key is not an error if present: DriverResilienceOptionsParser ignores unknown keys, so
// a ResilienceConfig blob still carrying "recycleIntervalSeconds" parses cleanly and the value is
// simply dropped, which is what it already did in effect.
/// <summary>
/// Look up the effective policy for a capability, falling back to tier defaults when no
src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceOptionsParser.cs
+7 -17
View File
@@ -94,25 +94,15 @@ public static class DriverResilienceOptionsParser
}
}
// Scheduled recycle is Tier C only — reject a configured interval on Tier A/B as a
// misconfiguration surface rather than silently honouring it (recycling an in-process
// driver would kill every OPC UA session + every co-hosted driver).
int? recycleIntervalSeconds = null;
if (shape.RecycleIntervalSeconds is int secs)
{
if (secs <= 0)
AppendDiagnostic(ref parseDiagnostic, $"RecycleIntervalSeconds must be positive; got {secs} — ignoring.");
else if (tier != DriverTier.C)
AppendDiagnostic(ref parseDiagnostic, $"RecycleIntervalSeconds is Tier C only; Tier {tier} driver will not scheduled-recycle.");
else
recycleIntervalSeconds = secs;
}
// No recycle-interval handling: the scheduled-recycle machinery was deleted in Gitea #522
// (nothing constructed it, and IDriverSupervisor — which would have performed the recycle —
// had no implementations). A "recycleIntervalSeconds" key surviving in an existing
// ResilienceConfig blob is harmless: the shape no longer declares it, and unknown JSON
// properties are ignored, so the config still parses and the value is dropped.
return new DriverResilienceOptions
{
Tier = tier,
CapabilityPolicies = merged,
RecycleIntervalSeconds = recycleIntervalSeconds,
};
}
@@ -196,8 +186,8 @@ public static class DriverResilienceOptionsParser
private sealed class ResilienceConfigShape
{
/// <summary>Gets or sets the scheduled recycle interval in seconds.</summary>
public int? RecycleIntervalSeconds { get; set; }
// No RecycleIntervalSeconds — see the note at the return site. Deserialization ignores unknown
// properties, so an existing blob carrying the key still binds.
/// <summary>Gets or sets the per-capability resilience policies.</summary>
public Dictionary<string, CapabilityPolicyShape>? CapabilityPolicies { get; set; }
}
src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryRecycle.cs
-72
View File
@@ -1,72 +0,0 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Stability;
/// <summary>
/// Tier C only process-recycle companion to <see cref="MemoryTracking"/>. On a
/// <see cref="MemoryTrackingAction.HardBreach"/> signal, invokes the supplied
/// <see cref="IDriverSupervisor"/> to restart the out-of-process Host.
/// </summary>
/// <remarks>
/// Per <c>docs/v2/plan.md</c>. Tier A/B hard-breach on an in-process
/// driver would kill every OPC UA session and every co-hosted driver, so for Tier A/B this
/// class logs a <b>promotion-to-Tier-C recommendation</b> and does NOT invoke any supervisor.
/// A future tier-migration workflow acts on the recommendation.
/// </remarks>
public sealed class MemoryRecycle
{
private readonly DriverTier _tier;
private readonly IDriverSupervisor? _supervisor;
private readonly ILogger<MemoryRecycle> _logger;
/// <summary>Initializes a new instance of the <see cref="MemoryRecycle"/> class.</summary>
/// <param name="tier">The driver tier.</param>
/// <param name="supervisor">Optional supervisor for process recycling.</param>
/// <param name="logger">Logger for recycling events.</param>
public MemoryRecycle(DriverTier tier, IDriverSupervisor? supervisor, ILogger<MemoryRecycle> logger)
{
_tier = tier;
_supervisor = supervisor;
_logger = logger;
}
/// <summary>
/// Handle a <see cref="MemoryTracking"/> classification for the driver. For Tier C with a
/// wired supervisor, <c>HardBreach</c> triggers <see cref="IDriverSupervisor.RecycleAsync"/>.
/// All other combinations are no-ops with respect to process state (soft breaches + Tier A/B
/// hard breaches just log).
/// </summary>
/// <param name="action">The memory tracking action.</param>
/// <param name="footprintBytes">The current process footprint in bytes.</param>
/// <param name="cancellationToken">Cancellation token for the operation.</param>
/// <returns>True when a recycle was requested; false otherwise.</returns>
public async Task<bool> HandleAsync(MemoryTrackingAction action, long footprintBytes, CancellationToken cancellationToken)
{
switch (action)
{
case MemoryTrackingAction.SoftBreach:
_logger.LogWarning(
"Memory soft-breach on driver {DriverId}: footprint={Footprint:N0} bytes, tier={Tier}. Surfaced to Admin; no action.",
_supervisor?.DriverInstanceId ?? "(unknown)", footprintBytes, _tier);
return false;
case MemoryTrackingAction.HardBreach when _tier == DriverTier.C && _supervisor is not null:
_logger.LogError(
"Memory hard-breach on Tier C driver {DriverId}: footprint={Footprint:N0} bytes. Requesting supervisor recycle.",
_supervisor.DriverInstanceId, footprintBytes);
await _supervisor.RecycleAsync($"Memory hard-breach: {footprintBytes} bytes", cancellationToken).ConfigureAwait(false);
return true;
case MemoryTrackingAction.HardBreach:
_logger.LogError(
"Memory hard-breach on Tier {Tier} in-process driver {DriverId}: footprint={Footprint:N0} bytes. " +
"Recommending promotion to Tier C; NOT auto-killing (decisions #74, #145).",
_tier, _supervisor?.DriverInstanceId ?? "(unknown)", footprintBytes);
return false;
default:
return false;
}
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryTracking.cs
-144
View File
@@ -1,144 +0,0 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Stability;
/// <summary>
/// Tier-agnostic memory-footprint tracker. Captures the post-initialize <b>baseline</b>
/// from the first samples after <c>IDriver.InitializeAsync</c>, then classifies each
/// subsequent sample against a hybrid soft/hard threshold per
/// <c>docs/v2/plan.md</c> — <c>soft = max(multiplier × baseline, baseline + floor)</c>,
/// <c>hard = 2 × soft</c>.
/// </summary>
/// <remarks>
/// <para>This tracker <b>never kills a process</b>. Soft and hard breaches
/// log + surface to the Admin UI via <c>DriverInstanceResilienceStatus</c>. The matching
/// process-level recycle protection lives in a separate <c>MemoryRecycle</c> that activates
/// for Tier C drivers only (where the driver runs out-of-process behind a supervisor that
/// can safely restart it without tearing down the OPC UA session or co-hosted in-proc
/// drivers).</para>
///
/// <para>Baseline capture: the tracker starts in <see cref="TrackingPhase.WarmingUp"/> for
/// <see cref="BaselineWindow"/> (default 5 min). During that window samples are collected;
/// the baseline is computed as the median once the window elapses. Before that point every
/// classification returns <see cref="MemoryTrackingAction.Warming"/>.</para>
/// </remarks>
public sealed class MemoryTracking
{
private readonly DriverTier _tier;
private readonly TimeSpan _baselineWindow;
private readonly List<long> _warmupSamples = [];
private long _baselineBytes;
private TrackingPhase _phase = TrackingPhase.WarmingUp;
private DateTime? _warmupStartUtc;
/// <summary>Tier-default multiplier/floor constants.</summary>
/// <param name="tier">The driver tier.</param>
/// <returns>The multiplier and floor-bytes constants for the tier.</returns>
public static (int Multiplier, long FloorBytes) GetTierConstants(DriverTier tier) => tier switch
{
DriverTier.A => (Multiplier: 3, FloorBytes: 50L * 1024 * 1024),
DriverTier.B => (Multiplier: 3, FloorBytes: 100L * 1024 * 1024),
DriverTier.C => (Multiplier: 2, FloorBytes: 500L * 1024 * 1024),
_ => throw new ArgumentOutOfRangeException(nameof(tier), tier, $"No memory-tracking constants defined for tier {tier}."),
};
/// <summary>Window over which post-init samples are collected to compute the baseline.</summary>
public TimeSpan BaselineWindow => _baselineWindow;
/// <summary>Current phase: <see cref="TrackingPhase.WarmingUp"/> or <see cref="TrackingPhase.Steady"/>.</summary>
public TrackingPhase Phase => _phase;
/// <summary>Captured baseline; 0 until warmup completes.</summary>
public long BaselineBytes => _baselineBytes;
/// <summary>Effective soft threshold (zero while warming up).</summary>
public long SoftThresholdBytes => _baselineBytes == 0 ? 0 : ComputeSoft(_tier, _baselineBytes);
/// <summary>Effective hard threshold = 2 × soft (zero while warming up).</summary>
public long HardThresholdBytes => _baselineBytes == 0 ? 0 : ComputeSoft(_tier, _baselineBytes) * 2;
/// <summary>Initializes a new instance of the <see cref="MemoryTracking"/> class.</summary>
/// <param name="tier">The driver tier for threshold constants.</param>
/// <param name="baselineWindow">Optional custom baseline window duration (default 5 minutes).</param>
public MemoryTracking(DriverTier tier, TimeSpan? baselineWindow = null)
{
_tier = tier;
_baselineWindow = baselineWindow ?? TimeSpan.FromMinutes(5);
}
/// <summary>
/// Submit a memory-footprint sample. Returns the action the caller should surface.
/// During warmup, always returns <see cref="MemoryTrackingAction.Warming"/> and accumulates
/// samples; once the window elapses the first steady-phase sample triggers baseline capture
/// (median of warmup samples).
/// </summary>
/// <param name="footprintBytes">The current memory footprint in bytes.</param>
/// <param name="utcNow">The current UTC time.</param>
/// <returns>The tracking action the caller should surface for this sample.</returns>
public MemoryTrackingAction Sample(long footprintBytes, DateTime utcNow)
{
if (_phase == TrackingPhase.WarmingUp)
{
_warmupStartUtc ??= utcNow;
_warmupSamples.Add(footprintBytes);
if (utcNow - _warmupStartUtc.Value >= _baselineWindow && _warmupSamples.Count > 0)
{
_baselineBytes = ComputeMedian(_warmupSamples);
_phase = TrackingPhase.Steady;
}
else
{
return MemoryTrackingAction.Warming;
}
}
if (footprintBytes >= HardThresholdBytes) return MemoryTrackingAction.HardBreach;
if (footprintBytes >= SoftThresholdBytes) return MemoryTrackingAction.SoftBreach;
return MemoryTrackingAction.None;
}
private static long ComputeSoft(DriverTier tier, long baseline)
{
var (multiplier, floor) = GetTierConstants(tier);
return Math.Max(multiplier * baseline, baseline + floor);
}
private static long ComputeMedian(List<long> samples)
{
var sorted = samples.Order().ToArray();
var mid = sorted.Length / 2;
return sorted.Length % 2 == 1
? sorted[mid]
: (sorted[mid - 1] + sorted[mid]) / 2;
}
}
/// <summary>Phase of a <see cref="MemoryTracking"/> lifecycle.</summary>
public enum TrackingPhase
{
/// <summary>Collecting post-init samples; baseline not yet computed.</summary>
WarmingUp,
/// <summary>Baseline captured; every sample classified against soft/hard thresholds.</summary>
Steady,
}
/// <summary>Classification the tracker returns per sample.</summary>
public enum MemoryTrackingAction
{
/// <summary>Baseline not yet captured; sample collected, no threshold check.</summary>
Warming,
/// <summary>Below soft threshold.</summary>
None,
/// <summary>Between soft and hard thresholds — log + surface, no action.</summary>
SoftBreach,
/// <summary>
/// ≥ hard threshold. Log + surface + (Tier C only, via <c>MemoryRecycle</c>) request
/// process recycle via the driver supervisor. Tier A/B breach never invokes any
/// kill path.
/// </summary>
HardBreach,
}
src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/ScheduledRecycleScheduler.cs
-92
View File
@@ -1,92 +0,0 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Stability;
/// <summary>
/// Tier C opt-in periodic-recycle driver.
/// A tick method advanced by the caller (fed by a background timer in prod; by test clock
/// in unit tests) decides whether the configured interval has elapsed and, if so, drives the
/// supplied <see cref="IDriverSupervisor"/> to recycle the Host.
/// </summary>
/// <remarks>
/// Tier A/B drivers MUST NOT use this class — scheduled recycle for in-process drivers would
/// kill every OPC UA session and every co-hosted driver. The ctor throws when constructed
/// with any tier other than C to make the misuse structurally impossible.
///
/// <para>Keeps no background thread of its own — callers invoke <see cref="TickAsync"/> on
/// their ambient scheduler tick (Phase 6.1 Stream C's health-endpoint host runs one). That
/// decouples the unit under test from wall-clock time and thread-pool scheduling.</para>
/// </remarks>
public sealed class ScheduledRecycleScheduler
{
private readonly TimeSpan _recycleInterval;
private readonly IDriverSupervisor _supervisor;
private readonly ILogger<ScheduledRecycleScheduler> _logger;
private DateTime _nextRecycleUtc;
/// <summary>
/// Construct the scheduler for a Tier C driver. Throws if <paramref name="tier"/> isn't C.
/// </summary>
/// <param name="tier">Driver tier; must be <see cref="DriverTier.C"/>.</param>
/// <param name="recycleInterval">Interval between recycles (e.g. 7 days).</param>
/// <param name="startUtc">Anchor time; next recycle fires at <paramref name="startUtc"/> + <paramref name="recycleInterval"/>.</param>
/// <param name="supervisor">Supervisor that performs the actual recycle.</param>
/// <param name="logger">Diagnostic sink.</param>
public ScheduledRecycleScheduler(
DriverTier tier,
TimeSpan recycleInterval,
DateTime startUtc,
IDriverSupervisor supervisor,
ILogger<ScheduledRecycleScheduler> logger)
{
if (tier != DriverTier.C)
throw new ArgumentException(
$"ScheduledRecycleScheduler is Tier C only (got {tier}). " +
"In-process drivers must not use scheduled recycle; see decisions #74 and #145.",
nameof(tier));
if (recycleInterval <= TimeSpan.Zero)
throw new ArgumentException("RecycleInterval must be positive.", nameof(recycleInterval));
_recycleInterval = recycleInterval;
_supervisor = supervisor;
_logger = logger;
_nextRecycleUtc = startUtc + recycleInterval;
}
/// <summary>Next scheduled recycle UTC. Advances by <see cref="RecycleInterval"/> on each fire.</summary>
public DateTime NextRecycleUtc => _nextRecycleUtc;
/// <summary>Recycle interval this scheduler was constructed with.</summary>
public TimeSpan RecycleInterval => _recycleInterval;
/// <summary>
/// Tick the scheduler forward. If <paramref name="utcNow"/> is past
/// <see cref="NextRecycleUtc"/>, requests a recycle from the supervisor and advances
/// <see cref="NextRecycleUtc"/> by exactly one interval. Returns true when a recycle fired.
/// </summary>
/// <param name="utcNow">The current UTC time.</param>
/// <param name="cancellationToken">The cancellation token.</param>
/// <returns>True if a recycle was triggered; false otherwise.</returns>
public async Task<bool> TickAsync(DateTime utcNow, CancellationToken cancellationToken)
{
if (utcNow < _nextRecycleUtc)
return false;
_logger.LogInformation(
"Scheduled recycle due for Tier C driver {DriverId} at {Now:o}; advancing next to {Next:o}.",
_supervisor.DriverInstanceId, utcNow, _nextRecycleUtc + _recycleInterval);
await _supervisor.RecycleAsync("Scheduled periodic recycle", cancellationToken).ConfigureAwait(false);
_nextRecycleUtc += _recycleInterval;
return true;
}
/// <summary>Request an immediate recycle outside the schedule (e.g. MemoryRecycle hard-breach escalation).</summary>
/// <param name="reason">The reason for requesting an immediate recycle.</param>
/// <param name="cancellationToken">The cancellation token.</param>
/// <returns>A task representing the asynchronous recycle operation.</returns>
public Task RequestRecycleNowAsync(string reason, CancellationToken cancellationToken) =>
_supervisor.RecycleAsync(reason, cancellationToken);
}