Phase 6.1 Stream E (data layer) — DriverInstanceResilienceStatus entity + DriverResilienceStatusTracker + EF migration

Ships the data + runtime layer of Stream E. The SignalR hub and Blazor /hosts
page refresh (E.2-E.3) are follow-up work paired with the visual-compliance
review per Phase 6.4 patterns — documented as a deferred follow-up below.

Configuration:
- New entity DriverInstanceResilienceStatus with:
  DriverInstanceId, HostName (composite PK),
  LastCircuitBreakerOpenUtc, ConsecutiveFailures, CurrentBulkheadDepth,
  LastRecycleUtc, BaselineFootprintBytes, CurrentFootprintBytes,
  LastSampledUtc.
- Separate from DriverHostStatus (per-host connectivity view) so a Running
  host that has tripped its breaker or is nearing its memory ceiling shows up
  distinctly on Admin /hosts. Admin page left-joins both for display.
- OtOpcUaConfigDbContext + Fluent-API config + IX_DriverResilience_LastSampled
  index for the stale-sample filter query.
- EF migration: 20260419124034_AddDriverInstanceResilienceStatus.

Core.Resilience:
- DriverResilienceStatusTracker — process-singleton in-memory tracker keyed on
  (DriverInstanceId, HostName). CapabilityInvoker + MemoryTracking +
  MemoryRecycle callers record failure/success/breaker-open/recycle/footprint
  events; a HostedService (Stream E.2 follow-up) samples this tracker every
  5 s and persists to the DB. Pure in-memory keeps tests fast + the core
  free of EF/SQL dependencies.

Tests:
- DriverResilienceStatusTrackerTests (9 new, all pass): tryget-before-write
  returns null; failures accumulate; success resets; breaker/recycle/footprint
  fields populate; per-host isolation; snapshot returns all pairs; concurrent
  writes don't lose counts.
- SchemaComplianceTests: expected-tables list updated to include the new
  DriverInstanceResilienceStatus table.

Full solution dotnet test: 1042 passing (baseline 906, +136 for Phase 6.1 so
far across Streams A/B/C/D/E.1). Pre-existing Client.CLI Subscribe flake
unchanged.

Deferred to follow-up PR (E.2/E.3):
- ResilienceStatusPublisher HostedService that samples DriverResilienceStatusTracker
  every 5 s + upserts DriverInstanceResilienceStatus rows.
- Admin FleetStatusHub SignalR hub pushing LastCircuitBreakerOpenUtc /
  CurrentBulkheadDepth / LastRecycleUtc on change.
- Admin /hosts Blazor column additions (red badge when
  ConsecutiveFailures > breakerThreshold / 2). Visual-compliance reviewer
  signoff alongside Phase 6.4 admin-ui patterns.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-04-19 08:47:43 -04:00
parent 8d81715079
commit cbcaf6593a
8 changed files with 1650 additions and 0 deletions

View File

@@ -0,0 +1,104 @@
using System.Collections.Concurrent;
namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
/// <summary>
/// Process-singleton tracker of live resilience counters per
/// <c>(DriverInstanceId, HostName)</c>. Populated by the CapabilityInvoker and the
/// MemoryTracking layer; consumed by a HostedService that periodically persists a
/// snapshot to the <c>DriverInstanceResilienceStatus</c> table for Admin <c>/hosts</c>.
/// </summary>
/// <remarks>
/// Per Phase 6.1 Stream E. No DB dependency here — the tracker is pure in-memory so
/// tests can exercise it without EF Core or SQL Server. The HostedService that writes
/// snapshots lives in the Server project (Stream E.2); the actual SignalR push + Blazor
/// page refresh (E.3) lands in a follow-up visual-review PR.
/// </remarks>
public sealed class DriverResilienceStatusTracker
{
private readonly ConcurrentDictionary<StatusKey, ResilienceStatusSnapshot> _status = new();
/// <summary>Record a Polly pipeline failure for <paramref name="hostName"/>.</summary>
public void RecordFailure(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { ConsecutiveFailures = 1, LastSampledUtc = utcNow },
(_, existing) => existing with
{
ConsecutiveFailures = existing.ConsecutiveFailures + 1,
LastSampledUtc = utcNow,
});
}
/// <summary>Reset the consecutive-failure count on a successful pipeline execution.</summary>
public void RecordSuccess(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { ConsecutiveFailures = 0, LastSampledUtc = utcNow },
(_, existing) => existing with
{
ConsecutiveFailures = 0,
LastSampledUtc = utcNow,
});
}
/// <summary>Record a circuit-breaker open event.</summary>
public void RecordBreakerOpen(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { LastBreakerOpenUtc = utcNow, LastSampledUtc = utcNow },
(_, existing) => existing with { LastBreakerOpenUtc = utcNow, LastSampledUtc = utcNow });
}
/// <summary>Record a process recycle event (Tier C only).</summary>
public void RecordRecycle(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { LastRecycleUtc = utcNow, LastSampledUtc = utcNow },
(_, existing) => existing with { LastRecycleUtc = utcNow, LastSampledUtc = utcNow });
}
/// <summary>Capture / update the MemoryTracking-supplied baseline + current footprint.</summary>
public void RecordFootprint(string driverInstanceId, string hostName, long baselineBytes, long currentBytes, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot
{
BaselineFootprintBytes = baselineBytes,
CurrentFootprintBytes = currentBytes,
LastSampledUtc = utcNow,
},
(_, existing) => existing with
{
BaselineFootprintBytes = baselineBytes,
CurrentFootprintBytes = currentBytes,
LastSampledUtc = utcNow,
});
}
/// <summary>Snapshot of a specific (instance, host) pair; null if no counters recorded yet.</summary>
public ResilienceStatusSnapshot? TryGet(string driverInstanceId, string hostName) =>
_status.TryGetValue(new StatusKey(driverInstanceId, hostName), out var snapshot) ? snapshot : null;
/// <summary>Copy of every currently-tracked (instance, host, snapshot) triple. Safe under concurrent writes.</summary>
public IReadOnlyList<(string DriverInstanceId, string HostName, ResilienceStatusSnapshot Snapshot)> Snapshot() =>
_status.Select(kvp => (kvp.Key.DriverInstanceId, kvp.Key.HostName, kvp.Value)).ToList();
private readonly record struct StatusKey(string DriverInstanceId, string HostName);
}
/// <summary>Snapshot of the resilience counters for one <c>(DriverInstanceId, HostName)</c> pair.</summary>
public sealed record ResilienceStatusSnapshot
{
public int ConsecutiveFailures { get; init; }
public DateTime? LastBreakerOpenUtc { get; init; }
public DateTime? LastRecycleUtc { get; init; }
public long BaselineFootprintBytes { get; init; }
public long CurrentFootprintBytes { get; init; }
public DateTime LastSampledUtc { get; init; }
}