Compare commits
12 Commits
v2-release
...
phase-6-1-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7b50118b68 | ||
| eac457fa7c | |||
|
|
c1cab33e38 | ||
| 0c903ff4e0 | |||
|
|
c4a92f424a | ||
| 510e488ea4 | |||
| 8994e73a0b | |||
|
|
e71f44603c | ||
|
|
c4824bea12 | ||
| e588c4f980 | |||
|
|
84fe88fadb | ||
| 59f793f87c |
@@ -1,7 +1,7 @@
|
||||
# v2 Release Readiness
|
||||
|
||||
> **Last updated**: 2026-04-19 (release blockers #1 + #2 closed; Phase 6.3 redundancy runtime is the last)
|
||||
> **Status**: **NOT YET RELEASE-READY** — one of three release blockers remains (Phase 6.3 Streams A/C/F redundancy-coordinator + OPC UA node wiring + client interop).
|
||||
> **Last updated**: 2026-04-19 (all three release blockers CLOSED — Phase 6.3 Streams A/C core shipped)
|
||||
> **Status**: **RELEASE-READY (code-path)** for v2 GA — all three code-path release blockers are closed. Remaining work is manual (client interop matrix, deployment checklist signoff, OPC UA CTT pass) + hardening follow-ups; see exit-criteria checklist below.
|
||||
|
||||
This doc is the single view of where v2 stands against its release criteria. Update it whenever a deferred follow-up closes or a new release blocker is discovered.
|
||||
|
||||
@@ -52,17 +52,19 @@ Remaining follow-ups (hardening, not release-blocking):
|
||||
- A `HostedService` that polls `sp_GetCurrentGenerationForCluster` periodically so peer-published generations land in this node's cache without a restart.
|
||||
- Richer snapshot payload via `sp_GetGenerationContent` so fallback can serve the full generation content (DriverInstance enumeration, ACL rows, etc.) from the sealed cache alone.
|
||||
|
||||
### Redundancy — Phase 6.3 Streams A/C/F (tasks #145, #147, #150)
|
||||
### ~~Redundancy — Phase 6.3 Streams A/C core~~ (tasks #145 + #147 — **CLOSED** 2026-04-19, PRs #98–99)
|
||||
|
||||
`ServiceLevelCalculator` + `RecoveryStateManager` + `ApplyLeaseRegistry` exist as pure logic. **No code invokes them at runtime.** The OPC UA server still publishes a static `ServiceLevel`; `ServerUriArray` still carries only self; no coordinator reads cluster topology; no peer probing.
|
||||
**Closed**. The runtime orchestration layer now exists end-to-end:
|
||||
|
||||
Closing this requires:
|
||||
- `RedundancyCoordinator` reads `ClusterNode` + peer list at startup (Stream A shipped in PR #98). Invariants enforced: 1-2 nodes (decision #83), unique ApplicationUri (#86), ≤1 Primary in Warm/Hot (#84). Startup fails fast on violation; runtime refresh logs + flips `IsTopologyValid=false` so the calculator falls to band 2 without tearing down.
|
||||
- `RedundancyStatePublisher` orchestrates topology + apply lease + recovery state + peer reachability through `ServiceLevelCalculator` + emits `OnStateChanged` / `OnServerUriArrayChanged` edge-triggered events (Stream C core shipped in PR #99). The OPC UA `ServiceLevel` Byte variable + `ServerUriArray` String[] variable subscribe to these events.
|
||||
|
||||
- `RedundancyCoordinator` singleton reads `ClusterNode` + peer list at startup (Stream A).
|
||||
- `PeerHttpProbeLoop` + `PeerUaProbeLoop` feed the calculator.
|
||||
- OPC UA node wiring: `ServiceLevel` becomes a live `BaseDataVariable` on calculator observer output; `ServerUriArray` includes self + peers; `RedundancySupport` static from `RedundancyMode` (Stream C).
|
||||
- `sp_PublishGeneration` wraps in `await using var lease = coordinator.BeginApplyLease(...)` so the `PrimaryMidApply` band fires during actual publishes.
|
||||
- Client interop matrix validation against Ignition / Kepware / Aveva OI Gateway (Stream F).
|
||||
Remaining Phase 6.3 surfaces (hardening, not release-blocking):
|
||||
|
||||
- `PeerHttpProbeLoop` + `PeerUaProbeLoop` HostedServices that poll the peer + write to `PeerReachabilityTracker` on each tick. Without these the publisher sees `PeerReachability.Unknown` for every peer → Isolated-Primary band (230) even when the peer is up. Safe default (retains authority) but not the full non-transparent-redundancy UX.
|
||||
- OPC UA variable-node wiring layer: bind the `ServiceLevel` Byte node + `ServerUriArray` String[] node to the publisher's events via `BaseDataVariable.OnReadValue` / direct value push. Scoped follow-up on the Opc.Ua.Server stack integration.
|
||||
- `sp_PublishGeneration` wraps its apply in `await using var lease = coordinator.BeginApplyLease(...)` so the `PrimaryMidApply` band (200) fires during actual publishes (task #148 part 2).
|
||||
- Client interop matrix validation — Ignition / Kepware / Aveva OI Gateway (Stream F, task #150). Manual + doc-only work; doesn't block code ship.
|
||||
|
||||
### Remaining drivers (task #120)
|
||||
|
||||
@@ -98,6 +100,7 @@ v2 GA requires all of the following:
|
||||
|
||||
## Change log
|
||||
|
||||
- **2026-04-19** — Release blocker #3 **closed** (PRs #98–99). Phase 6.3 Streams A + C core shipped: `ClusterTopologyLoader` + `RedundancyCoordinator` + `RedundancyStatePublisher` + `PeerReachabilityTracker`. Code-path release blockers all closed; remaining Phase 6.3 surfaces (peer-probe HostedServices, OPC UA variable-node binding, sp_PublishGeneration lease wrap, client interop matrix) are hardening follow-ups.
|
||||
- **2026-04-19** — Release blocker #2 **closed** (PR #96). `SealedBootstrap` consumes `ResilientConfigReader` + `GenerationSealedCache` + `StaleConfigFlag`; `/healthz` now surfaces the stale flag. Remaining follow-ups (periodic poller + richer snapshot payload) downgraded to hardening.
|
||||
- **2026-04-19** — Release blocker #1 **closed** (PR #94). `AuthorizationGate` wired into `DriverNodeManager` Read / Write / HistoryRead dispatch. Remaining Stream C surfaces (Browse / Subscribe / Alarm / Call + finer-grained scope resolution) downgraded to hardening follow-ups — no longer release-blocking.
|
||||
- **2026-04-19** — Phase 6.4 data layer merged (PRs #91–92). Phase 6 core complete. Capstone doc created.
|
||||
|
||||
@@ -27,6 +27,24 @@ public sealed class DriverInstance
|
||||
/// <summary>Schemaless per-driver-type JSON config. Validated against registered JSON schema at draft-publish time (decision #91).</summary>
|
||||
public required string DriverConfig { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional per-instance overrides for the Phase 6.1 shared Polly resilience pipeline.
|
||||
/// Null = use the driver's tier defaults (decision #143). When populated, expected shape:
|
||||
/// <code>
|
||||
/// {
|
||||
/// "bulkheadMaxConcurrent": 16,
|
||||
/// "bulkheadMaxQueue": 64,
|
||||
/// "capabilityPolicies": {
|
||||
/// "Read": { "timeoutSeconds": 5, "retryCount": 5, "breakerFailureThreshold": 3 },
|
||||
/// "Write": { "timeoutSeconds": 5, "retryCount": 0, "breakerFailureThreshold": 5 }
|
||||
/// }
|
||||
/// }
|
||||
/// </code>
|
||||
/// Parsed at startup by <c>DriverResilienceOptionsParser</c>; every key is optional +
|
||||
/// unrecognised keys are ignored so future shapes land without a migration.
|
||||
/// </summary>
|
||||
public string? ResilienceConfig { get; set; }
|
||||
|
||||
public ConfigGeneration? Generation { get; set; }
|
||||
public ServerCluster? Cluster { get; set; }
|
||||
}
|
||||
|
||||
1347
src/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260419161932_AddDriverInstanceResilienceConfig.Designer.cs
generated
Normal file
1347
src/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260419161932_AddDriverInstanceResilienceConfig.Designer.cs
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,37 @@
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class AddDriverInstanceResilienceConfig : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ResilienceConfig",
|
||||
table: "DriverInstance",
|
||||
type: "nvarchar(max)",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddCheckConstraint(
|
||||
name: "CK_DriverInstance_ResilienceConfig_IsJson",
|
||||
table: "DriverInstance",
|
||||
sql: "ResilienceConfig IS NULL OR ISJSON(ResilienceConfig) = 1");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropCheckConstraint(
|
||||
name: "CK_DriverInstance_ResilienceConfig_IsJson",
|
||||
table: "DriverInstance");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ResilienceConfig",
|
||||
table: "DriverInstance");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -413,6 +413,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
||||
.HasMaxLength(64)
|
||||
.HasColumnType("nvarchar(64)");
|
||||
|
||||
b.Property<string>("ResilienceConfig")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.HasKey("DriverInstanceRowId");
|
||||
|
||||
b.HasIndex("ClusterId");
|
||||
@@ -431,6 +434,8 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
||||
b.ToTable("DriverInstance", null, t =>
|
||||
{
|
||||
t.HasCheckConstraint("CK_DriverInstance_DriverConfig_IsJson", "ISJSON(DriverConfig) = 1");
|
||||
|
||||
t.HasCheckConstraint("CK_DriverInstance_ResilienceConfig_IsJson", "ResilienceConfig IS NULL OR ISJSON(ResilienceConfig) = 1");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -251,6 +251,8 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
|
||||
{
|
||||
t.HasCheckConstraint("CK_DriverInstance_DriverConfig_IsJson",
|
||||
"ISJSON(DriverConfig) = 1");
|
||||
t.HasCheckConstraint("CK_DriverInstance_ResilienceConfig_IsJson",
|
||||
"ResilienceConfig IS NULL OR ISJSON(ResilienceConfig) = 1");
|
||||
});
|
||||
e.HasKey(x => x.DriverInstanceRowId);
|
||||
e.Property(x => x.DriverInstanceRowId).HasDefaultValueSql("NEWSEQUENTIALID()");
|
||||
@@ -260,6 +262,7 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
|
||||
e.Property(x => x.Name).HasMaxLength(128);
|
||||
e.Property(x => x.DriverType).HasMaxLength(32);
|
||||
e.Property(x => x.DriverConfig).HasColumnType("nvarchar(max)");
|
||||
e.Property(x => x.ResilienceConfig).HasColumnType("nvarchar(max)");
|
||||
|
||||
e.HasOne(x => x.Generation).WithMany().HasForeignKey(x => x.GenerationId).OnDelete(DeleteBehavior.Restrict);
|
||||
e.HasOne(x => x.Cluster).WithMany().HasForeignKey(x => x.ClusterId).OnDelete(DeleteBehavior.Restrict);
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.OpcUa;
|
||||
|
||||
/// <summary>
|
||||
/// Phase 6.4 Stream D: materializes the OPC 40010 Machinery companion-spec Identification
|
||||
/// sub-folder under an Equipment node. Reads the nine decision-#139 columns off the
|
||||
/// <see cref="Equipment"/> row and emits one property per non-null field.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Pure-function shape — testable without a real OPC UA node manager. The caller
|
||||
/// passes the builder scoped to the Equipment node; this class handles the Identification
|
||||
/// sub-folder creation + per-field <see cref="IAddressSpaceBuilder.AddProperty"/> calls.</para>
|
||||
///
|
||||
/// <para>ACL binding: the sub-folder + its properties inherit the Equipment scope's
|
||||
/// grants (no new scope level). Phase 6.2's trie treats them as part of the Equipment
|
||||
/// ScopeId — a user with Equipment-level grant reads Identification; a user without the
|
||||
/// grant gets BadUserAccessDenied on both the Equipment node + its Identification variables.
|
||||
/// See <c>docs/v2/acl-design.md</c> §Identification cross-reference.</para>
|
||||
///
|
||||
/// <para>The nine fields per decision #139 are exposed exactly when they carry a non-null
|
||||
/// value. A row with all nine null produces no Identification sub-folder at all — the
|
||||
/// caller can use <see cref="HasAnyFields(Equipment)"/> to skip the Folder call entirely
|
||||
/// and avoid a pointless empty folder appearing in browse trees.</para>
|
||||
/// </remarks>
|
||||
public static class IdentificationFolderBuilder
|
||||
{
|
||||
/// <summary>Browse + display name of the sub-folder — fixed per OPC 40010 convention.</summary>
|
||||
public const string FolderName = "Identification";
|
||||
|
||||
/// <summary>
|
||||
/// Canonical decision #139 field set exposed in the Identification sub-folder. Order
|
||||
/// matches the decision-log entry so any browse-order reader can cross-reference
|
||||
/// without re-sorting.
|
||||
/// </summary>
|
||||
public static IReadOnlyList<string> FieldNames { get; } = new[]
|
||||
{
|
||||
"Manufacturer", "Model", "SerialNumber",
|
||||
"HardwareRevision", "SoftwareRevision",
|
||||
"YearOfConstruction", "AssetLocation",
|
||||
"ManufacturerUri", "DeviceManualUri",
|
||||
};
|
||||
|
||||
/// <summary>True when the equipment row has at least one non-null Identification field.</summary>
|
||||
public static bool HasAnyFields(Equipment equipment)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(equipment);
|
||||
return equipment.Manufacturer is not null
|
||||
|| equipment.Model is not null
|
||||
|| equipment.SerialNumber is not null
|
||||
|| equipment.HardwareRevision is not null
|
||||
|| equipment.SoftwareRevision is not null
|
||||
|| equipment.YearOfConstruction is not null
|
||||
|| equipment.AssetLocation is not null
|
||||
|| equipment.ManufacturerUri is not null
|
||||
|| equipment.DeviceManualUri is not null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Build the Identification sub-folder under <paramref name="equipmentBuilder"/>. No-op
|
||||
/// when every field is null. Returns the sub-folder builder (or null when no-op) so
|
||||
/// callers can attach additional nodes underneath if needed.
|
||||
/// </summary>
|
||||
public static IAddressSpaceBuilder? Build(IAddressSpaceBuilder equipmentBuilder, Equipment equipment)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(equipmentBuilder);
|
||||
ArgumentNullException.ThrowIfNull(equipment);
|
||||
|
||||
if (!HasAnyFields(equipment)) return null;
|
||||
|
||||
var folder = equipmentBuilder.Folder(FolderName, FolderName);
|
||||
AddIfPresent(folder, "Manufacturer", DriverDataType.String, equipment.Manufacturer);
|
||||
AddIfPresent(folder, "Model", DriverDataType.String, equipment.Model);
|
||||
AddIfPresent(folder, "SerialNumber", DriverDataType.String, equipment.SerialNumber);
|
||||
AddIfPresent(folder, "HardwareRevision", DriverDataType.String, equipment.HardwareRevision);
|
||||
AddIfPresent(folder, "SoftwareRevision", DriverDataType.String, equipment.SoftwareRevision);
|
||||
AddIfPresent(folder, "YearOfConstruction", DriverDataType.Int32,
|
||||
equipment.YearOfConstruction is null ? null : (object)(int)equipment.YearOfConstruction.Value);
|
||||
AddIfPresent(folder, "AssetLocation", DriverDataType.String, equipment.AssetLocation);
|
||||
AddIfPresent(folder, "ManufacturerUri", DriverDataType.String, equipment.ManufacturerUri);
|
||||
AddIfPresent(folder, "DeviceManualUri", DriverDataType.String, equipment.DeviceManualUri);
|
||||
return folder;
|
||||
}
|
||||
|
||||
private static void AddIfPresent(IAddressSpaceBuilder folder, string name, DriverDataType dataType, object? value)
|
||||
{
|
||||
if (value is null) return;
|
||||
folder.AddProperty(name, dataType, value);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
|
||||
/// <summary>
|
||||
/// Parses the <c>DriverInstance.ResilienceConfig</c> JSON column into a
|
||||
/// <see cref="DriverResilienceOptions"/> instance layered on top of the tier defaults.
|
||||
/// Every key in the JSON is optional; missing keys fall back to the tier defaults from
|
||||
/// <see cref="DriverResilienceOptions.GetTierDefaults(DriverTier)"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Example JSON shape per Phase 6.1 Stream A.2:</para>
|
||||
/// <code>
|
||||
/// {
|
||||
/// "bulkheadMaxConcurrent": 16,
|
||||
/// "bulkheadMaxQueue": 64,
|
||||
/// "capabilityPolicies": {
|
||||
/// "Read": { "timeoutSeconds": 5, "retryCount": 5, "breakerFailureThreshold": 3 },
|
||||
/// "Write": { "timeoutSeconds": 5, "retryCount": 0, "breakerFailureThreshold": 5 }
|
||||
/// }
|
||||
/// }
|
||||
/// </code>
|
||||
///
|
||||
/// <para>Unrecognised keys + values are ignored so future shapes land without a migration.
|
||||
/// Per-capability overrides are layered on top of tier defaults — a partial policy (only
|
||||
/// some of TimeoutSeconds/RetryCount/BreakerFailureThreshold) fills in the other fields
|
||||
/// from the tier default for that capability.</para>
|
||||
///
|
||||
/// <para>Parser failures (malformed JSON, type mismatches) fall back to pure tier defaults
|
||||
/// + surface through an out-parameter diagnostic. Callers may log the diagnostic but should
|
||||
/// NOT fail driver startup — a misconfigured ResilienceConfig should never brick a
|
||||
/// working driver.</para>
|
||||
/// </remarks>
|
||||
public static class DriverResilienceOptionsParser
|
||||
{
|
||||
private static readonly JsonSerializerOptions JsonOpts = new()
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
AllowTrailingCommas = true,
|
||||
ReadCommentHandling = JsonCommentHandling.Skip,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Parse the JSON payload layered on <paramref name="tier"/>'s defaults. Returns the
|
||||
/// effective options; <paramref name="parseDiagnostic"/> is null on success, or a
|
||||
/// human-readable error message when the JSON was malformed (options still returned
|
||||
/// = tier defaults).
|
||||
/// </summary>
|
||||
public static DriverResilienceOptions ParseOrDefaults(
|
||||
DriverTier tier,
|
||||
string? resilienceConfigJson,
|
||||
out string? parseDiagnostic)
|
||||
{
|
||||
parseDiagnostic = null;
|
||||
var baseDefaults = DriverResilienceOptions.GetTierDefaults(tier);
|
||||
var baseOptions = new DriverResilienceOptions { Tier = tier, CapabilityPolicies = baseDefaults };
|
||||
|
||||
if (string.IsNullOrWhiteSpace(resilienceConfigJson))
|
||||
return baseOptions;
|
||||
|
||||
ResilienceConfigShape? shape;
|
||||
try
|
||||
{
|
||||
shape = JsonSerializer.Deserialize<ResilienceConfigShape>(resilienceConfigJson, JsonOpts);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
parseDiagnostic = $"ResilienceConfig JSON malformed; falling back to tier {tier} defaults. Detail: {ex.Message}";
|
||||
return baseOptions;
|
||||
}
|
||||
|
||||
if (shape is null) return baseOptions;
|
||||
|
||||
var merged = new Dictionary<DriverCapability, CapabilityPolicy>(baseDefaults);
|
||||
if (shape.CapabilityPolicies is not null)
|
||||
{
|
||||
foreach (var (capName, overridePolicy) in shape.CapabilityPolicies)
|
||||
{
|
||||
if (!Enum.TryParse<DriverCapability>(capName, ignoreCase: true, out var capability))
|
||||
{
|
||||
parseDiagnostic ??= $"Unknown capability '{capName}' in ResilienceConfig; skipped.";
|
||||
continue;
|
||||
}
|
||||
|
||||
var basePolicy = merged[capability];
|
||||
merged[capability] = new CapabilityPolicy(
|
||||
TimeoutSeconds: overridePolicy.TimeoutSeconds ?? basePolicy.TimeoutSeconds,
|
||||
RetryCount: overridePolicy.RetryCount ?? basePolicy.RetryCount,
|
||||
BreakerFailureThreshold: overridePolicy.BreakerFailureThreshold ?? basePolicy.BreakerFailureThreshold);
|
||||
}
|
||||
}
|
||||
|
||||
return new DriverResilienceOptions
|
||||
{
|
||||
Tier = tier,
|
||||
CapabilityPolicies = merged,
|
||||
BulkheadMaxConcurrent = shape.BulkheadMaxConcurrent ?? baseOptions.BulkheadMaxConcurrent,
|
||||
BulkheadMaxQueue = shape.BulkheadMaxQueue ?? baseOptions.BulkheadMaxQueue,
|
||||
};
|
||||
}
|
||||
|
||||
private sealed class ResilienceConfigShape
|
||||
{
|
||||
public int? BulkheadMaxConcurrent { get; set; }
|
||||
public int? BulkheadMaxQueue { get; set; }
|
||||
public Dictionary<string, CapabilityPolicyShape>? CapabilityPolicies { get; set; }
|
||||
}
|
||||
|
||||
private sealed class CapabilityPolicyShape
|
||||
{
|
||||
public int? TimeoutSeconds { get; set; }
|
||||
public int? RetryCount { get; set; }
|
||||
public int? BreakerFailureThreshold { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Stability;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Hosting;
|
||||
|
||||
/// <summary>
|
||||
/// Drives one or more <see cref="ScheduledRecycleScheduler"/> instances on a fixed tick
|
||||
/// cadence. Closes Phase 6.1 Stream B.4 by turning the shipped-as-pure-logic scheduler
|
||||
/// into a running background feature.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Registered as a singleton in Program.cs. Each Tier C driver instance that wants a
|
||||
/// scheduled recycle registers its scheduler via
|
||||
/// <see cref="AddScheduler(ScheduledRecycleScheduler)"/> at startup. The hosted service
|
||||
/// wakes every <see cref="TickInterval"/> (default 1 min) and calls
|
||||
/// <see cref="ScheduledRecycleScheduler.TickAsync"/> on each registered scheduler.</para>
|
||||
///
|
||||
/// <para>Scheduler registration is closed after <see cref="ExecuteAsync"/> starts — callers
|
||||
/// must register before the host starts, typically during DI setup. Adding a scheduler
|
||||
/// mid-flight throws to avoid confusing "some ticks saw my scheduler, some didn't" races.</para>
|
||||
/// </remarks>
|
||||
public sealed class ScheduledRecycleHostedService : BackgroundService
|
||||
{
|
||||
private readonly List<ScheduledRecycleScheduler> _schedulers = [];
|
||||
private readonly ILogger<ScheduledRecycleHostedService> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private bool _started;
|
||||
|
||||
/// <summary>How often <see cref="ScheduledRecycleScheduler.TickAsync"/> fires on each registered scheduler.</summary>
|
||||
public TimeSpan TickInterval { get; }
|
||||
|
||||
public ScheduledRecycleHostedService(
|
||||
ILogger<ScheduledRecycleHostedService> logger,
|
||||
TimeProvider? timeProvider = null,
|
||||
TimeSpan? tickInterval = null)
|
||||
{
|
||||
_logger = logger;
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
TickInterval = tickInterval ?? TimeSpan.FromMinutes(1);
|
||||
}
|
||||
|
||||
/// <summary>Register a scheduler to drive. Must be called before the host starts.</summary>
|
||||
public void AddScheduler(ScheduledRecycleScheduler scheduler)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(scheduler);
|
||||
if (_started)
|
||||
throw new InvalidOperationException(
|
||||
"Cannot register a ScheduledRecycleScheduler after the hosted service has started. " +
|
||||
"Register all schedulers during DI configuration / startup.");
|
||||
_schedulers.Add(scheduler);
|
||||
}
|
||||
|
||||
/// <summary>Snapshot of the current tick count — diagnostics only.</summary>
|
||||
public int TickCount { get; private set; }
|
||||
|
||||
/// <summary>Snapshot of the number of registered schedulers — diagnostics only.</summary>
|
||||
public int SchedulerCount => _schedulers.Count;
|
||||
|
||||
public override Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
_started = true;
|
||||
return base.StartAsync(cancellationToken);
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"ScheduledRecycleHostedService starting — {Count} scheduler(s), tick interval = {Interval}",
|
||||
_schedulers.Count, TickInterval);
|
||||
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(TickInterval, _timeProvider, stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
await TickOnceAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
_logger.LogInformation("ScheduledRecycleHostedService stopping after {TickCount} tick(s).", TickCount);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Execute one scheduler tick against every registered scheduler. Factored out of the
|
||||
/// <see cref="ExecuteAsync"/> loop so tests can drive it directly without needing to
|
||||
/// synchronize with <see cref="Task.Delay(TimeSpan, TimeProvider, CancellationToken)"/>.
|
||||
/// </summary>
|
||||
public async Task TickOnceAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var now = _timeProvider.GetUtcNow().UtcDateTime;
|
||||
TickCount++;
|
||||
|
||||
foreach (var scheduler in _schedulers)
|
||||
{
|
||||
try
|
||||
{
|
||||
var fired = await scheduler.TickAsync(now, cancellationToken).ConfigureAwait(false);
|
||||
if (fired)
|
||||
_logger.LogInformation("Scheduled recycle fired at {Now:o}; next = {Next:o}",
|
||||
now, scheduler.NextRecycleUtc);
|
||||
}
|
||||
catch (OperationCanceledException) { throw; }
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A single scheduler fault must not take down the rest — log + continue.
|
||||
_logger.LogError(ex,
|
||||
"ScheduledRecycleScheduler tick failed at {Now:o}; continuing to other schedulers.", now);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,8 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
|
||||
private readonly AuthorizationGate? _authzGate;
|
||||
private readonly NodeScopeResolver? _scopeResolver;
|
||||
private readonly StaleConfigFlag? _staleConfigFlag;
|
||||
private readonly Func<string, ZB.MOM.WW.OtOpcUa.Core.Abstractions.DriverTier>? _tierLookup;
|
||||
private readonly Func<string, string?>? _resilienceConfigLookup;
|
||||
private readonly ILoggerFactory _loggerFactory;
|
||||
private readonly ILogger<OpcUaApplicationHost> _logger;
|
||||
private ApplicationInstance? _application;
|
||||
@@ -39,7 +41,9 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
|
||||
DriverResiliencePipelineBuilder? pipelineBuilder = null,
|
||||
AuthorizationGate? authzGate = null,
|
||||
NodeScopeResolver? scopeResolver = null,
|
||||
StaleConfigFlag? staleConfigFlag = null)
|
||||
StaleConfigFlag? staleConfigFlag = null,
|
||||
Func<string, ZB.MOM.WW.OtOpcUa.Core.Abstractions.DriverTier>? tierLookup = null,
|
||||
Func<string, string?>? resilienceConfigLookup = null)
|
||||
{
|
||||
_options = options;
|
||||
_driverHost = driverHost;
|
||||
@@ -48,6 +52,8 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
|
||||
_authzGate = authzGate;
|
||||
_scopeResolver = scopeResolver;
|
||||
_staleConfigFlag = staleConfigFlag;
|
||||
_tierLookup = tierLookup;
|
||||
_resilienceConfigLookup = resilienceConfigLookup;
|
||||
_loggerFactory = loggerFactory;
|
||||
_logger = logger;
|
||||
}
|
||||
@@ -75,7 +81,8 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
|
||||
$"OPC UA application certificate could not be validated or created in {_options.PkiStoreRoot}");
|
||||
|
||||
_server = new OtOpcUaServer(_driverHost, _authenticator, _pipelineBuilder, _loggerFactory,
|
||||
authzGate: _authzGate, scopeResolver: _scopeResolver);
|
||||
authzGate: _authzGate, scopeResolver: _scopeResolver,
|
||||
tierLookup: _tierLookup, resilienceConfigLookup: _resilienceConfigLookup);
|
||||
await _application.Start(_server).ConfigureAwait(false);
|
||||
|
||||
_logger.LogInformation("OPC UA server started — endpoint={Endpoint} driverCount={Count}",
|
||||
|
||||
@@ -23,6 +23,8 @@ public sealed class OtOpcUaServer : StandardServer
|
||||
private readonly DriverResiliencePipelineBuilder _pipelineBuilder;
|
||||
private readonly AuthorizationGate? _authzGate;
|
||||
private readonly NodeScopeResolver? _scopeResolver;
|
||||
private readonly Func<string, DriverTier>? _tierLookup;
|
||||
private readonly Func<string, string?>? _resilienceConfigLookup;
|
||||
private readonly ILoggerFactory _loggerFactory;
|
||||
private readonly List<DriverNodeManager> _driverNodeManagers = new();
|
||||
|
||||
@@ -32,13 +34,17 @@ public sealed class OtOpcUaServer : StandardServer
|
||||
DriverResiliencePipelineBuilder pipelineBuilder,
|
||||
ILoggerFactory loggerFactory,
|
||||
AuthorizationGate? authzGate = null,
|
||||
NodeScopeResolver? scopeResolver = null)
|
||||
NodeScopeResolver? scopeResolver = null,
|
||||
Func<string, DriverTier>? tierLookup = null,
|
||||
Func<string, string?>? resilienceConfigLookup = null)
|
||||
{
|
||||
_driverHost = driverHost;
|
||||
_authenticator = authenticator;
|
||||
_pipelineBuilder = pipelineBuilder;
|
||||
_authzGate = authzGate;
|
||||
_scopeResolver = scopeResolver;
|
||||
_tierLookup = tierLookup;
|
||||
_resilienceConfigLookup = resilienceConfigLookup;
|
||||
_loggerFactory = loggerFactory;
|
||||
}
|
||||
|
||||
@@ -59,10 +65,16 @@ public sealed class OtOpcUaServer : StandardServer
|
||||
if (driver is null) continue;
|
||||
|
||||
var logger = _loggerFactory.CreateLogger<DriverNodeManager>();
|
||||
// Per-driver resilience options: default Tier A pending Stream B.1 which wires
|
||||
// per-type tiers into DriverTypeRegistry. Read ResilienceConfig JSON from the
|
||||
// DriverInstance row in a follow-up PR; for now every driver gets Tier A defaults.
|
||||
var options = new DriverResilienceOptions { Tier = DriverTier.A };
|
||||
// Per-driver resilience options: tier comes from lookup (Phase 6.1 Stream B.1
|
||||
// DriverTypeRegistry in the prod wire-up) or falls back to Tier A. ResilienceConfig
|
||||
// JSON comes from the DriverInstance row via the optional lookup Func; parser
|
||||
// layers JSON overrides on top of tier defaults (Phase 6.1 Stream A.2).
|
||||
var tier = _tierLookup?.Invoke(driver.DriverType) ?? DriverTier.A;
|
||||
var resilienceJson = _resilienceConfigLookup?.Invoke(driver.DriverInstanceId);
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(tier, resilienceJson, out var diag);
|
||||
if (diag is not null)
|
||||
logger.LogWarning("ResilienceConfig parse diagnostic for driver {DriverId}: {Diag}", driver.DriverInstanceId, diag);
|
||||
|
||||
var invoker = new CapabilityInvoker(_pipelineBuilder, driver.DriverInstanceId, () => options, driver.DriverType);
|
||||
var manager = new DriverNodeManager(server, configuration, driver, invoker, logger,
|
||||
authzGate: _authzGate, scopeResolver: _scopeResolver);
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Pure-function mapper from the shared config DB's <see cref="ServerCluster"/> +
|
||||
/// <see cref="ClusterNode"/> rows to an immutable <see cref="RedundancyTopology"/>.
|
||||
/// Validates Phase 6.3 Stream A.1 invariants and throws
|
||||
/// <see cref="InvalidTopologyException"/> on violation so the coordinator can fail startup
|
||||
/// fast with a clear message rather than boot into an ambiguous state.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Stateless — the caller owns the DB round-trip + hands rows in. Keeping it pure makes
|
||||
/// the invariant matrix testable without EF or SQL Server.
|
||||
/// </remarks>
|
||||
public static class ClusterTopologyLoader
|
||||
{
|
||||
/// <summary>Build a topology snapshot for the given self node. Throws on invariant violation.</summary>
|
||||
public static RedundancyTopology Load(string selfNodeId, ServerCluster cluster, IReadOnlyList<ClusterNode> nodes)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(selfNodeId);
|
||||
ArgumentNullException.ThrowIfNull(cluster);
|
||||
ArgumentNullException.ThrowIfNull(nodes);
|
||||
|
||||
ValidateClusterShape(cluster, nodes);
|
||||
ValidateUniqueApplicationUris(nodes);
|
||||
ValidatePrimaryCount(cluster, nodes);
|
||||
|
||||
var self = nodes.FirstOrDefault(n => string.Equals(n.NodeId, selfNodeId, StringComparison.OrdinalIgnoreCase))
|
||||
?? throw new InvalidTopologyException(
|
||||
$"Self node '{selfNodeId}' is not a member of cluster '{cluster.ClusterId}'. " +
|
||||
$"Members: {string.Join(", ", nodes.Select(n => n.NodeId))}.");
|
||||
|
||||
var peers = nodes
|
||||
.Where(n => !string.Equals(n.NodeId, selfNodeId, StringComparison.OrdinalIgnoreCase))
|
||||
.Select(n => new RedundancyPeer(
|
||||
NodeId: n.NodeId,
|
||||
Role: n.RedundancyRole,
|
||||
Host: n.Host,
|
||||
OpcUaPort: n.OpcUaPort,
|
||||
DashboardPort: n.DashboardPort,
|
||||
ApplicationUri: n.ApplicationUri))
|
||||
.ToList();
|
||||
|
||||
return new RedundancyTopology(
|
||||
ClusterId: cluster.ClusterId,
|
||||
SelfNodeId: self.NodeId,
|
||||
SelfRole: self.RedundancyRole,
|
||||
Mode: cluster.RedundancyMode,
|
||||
Peers: peers,
|
||||
SelfApplicationUri: self.ApplicationUri);
|
||||
}
|
||||
|
||||
private static void ValidateClusterShape(ServerCluster cluster, IReadOnlyList<ClusterNode> nodes)
|
||||
{
|
||||
if (nodes.Count == 0)
|
||||
throw new InvalidTopologyException($"Cluster '{cluster.ClusterId}' has zero nodes.");
|
||||
|
||||
// Decision #83 — v2.0 caps clusters at two nodes.
|
||||
if (nodes.Count > 2)
|
||||
throw new InvalidTopologyException(
|
||||
$"Cluster '{cluster.ClusterId}' has {nodes.Count} nodes. v2.0 supports at most 2 nodes per cluster (decision #83).");
|
||||
|
||||
// Every node must belong to the given cluster.
|
||||
var wrongCluster = nodes.FirstOrDefault(n =>
|
||||
!string.Equals(n.ClusterId, cluster.ClusterId, StringComparison.OrdinalIgnoreCase));
|
||||
if (wrongCluster is not null)
|
||||
throw new InvalidTopologyException(
|
||||
$"Node '{wrongCluster.NodeId}' belongs to cluster '{wrongCluster.ClusterId}', not '{cluster.ClusterId}'.");
|
||||
}
|
||||
|
||||
private static void ValidateUniqueApplicationUris(IReadOnlyList<ClusterNode> nodes)
|
||||
{
|
||||
var dup = nodes
|
||||
.GroupBy(n => n.ApplicationUri, StringComparer.Ordinal)
|
||||
.FirstOrDefault(g => g.Count() > 1);
|
||||
if (dup is not null)
|
||||
throw new InvalidTopologyException(
|
||||
$"Nodes {string.Join(", ", dup.Select(n => n.NodeId))} share ApplicationUri '{dup.Key}'. " +
|
||||
$"OPC UA Part 4 requires unique ApplicationUri per server — clients pin trust here (decision #86).");
|
||||
}
|
||||
|
||||
private static void ValidatePrimaryCount(ServerCluster cluster, IReadOnlyList<ClusterNode> nodes)
|
||||
{
|
||||
// Standalone mode: any role is fine. Warm / Hot: at most one Primary per cluster.
|
||||
if (cluster.RedundancyMode == RedundancyMode.None) return;
|
||||
|
||||
var primaries = nodes.Count(n => n.RedundancyRole == RedundancyRole.Primary);
|
||||
if (primaries > 1)
|
||||
throw new InvalidTopologyException(
|
||||
$"Cluster '{cluster.ClusterId}' has {primaries} Primary nodes in redundancy mode {cluster.RedundancyMode}. " +
|
||||
$"At most one Primary per cluster (decision #84). Runtime detects and demotes both to ServiceLevel 2 " +
|
||||
$"per the 8-state matrix; startup fails fast to surface the misconfiguration earlier.");
|
||||
}
|
||||
}
|
||||
42
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/PeerReachability.cs
Normal file
42
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/PeerReachability.cs
Normal file
@@ -0,0 +1,42 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Latest observed reachability of the peer node per the Phase 6.3 Stream B.1/B.2 two-layer
|
||||
/// probe model. HTTP layer is the fast-fail; UA layer is authoritative.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Fed into the <see cref="ServiceLevelCalculator"/> as <c>peerHttpHealthy</c> +
|
||||
/// <c>peerUaHealthy</c>. The concrete probe loops (<c>PeerHttpProbeLoop</c> +
|
||||
/// <c>PeerUaProbeLoop</c>) live in a Stream B runtime follow-up — this type is the
|
||||
/// contract the publisher reads; probers write via
|
||||
/// <see cref="PeerReachabilityTracker"/>.
|
||||
/// </remarks>
|
||||
public sealed record PeerReachability(bool HttpHealthy, bool UaHealthy)
|
||||
{
|
||||
public static readonly PeerReachability Unknown = new(false, false);
|
||||
public static readonly PeerReachability FullyHealthy = new(true, true);
|
||||
|
||||
/// <summary>True when both probes report healthy — the <c>ServiceLevelCalculator</c>'s peerReachable gate.</summary>
|
||||
public bool BothHealthy => HttpHealthy && UaHealthy;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Thread-safe holder of the latest <see cref="PeerReachability"/> per peer NodeId. Probe
|
||||
/// loops call <see cref="Update"/>; the <see cref="RedundancyStatePublisher"/> reads via
|
||||
/// <see cref="Get"/>.
|
||||
/// </summary>
|
||||
public sealed class PeerReachabilityTracker
|
||||
{
|
||||
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, PeerReachability> _byPeer =
|
||||
new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public void Update(string peerNodeId, PeerReachability reachability)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(peerNodeId);
|
||||
_byPeer[peerNodeId] = reachability ?? throw new ArgumentNullException(nameof(reachability));
|
||||
}
|
||||
|
||||
/// <summary>Current reachability for a peer. Returns <see cref="PeerReachability.Unknown"/> when not yet probed.</summary>
|
||||
public PeerReachability Get(string peerNodeId) =>
|
||||
_byPeer.TryGetValue(peerNodeId, out var r) ? r : PeerReachability.Unknown;
|
||||
}
|
||||
107
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/RedundancyCoordinator.cs
Normal file
107
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/RedundancyCoordinator.cs
Normal file
@@ -0,0 +1,107 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Process-singleton holder of the current <see cref="RedundancyTopology"/>. Reads the
|
||||
/// shared config DB at <see cref="InitializeAsync"/> time + re-reads on
|
||||
/// <see cref="RefreshAsync"/> (called after <c>sp_PublishGeneration</c> completes so
|
||||
/// operator role-swaps take effect without a process restart).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Per Phase 6.3 Stream A.1-A.2. The coordinator is the source of truth for the
|
||||
/// <see cref="ServiceLevelCalculator"/> inputs: role (from topology), peer reachability
|
||||
/// (from peer-probe loops — Stream B.1/B.2 follow-up), apply-in-progress (from
|
||||
/// <see cref="ApplyLeaseRegistry"/>), topology-valid (from invariant checks at load time
|
||||
/// + runtime detection of conflicting peer claims).</para>
|
||||
///
|
||||
/// <para>Topology refresh is CAS-style: a new <see cref="RedundancyTopology"/> instance
|
||||
/// replaces the old one atomically via <see cref="Interlocked.Exchange{T}"/>. Readers
|
||||
/// always see a coherent snapshot — never a partial transition.</para>
|
||||
/// </remarks>
|
||||
public sealed class RedundancyCoordinator
|
||||
{
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbContextFactory;
|
||||
private readonly ILogger<RedundancyCoordinator> _logger;
|
||||
private readonly string _selfNodeId;
|
||||
private readonly string _selfClusterId;
|
||||
private RedundancyTopology? _current;
|
||||
private bool _topologyValid = true;
|
||||
|
||||
public RedundancyCoordinator(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> dbContextFactory,
|
||||
ILogger<RedundancyCoordinator> logger,
|
||||
string selfNodeId,
|
||||
string selfClusterId)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(selfNodeId);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(selfClusterId);
|
||||
|
||||
_dbContextFactory = dbContextFactory;
|
||||
_logger = logger;
|
||||
_selfNodeId = selfNodeId;
|
||||
_selfClusterId = selfClusterId;
|
||||
}
|
||||
|
||||
/// <summary>Last-loaded topology; null before <see cref="InitializeAsync"/> completes.</summary>
|
||||
public RedundancyTopology? Current => Volatile.Read(ref _current);
|
||||
|
||||
/// <summary>
|
||||
/// True when the last load/refresh completed without an invariant violation; false
|
||||
/// forces <see cref="ServiceLevelCalculator"/> into the <see cref="ServiceLevelBand.InvalidTopology"/>
|
||||
/// band regardless of other inputs.
|
||||
/// </summary>
|
||||
public bool IsTopologyValid => Volatile.Read(ref _topologyValid);
|
||||
|
||||
/// <summary>Load the topology for the first time. Throws on invariant violation.</summary>
|
||||
public async Task InitializeAsync(CancellationToken ct)
|
||||
{
|
||||
await RefreshInternalAsync(throwOnInvalid: true, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Re-read the topology from the shared DB. Called after <c>sp_PublishGeneration</c>
|
||||
/// completes or after an Admin-triggered role-swap. Never throws — on invariant
|
||||
/// violation it logs + flips <see cref="IsTopologyValid"/> false so the calculator
|
||||
/// returns <see cref="ServiceLevelBand.InvalidTopology"/> = 2.
|
||||
/// </summary>
|
||||
public async Task RefreshAsync(CancellationToken ct)
|
||||
{
|
||||
await RefreshInternalAsync(throwOnInvalid: false, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task RefreshInternalAsync(bool throwOnInvalid, CancellationToken ct)
|
||||
{
|
||||
await using var db = await _dbContextFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
|
||||
|
||||
var cluster = await db.ServerClusters.AsNoTracking()
|
||||
.FirstOrDefaultAsync(c => c.ClusterId == _selfClusterId, ct).ConfigureAwait(false)
|
||||
?? throw new InvalidTopologyException($"Cluster '{_selfClusterId}' not found in config DB.");
|
||||
|
||||
var nodes = await db.ClusterNodes.AsNoTracking()
|
||||
.Where(n => n.ClusterId == _selfClusterId && n.Enabled)
|
||||
.ToListAsync(ct).ConfigureAwait(false);
|
||||
|
||||
try
|
||||
{
|
||||
var topology = ClusterTopologyLoader.Load(_selfNodeId, cluster, nodes);
|
||||
Volatile.Write(ref _current, topology);
|
||||
Volatile.Write(ref _topologyValid, true);
|
||||
_logger.LogInformation(
|
||||
"Redundancy topology loaded: cluster={Cluster} self={Self} role={Role} mode={Mode} peers={PeerCount}",
|
||||
topology.ClusterId, topology.SelfNodeId, topology.SelfRole, topology.Mode, topology.PeerCount);
|
||||
}
|
||||
catch (InvalidTopologyException ex)
|
||||
{
|
||||
Volatile.Write(ref _topologyValid, false);
|
||||
_logger.LogError(ex,
|
||||
"Redundancy topology invariant violation for cluster {Cluster}: {Reason}",
|
||||
_selfClusterId, ex.Message);
|
||||
if (throwOnInvalid) throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Orchestrates Phase 6.3 Stream C: feeds the <see cref="ServiceLevelCalculator"/> with the
|
||||
/// current (topology, peer reachability, apply-in-progress, recovery dwell, self health)
|
||||
/// inputs and emits the resulting <see cref="byte"/> + labelled <see cref="ServiceLevelBand"/>
|
||||
/// to subscribers. The OPC UA <c>ServiceLevel</c> variable node consumes this via
|
||||
/// <see cref="OnStateChanged"/> on every tick.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Pure orchestration — no background timer, no OPC UA stack dep. The caller (a
|
||||
/// HostedService in a future PR, or a test) drives <see cref="ComputeAndPublish"/> at
|
||||
/// whatever cadence is appropriate. Each call reads the inputs + recomputes the ServiceLevel
|
||||
/// byte; state is fired on the <see cref="OnStateChanged"/> event when the byte differs from
|
||||
/// the last emitted value (edge-triggered). The <see cref="OnServerUriArrayChanged"/> event
|
||||
/// fires whenever the topology's <c>ServerUriArray</c> content changes.
|
||||
/// </remarks>
|
||||
public sealed class RedundancyStatePublisher
|
||||
{
|
||||
private readonly RedundancyCoordinator _coordinator;
|
||||
private readonly ApplyLeaseRegistry _leases;
|
||||
private readonly RecoveryStateManager _recovery;
|
||||
private readonly PeerReachabilityTracker _peers;
|
||||
private readonly Func<bool> _selfHealthy;
|
||||
private readonly Func<bool> _operatorMaintenance;
|
||||
private byte _lastByte = 255; // start at Authoritative — harmless before first tick
|
||||
private IReadOnlyList<string>? _lastServerUriArray;
|
||||
|
||||
public RedundancyStatePublisher(
|
||||
RedundancyCoordinator coordinator,
|
||||
ApplyLeaseRegistry leases,
|
||||
RecoveryStateManager recovery,
|
||||
PeerReachabilityTracker peers,
|
||||
Func<bool>? selfHealthy = null,
|
||||
Func<bool>? operatorMaintenance = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(coordinator);
|
||||
ArgumentNullException.ThrowIfNull(leases);
|
||||
ArgumentNullException.ThrowIfNull(recovery);
|
||||
ArgumentNullException.ThrowIfNull(peers);
|
||||
|
||||
_coordinator = coordinator;
|
||||
_leases = leases;
|
||||
_recovery = recovery;
|
||||
_peers = peers;
|
||||
_selfHealthy = selfHealthy ?? (() => true);
|
||||
_operatorMaintenance = operatorMaintenance ?? (() => false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fires with the current ServiceLevel byte + band on every call to
|
||||
/// <see cref="ComputeAndPublish"/> when the byte differs from the previously-emitted one.
|
||||
/// </summary>
|
||||
public event Action<ServiceLevelSnapshot>? OnStateChanged;
|
||||
|
||||
/// <summary>
|
||||
/// Fires when the cluster's ServerUriArray (self + peers) content changes — e.g. an
|
||||
/// operator adds or removes a peer. Consumer is the OPC UA <c>ServerUriArray</c>
|
||||
/// variable node in Stream C.2.
|
||||
/// </summary>
|
||||
public event Action<IReadOnlyList<string>>? OnServerUriArrayChanged;
|
||||
|
||||
/// <summary>Snapshot of the last-published ServiceLevel byte — diagnostics + tests.</summary>
|
||||
public byte LastByte => _lastByte;
|
||||
|
||||
/// <summary>
|
||||
/// Compute the current ServiceLevel + emit change events if anything moved. Caller
|
||||
/// drives cadence — a 1 s tick in production is reasonable; tests drive it directly.
|
||||
/// </summary>
|
||||
public ServiceLevelSnapshot ComputeAndPublish()
|
||||
{
|
||||
var topology = _coordinator.Current;
|
||||
if (topology is null)
|
||||
{
|
||||
// Not yet initialized — surface NoData so clients don't treat us as authoritative.
|
||||
return Emit((byte)ServiceLevelBand.NoData, null);
|
||||
}
|
||||
|
||||
// Aggregate peer reachability. For 2-node v2.0 clusters there is at most one peer;
|
||||
// treat "all peers healthy" as the boolean input to the calculator.
|
||||
var peerReachable = topology.Peers.All(p => _peers.Get(p.NodeId).BothHealthy);
|
||||
var peerUaHealthy = topology.Peers.All(p => _peers.Get(p.NodeId).UaHealthy);
|
||||
var peerHttpHealthy = topology.Peers.All(p => _peers.Get(p.NodeId).HttpHealthy);
|
||||
|
||||
var role = MapRole(topology.SelfRole);
|
||||
|
||||
var value = ServiceLevelCalculator.Compute(
|
||||
role: role,
|
||||
selfHealthy: _selfHealthy(),
|
||||
peerUaHealthy: peerUaHealthy,
|
||||
peerHttpHealthy: peerHttpHealthy,
|
||||
applyInProgress: _leases.IsApplyInProgress,
|
||||
recoveryDwellMet: _recovery.IsDwellMet(),
|
||||
topologyValid: _coordinator.IsTopologyValid,
|
||||
operatorMaintenance: _operatorMaintenance());
|
||||
|
||||
MaybeFireServerUriArray(topology);
|
||||
return Emit(value, topology);
|
||||
}
|
||||
|
||||
private static RedundancyRole MapRole(RedundancyRole role) => role switch
|
||||
{
|
||||
// Standalone is serving; treat as Primary for the matrix since the calculator
|
||||
// already special-cases Standalone inside its Compute.
|
||||
RedundancyRole.Primary => RedundancyRole.Primary,
|
||||
RedundancyRole.Secondary => RedundancyRole.Secondary,
|
||||
_ => RedundancyRole.Standalone,
|
||||
};
|
||||
|
||||
private ServiceLevelSnapshot Emit(byte value, RedundancyTopology? topology)
|
||||
{
|
||||
var snap = new ServiceLevelSnapshot(
|
||||
Value: value,
|
||||
Band: ServiceLevelCalculator.Classify(value),
|
||||
Topology: topology);
|
||||
|
||||
if (value != _lastByte)
|
||||
{
|
||||
_lastByte = value;
|
||||
OnStateChanged?.Invoke(snap);
|
||||
}
|
||||
return snap;
|
||||
}
|
||||
|
||||
private void MaybeFireServerUriArray(RedundancyTopology topology)
|
||||
{
|
||||
var current = topology.ServerUriArray();
|
||||
if (_lastServerUriArray is null || !current.SequenceEqual(_lastServerUriArray, StringComparer.Ordinal))
|
||||
{
|
||||
_lastServerUriArray = current;
|
||||
OnServerUriArrayChanged?.Invoke(current);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Per-tick output of <see cref="RedundancyStatePublisher.ComputeAndPublish"/>.</summary>
|
||||
public sealed record ServiceLevelSnapshot(
|
||||
byte Value,
|
||||
ServiceLevelBand Band,
|
||||
RedundancyTopology? Topology);
|
||||
@@ -0,0 +1,55 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Snapshot of the cluster topology the <see cref="RedundancyCoordinator"/> holds. Read
|
||||
/// once at startup + refreshed on publish-generation notification. Immutable — every
|
||||
/// refresh produces a new instance so observers can compare identity-equality to detect
|
||||
/// topology change.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Per Phase 6.3 Stream A.1. Invariants enforced by the loader (see
|
||||
/// <see cref="ClusterTopologyLoader"/>): at most one Primary per cluster for
|
||||
/// WarmActive/Hot redundancy modes; every node has a unique ApplicationUri (OPC UA
|
||||
/// Part 4 requirement — clients pin trust here); at most 2 nodes total per cluster
|
||||
/// (decision #83).
|
||||
/// </remarks>
|
||||
public sealed record RedundancyTopology(
|
||||
string ClusterId,
|
||||
string SelfNodeId,
|
||||
RedundancyRole SelfRole,
|
||||
RedundancyMode Mode,
|
||||
IReadOnlyList<RedundancyPeer> Peers,
|
||||
string SelfApplicationUri)
|
||||
{
|
||||
/// <summary>Peer count — 0 for a standalone (single-node) cluster, 1 for v2 two-node clusters.</summary>
|
||||
public int PeerCount => Peers.Count;
|
||||
|
||||
/// <summary>
|
||||
/// ServerUriArray shape per OPC UA Part 4 §6.6.2.2 — self first, peers in stable
|
||||
/// deterministic order (lexicographic by NodeId), self's ApplicationUri always at index 0.
|
||||
/// </summary>
|
||||
public IReadOnlyList<string> ServerUriArray() =>
|
||||
new[] { SelfApplicationUri }
|
||||
.Concat(Peers.OrderBy(p => p.NodeId, StringComparer.OrdinalIgnoreCase).Select(p => p.ApplicationUri))
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>One peer in the cluster (every node other than self).</summary>
|
||||
/// <param name="NodeId">Peer's stable logical NodeId (e.g. <c>"LINE3-OPCUA-B"</c>).</param>
|
||||
/// <param name="Role">Peer's declared redundancy role from the shared config DB.</param>
|
||||
/// <param name="Host">Peer's hostname / IP — drives the health-probe target.</param>
|
||||
/// <param name="OpcUaPort">Peer's OPC UA endpoint port.</param>
|
||||
/// <param name="DashboardPort">Peer's dashboard / health-endpoint port.</param>
|
||||
/// <param name="ApplicationUri">Peer's declared ApplicationUri (carried in <see cref="RedundancyTopology.ServerUriArray"/>).</param>
|
||||
public sealed record RedundancyPeer(
|
||||
string NodeId,
|
||||
RedundancyRole Role,
|
||||
string Host,
|
||||
int OpcUaPort,
|
||||
int DashboardPort,
|
||||
string ApplicationUri);
|
||||
|
||||
/// <summary>Thrown when the loader detects a topology-invariant violation at startup or refresh.</summary>
|
||||
public sealed class InvalidTopologyException(string message) : Exception(message);
|
||||
@@ -78,6 +78,7 @@ WHERE i.is_unique = 1 AND i.has_filter = 1;",
|
||||
"CK_ServerCluster_RedundancyMode_NodeCount",
|
||||
"CK_Device_DeviceConfig_IsJson",
|
||||
"CK_DriverInstance_DriverConfig_IsJson",
|
||||
"CK_DriverInstance_ResilienceConfig_IsJson",
|
||||
"CK_PollGroup_IntervalMs_Min",
|
||||
"CK_Tag_TagConfig_IsJson",
|
||||
"CK_ConfigAuditLog_DetailsJson_IsJson",
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.OpcUa;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class IdentificationFolderBuilderTests
|
||||
{
|
||||
private sealed class RecordingBuilder : IAddressSpaceBuilder
|
||||
{
|
||||
public List<(string BrowseName, string DisplayName)> Folders { get; } = [];
|
||||
public List<(string BrowseName, DriverDataType DataType, object? Value)> Properties { get; } = [];
|
||||
|
||||
public IAddressSpaceBuilder Folder(string browseName, string displayName)
|
||||
{
|
||||
Folders.Add((browseName, displayName));
|
||||
return this; // flat recording — identification fields land in the same bucket
|
||||
}
|
||||
|
||||
public IVariableHandle Variable(string browseName, string displayName, DriverAttributeInfo attributeInfo)
|
||||
=> throw new NotSupportedException("Identification fields use AddProperty, not Variable");
|
||||
|
||||
public void AddProperty(string browseName, DriverDataType dataType, object? value)
|
||||
=> Properties.Add((browseName, dataType, value));
|
||||
}
|
||||
|
||||
private static Equipment EmptyEquipment() => new()
|
||||
{
|
||||
EquipmentId = "EQ-000000000001",
|
||||
DriverInstanceId = "drv-1",
|
||||
UnsLineId = "line-1",
|
||||
Name = "eq-1",
|
||||
MachineCode = "machine_001",
|
||||
};
|
||||
|
||||
private static Equipment FullyPopulatedEquipment() => new()
|
||||
{
|
||||
EquipmentId = "EQ-000000000001",
|
||||
DriverInstanceId = "drv-1",
|
||||
UnsLineId = "line-1",
|
||||
Name = "eq-1",
|
||||
MachineCode = "machine_001",
|
||||
Manufacturer = "Siemens",
|
||||
Model = "S7-1500",
|
||||
SerialNumber = "SN-12345",
|
||||
HardwareRevision = "Rev-A",
|
||||
SoftwareRevision = "Fw-2.3.1",
|
||||
YearOfConstruction = 2023,
|
||||
AssetLocation = "Warsaw-West/Bldg-3",
|
||||
ManufacturerUri = "https://siemens.example",
|
||||
DeviceManualUri = "https://siemens.example/manual",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void HasAnyFields_AllNull_ReturnsFalse()
|
||||
{
|
||||
IdentificationFolderBuilder.HasAnyFields(EmptyEquipment()).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HasAnyFields_OneNonNull_ReturnsTrue()
|
||||
{
|
||||
var eq = EmptyEquipment();
|
||||
eq.SerialNumber = "SN-1";
|
||||
IdentificationFolderBuilder.HasAnyFields(eq).ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Build_AllNull_ReturnsNull_AndDoesNotEmit_Folder()
|
||||
{
|
||||
var builder = new RecordingBuilder();
|
||||
|
||||
var result = IdentificationFolderBuilder.Build(builder, EmptyEquipment());
|
||||
|
||||
result.ShouldBeNull();
|
||||
builder.Folders.ShouldBeEmpty("no Identification folder when every field is null");
|
||||
builder.Properties.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Build_FullyPopulated_EmitsAllNineFields()
|
||||
{
|
||||
var builder = new RecordingBuilder();
|
||||
|
||||
var result = IdentificationFolderBuilder.Build(builder, FullyPopulatedEquipment());
|
||||
|
||||
result.ShouldNotBeNull();
|
||||
builder.Folders.ShouldContain(f => f.BrowseName == "Identification");
|
||||
builder.Properties.Count.ShouldBe(9);
|
||||
builder.Properties.Select(p => p.BrowseName).ShouldBe(
|
||||
["Manufacturer", "Model", "SerialNumber",
|
||||
"HardwareRevision", "SoftwareRevision",
|
||||
"YearOfConstruction", "AssetLocation",
|
||||
"ManufacturerUri", "DeviceManualUri"],
|
||||
"property order matches decision #139 exactly");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Build_OnlyNonNull_Are_Emitted()
|
||||
{
|
||||
var eq = EmptyEquipment();
|
||||
eq.Manufacturer = "Siemens";
|
||||
eq.SerialNumber = "SN-1";
|
||||
eq.YearOfConstruction = 2024;
|
||||
var builder = new RecordingBuilder();
|
||||
|
||||
IdentificationFolderBuilder.Build(builder, eq);
|
||||
|
||||
builder.Properties.Count.ShouldBe(3, "only the 3 non-null fields are exposed");
|
||||
builder.Properties.Select(p => p.BrowseName).ShouldBe(
|
||||
["Manufacturer", "SerialNumber", "YearOfConstruction"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void YearOfConstruction_Maps_Short_To_Int32_DriverDataType()
|
||||
{
|
||||
var eq = EmptyEquipment();
|
||||
eq.YearOfConstruction = 2023;
|
||||
var builder = new RecordingBuilder();
|
||||
|
||||
IdentificationFolderBuilder.Build(builder, eq);
|
||||
|
||||
var prop = builder.Properties.Single(p => p.BrowseName == "YearOfConstruction");
|
||||
prop.DataType.ShouldBe(DriverDataType.Int32);
|
||||
prop.Value.ShouldBe(2023, "short is widened to int for OPC UA Int32 representation");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Build_StringValues_RoundTrip()
|
||||
{
|
||||
var eq = FullyPopulatedEquipment();
|
||||
var builder = new RecordingBuilder();
|
||||
|
||||
IdentificationFolderBuilder.Build(builder, eq);
|
||||
|
||||
builder.Properties.Single(p => p.BrowseName == "Manufacturer").Value.ShouldBe("Siemens");
|
||||
builder.Properties.Single(p => p.BrowseName == "DeviceManualUri").Value.ShouldBe("https://siemens.example/manual");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FieldNames_Match_Decision139_Exactly()
|
||||
{
|
||||
IdentificationFolderBuilder.FieldNames.ShouldBe(
|
||||
["Manufacturer", "Model", "SerialNumber",
|
||||
"HardwareRevision", "SoftwareRevision",
|
||||
"YearOfConstruction", "AssetLocation",
|
||||
"ManufacturerUri", "DeviceManualUri"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FolderName_Is_Identification()
|
||||
{
|
||||
IdentificationFolderBuilder.FolderName.ShouldBe("Identification");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Resilience;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class DriverResilienceOptionsParserTests
|
||||
{
|
||||
[Fact]
|
||||
public void NullJson_ReturnsPureTierDefaults()
|
||||
{
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, null, out var diag);
|
||||
|
||||
diag.ShouldBeNull();
|
||||
options.Tier.ShouldBe(DriverTier.A);
|
||||
options.Resolve(DriverCapability.Read).ShouldBe(
|
||||
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhitespaceJson_ReturnsDefaults()
|
||||
{
|
||||
DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.B, " ", out var diag);
|
||||
diag.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MalformedJson_FallsBack_WithDiagnostic()
|
||||
{
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, "{not json", out var diag);
|
||||
|
||||
diag.ShouldNotBeNull();
|
||||
diag.ShouldContain("malformed");
|
||||
options.Tier.ShouldBe(DriverTier.A);
|
||||
options.Resolve(DriverCapability.Read).ShouldBe(
|
||||
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EmptyObject_ReturnsDefaults()
|
||||
{
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, "{}", out var diag);
|
||||
|
||||
diag.ShouldBeNull();
|
||||
options.Resolve(DriverCapability.Write).ShouldBe(
|
||||
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Write]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReadOverride_MergedIntoTierDefaults()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"capabilityPolicies": {
|
||||
"Read": { "timeoutSeconds": 5, "retryCount": 7, "breakerFailureThreshold": 2 }
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, json, out var diag);
|
||||
|
||||
diag.ShouldBeNull();
|
||||
var read = options.Resolve(DriverCapability.Read);
|
||||
read.TimeoutSeconds.ShouldBe(5);
|
||||
read.RetryCount.ShouldBe(7);
|
||||
read.BreakerFailureThreshold.ShouldBe(2);
|
||||
|
||||
// Other capabilities untouched
|
||||
options.Resolve(DriverCapability.Write).ShouldBe(
|
||||
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Write]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PartialPolicy_FillsMissingFieldsFromTierDefault()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"capabilityPolicies": {
|
||||
"Read": { "retryCount": 10 }
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, json, out _);
|
||||
|
||||
var read = options.Resolve(DriverCapability.Read);
|
||||
var tierDefault = DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read];
|
||||
read.RetryCount.ShouldBe(10);
|
||||
read.TimeoutSeconds.ShouldBe(tierDefault.TimeoutSeconds, "partial override; timeout falls back to tier default");
|
||||
read.BreakerFailureThreshold.ShouldBe(tierDefault.BreakerFailureThreshold);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BulkheadOverrides_AreHonored()
|
||||
{
|
||||
var json = """
|
||||
{ "bulkheadMaxConcurrent": 100, "bulkheadMaxQueue": 500 }
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.B, json, out _);
|
||||
|
||||
options.BulkheadMaxConcurrent.ShouldBe(100);
|
||||
options.BulkheadMaxQueue.ShouldBe(500);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnknownCapability_Surfaces_InDiagnostic_ButDoesNotFail()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"capabilityPolicies": {
|
||||
"InventedCapability": { "timeoutSeconds": 99 }
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, json, out var diag);
|
||||
|
||||
diag.ShouldNotBeNull();
|
||||
diag.ShouldContain("InventedCapability");
|
||||
// Known capabilities untouched.
|
||||
options.Resolve(DriverCapability.Read).ShouldBe(
|
||||
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PropertyNames_AreCaseInsensitive()
|
||||
{
|
||||
var json = """
|
||||
{ "BULKHEADMAXCONCURRENT": 42 }
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, json, out _);
|
||||
|
||||
options.BulkheadMaxConcurrent.ShouldBe(42);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CapabilityName_IsCaseInsensitive()
|
||||
{
|
||||
var json = """
|
||||
{ "capabilityPolicies": { "read": { "retryCount": 99 } } }
|
||||
""";
|
||||
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(DriverTier.A, json, out var diag);
|
||||
|
||||
diag.ShouldBeNull();
|
||||
options.Resolve(DriverCapability.Read).RetryCount.ShouldBe(99);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(DriverTier.A)]
|
||||
[InlineData(DriverTier.B)]
|
||||
[InlineData(DriverTier.C)]
|
||||
public void EveryTier_WithEmptyJson_RoundTrips_Its_Defaults(DriverTier tier)
|
||||
{
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(tier, "{}", out var diag);
|
||||
|
||||
diag.ShouldBeNull();
|
||||
options.Tier.ShouldBe(tier);
|
||||
foreach (var cap in Enum.GetValues<DriverCapability>())
|
||||
options.Resolve(cap).ShouldBe(DriverResilienceOptions.GetTierDefaults(tier)[cap]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Tests;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class ClusterTopologyLoaderTests
|
||||
{
|
||||
private static ServerCluster Cluster(RedundancyMode mode = RedundancyMode.Warm) => new()
|
||||
{
|
||||
ClusterId = "c1",
|
||||
Name = "Warsaw-West",
|
||||
Enterprise = "zb",
|
||||
Site = "warsaw-west",
|
||||
RedundancyMode = mode,
|
||||
CreatedBy = "test",
|
||||
};
|
||||
|
||||
private static ClusterNode Node(string id, RedundancyRole role, string host, int port = 4840, string? appUri = null) => new()
|
||||
{
|
||||
NodeId = id,
|
||||
ClusterId = "c1",
|
||||
RedundancyRole = role,
|
||||
Host = host,
|
||||
OpcUaPort = port,
|
||||
ApplicationUri = appUri ?? $"urn:{host}:OtOpcUa",
|
||||
CreatedBy = "test",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void SingleNode_Standalone_Loads()
|
||||
{
|
||||
var cluster = Cluster(RedundancyMode.None);
|
||||
var nodes = new[] { Node("A", RedundancyRole.Standalone, "hostA") };
|
||||
|
||||
var topology = ClusterTopologyLoader.Load("A", cluster, nodes);
|
||||
|
||||
topology.SelfNodeId.ShouldBe("A");
|
||||
topology.SelfRole.ShouldBe(RedundancyRole.Standalone);
|
||||
topology.Peers.ShouldBeEmpty();
|
||||
topology.SelfApplicationUri.ShouldBe("urn:hostA:OtOpcUa");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TwoNode_Cluster_LoadsSelfAndPeer()
|
||||
{
|
||||
var cluster = Cluster();
|
||||
var nodes = new[]
|
||||
{
|
||||
Node("A", RedundancyRole.Primary, "hostA"),
|
||||
Node("B", RedundancyRole.Secondary, "hostB"),
|
||||
};
|
||||
|
||||
var topology = ClusterTopologyLoader.Load("A", cluster, nodes);
|
||||
|
||||
topology.SelfNodeId.ShouldBe("A");
|
||||
topology.SelfRole.ShouldBe(RedundancyRole.Primary);
|
||||
topology.Peers.Count.ShouldBe(1);
|
||||
topology.Peers[0].NodeId.ShouldBe("B");
|
||||
topology.Peers[0].Role.ShouldBe(RedundancyRole.Secondary);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ServerUriArray_Puts_Self_First_Peers_SortedLexicographically()
|
||||
{
|
||||
var cluster = Cluster();
|
||||
var nodes = new[]
|
||||
{
|
||||
Node("A", RedundancyRole.Primary, "hostA", appUri: "urn:A"),
|
||||
Node("B", RedundancyRole.Secondary, "hostB", appUri: "urn:B"),
|
||||
};
|
||||
|
||||
var topology = ClusterTopologyLoader.Load("A", cluster, nodes);
|
||||
|
||||
topology.ServerUriArray().ShouldBe(["urn:A", "urn:B"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EmptyNodes_Throws()
|
||||
{
|
||||
Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A", Cluster(), []));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfNotInCluster_Throws()
|
||||
{
|
||||
var nodes = new[] { Node("B", RedundancyRole.Primary, "hostB") };
|
||||
|
||||
Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A-missing", Cluster(), nodes));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ThreeNodeCluster_Rejected_Per_Decision83()
|
||||
{
|
||||
var nodes = new[]
|
||||
{
|
||||
Node("A", RedundancyRole.Primary, "hostA"),
|
||||
Node("B", RedundancyRole.Secondary, "hostB"),
|
||||
Node("C", RedundancyRole.Secondary, "hostC"),
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A", Cluster(), nodes));
|
||||
ex.Message.ShouldContain("decision #83");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DuplicateApplicationUri_Rejected()
|
||||
{
|
||||
var nodes = new[]
|
||||
{
|
||||
Node("A", RedundancyRole.Primary, "hostA", appUri: "urn:shared"),
|
||||
Node("B", RedundancyRole.Secondary, "hostB", appUri: "urn:shared"),
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A", Cluster(), nodes));
|
||||
ex.Message.ShouldContain("ApplicationUri");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TwoPrimaries_InWarmMode_Rejected()
|
||||
{
|
||||
var nodes = new[]
|
||||
{
|
||||
Node("A", RedundancyRole.Primary, "hostA"),
|
||||
Node("B", RedundancyRole.Primary, "hostB"),
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A", Cluster(RedundancyMode.Warm), nodes));
|
||||
ex.Message.ShouldContain("2 Primary");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CrossCluster_Node_Rejected()
|
||||
{
|
||||
var foreign = Node("B", RedundancyRole.Secondary, "hostB");
|
||||
foreign.ClusterId = "c-other";
|
||||
|
||||
var nodes = new[] { Node("A", RedundancyRole.Primary, "hostA"), foreign };
|
||||
|
||||
Should.Throw<InvalidTopologyException>(
|
||||
() => ClusterTopologyLoader.Load("A", Cluster(), nodes));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void None_Mode_Allows_Any_Role_Mix()
|
||||
{
|
||||
// Standalone clusters don't enforce Primary-count; operator can pick anything.
|
||||
var cluster = Cluster(RedundancyMode.None);
|
||||
var nodes = new[] { Node("A", RedundancyRole.Primary, "hostA") };
|
||||
|
||||
var topology = ClusterTopologyLoader.Load("A", cluster, nodes);
|
||||
|
||||
topology.Mode.ShouldBe(RedundancyMode.None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.Server.Redundancy;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Tests;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class RedundancyStatePublisherTests : IDisposable
|
||||
{
|
||||
private readonly OtOpcUaConfigDbContext _db;
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||
|
||||
public RedundancyStatePublisherTests()
|
||||
{
|
||||
var options = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||
.UseInMemoryDatabase($"redundancy-publisher-{Guid.NewGuid():N}")
|
||||
.Options;
|
||||
_db = new OtOpcUaConfigDbContext(options);
|
||||
_dbFactory = new DbContextFactory(options);
|
||||
}
|
||||
|
||||
public void Dispose() => _db.Dispose();
|
||||
|
||||
private sealed class DbContextFactory(DbContextOptions<OtOpcUaConfigDbContext> options)
|
||||
: IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext() => new(options);
|
||||
}
|
||||
|
||||
private async Task<RedundancyCoordinator> SeedAndInitialize(string selfNodeId, params (string id, RedundancyRole role, string appUri)[] nodes)
|
||||
{
|
||||
var cluster = new ServerCluster
|
||||
{
|
||||
ClusterId = "c1",
|
||||
Name = "Warsaw-West",
|
||||
Enterprise = "zb",
|
||||
Site = "warsaw-west",
|
||||
RedundancyMode = nodes.Length == 1 ? RedundancyMode.None : RedundancyMode.Warm,
|
||||
CreatedBy = "test",
|
||||
};
|
||||
_db.ServerClusters.Add(cluster);
|
||||
foreach (var (id, role, appUri) in nodes)
|
||||
{
|
||||
_db.ClusterNodes.Add(new ClusterNode
|
||||
{
|
||||
NodeId = id,
|
||||
ClusterId = "c1",
|
||||
RedundancyRole = role,
|
||||
Host = id.ToLowerInvariant(),
|
||||
ApplicationUri = appUri,
|
||||
CreatedBy = "test",
|
||||
});
|
||||
}
|
||||
await _db.SaveChangesAsync();
|
||||
|
||||
var coordinator = new RedundancyCoordinator(_dbFactory, NullLogger<RedundancyCoordinator>.Instance, selfNodeId, "c1");
|
||||
await coordinator.InitializeAsync(CancellationToken.None);
|
||||
return coordinator;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BeforeInit_Publishes_NoData()
|
||||
{
|
||||
// Coordinator not initialized — current topology is null.
|
||||
var coordinator = new RedundancyCoordinator(_dbFactory, NullLogger<RedundancyCoordinator>.Instance, "A", "c1");
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), new PeerReachabilityTracker());
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Band.ShouldBe(ServiceLevelBand.NoData);
|
||||
snap.Value.ShouldBe((byte)1);
|
||||
await Task.Yield();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AuthoritativePrimary_WhenHealthyAndPeerReachable()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.FullyHealthy);
|
||||
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), peers);
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Value.ShouldBe((byte)255);
|
||||
snap.Band.ShouldBe(ServiceLevelBand.AuthoritativePrimary);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task IsolatedPrimary_WhenPeerUnreachable_RetainsAuthority()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.Unknown);
|
||||
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), peers);
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Value.ShouldBe((byte)230);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MidApply_WhenLeaseOpen_Dominates()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var leases = new ApplyLeaseRegistry();
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.FullyHealthy);
|
||||
|
||||
await using var lease = leases.BeginApplyLease(1, Guid.NewGuid());
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, leases, new RecoveryStateManager(), peers);
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Value.ShouldBe((byte)200);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SelfUnhealthy_Returns_NoData()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.FullyHealthy);
|
||||
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), peers,
|
||||
selfHealthy: () => false);
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Value.ShouldBe((byte)1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task OnStateChanged_FiresOnly_OnValueChange()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.FullyHealthy);
|
||||
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), peers);
|
||||
|
||||
var emitCount = 0;
|
||||
byte? lastEmitted = null;
|
||||
publisher.OnStateChanged += snap => { emitCount++; lastEmitted = snap.Value; };
|
||||
|
||||
publisher.ComputeAndPublish(); // first tick — emits 255 since _lastByte was seeded at 255; no change
|
||||
peers.Update("B", PeerReachability.Unknown);
|
||||
publisher.ComputeAndPublish(); // 255 → 230 transition — emits
|
||||
publisher.ComputeAndPublish(); // still 230 — no emit
|
||||
|
||||
emitCount.ShouldBe(1);
|
||||
lastEmitted.ShouldBe((byte)230);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task OnServerUriArrayChanged_FiresOnce_PerTopology()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Primary, "urn:A"),
|
||||
("B", RedundancyRole.Secondary, "urn:B"));
|
||||
var peers = new PeerReachabilityTracker();
|
||||
peers.Update("B", PeerReachability.FullyHealthy);
|
||||
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), peers);
|
||||
|
||||
var emits = new List<IReadOnlyList<string>>();
|
||||
publisher.OnServerUriArrayChanged += arr => emits.Add(arr);
|
||||
|
||||
publisher.ComputeAndPublish();
|
||||
publisher.ComputeAndPublish();
|
||||
publisher.ComputeAndPublish();
|
||||
|
||||
emits.Count.ShouldBe(1, "ServerUriArray event is edge-triggered on topology content change");
|
||||
emits[0].ShouldBe(["urn:A", "urn:B"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Standalone_Cluster_IsAuthoritative_When_Healthy()
|
||||
{
|
||||
var coordinator = await SeedAndInitialize("A",
|
||||
("A", RedundancyRole.Standalone, "urn:A"));
|
||||
var publisher = new RedundancyStatePublisher(
|
||||
coordinator, new ApplyLeaseRegistry(), new RecoveryStateManager(), new PeerReachabilityTracker());
|
||||
|
||||
var snap = publisher.ComputeAndPublish();
|
||||
|
||||
snap.Value.ShouldBe((byte)255);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Stability;
|
||||
using ZB.MOM.WW.OtOpcUa.Server.Hosting;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Server.Tests;
|
||||
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class ScheduledRecycleHostedServiceTests
|
||||
{
|
||||
private static readonly DateTime T0 = new(2026, 4, 19, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private sealed class FakeClock : TimeProvider
|
||||
{
|
||||
public DateTime Utc { get; set; } = T0;
|
||||
public override DateTimeOffset GetUtcNow() => new(Utc, TimeSpan.Zero);
|
||||
}
|
||||
|
||||
private sealed class FakeSupervisor : IDriverSupervisor
|
||||
{
|
||||
public string DriverInstanceId => "tier-c-fake";
|
||||
public int RecycleCount { get; private set; }
|
||||
public Task RecycleAsync(string reason, CancellationToken cancellationToken)
|
||||
{
|
||||
RecycleCount++;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class ThrowingSupervisor : IDriverSupervisor
|
||||
{
|
||||
public string DriverInstanceId => "tier-c-throws";
|
||||
public Task RecycleAsync(string reason, CancellationToken cancellationToken)
|
||||
=> throw new InvalidOperationException("supervisor unavailable");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TickOnce_BeforeInterval_DoesNotFire()
|
||||
{
|
||||
var clock = new FakeClock();
|
||||
var supervisor = new FakeSupervisor();
|
||||
var scheduler = new ScheduledRecycleScheduler(
|
||||
DriverTier.C, TimeSpan.FromMinutes(5), T0, supervisor,
|
||||
NullLogger<ScheduledRecycleScheduler>.Instance);
|
||||
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance, clock);
|
||||
host.AddScheduler(scheduler);
|
||||
|
||||
clock.Utc = T0.AddMinutes(1);
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
|
||||
supervisor.RecycleCount.ShouldBe(0);
|
||||
host.TickCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TickOnce_AfterInterval_Fires()
|
||||
{
|
||||
var clock = new FakeClock();
|
||||
var supervisor = new FakeSupervisor();
|
||||
var scheduler = new ScheduledRecycleScheduler(
|
||||
DriverTier.C, TimeSpan.FromMinutes(5), T0, supervisor,
|
||||
NullLogger<ScheduledRecycleScheduler>.Instance);
|
||||
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance, clock);
|
||||
host.AddScheduler(scheduler);
|
||||
|
||||
clock.Utc = T0.AddMinutes(6);
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
|
||||
supervisor.RecycleCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TickOnce_MultipleTicks_AccumulateCount()
|
||||
{
|
||||
var clock = new FakeClock();
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance, clock);
|
||||
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
|
||||
host.TickCount.ShouldBe(3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AddScheduler_AfterStart_Throws()
|
||||
{
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance);
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.Cancel();
|
||||
|
||||
await host.StartAsync(cts.Token); // flips _started true even with cancelled token
|
||||
await host.StopAsync(CancellationToken.None);
|
||||
|
||||
var scheduler = new ScheduledRecycleScheduler(
|
||||
DriverTier.C, TimeSpan.FromMinutes(5), DateTime.UtcNow, new FakeSupervisor(),
|
||||
NullLogger<ScheduledRecycleScheduler>.Instance);
|
||||
|
||||
Should.Throw<InvalidOperationException>(() => host.AddScheduler(scheduler));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task OneSchedulerThrowing_DoesNotStopOthers()
|
||||
{
|
||||
var clock = new FakeClock();
|
||||
var good = new FakeSupervisor();
|
||||
var bad = new ThrowingSupervisor();
|
||||
|
||||
var goodSch = new ScheduledRecycleScheduler(
|
||||
DriverTier.C, TimeSpan.FromMinutes(5), T0, good,
|
||||
NullLogger<ScheduledRecycleScheduler>.Instance);
|
||||
var badSch = new ScheduledRecycleScheduler(
|
||||
DriverTier.C, TimeSpan.FromMinutes(5), T0, bad,
|
||||
NullLogger<ScheduledRecycleScheduler>.Instance);
|
||||
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance, clock);
|
||||
host.AddScheduler(badSch);
|
||||
host.AddScheduler(goodSch);
|
||||
|
||||
clock.Utc = T0.AddMinutes(6);
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
|
||||
good.RecycleCount.ShouldBe(1, "a faulting scheduler must not poison its neighbours");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SchedulerCount_MatchesAdded()
|
||||
{
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance);
|
||||
var sup = new FakeSupervisor();
|
||||
host.AddScheduler(new ScheduledRecycleScheduler(DriverTier.C, TimeSpan.FromMinutes(5), DateTime.UtcNow, sup, NullLogger<ScheduledRecycleScheduler>.Instance));
|
||||
host.AddScheduler(new ScheduledRecycleScheduler(DriverTier.C, TimeSpan.FromMinutes(10), DateTime.UtcNow, sup, NullLogger<ScheduledRecycleScheduler>.Instance));
|
||||
|
||||
host.SchedulerCount.ShouldBe(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptyScheduler_List_TicksCleanly()
|
||||
{
|
||||
var clock = new FakeClock();
|
||||
var host = new ScheduledRecycleHostedService(NullLogger<ScheduledRecycleHostedService>.Instance, clock);
|
||||
|
||||
// No registered schedulers — tick is a no-op + counter still advances.
|
||||
await host.TickOnceAsync(CancellationToken.None);
|
||||
|
||||
host.TickCount.ShouldBe(1);
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@
|
||||
<PackageReference Include="Shouldly" Version="4.3.0"/>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0"/>
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0"/>
|
||||
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.0"/>
|
||||
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Client" Version="1.5.374.126"/>
|
||||
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.2">
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
|
||||
Reference in New Issue
Block a user