30d0697c28
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
130 lines
6.1 KiB
C#
130 lines
6.1 KiB
C#
using Akka.Actor;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
|
|
|
/// <summary>
|
|
/// Gitea #488 — the periodic desired-vs-actual subscription reconcile that rides
|
|
/// <see cref="DriverInstanceActor"/>'s existing <c>health-poll</c> tick.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// The hole it closes: the desired ref set is re-applied on a <b>Connected transition</b> and on a
|
|
/// deploy's <c>SetDesiredSubscriptions</c>, and nowhere else. A subscription lost while the driver
|
|
/// <i>stayed</i> Connected — the shape reproduced here, a subscribe that failed and was never
|
|
/// retried — left the actor permanently subscribed to nothing while reporting Healthy.
|
|
/// </para>
|
|
/// <para>
|
|
/// All three tests drive a short <c>healthPollInterval</c> rather than the production 30 s. The
|
|
/// two absence assertions are bounded by a positive control (a counted number of ticks provably
|
|
/// processed), never by a bare sleep.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class DriverInstanceActorSubscriptionReconcileTests : RuntimeActorTestBase
|
|
{
|
|
private static readonly TimeSpan FastPoll = TimeSpan.FromMilliseconds(100);
|
|
|
|
/// <summary>
|
|
/// The reconcile itself: the first subscribe fails, leaving the actor Connected with a desired set
|
|
/// and no handle. Nothing else would ever retry it — no reconnect, no deploy — so a second
|
|
/// subscribe can only come from the periodic reconcile.
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_lost_subscription_is_re_established_while_still_connected()
|
|
{
|
|
var driver = new SubscribableStubDriver { SubscribeFailuresBeforeSuccess = 1 };
|
|
var actor = Sys.ActorOf(DriverInstanceActor.Props(driver, healthPollInterval: FastPoll));
|
|
|
|
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
|
|
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
|
|
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
|
|
|
|
// The connect-path subscribe runs and throws, so the handle stays null.
|
|
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
|
|
|
|
// Only the reconcile can produce this second attempt — and it must succeed this time.
|
|
AwaitCondition(() => driver.SubscribeCount >= 2, PresenceBudget);
|
|
driver.LastSubscribedRefs.ShouldBe(new[] { "tag-a" });
|
|
}
|
|
|
|
/// <summary>
|
|
/// A healthy subscription is left alone — the reconcile must not re-subscribe on every tick, which
|
|
/// would churn the backend every 30 s in production for no reason.
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_healthy_subscription_is_not_re_subscribed_on_every_tick()
|
|
{
|
|
var publisher = new CountingHealthPublisher();
|
|
var driver = new SubscribableStubDriver();
|
|
var actor = Sys.ActorOf(DriverInstanceActor.Props(
|
|
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
|
|
|
|
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
|
|
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
|
|
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
|
|
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
|
|
|
|
// POSITIVE CONTROL for the absence below. Every health-poll tick publishes a snapshot, so the
|
|
// publisher count is a direct observation of ticks PROCESSED. Waiting for several of them is what
|
|
// makes "SubscribeCount is still 1" mean "the reconcile ran and declined" rather than "the timer
|
|
// had not fired yet" — the latter would pass instantly and prove nothing.
|
|
var before = publisher.Count;
|
|
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
|
|
|
|
driver.SubscribeCount.ShouldBe(1,
|
|
"the handle is live, so the reconcile must leave it alone");
|
|
}
|
|
|
|
/// <summary>
|
|
/// A driver that is not <see cref="ISubscribable"/> can still be handed a desired set. Reconciling
|
|
/// it would tell <c>Subscribe</c> every tick and fail every tick, forever — so the reconcile is
|
|
/// gated on the capability.
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_non_subscribable_driver_is_never_reconciled()
|
|
{
|
|
var publisher = new CountingHealthPublisher();
|
|
var driver = new StubDriver(); // IDriver only — no ISubscribable
|
|
var actor = Sys.ActorOf(DriverInstanceActor.Props(
|
|
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
|
|
|
|
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
|
|
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
|
|
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
|
|
|
|
// Same positive control: assert the reconcile stays silent across several PROCESSED ticks.
|
|
// EventFilter bounds the absence to this block, and the reconcile announces itself at Info —
|
|
// the first test above is the proof that this message does appear when the reconcile fires.
|
|
EventFilter.Info(contains: "periodic reconcile").Expect(0, () =>
|
|
{
|
|
var before = publisher.Count;
|
|
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
|
|
});
|
|
}
|
|
|
|
/// <summary>Counts health-snapshot publishes, which is one per <c>health-poll</c> tick once the actor
|
|
/// is running — the observable that turns "time passed" into "ticks were processed".</summary>
|
|
private sealed class CountingHealthPublisher : IDriverHealthPublisher
|
|
{
|
|
private int _count;
|
|
|
|
/// <summary>Number of snapshots published so far.</summary>
|
|
public int Count => Volatile.Read(ref _count);
|
|
|
|
/// <inheritdoc />
|
|
public void Publish(
|
|
string clusterId,
|
|
string driverInstanceId,
|
|
DriverHealth health,
|
|
int errorCount5Min,
|
|
DateTime? rediscoveryNeededUtc = null,
|
|
string? rediscoveryReason = null,
|
|
IReadOnlyList<HostConnectivityStatus>? hostStatuses = null)
|
|
=> Interlocked.Increment(ref _count);
|
|
}
|
|
}
|