fix(#662 review): back off alarm retries, contain failed OpcUaClient alarm subscribes

- Reconcile retries of a failing alarm subscribe back off (skip 2^(n-1)-1
  ticks, cap 7 => >= every 4 min at the 30 s tick) so a hanging subscribe
  cannot hold the mailbox every tick; the stack is logged once per streak.
- The unreachable handler re-attach logs at Warning (it only fires on a
  regression).
- OpcUaClient: a failed SubscribeAlarmsAsync detaches its notification
  handler and bounds the session cleanup to 5 s under _gate;
  OnEventNotification drops events from subscriptions not registered in
  _alarmSubscriptions, so a stray that survives cleanup cannot deliver.
- The retry test also covers 3 failures (crosses the backoff); its tick
  positive control documents its dependence on the stub's changing health.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-09 07:35:50 -04:00
parent 5067509dd4
commit 28ca04e466
3 changed files with 75 additions and 17 deletions
@@ -261,10 +261,14 @@ public sealed class DriverInstanceActorNativeAlarmTests : RuntimeActorTestBase
/// reconcile looked at data only, so the feed stayed gated for the life of the connection. Nothing
/// else here can produce the second call: no reconnect, no deploy.
/// </summary>
[Fact]
public void A_failed_alarm_subscribe_is_retried_by_the_periodic_reconcile()
/// <param name="failures">Leading failures; 3 crosses the backoff (skips 0, 1, then 3 ticks), proving the
/// retries keep coming rather than giving up.</param>
[Theory]
[InlineData(1)]
[InlineData(3)]
public void A_failed_alarm_subscribe_is_retried_by_the_periodic_reconcile(int failures)
{
var driver = new AlarmSourceStubDriver { AlarmSubscribeFailuresBeforeSuccess = 1 };
var driver = new AlarmSourceStubDriver { AlarmSubscribeFailuresBeforeSuccess = failures };
var publisher = new CountingHealthPublisher();
var parent = CreateTestProbe();
parent.IgnoreMessages(m => m is DriverInstanceActor.ConnectivityChanged);
@@ -276,7 +280,7 @@ public sealed class DriverInstanceActorNativeAlarmTests : RuntimeActorTestBase
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
AwaitCondition(() => driver.SubscribeAlarmsCallCount >= 1, PresenceBudget);
AwaitCondition(() => driver.SubscribeAlarmsCallCount >= 2, PresenceBudget);
AwaitCondition(() => driver.SubscribeAlarmsCallCount >= failures + 1, PresenceBudget);
driver.LastAlarmRefs.ShouldBe(new[] { "Temp.HiHi" });
driver.RaiseAlarm(new AlarmEventArgs(
@@ -287,7 +291,9 @@ public sealed class DriverInstanceActorNativeAlarmTests : RuntimeActorTestBase
// Converged: a live, matching handle is left alone on later ticks. POSITIVE CONTROL: each tick
// publishes one health snapshot, so waiting for four more makes the unchanged count mean "the
// reconcile ran and declined", not "no tick had fired yet".
// reconcile ran and declined", not "no tick had fired yet". It relies on the base StubDriver's
// GetHealth() stamping UtcNow, which defeats the publish dedup so EVERY tick publishes — a stub with
// stable health would publish once and this wait would time out (loudly, not vacuously).
var settled = driver.SubscribeAlarmsCallCount;
var before = publisher.Count;
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);