Commit Graph

5 Commits

Author SHA1 Message Date
Joseph Doherty c569f80694 fix(717): Warm crash survivor stays Active through auto-down; verdict reads its own events
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:

1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
   snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
   Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
   after the hand-over retries. A Primary that is Down, or that this node saw removed, now
   counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
   Leaving/Exiting, or that this node never saw, is not.

2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
   to the same events), which can lag the event being handled. Live, the survivor stayed
   Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
   tick). The Warm verdict now uses a membership view built from this actor's own
   member/reachability subscription, falling back to the read view only before the first event.

Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 04:36:58 -05:00
Joseph Doherty 7829b937d1 feat(717): Warm non-transparent redundancy, default-off (T1-T10)
OpcUa:Redundancy:Mode (Unset default = today's server exactly; None; Warm; Hot and unknown values fail start).
- T1 options + validator + effective-URI check + boot banner; Warm needs the driver role.
- T2 metadata in OnNodeManagerStartedAsync: Warm -> NonTransparentRedundancyType, read-only
  ServerUriArray [self, peers], RedundancySupport=Warm, boot ServiceLevel 1; None -> RedundancySupport=None
  and the SDK's orphan i=11314 deleted; Unset untouched.
- T3 RedundancyStateChanged.Revision (up-number, incarnation tick, sequence); subscribers drop older
  revisions under Warm (OpcUaPublishActor, DriverHostActor); unrevisioned never makes a node Active.
- T4 WarmRoleAuthority + ComputeWarm bands (Standby 90/50, Active 250/200/100), PromoteWhilePrimaryUnreachable.
- T5 IStandbyProjection + DeferredStandbyProjection; OpcUaPublishActor runs promotion/demotion in the spec
  4.7 order (Active level last and retried; Standby level first, data degraded regardless).
- T6 EquipmentVariableState read projection (BadOutOfService/null); Warm Standby refuses writes BadInvalidState.
- T7 D4 mechanism adopted after T0: ordinary-tag marker on MonitoredItemNotification.Message validated by
  (ClientHandle, DataValue); first-delivery decision keyed by (SubscriptionId, SequenceNumber); the SDK's
  cached copy is never mutated.
- T8 queue current/BadOutOfService to live ordinary-tag items ignoring filters.
- T9 alarm stamp/deliver split (suppressed on Standby), Part 9 methods refused on Standby, promotion
  refresh via ISubscription.ConditionRefresh2Async on every live event item.
- T10 deploy-time RedundancyMode cross-check + six instruments.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:39:50 -05:00
Joseph Doherty b211911f98 test: migrate AdminUI/ControlPlane/Runtime/Host tests to xunit.v3 (#756)
v2-ci / unit-tests (push) Failing after 13m22s
Akka.TestKit.Xunit2 -> Akka.TestKit.Xunit 1.5.62 (xunit.v3 TestKit; needs xunit.v3 3.2.2, set via
VersionOverride on the four projects). Removes the xunit 2.9.3 pin. IAsyncLifetime returns ValueTask,
SqlServerFactAttribute gains the source-info ctor, and xUnit1051 call sites pass
TestContext.Current.CancellationToken. Test counts unchanged.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 08:57:33 -05:00
Joseph Doherty c86c7cbf35 test: shared actor test bases poll AwaitCondition every 20 ms, not max/10 (lmxopcua#775)
v2-ci / unit-tests (push) Has been cancelled
Akka TestKit's AwaitCondition without an interval polls every max/10 of the
DILATED budget. Measured here: a condition that turned true at 2 s under a
15 s budget returned after 3003 ms at timefactor 1 (3 polls) and 4502 ms at
timefactor 3 (2 polls). AwaitAssert / AwaitAssertAsync are unaffected (a
fixed 100 ms: 21 polls in ~2 s at both factors).

RuntimeActorTestBase and ControlPlaneActorTestBase now declare
AwaitCondition overloads that hide TestKit's and default the interval to
PollInterval (20 ms), so every subclass call without an interval binds to
them (measured after: returns 5-22 ms after the condition holds, at both
factors). PresenceBudget's doc no longer claims the helpers return "the
instant the condition holds" - it states the per-helper poll behaviour.

Bare-TestKit classes with AwaitCondition calls get an explicit interval:
TelemetryDialDeadlineTests / TelemetryDialSupervisorTests (next to their own
undilated 5 s / 8 s WaitForCount deadlines and a 1 s dial deadline) and
OpcUaEndpointStepDownActorTests (RuntimeActorTestBase.PollInterval, now
public like PresenceBudget). ContinuousHistorizationRecorderTypedValueTests'
AwaitConditionAsync was left alone (historian code is being changed in
parallel).

Suite results (Release, 0 warnings):
- Runtime.Tests 946/3 skipped/0 failed at timefactor 3 (1 m 45 s, was
  2 m 57 s) and 1 (1 m 18 s, was 2 m 57 s);
- ControlPlane.Tests 319/0 failed at 3 (1 m 51 s) and 1 (1 m 11 s);
- AdminUI.Tests Telemetry 15/0 failed at 3 and 1.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:29:33 -05:00
Joseph Doherty acfb4fbaca fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
2026-11-09 12:16:37 -05:00