Flaky under load: NodeManagerLockTimeBenchmark 50k-node budget; OpcUaClientDriverWire host-status test #769

New Issue

2026-11-09T14:09:26-05:00

dohertj2 commented

2026-11-09 14:09:26 -05:00

Load-sensitive tests seen failing once each during full runs (both pass in isolation):

NodeManagerLockTimeBenchmarkTests.Removing_200_equipment_from_a_50k_node_space_stays_within_budget (OpcUaServer.Tests) — wall-clock budget.
OpcUaClientDriverWireTests.Host_status_is_Running_then_leaves_Running_when_the_server_stops (OpcUaServer.IntegrationTests) — passed 3/3 alone.
Make the benchmark budget relative/skippable under CI load or move it to a perf category; give the wire test a signal-based wait.

Load-sensitive tests seen failing once each during full runs (both pass in isolation): - `NodeManagerLockTimeBenchmarkTests.Removing_200_equipment_from_a_50k_node_space_stays_within_budget` (OpcUaServer.Tests) — wall-clock budget. - `OpcUaClientDriverWireTests.Host_status_is_Running_then_leaves_Running_when_the_server_stops` (OpcUaServer.IntegrationTests) — passed 3/3 alone. Make the benchmark budget relative/skippable under CI load or move it to a perf category; give the wire test a signal-based wait.

dohertj2 added the type:test priority:p3 labels 2026-11-09 14:09:26 -05:00

dohertj2 commented

2026-11-09 14:18:43 -05:00

Mitigated on master acc30576 — not proven (neither flake reproduced locally): NodeManagerLockTimeBenchmark budget 1 s → 5 s (measured 5–60 ms, still catches an order-of-magnitude regression; keeps Category=Benchmark); the wire host-status test already waits on a TaskCompletionSource, ceiling 30 s → 120 s. Keeping open until a few hosted CI runs pass without either failing.

Mitigated on master acc30576 — not proven (neither flake reproduced locally): NodeManagerLockTimeBenchmark budget 1 s → 5 s (measured 5–60 ms, still catches an order-of-magnitude regression; keeps Category=Benchmark); the wire host-status test already waits on a TaskCompletionSource, ceiling 30 s → 120 s. Keeping open until a few hosted CI runs pass without either failing.

dohertj2 commented

2026-11-09 20:56:47 -05:00

Another load-sensitive failure, seen 2026-11-09 during the #761 fixture gate on master 1b9c36f3: Driver.AbCip.IntegrationTests PollLatencyComparisonTests.Native_poll_p95_is_within_ten_percent_of_libplctag failed once with 'connection reset by peer' from the shared ab_server fixture on 10.100.0.35, then passed 2/2 on rerun. It's a timing comparison against a shared fixture, so the same remedy applies: put it in a perf category, or retry on a fixture transport reset.

Another load-sensitive failure, seen 2026-11-09 during the #761 fixture gate on master `1b9c36f3`: `Driver.AbCip.IntegrationTests` `PollLatencyComparisonTests.Native_poll_p95_is_within_ten_percent_of_libplctag` failed once with 'connection reset by peer' from the shared ab_server fixture on 10.100.0.35, then passed 2/2 on rerun. It's a timing comparison against a shared fixture, so the same remedy applies: put it in a perf category, or retry on a fixture transport reset.

dohertj2 referenced this issue

2026-11-09 21:10:24 -05:00

Review remediation 2026-10-21: run the Wave-4 integration gates (lanes A–G, H1, drivers) #761

dohertj2 referenced this issue

2026-11-09 21:39:30 -05:00

Gate findings: deploy TimedOut has null failureReason; quarantine logged as 'unknown' and shown as BadInternalError; rig PSK 'REVOKED' warning vs working dispatch #784

dohertj2 commented

2026-11-09 21:59:37 -05:00

Caught the unnamed Runtime.Tests flake seen several times this session: VirtualTagHostActorTests.Child_is_respawned_after_unexpected_termination, 1 failure in 5 full runs (a 15 s timeout waiting for RegisterInterest). It's a test race: the test waits for the child to die but not for the host to handle its own Terminated (which prunes _children), so a re-apply that lands first spawns nothing. Fixed on master f7177357: the re-apply is retried inside AwaitAssert. 30/30 isolated runs pass.
The product side of the same race is minor: an apply that arrives at the moment a child dies doesn't restart that child until the next apply. Not changed.
Still open here: the NodeManagerLockTimeBenchmark and OpcUaClient wire-test mitigations need a few hosted CI runs, and the AbCip latency-comparison flake noted above.

Caught the unnamed Runtime.Tests flake seen several times this session: `VirtualTagHostActorTests.Child_is_respawned_after_unexpected_termination`, 1 failure in 5 full runs (a 15 s timeout waiting for `RegisterInterest`). It's a test race: the test waits for the child to die but not for the host to handle its own `Terminated` (which prunes `_children`), so a re-apply that lands first spawns nothing. Fixed on master `f7177357`: the re-apply is retried inside AwaitAssert. 30/30 isolated runs pass. The product side of the same race is minor: an apply that arrives at the moment a child dies doesn't restart that child until the next apply. Not changed. Still open here: the NodeManagerLockTimeBenchmark and OpcUaClient wire-test mitigations need a few hosted CI runs, and the AbCip latency-comparison flake noted above.

dohertj2 commented

2026-11-10 05:39:33 -05:00

Fixed on master 11a5c1d4 (tests only, no product code). Root causes / fixes:

NodeManagerLockTimeBenchmark: wall-clock budget now enforced only with OTOPCUA_BENCH=1 (opt-in like OTOPCUA_SCRIPT_PERF); default run still executes both teardowns and asserts counts/SDK handles/re-population, logs timings. Mutation: budget 1 ms -> default passes, OTOPCUA_BENCH=1 fails both legs.
OpcUaClientDriverWire host-status: the old TCS could fire on a transient non-Running keep-alive tick before the server was gone and the final assert then raced a return to Running. Now a semaphore-signalled loop accepts only a non-Running state observed after the rig is disposed. Mutation (keep-alive loss no longer transitions to Stopped) -> fails with 'never left Running'.
AbCip PollLatencyComparison (LiveIntegration, skipped without AB_DIFF_ENDPOINT): retried once on a fixture 'connection reset by peer'; p95 criterion enforced only with AB_LAT_ENFORCE=1, always reported. Not runnable here (needs the fixture); compiles clean.
ScriptRunGuardTests.Three_consecutive_timeouts (+2 siblings using the same script): the 'brief spin' was a 250 ms wall-clock loop racing the 60 ms budget, so a starved box could let it finish inside the budget (no timeout). Orphans are now held by a ctx.Released signal and released/drained explicitly; the wall-clock stuck-orphan rule is disabled in the tests that pin strike counting. Mutation (quarantine threshold +1000) -> 6 guard tests fail.
DependencyMuxReplayTests.Retention: positive control sat inside a 200 ms wall-clock grace. Split into a 1 h-grace survive test and a release test whose control is asserted before unregister. Mutations (sweep never releases / release forgets at once) -> the matching test fails.

Evidence: with 24 busy-loop processes on 14 cores and AKKA_TEST_TIMEFACTOR=3, 20/20 passes each for ScriptRunGuardTests (class), DependencyMuxReplayTests (class), NodeManagerLockTimeBenchmarkTests (class) and the wire host-status test. Release builds 0 warnings; full runs green: Core.Scripting 221, Runtime 996, OpcUaServer.Tests 1027, AbCip.Integration 28 (28 skipped, fixture-gated), OpcUaServer.IntegrationTests 133. Closing on the loop-under-load evidence (hosted CI confirmation not required by the owner).

Fixed on master `11a5c1d4` (tests only, no product code). Root causes / fixes: - NodeManagerLockTimeBenchmark: wall-clock budget now enforced only with `OTOPCUA_BENCH=1` (opt-in like `OTOPCUA_SCRIPT_PERF`); default run still executes both teardowns and asserts counts/SDK handles/re-population, logs timings. Mutation: budget 1 ms -> default passes, `OTOPCUA_BENCH=1` fails both legs. - OpcUaClientDriverWire host-status: the old TCS could fire on a transient non-Running keep-alive tick *before* the server was gone and the final assert then raced a return to Running. Now a semaphore-signalled loop accepts only a non-Running state observed after the rig is disposed. Mutation (keep-alive loss no longer transitions to Stopped) -> fails with 'never left Running'. - AbCip PollLatencyComparison (LiveIntegration, skipped without AB_DIFF_ENDPOINT): retried once on a fixture 'connection reset by peer'; p95 criterion enforced only with `AB_LAT_ENFORCE=1`, always reported. Not runnable here (needs the fixture); compiles clean. - ScriptRunGuardTests.Three_consecutive_timeouts (+2 siblings using the same script): the 'brief spin' was a 250 ms wall-clock loop racing the 60 ms budget, so a starved box could let it finish inside the budget (no timeout). Orphans are now held by a `ctx.Released` signal and released/drained explicitly; the wall-clock stuck-orphan rule is disabled in the tests that pin strike counting. Mutation (quarantine threshold +1000) -> 6 guard tests fail. - DependencyMuxReplayTests.Retention: positive control sat inside a 200 ms wall-clock grace. Split into a 1 h-grace survive test and a release test whose control is asserted before unregister. Mutations (sweep never releases / release forgets at once) -> the matching test fails. Evidence: with 24 busy-loop processes on 14 cores and AKKA_TEST_TIMEFACTOR=3, 20/20 passes each for ScriptRunGuardTests (class), DependencyMuxReplayTests (class), NodeManagerLockTimeBenchmarkTests (class) and the wire host-status test. Release builds 0 warnings; full runs green: Core.Scripting 221, Runtime 996, OpcUaServer.Tests 1027, AbCip.Integration 28 (28 skipped, fixture-gated), OpcUaServer.IntegrationTests 133. Closing on the loop-under-load evidence (hosted CI confirmation not required by the owner).

dohertj2 closed this issue

2026-11-10 05:39:34 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#769