dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Current master CI has 18 unit failures across transport, Gateway and actor suites #707

New Issue
Closed
opened 2026-10-13 09:06:32 -04:00 by dohertj2 · 4 comments
dohertj2 commented 2026-10-13 09:06:32 -04:00
Owner
Copy Link
Copy Source

Current master CI run952 at7b238750 completed failure. Build, README drift and artifact upload passed; minimum-test floor passed14,630>=12,000. Unit job3544 reported18 failures across AB scheduler/transport, Modbus/S7 reconnect, Gateway bounds/catalog/paging, runtime alarm/watchdog/host/OPCUA and control-plane bootstrap tests. Error evidence includes cancellation, condition-never-met and loopback gRPC ConnectTimeout; runner contention is a hypothesis, not an established root cause.

Run: https://gitea.dohertylan.com/dohertj2/lmxopcua/actions/runs/952

Failing cases:

  • ZB.MOM.WW.OtOpcUa.Driver.Ab.Transport.Tests.Cip.RequestSchedulerTests.Channel_failure_faults_every_waiter_in_the_batch
  • ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests.ModbusTcpReconnectTests.SendAsync_AfterResponseTimeout_TearsDownAndReconnects
  • ZB.MOM.WW.OtOpcUa.Driver.S7.Tests.S7DriverReconnectTests.Cancelled_write_marks_handle_dead_and_next_read_reopens
  • ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipTransportSwitchTests.Probe_with_no_transport_uses_the_node_default
  • ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipAlmaProjectionTests.A_bad_severity_read_changes_nothing
  • ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipTransportSwitchTests.Probe_with_no_transport_resolves_like_the_driver_to_native
  • ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayEventBoundsTests.A_start_only_bound_of_UtcMax_reaches_the_wire_intact_through_the_published_client
  • ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayTagCatalogWireTests.A_catalog_stream_fault_after_a_matching_name_is_not_cached_as_presence
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverInstanceActorCommsLossWatchdogTests.A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorUnreadableArtifactTests.Dispatch_whose_artifact_reads_back_empty_keeps_the_drivers_and_their_subscriptions
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.NativeAlarmRestatementEndToEndTests.OpcUaClient_respawn_restating_an_inactive_acked_alarm_writes_no_Activated_row
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.ScriptedAlarms.ScriptedAlarmHostActorTests.AlarmCommand_add_comment_empty_text_is_rejected_not_driven(emptyComment: "")
  • ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayCeilingPagingTests.A_cold_end_only_read_with_a_huge_page_shrinks_the_page_not_the_walk(ceiling: 1000, n: 5000, maxReads: 10)
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.ScriptedAlarms.ScriptedAlarmHostActorTests.Unknown_role_single_driver_publishes_alerts
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.OpcUa.OpcUaPublishActorRebuildTests.Rebuild_direct_mode_with_dbFactory_and_deploymentId_still_loads_and_applies
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorNativeAlarmTests.Driver_disconnect_and_reconnect_annotate_native_condition_quality
  • ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorDegradedAckTests.No_outcome_within_the_timeout_acks_degraded_never_clean
  • ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication.CentralCommunicationActorTests.Bootstrap_retries_stop_once_the_gate_is_published

Acceptance: reproduce affected cases in isolation and under the actual CI execution/parallelism settings; correlate permitted runner CPU/memory/timing evidence; distinguish product regressions from fixture startup/resource timing; add deterministic regressions for verified defects without weakening assertions or the minimum execution floor; verify complete CI green at the resulting commit. No live hardware or ad-hoc remote execution required/authorized by this tracker. #606 remains open for engineering/green-CI acceptance.

Current master CI run952 at7b238750 completed failure. Build, README drift and artifact upload passed; minimum-test floor passed14,630>=12,000. Unit job3544 reported18 failures across AB scheduler/transport, Modbus/S7 reconnect, Gateway bounds/catalog/paging, runtime alarm/watchdog/host/OPCUA and control-plane bootstrap tests. Error evidence includes cancellation, condition-never-met and loopback gRPC ConnectTimeout; runner contention is a hypothesis, not an established root cause. Run: https://gitea.dohertylan.com/dohertj2/lmxopcua/actions/runs/952 Failing cases: - ZB.MOM.WW.OtOpcUa.Driver.Ab.Transport.Tests.Cip.RequestSchedulerTests.Channel_failure_faults_every_waiter_in_the_batch - ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests.ModbusTcpReconnectTests.SendAsync_AfterResponseTimeout_TearsDownAndReconnects - ZB.MOM.WW.OtOpcUa.Driver.S7.Tests.S7DriverReconnectTests.Cancelled_write_marks_handle_dead_and_next_read_reopens - ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipTransportSwitchTests.Probe_with_no_transport_uses_the_node_default - ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipAlmaProjectionTests.A_bad_severity_read_changes_nothing - ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.AbCipTransportSwitchTests.Probe_with_no_transport_resolves_like_the_driver_to_native - ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayEventBoundsTests.A_start_only_bound_of_UtcMax_reaches_the_wire_intact_through_the_published_client - ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayTagCatalogWireTests.A_catalog_stream_fault_after_a_matching_name_is_not_cached_as_presence - ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverInstanceActorCommsLossWatchdogTests.A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again - ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorUnreadableArtifactTests.Dispatch_whose_artifact_reads_back_empty_keeps_the_drivers_and_their_subscriptions - ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.NativeAlarmRestatementEndToEndTests.OpcUaClient_respawn_restating_an_inactive_acked_alarm_writes_no_Activated_row - ZB.MOM.WW.OtOpcUa.Runtime.Tests.ScriptedAlarms.ScriptedAlarmHostActorTests.AlarmCommand_add_comment_empty_text_is_rejected_not_driven(emptyComment: "") - ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.GatewayCeilingPagingTests.A_cold_end_only_read_with_a_huge_page_shrinks_the_page_not_the_walk(ceiling: 1000, n: 5000, maxReads: 10) - ZB.MOM.WW.OtOpcUa.Runtime.Tests.ScriptedAlarms.ScriptedAlarmHostActorTests.Unknown_role_single_driver_publishes_alerts - ZB.MOM.WW.OtOpcUa.Runtime.Tests.OpcUa.OpcUaPublishActorRebuildTests.Rebuild_direct_mode_with_dbFactory_and_deploymentId_still_loads_and_applies - ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorNativeAlarmTests.Driver_disconnect_and_reconnect_annotate_native_condition_quality - ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers.DriverHostActorDegradedAckTests.No_outcome_within_the_timeout_acks_degraded_never_clean - ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication.CentralCommunicationActorTests.Bootstrap_retries_stop_once_the_gate_is_published Acceptance: reproduce affected cases in isolation and under the actual CI execution/parallelism settings; correlate permitted runner CPU/memory/timing evidence; distinguish product regressions from fixture startup/resource timing; add deterministic regressions for verified defects without weakening assertions or the minimum execution floor; verify complete CI green at the resulting commit. No live hardware or ad-hoc remote execution required/authorized by this tracker. #606 remains open for engineering/green-CI acceptance.
dohertj2 commented 2026-10-13 09:16:12 -04:00
Author
Owner
Copy Link
Copy Source

Seven selected hermetic regression cases passed in isolation on the unchanged production/test tree: S7 cancellation (1), Modbus reconnect (1), Gateway UTC maximum bound (1), catalog fault after matching row (1), and huge end-only paging (3). Five sequential Release invocations completed successfully with 0 failures and 0 skips; TRX counters inspected. CI run #952 reported load 118.42/120.72/84.85 on 22 CPUs and fully used 8 GiB swap. These results support resource contention as a hypothesis, but do not establish the cause of all 18 CI failures. Investigating bounded workflow concurrency without loosening assertions, adding retries, or dropping SlowUnit coverage.

Seven selected hermetic regression cases passed in isolation on the unchanged production/test tree: S7 cancellation (1), Modbus reconnect (1), Gateway UTC maximum bound (1), catalog fault after matching row (1), and huge end-only paging (3). Five sequential Release invocations completed successfully with 0 failures and 0 skips; TRX counters inspected. CI run #952 reported load 118.42/120.72/84.85 on 22 CPUs and fully used 8 GiB swap. These results support resource contention as a hypothesis, but do not establish the cause of all 18 CI failures. Investigating bounded workflow concurrency without loosening assertions, adding retries, or dropping SlowUnit coverage.
dohertj2 commented 2026-10-13 10:06:45 -04:00
Author
Owner
Copy Link
Copy Source

Original capped offline run completed: restore/build passed; full suite14,630 passed/1failed/572skipped, terminalexit1; executedfloor14,631>=12,000 passed. Only failure was ControlPlane bootstrap cancellation wall-clock race (#710); Runtime827 passed. This proves the old run is not green. The reviewed #709/#710 test fixes are now being verified together with the three-file CI resource-limit patch in one new hermetic full run. No assertions weakened, no added retries, and SlowUnit remains included. Publishing/closure awaits corrected full run and hostedCI.

Original capped offline run completed: restore/build passed; full suite14,630 passed/1failed/572skipped, terminalexit1; executedfloor14,631>=12,000 passed. Only failure was ControlPlane bootstrap cancellation wall-clock race (#710); Runtime827 passed. This proves the old run is not green. The reviewed #709/#710 test fixes are now being verified together with the three-file CI resource-limit patch in one new hermetic full run. No assertions weakened, no added retries, and SlowUnit remains included. Publishing/closure awaits corrected full run and hostedCI.
dohertj2 commented 2026-10-13 10:53:28 -04:00
Author
Owner
Copy Link
Copy Source

Combined reviewed test/CI fixes published in PR #711, head 76bdca90e9. Complete local coverage: 14,631 passed, 572 skipped, zero failures across 62 assembly TRX files; 12,000-test floor passed. Completed results from the interrupted invocation were retained and only five unfinished assemblies resumed with identical settings/code. Cancellation mutation verified RED; independent combined review clear. Hosted CI run 975 is active; issues stay open pending CI and merge.

Combined reviewed test/CI fixes published in PR #711, head 76bdca90e94e814e06ef10b78271e2b492b2ba32. Complete local coverage: 14,631 passed, 572 skipped, zero failures across 62 assembly TRX files; 12,000-test floor passed. Completed results from the interrupted invocation were retained and only five unfinished assemblies resumed with identical settings/code. Cancellation mutation verified RED; independent combined review clear. Hosted CI run 975 is active; issues stay open pending CI and merge.
dohertj2 commented 2026-10-13 11:55:37 -04:00
Author
Owner
Copy Link
Copy Source

Merged PR #711 as 19bc13bf6a after hosted CI run 975 passed on exact head 76bdca90e9. Merged tree equals tested tree. Restore, Release build, README drift check, serialized full test step, 12,000 executed-test floor and TRX upload all succeeded; gate counted 14,630 executed tests across 62 reports. Local coverage also passed with 14,631 tests and 572 skips. Existing skip-summary diagnostic defect is tracked separately in #712. No retries or assertion deadline changes.

Merged PR #711 as 19bc13bf6a408e795de656f97b9bc5c2b976f5e5 after [hosted CI run 975](https://gitea.dohertylan.com/dohertj2/lmxopcua/actions/runs/975) passed on exact head 76bdca90e94e814e06ef10b78271e2b492b2ba32. Merged tree equals tested tree. Restore, Release build, README drift check, serialized full test step, 12,000 executed-test floor and TRX upload all succeeded; gate counted 14,630 executed tests across 62 reports. Local coverage also passed with 14,631 tests and 572 skips. Existing skip-summary diagnostic defect is tracked separately in #712. No retries or assertion deadline changes.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#707
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?