Intermittent restart/reconnect wire test failure in full solution; isolated retry passes #702

Closed
opened 2026-10-13 05:08:12 -04:00 by dohertj2 · 4 comments
Owner

New verification finding during the #701 tests-only batch on 2026-10-13. Full solution at master618d98e8 plus only Gateway cache test/fake changes failed one existing test: OpcUaClientDriverReconnectWireTests.A_server_restart_goes_Bad_then_the_watchdog_reinit_brings_the_value_back_Good (~18.46s); OPC UA integration assembly121passed/1failed. An immediate isolated rerun with console normal verbosity passed1/1 in14.6s. No reconnect production/fixture/test files changed in this batch.

The full run used quiet output and did not preserve the assertion stack. Root cause remains UNCONFIRMED. Read-only inspection found a possible brittle expectation: test line80 requires ConnectionLost reason BadServerHalted, while the production #676 contract also recognizes sustained BadSecureChannelClosed/BadConnectionClosed. Shutdown/keepalive ordering can affect observed transport status; alternatively a loss-notification timing condition may have failed. Do not weaken assertions without capturing the actual failure.

Acceptance: reproduce under the full integration load with detailed logger/TRX evidence; distinguish exact-status mismatch, duplicate/missing notification and recovery failures; fix only the proven cause while retaining BadNoCommunication/value retention/frozen read stamp/one notification/reinitialization recovery assertions. If multiple definitive loss reasons are legal, assert that semantic set and test each route separately. Verify isolated and full integration/solution runs. Related #676 is closed for the production behavior; this ticket tracks the new intermittent verification gap.

New verification finding during the #701 tests-only batch on 2026-10-13. Full solution at master618d98e8 plus only Gateway cache test/fake changes failed one existing test: OpcUaClientDriverReconnectWireTests.A_server_restart_goes_Bad_then_the_watchdog_reinit_brings_the_value_back_Good (~18.46s); OPC UA integration assembly121passed/1failed. An immediate isolated rerun with console normal verbosity passed1/1 in14.6s. No reconnect production/fixture/test files changed in this batch. The full run used quiet output and did not preserve the assertion stack. Root cause remains UNCONFIRMED. Read-only inspection found a possible brittle expectation: test line80 requires ConnectionLost reason BadServerHalted, while the production #676 contract also recognizes sustained BadSecureChannelClosed/BadConnectionClosed. Shutdown/keepalive ordering can affect observed transport status; alternatively a loss-notification timing condition may have failed. Do not weaken assertions without capturing the actual failure. Acceptance: reproduce under the full integration load with detailed logger/TRX evidence; distinguish exact-status mismatch, duplicate/missing notification and recovery failures; fix only the proven cause while retaining BadNoCommunication/value retention/frozen read stamp/one notification/reinitialization recovery assertions. If multiple definitive loss reasons are legal, assert that semantic set and test each route separately. Verify isolated and full integration/solution runs. Related #676 is closed for the production behavior; this ticket tracks the new intermittent verification gap.
Author
Owner

The full solution repeat with detailed console logging passed14,615/609skip/0fail across62suites, including the restart/reconnect test at14s and OPC UA integration122/122. This is evidence of intermittent behavior, not proof of a fix or of the suspected exact-status cause. #702 remains open; #701 coverage merged separately in PR703.

The full solution repeat with detailed console logging passed14,615/609skip/0fail across62suites, including the restart/reconnect test at14s and OPC UA integration122/122. This is evidence of intermittent behavior, not proof of a fix or of the suspected exact-status cause. #702 remains open; #701 coverage merged separately in PR703.
Author
Owner

Reproduced under the full OPC UA integration assembly with detailed console/TRX evidence on unchanged source. Actual assertion failure is line 80: expected reason containing BadServerHalted, received "upstream session keep-alive reported BadSessionIdInvalid". BadSessionIdInvalid is explicitly accepted by production IsDefinitiveSessionLoss; this establishes a test expectation defect. Prior down-state assertions (BadNoCommunication, retained value, unhealthy connection, frozen read stamp) passed. The test stops at that reason assertion, so this failed run does not prove subsequent recovery. No source edits. The fix should retain notification count/recovery assertions and accept the production semantic loss set rather than a single nondeterministic shutdown ordering.

Reproduced under the full OPC UA integration assembly with detailed console/TRX evidence on unchanged source. Actual assertion failure is line 80: expected reason containing BadServerHalted, received "upstream session keep-alive reported BadSessionIdInvalid". BadSessionIdInvalid is explicitly accepted by production IsDefinitiveSessionLoss; this establishes a test expectation defect. Prior down-state assertions (BadNoCommunication, retained value, unhealthy connection, frozen read stamp) passed. The test stops at that reason assertion, so this failed run does not prove subsequent recovery. No source edits. The fix should retain notification count/recovery assertions and accept the production semantic loss set rather than a single nondeterministic shutdown ordering.
Author
Owner

Full reproduction finished: 121 passed / 1 failed, 122 tests. TRX and detailed log retained. Only failure is the BadServerHalted vs BadSessionIdInvalid reason assertion described above. All four direct definitive statuses and sustained closed-channel routes already have deterministic unit coverage. No source edits; awaiting approval for the narrow test assertion/comment correction.

Full reproduction finished: 121 passed / 1 failed, 122 tests. TRX and detailed log retained. Only failure is the BadServerHalted vs BadSessionIdInvalid reason assertion described above. All four direct definitive statuses and sustained closed-channel routes already have deterministic unit coverage. No source edits; awaiting approval for the narrow test assertion/comment correction.
Author
Owner

Resolved and merged PR #704: fix dccab57d39, master 0ccfbbee5c. Merged tree matches the verified branch. Actual reproduction proved BadServerHalted-only expectation rejected valid BadSessionIdInvalid; explicit semantic set now accepted with every outage/count/recovery assertion preserved. No production changes.

Verification: isolated1/1, deterministic status routes30/30, full OPC UA integration122/122, full offline solution14,613 passed /611 skipped /zero failures, exit0. Independent review clear; diffcheckclean. Host integration266passed/14environment-fixture skips. Detailed logs/TRX retained; results document committed. Closing #702. Remaining native historian/live OPC UA verification gates are unrelated and remain open.

Resolved and merged PR #704: fix dccab57d394096b585009f16ae58cce41cebd862, master 0ccfbbee5ca418902fff7409e859ed7785332f05. Merged tree matches the verified branch. Actual reproduction proved BadServerHalted-only expectation rejected valid BadSessionIdInvalid; explicit semantic set now accepted with every outage/count/recovery assertion preserved. No production changes. Verification: isolated1/1, deterministic status routes30/30, full OPC UA integration122/122, full offline solution14,613 passed /611 skipped /zero failures, exit0. Independent review clear; diffcheckclean. Host integration266passed/14environment-fixture skips. Detailed logs/TRX retained; results document committed. Closing #702. Remaining native historian/live OPC UA verification gates are unrelated and remain open.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#702