Driver test fixtures leave owned probe loops without guaranteed teardown #726

Closed
opened 2026-10-13 16:11:03 -04:00 by dohertj2 · 8 comments
Owner

Read-only #725 task-warning audit found separate test-fixture ownership gaps at source head 8828e999 (same tested tree as merged master 1bf8c614). These are source-backed cleanup omissions, not inferred deadlocks or an established cause for historical CI failures.

Two successful test paths omit driver shutdown/disposal:

  • tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDriverLifecycleTests.cs:601 — Dropping_the_probe_cache_keeps_the_observation_index initializes a canned-client driver, then only checks probe-cache removal, observation index and Good/Healthy state. Default Probe.Enabled=true; MTConnectDriver.StartProbeLoopCore owns a CTS/task. Neither driver nor client is disposed here. No sample stream is inferred without subscriptions.
  • tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusLifecycleHygieneTests.cs:531 — DecodeBitArray_rejects_an_empty_bitmap_with_InvalidDataException initializes fake transport, reads the empty-bitmap response and asserts BadCommunicationError, without teardown. Default probing launches a background loop.

Four reinitialize fixture tests shut down only on the happy path: AbCipReinitializeReleasesStateTests, AbLegacyReinitializeReleasesStateTests, FocasReinitializeReleasesStateTests and TwinCATReinitializeReleasesStateTests. A startup timeout or assertion failure can bypass shutdown while the test-owned fake probe remains running. Their five CS4014 warnings are benign synchronous Shouldly assertion return values; the approved discard-only #725 batch does not fix this separate lifetime gap.

Acceptance:

  • Register/execute teardown for test-owned driver/client resources immediately, including initialization or assertion failures; use existing lifecycle semantics and preserve the primary test failure if cleanup also faults.
  • Preserve all reinitialize live-loop positive controls, old-loop completion/disposal assertions, replacement state and existing deadlines. Preserve MTConnect cache/index/health assertions and Modbus public empty-bitmap/BadCommunicationError path. Do not disable probing merely to hide ownership.
  • Verify teardown with bounded fake-client/loop evidence on normal and injected-failure paths. No external devices, configuration changes or production behavior changes.
  • Run affected existing suites and hosted CI, review and publish the bounded test-only fix. Record any residual SDK/driver cleanup limit rather than claiming stop/join behavior that the implementation does not establish.

Duplicate searches for the fixture class names returned no existing ticket. Evidence: work/lmx725-task-warning-audit.md, root source review of initialized fixture helpers/default options/probe-start code. Implementation needs a separate approved design; no fixture changes or runtime probes have been performed for this finding.

Read-only #725 task-warning audit found separate test-fixture ownership gaps at source head `8828e999` (same tested tree as merged master `1bf8c614`). These are source-backed cleanup omissions, not inferred deadlocks or an established cause for historical CI failures. Two successful test paths omit driver shutdown/disposal: - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDriverLifecycleTests.cs:601` — `Dropping_the_probe_cache_keeps_the_observation_index` initializes a canned-client driver, then only checks probe-cache removal, observation index and Good/Healthy state. Default Probe.Enabled=true; MTConnectDriver.StartProbeLoopCore owns a CTS/task. Neither driver nor client is disposed here. No sample stream is inferred without subscriptions. - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusLifecycleHygieneTests.cs:531` — `DecodeBitArray_rejects_an_empty_bitmap_with_InvalidDataException` initializes fake transport, reads the empty-bitmap response and asserts BadCommunicationError, without teardown. Default probing launches a background loop. Four reinitialize fixture tests shut down only on the happy path: AbCipReinitializeReleasesStateTests, AbLegacyReinitializeReleasesStateTests, FocasReinitializeReleasesStateTests and TwinCATReinitializeReleasesStateTests. A startup timeout or assertion failure can bypass shutdown while the test-owned fake probe remains running. Their five CS4014 warnings are benign synchronous Shouldly assertion return values; the approved discard-only #725 batch does not fix this separate lifetime gap. Acceptance: - [ ] Register/execute teardown for test-owned driver/client resources immediately, including initialization or assertion failures; use existing lifecycle semantics and preserve the primary test failure if cleanup also faults. - [ ] Preserve all reinitialize live-loop positive controls, old-loop completion/disposal assertions, replacement state and existing deadlines. Preserve MTConnect cache/index/health assertions and Modbus public empty-bitmap/BadCommunicationError path. Do not disable probing merely to hide ownership. - [ ] Verify teardown with bounded fake-client/loop evidence on normal and injected-failure paths. No external devices, configuration changes or production behavior changes. - [ ] Run affected existing suites and hosted CI, review and publish the bounded test-only fix. Record any residual SDK/driver cleanup limit rather than claiming stop/join behavior that the implementation does not establish. Duplicate searches for the fixture class names returned no existing ticket. Evidence: `work/lmx725-task-warning-audit.md`, root source review of initialized fixture helpers/default options/probe-start code. Implementation needs a separate approved design; no fixture changes or runtime probes have been performed for this finding.
Author
Owner

Source/design review completed; bounded implementation approval requested, not yet received.

Register cleanup before Initialize, including MTConnect NewDriver construction in the scoped cache-drop test instead of returning only after InitializedDriverAsync. Proposed one small linked test-only body/shutdown/report helper shared by the six affected test projects. Preserve exact primary exception/stack; report secondary cleanup failure without replacing it; cleanup-only failure fails the test. Preserve all existing assertions, enabled-probe defaults, body tokens and deadlines. Deterministic fake-resource failure paths plus helper fault-policy tests are needed; normal success runs alone do not prove failure-path cleanup.

Root reviewed exact Modbus/MTConnect teardown source: Modbus cancels CTS and awaits poll/transport disposal but holds no probe/reprobe task to join. Test cancellation/disposal and fake request completion separately; do not claim authoritative probe-task completion. MTConnect retains its task and uses a bounded stop wait that can log/continue. Other drivers also have distinct timeout/fault policies. No production lifecycle changes are proposed.

If approved, build/run affected offline suites, review and commit locally; publish after PR #724 clears CI/merge, alongside already verified #725 explicit-discard cleanup. No changes have been implemented for #726, no runtime verification or external calls.

Evidence: work/lmx726-fixture-cleanup-design-audit.md. Prior #725 discard-only commit5310a7e0 remains clean and unpublished.

Source/design review completed; bounded implementation approval requested, not yet received. Register cleanup before Initialize, including MTConnect NewDriver construction in the scoped cache-drop test instead of returning only after InitializedDriverAsync. Proposed one small linked test-only body/shutdown/report helper shared by the six affected test projects. Preserve exact primary exception/stack; report secondary cleanup failure without replacing it; cleanup-only failure fails the test. Preserve all existing assertions, enabled-probe defaults, body tokens and deadlines. Deterministic fake-resource failure paths plus helper fault-policy tests are needed; normal success runs alone do not prove failure-path cleanup. Root reviewed exact Modbus/MTConnect teardown source: Modbus cancels CTS and awaits poll/transport disposal but holds no probe/reprobe task to join. Test cancellation/disposal and fake request completion separately; do not claim authoritative probe-task completion. MTConnect retains its task and uses a bounded stop wait that can log/continue. Other drivers also have distinct timeout/fault policies. No production lifecycle changes are proposed. If approved, build/run affected offline suites, review and commit locally; publish after PR #724 clears CI/merge, alongside already verified #725 explicit-discard cleanup. No changes have been implemented for #726, no runtime verification or external calls. Evidence: work/lmx726-fixture-cleanup-design-audit.md. Prior #725 discard-only commit5310a7e0 remains clean and unpublished.
Author
Owner

Owner approved the bounded test-only cleanup design. Implementation is underway on the isolated branch based on local5310a7e0, retaining the separate #725 assertion-discard commit. Scope: one linked cleanup operation in six fake driver projects; preserve primary exception identity/stack, report secondary cleanup faults, propagate cleanup-only faults; cleanup registered before initialization; existing assertions/probe defaults/tokens/deadlines retained. Verification will include deterministic failure/resource checks and six offline project suites. Modbus evidence will distinguish cancellation/disposal/request completion from the unproven probe-task join. No production changes or live calls. Publication remains behind PR #724 final-head CI and merge; this issue stays open until reviewed/published verification completes.

Owner approved the bounded test-only cleanup design. Implementation is underway on the isolated branch based on local5310a7e0, retaining the separate #725 assertion-discard commit. Scope: one linked cleanup operation in six fake driver projects; preserve primary exception identity/stack, report secondary cleanup faults, propagate cleanup-only faults; cleanup registered before initialization; existing assertions/probe defaults/tokens/deadlines retained. Verification will include deterministic failure/resource checks and six offline project suites. Modbus evidence will distinguish cancellation/disposal/request completion from the unproven probe-task join. No production changes or live calls. Publication remains behind PR #724 final-head CI and merge; this issue stays open until reviewed/published verification completes.
Author
Owner

The approved test-only cleanup is independently reviewed and locally committed eed7cc7d, separately atop #725 discard commit5310a7e0.14test files only; no production/dependency changes. Final offline suites: AbCip752,AbLegacy597,FOCAS511,TwinCAT363,MTConnect635,Modbus465:3323passed/0failed/0skipped. Parent parsed final individual TRX outcomes and checked original assertion lines are retained. Helper executable stub RED5/5 then GREEN5/5; cleanup-removal mutations failed the resource checks in every affected project, then restored full suites passed. Review spec+quality PASS/noP1P2. Modbus proves request cancellation/completion and transport disposal, not background probe-task join; drivers with bounded joins were proven only in these fake scenarios. Existing warning groups remain #725. No push yet: publication waits on PR724 final-head CI/merge, then merged-main integration and final hosted gate. Issue remains open.

The approved test-only cleanup is independently reviewed and locally committed eed7cc7d, separately atop #725 discard commit5310a7e0.14test files only; no production/dependency changes. Final offline suites: AbCip752,AbLegacy597,FOCAS511,TwinCAT363,MTConnect635,Modbus465:3323passed/0failed/0skipped. Parent parsed final individual TRX outcomes and checked original assertion lines are retained. Helper executable stub RED5/5 then GREEN5/5; cleanup-removal mutations failed the resource checks in every affected project, then restored full suites passed. Review spec+quality PASS/noP1P2. Modbus proves request cancellation/completion and transport disposal, not background probe-task join; drivers with bounded joins were proven only in these fake scenarios. Existing warning groups remain #725. No push yet: publication waits on PR724 final-head CI/merge, then merged-main integration and final hosted gate. Issue remains open.
Author
Owner

Approved #725/#726 fixes are published in PR #730 at exact head 4f537a5652 after merging verified main c89912a9. The integrated tree 189a116a7b exactly matches the reviewed merge preflight; only the same 14 test-only files differ from main. Six offline affected suites passed 3,323 tests; cleanup/helper failure-policy mutations and independent review passed. Required final-head hosted run 988/job3609 is waiting behind main run987. Not merged or closed yet. #725 remains a partial warning cleanup; PR resolves only #726. No live calls or production changes.

Approved #725/#726 fixes are published in PR #730 at exact head 4f537a5652eec807ccdc020c7b3ab08a6435fdaf after merging verified main c89912a9. The integrated tree 189a116a7bb4aa713cb3a86212972d7cde5100ab exactly matches the reviewed merge preflight; only the same 14 test-only files differ from main. Six offline affected suites passed 3,323 tests; cleanup/helper failure-policy mutations and independent review passed. Required final-head hosted run 988/job3609 is waiting behind main run987. Not merged or closed yet. #725 remains a partial warning cleanup; PR resolves only #726. No live calls or production changes.
Author
Owner

Required PR730 final-head run988/job3609 has started at exact published head4f537a5652eec807ccdc020c7b3ab08a6435fdaf, observed00:05 UTC14October. Main987/job3608 first completed successfully at00:04:45: all10steps, Runtime837pass/3existing skips, 62TRX/14,656executed/573skips, floor>=12,000 and finalized unit-trx upload. That main result does not replace PR730 final-head acceptance. #726 remains open until run988 passes and PR730 is merged; #725 remains partial. No test retries, cancellations or new source changes.

Required PR730 final-head run988/job3609 has started at exact published head4f537a5652eec807ccdc020c7b3ab08a6435fdaf, observed00:05 UTC14October. Main987/job3608 first completed successfully at00:04:45: all10steps, Runtime837pass/3existing skips, 62TRX/14,656executed/573skips, floor>=12,000 and finalized unit-trx upload. That main result does not replace PR730 final-head acceptance. #726 remains open until run988 passes and PR730 is merged; #725 remains partial. No test retries, cancellations or new source changes.
Author
Owner

Required PR730 final-head hosted run988/job3609 at4f537a5652eec807ccdc020c7b3ab08a6435fdaf completed all six affected driver suites: AbCip752, AbLegacy597, FOCAS511, MTConnect635, Modbus465 and TwinCAT363; total3,323 passed/0failed/0skipped, observed00:24 UTC14October. Counts match the independently reviewed offline suite results. Build succeeded and emitted no CS4014 warnings. This is an affected-scope checkpoint, not full CI acceptance: server/remaining suites, test-floor and artifact gates still run. PR730 remains unmerged and #726 remains open pending that required gate. #725 remains partial; #727/#729 are independent unresolved scopes.

Required PR730 final-head hosted run988/job3609 at4f537a5652eec807ccdc020c7b3ab08a6435fdaf completed all six affected driver suites: AbCip752, AbLegacy597, FOCAS511, MTConnect635, Modbus465 and TwinCAT363; total3,323 passed/0failed/0skipped, observed00:24 UTC14October. Counts match the independently reviewed offline suite results. Build succeeded and emitted no CS4014 warnings. This is an affected-scope checkpoint, not full CI acceptance: server/remaining suites, test-floor and artifact gates still run. PR730 remains unmerged and #726 remains open pending that required gate. #725 remains partial; #727/#729 are independent unresolved scopes.
Author
Owner

PR730 merged as ebcbd81fca after required run988/job3609 passed all ten steps on exact final head4f537a5652eec807ccdc020c7b3ab08a6435fdaf. Merged tree189a116a7bb4aa713cb3a86212972d7cde5100ab equals the tested tree. Full gate:62TRX files,14,672executed tests,573individual skips, floor>=12,000 and finalized unit-trx artifact. Six affected suites:3,323passed/0failed/0skipped; helper exception policies/cleanup-removal mutations and independent review passed. Hosted build emitted zero CS4014 warnings. #726 is resolved and closed; #725 remains open for the other warning groups. Main checkout fast-forwarded with all ten user-owned file hashes freshly preserved. No production changes or live calls; Modbus evidence establishes cancellation/disposal/request completion, not a discarded probe-task join. #727/#728/#729 remain independent unresolved work.

PR730 merged as ebcbd81fcaa89e3a0bab8cc44d1d68049530f06d after required run988/job3609 passed all ten steps on exact final head4f537a5652eec807ccdc020c7b3ab08a6435fdaf. Merged tree189a116a7bb4aa713cb3a86212972d7cde5100ab equals the tested tree. Full gate:62TRX files,14,672executed tests,573individual skips, floor>=12,000 and finalized unit-trx artifact. Six affected suites:3,323passed/0failed/0skipped; helper exception policies/cleanup-removal mutations and independent review passed. Hosted build emitted zero CS4014 warnings. #726 is resolved and closed; #725 remains open for the other warning groups. Main checkout fast-forwarded with all ten user-owned file hashes freshly preserved. No production changes or live calls; Modbus evidence establishes cancellation/disposal/request completion, not a discarded probe-task join. #727/#728/#729 remain independent unresolved work.
Author
Owner

Additional merged-main verification: automatic run989/job3610 passed all ten steps at exact merged commit ebcbd81fca, completed02:07:59 UTC14October. Runtime837passed/0failed/3existing skips; all six affected driver suites3,323passed/0failed/0skipped. Floor62TRX/14,672executed/573individual skips>=12,000 and finalized unit-trx upload passed. This supports the completed #726 closure and merged first-five #725 slice; it does not resolve the remaining warnings or #727/#728/#729. No source changes, live calls, retries, cancellation or reruns were used for this observation.

Additional merged-main verification: automatic run989/job3610 passed all ten steps at exact merged commit ebcbd81fcaa89e3a0bab8cc44d1d68049530f06d, completed02:07:59 UTC14October. Runtime837passed/0failed/3existing skips; all six affected driver suites3,323passed/0failed/0skipped. Floor62TRX/14,672executed/573individual skips>=12,000 and finalized unit-trx upload passed. This supports the completed #726 closure and merged first-five #725 slice; it does not resolve the remaining warnings or #727/#728/#729. No source changes, live calls, retries, cancellation or reruns were used for this observation.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#726