Modbus background delegates read mutable CTS fields after lifecycle replacement #729

New Issue

2026-10-13T18:26:04-04:00

dohertj2 commented

2026-10-13 18:26:04 -04:00

Bounded source review at eed7cc7d75 found that Modbus background delegates do not snapshot their initialization’s CancellationToken value before scheduling.

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs:354–363 allocates _probeCts/_reprobeCts, then discards tasks scheduled as Task.Run(() => ProbeLoopAsync(_probeCts.Token), _probeCts.Token) and the analogous reprobe call. The scheduling token is evaluated immediately, but the callback later reads the mutable driver field. Initialize replacement at308–320 and teardown at1798–1819 cancel/dispose/reset those fields; another initialization installs new sources. The lifecycle gate serializes lifecycle bodies, not these worker delegates, and no probe/reprobe task handle is joined.

Local .NET10.0.1 reference docs and installed BCL decompilation establish the relevant boundary: cancellation can suppress a queued task before its start check, but does not bind a later field read after the task has committed to execution. If the worker pauses after that check, lifecycle teardown/replacement can advance before its token getter runs. Logical outcomes are a disposed-source getter exception, null-field dereference, or adoption of the replacement initialization’s token. The last case can start an older invocation with the new lifetime’s token. Capturing only the CTS object avoids field replacement/null but retains a disposed-getter window; a CancellationToken value snapshot is the narrow concept to evaluate.

This is source/contract evidence of a race window, not a reproduced runtime effect. No duplicate requests, unbounded leak, process crash, user-visible fault or historical CI cause is claimed. It is unrelated to #728’s OPC UA ConditionRefresh failure. #726/#727’s test-only cleanup scopes remain independent and must not absorb production changes.

Acceptance:

Obtain bounded deterministic evidence distinguishing correctly cancelled queued work from a delegate already committed before CTS-token retrieval; observe task outcome/token identity through shutdown and replacement. Avoid an unbounded timing stress test or a test that only mirrors the new implementation.
Present a bounded design before code changes. Evaluate capturing the per-initialization CancellationToken value before scheduling for both probe/reprobe. Preserve scheduling cancellation, lifecycle behavior, probe defaults and transport/health semantics; a task-join redesign is separate scope.
Verify queued cancellation, started-worker shutdown and replacement-generation identity with meaningful regression evidence, then affected offline suites and exact final-head CI; review before publishing/closure.
Record the discarded-task join/transport-lifetime limits rather than claiming universal shutdown completion. No live device calls are needed for the bounded evidence.

Current state=all searches for _probeCts returned no issue, CTS returned only #725–#727, and relevant closed #602/#630 bodies address different health/other-driver ownership work. Broad Modbus search was bounded to50rows, so no exhaustive historical duplicate claim is made. Evidence: work/lmx-modbus-token-lifetime-audit.md, primary local BCL/docs and parent source review. No tests, production edits or fixes have been performed for this finding.

Bounded source review at eed7cc7d75e4690fb409e0358249489547ace641 found that Modbus background delegates do not snapshot their initialization’s CancellationToken value before scheduling. `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs:354–363` allocates `_probeCts`/`_reprobeCts`, then discards tasks scheduled as `Task.Run(() => ProbeLoopAsync(_probeCts.Token), _probeCts.Token)` and the analogous reprobe call. The scheduling token is evaluated immediately, but the callback later reads the mutable driver field. Initialize replacement at308–320 and teardown at1798–1819 cancel/dispose/reset those fields; another initialization installs new sources. The lifecycle gate serializes lifecycle bodies, not these worker delegates, and no probe/reprobe task handle is joined. Local .NET10.0.1 reference docs and installed BCL decompilation establish the relevant boundary: cancellation can suppress a queued task before its start check, but does not bind a later field read after the task has committed to execution. If the worker pauses after that check, lifecycle teardown/replacement can advance before its token getter runs. Logical outcomes are a disposed-source getter exception, null-field dereference, or adoption of the replacement initialization’s token. The last case can start an older invocation with the new lifetime’s token. Capturing only the CTS object avoids field replacement/null but retains a disposed-getter window; a CancellationToken value snapshot is the narrow concept to evaluate. This is source/contract evidence of a race window, not a reproduced runtime effect. No duplicate requests, unbounded leak, process crash, user-visible fault or historical CI cause is claimed. It is unrelated to #728’s OPC UA ConditionRefresh failure. #726/#727’s test-only cleanup scopes remain independent and must not absorb production changes. Acceptance: - Obtain bounded deterministic evidence distinguishing correctly cancelled queued work from a delegate already committed before CTS-token retrieval; observe task outcome/token identity through shutdown and replacement. Avoid an unbounded timing stress test or a test that only mirrors the new implementation. - Present a bounded design before code changes. Evaluate capturing the per-initialization CancellationToken value before scheduling for both probe/reprobe. Preserve scheduling cancellation, lifecycle behavior, probe defaults and transport/health semantics; a task-join redesign is separate scope. - Verify queued cancellation, started-worker shutdown and replacement-generation identity with meaningful regression evidence, then affected offline suites and exact final-head CI; review before publishing/closure. - Record the discarded-task join/transport-lifetime limits rather than claiming universal shutdown completion. No live device calls are needed for the bounded evidence. Current state=all searches for `_probeCts` returned no issue, CTS returned only #725–#727, and relevant closed #602/#630 bodies address different health/other-driver ownership work. Broad Modbus search was bounded to50rows, so no exhaustive historical duplicate claim is made. Evidence: `work/lmx-modbus-token-lifetime-audit.md`, primary local BCL/docs and parent source review. No tests, production edits or fixes have been performed for this finding.

dohertj2 commented

2026-10-13 19:08:09 -04:00

Read-only follow-up found a candidate deterministic seam in local .NET10.0.1: synchronous TPL TaskStarted is emitted after the cancellation/start check and before delegate invocation. Correlating TaskScheduled on a controlled caller thread can identify the relevant outer task before queueing; ambiguous correlation must fail closed. Existing fake-transport callbacks occur too late. TaskCompleted exposes a matching fault flag but no exact exception object/type. No probe was executed, product code changed, or observed failure claimed. A bounded unchanged-driver hermetic shutdown/fault-flag probe is awaiting design approval; replacement-token and duplicate-activity claims are outside its scope. #727 cleanup remains independent.

dohertj2 commented

2026-10-14 05:21:20 -04:00

Approved bounded hermetic probe completed; this updates the earlier source-only status. #729 remains OPEN, with no production fix yet.

Against unchanged source 4f537a5652 (clean checkout; same product tree as merged main), one isolated process uniquely correlated TPL TaskScheduled/TaskStarted/TaskCompleted ID 1. The installed .NET 10.0.1 synchronous TaskStarted callback held the original worker after its cancellation/start check, before original delegate invocation. Initialization completed synchronously. Shutdown completed while the callback was held; _probeCts was null and the fake transport disposed before release. After release, matching TaskCompleted reported IsExceptional=true.

Evidence: exactly 1 armed schedule, 1 matching start, 1 matching completion; fake connect/send/dispose counts 1/0/1; no gate timeout, callback error or cleanup errors. Scratch build passed with 0 warnings/errors; the sole controlled execution exited 0 in 0.07s under an external 45s deadline. All gate/driver/listener cleanup completed. A separate agent reviewed the probe and result; root independently verified identity/order, execution count and unchanged source hashes.

Pinned CoreLib SHA256 522066c9ccb359e5d8c9aae255ea98548a0d4b274bcfae911dbbce23e39385c3; driver assembly SHA256 00eae29aba04428b6762097c7054361e22fc024671b9b014801ffb76dfd41184. Scratch evidence: work/lmx729-probe/Program.cs, work/lmx729-probe-result.json, work/lmx729-root-probe-verification.json, work/lmx729-probe-commands.jsonl; user report outputs/729-modbus-shutdown-probe-results.md. Initial cached restore failed on an unavailable app-host package; using the installed host with UseAppHost=false resolved restore without installation, before the single run.

Limits: TaskCompleted exposes only a fault flag, not an exact exception type. This proves matching-task exceptional completion under controlled shutdown-before-delegate release on this installed runtime. It does NOT prove replacement-token adoption, duplicate activity, a task join, crash or historical CI cause. No product edits, external calls or live device requests. Replacement/queued-cancellation regressions and a separately approved bounded production fix remain outstanding; #726/#727 and #728 remain independent.

Approved bounded hermetic probe completed; this updates the earlier source-only status. #729 remains OPEN, with no production fix yet. Against unchanged source 4f537a5652eec807ccdc020c7b3ab08a6435fdaf (clean checkout; same product tree as merged main), one isolated process uniquely correlated TPL TaskScheduled/TaskStarted/TaskCompleted ID 1. The installed .NET 10.0.1 synchronous TaskStarted callback held the original worker after its cancellation/start check, before original delegate invocation. Initialization completed synchronously. Shutdown completed while the callback was held; `_probeCts` was null and the fake transport disposed before release. After release, matching TaskCompleted reported IsExceptional=true. Evidence: exactly 1 armed schedule, 1 matching start, 1 matching completion; fake connect/send/dispose counts 1/0/1; no gate timeout, callback error or cleanup errors. Scratch build passed with 0 warnings/errors; the sole controlled execution exited 0 in 0.07s under an external 45s deadline. All gate/driver/listener cleanup completed. A separate agent reviewed the probe and result; root independently verified identity/order, execution count and unchanged source hashes. Pinned CoreLib SHA256 522066c9ccb359e5d8c9aae255ea98548a0d4b274bcfae911dbbce23e39385c3; driver assembly SHA256 00eae29aba04428b6762097c7054361e22fc024671b9b014801ffb76dfd41184. Scratch evidence: work/lmx729-probe/Program.cs, work/lmx729-probe-result.json, work/lmx729-root-probe-verification.json, work/lmx729-probe-commands.jsonl; user report outputs/729-modbus-shutdown-probe-results.md. Initial cached restore failed on an unavailable app-host package; using the installed host with UseAppHost=false resolved restore without installation, before the single run. Limits: TaskCompleted exposes only a fault flag, not an exact exception type. This proves matching-task exceptional completion under controlled shutdown-before-delegate release on this installed runtime. It does NOT prove replacement-token adoption, duplicate activity, a task join, crash or historical CI cause. No product edits, external calls or live device requests. Replacement/queued-cancellation regressions and a separately approved bounded production fix remain outstanding; #726/#727 and #728 remain independent.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#729