dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

ConditionRefresh wire test receives an extra Unspecified event in hosted CI #728

New Issue
Open
opened 2026-10-13 17:36:10 -04:00 by dohertj2 · 5 comments
dohertj2 commented 2026-10-13 17:36:10 -04:00
Owner
Copy Link
Copy Source

Hosted supporting run985/job3606 (exact head e1feb9fbb6) failed the existing loopback ConditionRefresh wire test at2026-10-13 21:32:19UTC:

tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverWireTests.cs:317
A_condition_already_active_before_subscribe_is_reported_by_ConditionRefresh

Expected [AlarmTransitionKind.Acknowledge]; actual [AlarmTransitionKind.Unspecified, AlarmTransitionKind.Acknowledge]. Assertion: “the second refresh re-delivered an unchanged state and must have emitted nothing”. The test verifies an active/unacked condition before subscribe, unchanged resubscription, and a positive upstream acknowledgement control.

This run is superseded by PR#724 head c4b4046832 and required run986/job3607; the required run is still waiting. The new head changes only freshness acceptance tests in addition to the existing #721/#722 scope. Neither head’s eventual result is a substitute for investigating this observed failure. No result for run986 is claimed.

Root cause is unproven. Source has at least two distinct routes to Unspecified: same Active/Acked with changed Message/Severity, or a projected non-condition event without state. Refresh markers are explicitly rejected. The test uses fixed settling delays and captures its baseline after the first Retrigger; ordering/delivery must be investigated rather than assumed. Existing alarm/refresh behavior fixes under closed #619/#587 are relevant history, not proof this is the same defect. State=all ConditionRefresh search found no separate open ticket for this failure; #717 is future Warm architecture, not this observed current-mode regression.

Acceptance:

  • Capture/trace the extra event’s origin, state, message/severity and timing relative to refresh boundaries and acknowledgement; determine test ordering vs product event/deduplication behavior.
  • Reproduce or add deterministic regression evidence for the confirmed cause. Preserve unchanged-refresh silence, marker/branch rejection, active-unacked restatement and exact acknowledgement positive-control assertions; do not accept extra events, add retries or inflate sleeps to mask failure.
  • Present a bounded design before implementation. Keep production changes out if the cause is only fixture synchronization; justify any product change with direct evidence.
  • Verify affected offline loopback/unit suites, review and require exact final-head hosted CI before merge/closure. A single unrelated green rerun does not prove this intermittent failure is resolved.

Read-only triage is in progress; no fixes, CI restarts/cancellations, external endpoint calls or historian writes have been performed. Evidence: captured authoritative work/lmx724-ci-watch-3606.log:2723–2739, parent source inspection, and forthcoming work/lmx-condition-refresh-failure-triage.md. No historical CI resource cause or production alarm loss is inferred from this one failure.

Hosted supporting run985/job3606 (exact head e1feb9fbb6e3a1cb78c95cdb31ee24635ab11a33) failed the existing loopback ConditionRefresh wire test at2026-10-13 21:32:19UTC: `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverWireTests.cs:317` `A_condition_already_active_before_subscribe_is_reported_by_ConditionRefresh` Expected `[AlarmTransitionKind.Acknowledge]`; actual `[AlarmTransitionKind.Unspecified, AlarmTransitionKind.Acknowledge]`. Assertion: “the second refresh re-delivered an unchanged state and must have emitted nothing”. The test verifies an active/unacked condition before subscribe, unchanged resubscription, and a positive upstream acknowledgement control. This run is superseded by PR#724 head c4b404683257da195cdf20e45a923a1b336be460 and required run986/job3607; the required run is still waiting. The new head changes only freshness acceptance tests in addition to the existing #721/#722 scope. Neither head’s eventual result is a substitute for investigating this observed failure. No result for run986 is claimed. Root cause is unproven. Source has at least two distinct routes to Unspecified: same Active/Acked with changed Message/Severity, or a projected non-condition event without state. Refresh markers are explicitly rejected. The test uses fixed settling delays and captures its baseline after the first Retrigger; ordering/delivery must be investigated rather than assumed. Existing alarm/refresh behavior fixes under closed #619/#587 are relevant history, not proof this is the same defect. State=all ConditionRefresh search found no separate open ticket for this failure; #717 is future Warm architecture, not this observed current-mode regression. Acceptance: - Capture/trace the extra event’s origin, state, message/severity and timing relative to refresh boundaries and acknowledgement; determine test ordering vs product event/deduplication behavior. - Reproduce or add deterministic regression evidence for the confirmed cause. Preserve unchanged-refresh silence, marker/branch rejection, active-unacked restatement and exact acknowledgement positive-control assertions; do not accept extra events, add retries or inflate sleeps to mask failure. - Present a bounded design before implementation. Keep production changes out if the cause is only fixture synchronization; justify any product change with direct evidence. - Verify affected offline loopback/unit suites, review and require exact final-head hosted CI before merge/closure. A single unrelated green rerun does not prove this intermittent failure is resolved. Read-only triage is in progress; no fixes, CI restarts/cancellations, external endpoint calls or historian writes have been performed. Evidence: captured authoritative `work/lmx724-ci-watch-3606.log:2723–2739`, parent source inspection, and forthcoming `work/lmx-condition-refresh-failure-triage.md`. No historical CI resource cause or production alarm loss is inferred from this one failure.
dohertj2 commented 2026-10-13 17:37:27 -04:00
Author
Owner
Copy Link
Copy Source

Read-only triage adds a third real Unspecified route: OpcUaClientDriver.AnnotateAlarmQuality (3252–3272) emits a quality-only annotation, called by communication-loss/recovery paths. The failing assertion records Kind only, so the extra event cannot yet be classified as a duplicate retained condition versus a quality annotation. Exact event QualityStatusCode/state/message/severity and PublishError status/subscription-ID/timing are missing from captured CI output.

A source-backed hypothesis: HandlePublishError (3039–3064) treats BadNoSubscription as definitive loss whenever any current tag/alarm subscription is tracked; same-session identity is checked, but this global status does not match subscriptionId. A delayed error during intentional last-subscription deletion could arrive after new subscription registration and annotate quality. This is not established as the hosted cause. Genuine loss handling must remain intact; do not suppress errors or weaken the exact acknowledgement/unchanged-refresh assertions merely to make CI pass.

The source-verified local loopback/synthetic exact test is being reproduced unchanged in a scrubbed environment with offline cached-package restore/build. No product/test edits, instrumentation, retries in CI, external calls or production alarm defect claim. Required run986 remains queued; supporting985 reports123 OPC UA integration passed/1failed and is still running other suites.

Read-only triage adds a third real Unspecified route: OpcUaClientDriver.AnnotateAlarmQuality (3252–3272) emits a quality-only annotation, called by communication-loss/recovery paths. The failing assertion records Kind only, so the extra event cannot yet be classified as a duplicate retained condition versus a quality annotation. Exact event QualityStatusCode/state/message/severity and PublishError status/subscription-ID/timing are missing from captured CI output. A source-backed hypothesis: HandlePublishError (3039–3064) treats BadNoSubscription as definitive loss whenever any current tag/alarm subscription is tracked; same-session identity is checked, but this global status does not match subscriptionId. A delayed error during intentional last-subscription deletion could arrive after new subscription registration and annotate quality. This is not established as the hosted cause. Genuine loss handling must remain intact; do not suppress errors or weaken the exact acknowledgement/unchanged-refresh assertions merely to make CI pass. The source-verified local loopback/synthetic exact test is being reproduced unchanged in a scrubbed environment with offline cached-package restore/build. No product/test edits, instrumentation, retries in CI, external calls or production alarm defect claim. Required run986 remains queued; supporting985 reports123 OPC UA integration passed/1failed and is still running other suites.
dohertj2 commented 2026-10-13 17:39:28 -04:00
Author
Owner
Copy Link
Copy Source

Unchanged exact local loopback reproduction completed at c4b40468: three attempts FAIL/PASS/FAIL, each one selected fact and zero skips. Both failures exactly match hosted [Unspecified,Acknowledge] atline317. Parent independently parsed all three TRX files. Offline project restore/build succeeded (zero build warnings/errors); five-key scrubbed environment, no source edits, no external fixtures; checkout remains clean. This establishes intermittent reproducibility, not cause or resolution. Initial three-attempt cap is exhausted.

Source trace confirms identical retained-condition content alone is deduplicated; candidates include changed message/severity, missing condition bool fields, or quality-only annotations. A delayed BadNoSubscription during intentional replacement remains a hypothesis. Bounded throwaway test-only metadata/session-error tracing design has been submitted for approval; no instrumentation or fix until approved. Preserve exact silence/acknowledgement assertions; no CI rerun/retry or sleep inflation. Full report work/lmx-condition-refresh-failure-triage.md.

Unchanged exact local loopback reproduction completed at c4b40468: three attempts FAIL/PASS/FAIL, each one selected fact and zero skips. Both failures exactly match hosted [Unspecified,Acknowledge] atline317. Parent independently parsed all three TRX files. Offline project restore/build succeeded (zero build warnings/errors); five-key scrubbed environment, no source edits, no external fixtures; checkout remains clean. This establishes intermittent reproducibility, not cause or resolution. Initial three-attempt cap is exhausted. Source trace confirms identical retained-condition content alone is deduplicated; candidates include changed message/severity, missing condition bool fields, or quality-only annotations. A delayed BadNoSubscription during intentional replacement remains a hypothesis. Bounded throwaway test-only metadata/session-error tracing design has been submitted for approval; no instrumentation or fix until approved. Preserve exact silence/acknowledgement assertions; no CI rerun/retry or sleep inflation. Full report work/lmx-condition-refresh-failure-triage.md.
dohertj2 commented 2026-10-13 18:00:04 -04:00
Author
Owner
Copy Link
Copy Source

Authoritative Actions API21:59UTC: supporting run985/job3606 completed FAILURE at21:58:35, exact earlier e1feb9f head. The one failed test is ConditionRefresh wire fact tracked in#728 (123integration passed/1failed); Runtime835passed/3existing skips and other suites passed. The always-run test-count guard succeeded:62TRX,14654executed,573individual skips, minimum12000. Artifact unit-trx upload succeeded. Floor/artifact success does not make the failed test run green.

Required run986/job3607 STARTED at21:58:38, exact final PR#724 head c4b4046832. It is in_progress with no conclusion. This new run is the required merge gate; no result is assumed. #728 unchanged local reproduction FAIL/PASS/FAIL and source hypotheses remain recorded; a bounded metadata probe awaits design approval. No product/test instrumentation or CI rerun/cancel/restart. PR remains open/unmerged;#721/#722 remain open, and local #725/#726 publication remains held until#724 clears its gate.

Authoritative Actions API21:59UTC: supporting run985/job3606 completed FAILURE at21:58:35, exact earlier e1feb9f head. The one failed test is ConditionRefresh wire fact tracked in#728 (123integration passed/1failed); Runtime835passed/3existing skips and other suites passed. The always-run test-count guard succeeded:62TRX,14654executed,573individual skips, minimum12000. Artifact unit-trx upload succeeded. Floor/artifact success does not make the failed test run green. Required run986/job3607 STARTED at21:58:38, exact final PR#724 head c4b404683257da195cdf20e45a923a1b336be460. It is in_progress with no conclusion. This new run is the required merge gate; no result is assumed. #728 unchanged local reproduction FAIL/PASS/FAIL and source hypotheses remain recorded; a bounded metadata probe awaits design approval. No product/test instrumentation or CI rerun/cancel/restart. PR remains open/unmerged;#721/#722 remain open, and local #725/#726 publication remains held until#724 clears its gate.
dohertj2 commented 2026-10-13 18:37:37 -04:00
Author
Owner
Copy Link
Copy Source

Required run986/job3607 at unchanged ConditionRefresh-source head c4b40468 reports OPC UA server integration124passed/0failed/0skipped at22:35:31UTC. Supporting985 had the same ConditionRefresh fact fail; unchanged local reproduction was FAIL/PASS/FAIL. This additional passing hosted run corroborates intermittency, not a fix. No diagnostic or product/test change has been made to the failing fact. #728 remains open pending approved tracing and cause-specific verification; do not close it on one green run.

Whole run986 is still in_progress; server unit and Runtime suites plus final count/artifact gates remain. #724’s merge gate is not yet satisfied.

Required run986/job3607 at unchanged ConditionRefresh-source head c4b40468 reports OPC UA server integration124passed/0failed/0skipped at22:35:31UTC. Supporting985 had the same ConditionRefresh fact fail; unchanged local reproduction was FAIL/PASS/FAIL. This additional passing hosted run corroborates intermittency, not a fix. No diagnostic or product/test change has been made to the failing fact. #728 remains open pending approved tracing and cause-specific verification; do not close it on one green run. Whole run986 is still in_progress; server unit and Runtime suites plus final count/artifact gates remain. #724’s merge gate is not yet satisfied.
dohertj2 commented 2026-10-13 19:41:48 -04:00
Author
Owner
Copy Link
Copy Source

Additional unchanged hosted observation: main run987/job3608 at merged head c89912a93b passed the OPC UA integration suite:124 passed/0 failed/0 skipped (8m54s), observed23:41 UTC. No ConditionRefresh source/assertion change was made. This corroborates intermittency alongside supporting CI985 failure and unchanged local FAIL/PASS/FAIL; it is not a verified cause or fix. #728 remains open, and the pending bounded metadata probe still requires its existing design approval. The whole main CI job remains active, and PR730 final-head acceptance has not started.

Additional unchanged hosted observation: main run987/job3608 at merged head c89912a93b87b1742dd049980780d5eb3f816c20 passed the OPC UA integration suite:124 passed/0 failed/0 skipped (8m54s), observed23:41 UTC. No ConditionRefresh source/assertion change was made. This corroborates intermittency alongside supporting CI985 failure and unchanged local FAIL/PASS/FAIL; it is not a verified cause or fix. #728 remains open, and the pending bounded metadata probe still requires its existing design approval. The whole main CI job remains active, and PR730 final-head acceptance has not started.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#728
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?