dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

ServiceLevel publication before SDK binding is dropped and unchanged state can remain deduplicated #719

New Issue
Closed
opened 2026-10-13 12:56:23 -04:00 by dohertj2 · 4 comments
dohertj2 commented 2026-10-13 12:56:23 -04:00
Owner
Copy Link
Copy Source

Source-evidenced startup race

DeferredServiceLevelPublisher routes pre-binding Publish values to NullServiceLevelPublisher and SetInner only swaps the reference; it does not retain/replay the current value. OpcUaPublishActor's ServiceLevelChanged handler records publication as successful and deduplicates subsequent identical values. OtOpcUaServerHostedService binds the SDK publisher only after server startup.

Thus a ServiceLevel calculated before binding can be discarded and the same current value suppressed after binding, leaving the SDK initial level until a different state is published. This is source analysis, not a reproduced production incident. The #717 loopback spike did not exercise actor/host startup.

Sources: src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredServiceLevelPublisher.cs; src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs ServiceLevelChanged handler; src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs SDK publisher binding.

Acceptance

  • Reproduce deterministically publication before binding followed by unchanged state after binding; show the latest state reaches the real publisher.
  • Define thread-safe ordering for concurrent Publish/SetInner, rebinding/unbinding, and publisher failure so an older level cannot overwrite newer state or be incorrectly marked delivered.
  • Prove client-visible startup ServiceLevel and binding handoff through appropriate host/loopback coverage; preserve existing health/election decisions.
  • For #717 Warm, ensure startup readiness cannot advertise healthy/active before the approved authority is established; do not claim this adapter fix alone implements Warm.
  • Review and run affected/full required tests, publish verified fix. No deployment/live probe authorized by this issue.

Related #717 and #615. Track separately because dropped late-bound publication also affects current deployments independent of future redundancy metadata.

## Source-evidenced startup race DeferredServiceLevelPublisher routes pre-binding Publish values to NullServiceLevelPublisher and SetInner only swaps the reference; it does not retain/replay the current value. OpcUaPublishActor's ServiceLevelChanged handler records publication as successful and deduplicates subsequent identical values. OtOpcUaServerHostedService binds the SDK publisher only after server startup. Thus a ServiceLevel calculated before binding can be discarded and the same current value suppressed after binding, leaving the SDK initial level until a different state is published. This is source analysis, not a reproduced production incident. The #717 loopback spike did not exercise actor/host startup. Sources: src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredServiceLevelPublisher.cs; src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs ServiceLevelChanged handler; src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs SDK publisher binding. ## Acceptance - Reproduce deterministically publication before binding followed by unchanged state after binding; show the latest state reaches the real publisher. - Define thread-safe ordering for concurrent Publish/SetInner, rebinding/unbinding, and publisher failure so an older level cannot overwrite newer state or be incorrectly marked delivered. - Prove client-visible startup ServiceLevel and binding handoff through appropriate host/loopback coverage; preserve existing health/election decisions. - For #717 Warm, ensure startup readiness cannot advertise healthy/active before the approved authority is established; do not claim this adapter fix alone implements Warm. - Review and run affected/full required tests, publish verified fix. No deployment/live probe authorized by this issue. Related #717 and #615. Track separately because dropped late-bound publication also affects current deployments independent of future redundancy metadata.
dohertj2 commented 2026-10-13 13:00:07 -04:00
Author
Owner
Copy Link
Copy Source

Read-only design review additionally confirmed two delivery-observation gaps: SdkServiceLevelPublisher logs/swallowing missing-node or write faults makes failure look successful, and actor dedup must be invalidated after any failed publish (success A, failed B, then A must deliver A again). Moving the remembered byte alone is insufficient. Current deferred tests assert safe pre-bind calls but do not cover binding replay.

Bounded design presented for approval: serialized latest-value retention/replay including0; failed-bind candidate/latest retained with visible failure; actor remembers successful value and invalidates dedup on failure; SDK propagation; cleanup of started OPC endpoint after replay failure preserving existing admin-host startup policy. Deterministic ordering/rebind/fault tests and actual loopback Read are required, followed by review and hosted CI. No implementation has started; no change to health arithmetic, election or mutation policies. First-ever readiness/default255 remains #717, not solved by replay alone.

Read-only design review additionally confirmed two delivery-observation gaps: SdkServiceLevelPublisher logs/swallowing missing-node or write faults makes failure look successful, and actor dedup must be invalidated after any failed publish (success A, failed B, then A must deliver A again). Moving the remembered byte alone is insufficient. Current deferred tests assert safe pre-bind calls but do not cover binding replay. Bounded design presented for approval: serialized latest-value retention/replay including0; failed-bind candidate/latest retained with visible failure; actor remembers successful value and invalidates dedup on failure; SDK propagation; cleanup of started OPC endpoint after replay failure preserving existing admin-host startup policy. Deterministic ordering/rebind/fault tests and actual loopback Read are required, followed by review and hosted CI. No implementation has started; no change to health arithmetic, election or mutation policies. First-ever readiness/default255 remains #717, not solved by replay alone.
dohertj2 commented 2026-10-13 13:27:12 -04:00
Author
Owner
Copy Link
Copy Source

Reviewed fix committed and pushed as c8e3e279b4 in PR #720. Affected suite verification: 2,381 passed, 4 existing skips, zero failures. Independent root loopback rerun passed both client ServiceLevel reads (100 and 0). Fault/mutation regressions cover cache retention, binding order, SDK visibility, dedup recovery and cleanup ownership. Real Host construction-failure test proves listener closure/admin continuation; replay/stop fault injection has the composition limits documented in the PR.

Hosted full CI remains required before merge and closure. No deployment or external device/historian calls. #717 Warm contract remains separate and open.

Reviewed fix committed and pushed as c8e3e279b4f9f1e8d420bb8adf83c5428e1b1b90 in PR #720. Affected suite verification: 2,381 passed, 4 existing skips, zero failures. Independent root loopback rerun passed both client ServiceLevel reads (100 and 0). Fault/mutation regressions cover cache retention, binding order, SDK visibility, dedup recovery and cleanup ownership. Real Host construction-failure test proves listener closure/admin continuation; replay/stop fault injection has the composition limits documented in the PR. Hosted full CI remains required before merge and closure. No deployment or external device/historian calls. #717 Warm contract remains separate and open.
dohertj2 commented 2026-10-13 13:58:27 -04:00
Author
Owner
Copy Link
Copy Source

PR #714 is now merged as 4f7a07fbd3 after full hosted run979 succeeded; #712/#606 are closed. PR #720 now includes that main history at head8828e999c41057ae7b9d84573228ab8caf71fe06. Its combined tree is a7ce6c7688, unchanged by the final ancestry-only main merge. All11 guard fixtures passed in the combined checkout.

Required final-head run983 is queued behind the already active prior-head run980. No manual cancellation/rerun occurred. Earlier-head CI will not replace the final-head gate. #719 remains open until verified merge.

PR #714 is now merged as 4f7a07fbd358b18cf441eba5ce80721f775babc4 after full hosted run979 succeeded; #712/#606 are closed. PR #720 now includes that main history at head8828e999c41057ae7b9d84573228ab8caf71fe06. Its combined tree is a7ce6c7688e5aa5e28c47519fd526c809306f93e, unchanged by the final ancestry-only main merge. All11 guard fixtures passed in the combined checkout. Required final-head run983 is queued behind the already active prior-head run980. No manual cancellation/rerun occurred. Earlier-head CI will not replace the final-head gate. #719 remains open until verified merge.
dohertj2 commented 2026-10-13 16:00:00 -04:00
Author
Owner
Copy Link
Copy Source

Resolved and merged in PR #720 as master 1bf8c614a9. Required final-head run #983/job3604 succeeded on 8828e999c4: all steps passed, 62 TRX reports, 14,648 executed tests, 573 individual skips, zero failures, minimum 12,000, and artifact upload finalized. Full Runtime passed 829 with three existing skips. Merged tree a7ce6c7688 exactly equals tested tree; issue closure verified.

Retained latest ServiceLevel including zero, serialized replay/binding and SDK failure visibility; actor dedup only after success and invalidates on failure. Startup binding cleanup preserves admin boot and retry ownership. Existing health arithmetic/election/mutation policies remain unchanged. Local real-client reads and ordering/failure regressions were independently reviewed. No deployment or live historian/device calls. Warm #717 remains separate and open.

Resolved and merged in PR #720 as master 1bf8c614a9fb9fb160a818fbb47bcf439b24ca00. Required final-head run #983/job3604 succeeded on 8828e999c41057ae7b9d84573228ab8caf71fe06: all steps passed, 62 TRX reports, 14,648 executed tests, 573 individual skips, zero failures, minimum 12,000, and artifact upload finalized. Full Runtime passed 829 with three existing skips. Merged tree a7ce6c7688e5aa5e28c47519fd526c809306f93e exactly equals tested tree; issue closure verified. Retained latest ServiceLevel including zero, serialized replay/binding and SDK failure visibility; actor dedup only after success and invalidates on failure. Startup binding cleanup preserves admin boot and retry ownership. Existing health arithmetic/election/mutation policies remain unchanged. Local real-client reads and ordering/failure regressions were independently reviewed. No deployment or live historian/device calls. Warm #717 remains separate and open.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#719
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?