Resolve standard non-transparent redundancy metadata deferred by current server #717
Assignees
dohertj2 (dohertj2)
Clear assignees
dohertj2 (dohertj2)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: dohertj2/lmxopcua#717
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Current contract and evidence (2026-10-13)
The owner selected Warm with an explicit degraded standby contract, and None for standalone deployments. Ordinary standby tag Reads and subscriptions must degrade; internal acquisition continues. Event subscriptions remain established while new standby alarm events are suppressed, with current retained conditions refreshed on promotion.
Two approved throwaway pinned-SDK 1.5.378.106 loopback probes completed and were independently rerun: standard graph Read/Browse/Translate for Warm and None; event admission/retention/promotion refresh (28 assertions). These prove supported SDK feasibility, not product implementation or CTT/client acceptance. Previously admitted events and Republish survived demotion; exact concurrent/in-flight boundaries and overflow were not verified.
Source audits recommend a per-tag OnSimpleReadValue projection callback and product alarm suppression before SDK field projection. The queue-only spike gate has late overflow/filtered-retain effects and must not be adopted unchanged. All ACL/direct/Server refresh paths need coverage. The owner decision on client-initiated standby ConditionRefresh is pending. Role snapshots have no ordered lease/revision, and current driver/config signals do not establish an existing global readiness contract.
Independent ServiceLevel binding/recovery work is published in #719 / PR #720, awaiting hosted CI; it retains/replays requests but does not synthesize a startup request when no snapshot exists. A newly found DB-less freshness-timer gap is tracked separately by #721.
Conversational architectural design, written spec and implementation plan approvals remain incomplete. No Warm product code/configuration, rollout, external probe, CTT or production-client acceptance has been performed. Keep all #615 gates open. Original finding and required work are preserved below.
Finding
Current docs/Redundancy.md:7 explicitly defers true Server.ServerRedundancy.ServerUriArray. OpcUaApplicationHost.PopulateServerArray populates Server.ServerArray (i=2254); the in-code ServerConfiguration builder does not set a redundancy mode. Source inspection found no initialization of the non-transparent redundancy object/property. A supplied application configuration may differ and must be checked before drawing deployment conclusions.
This is not a live reproduction. The runbook's i=2271 is also wrong: SDK standard constants map it to LocaleIdArray; true ServerUriArray is i=11314. Publishing peer URIs in ServerArray does not prove the standard redundancy object contract.
Standard / scope
OPC Foundation Part 5 §6.3.9 defines ServerUriArray as mandatory on NonTransparentRedundancyType and specifies the supported modes. Current product documentation advertises non-transparent warm/hot redundancy while deferring this property. #615's statement that all code-path blockers are closed therefore needs qualification.
Required work
No production implementation or live call was performed in this audit. Parent #615 remains open pending all original gates.
Read-only SDK feasibility update: pinned 1.5.378.106 package metadata identifies source commit 99044fdbb638087d62f1e16580e9f73bfda364e2. Its APIs expose StandardServer.CreateMainNodeManagerFactory and ConfigurationNodeManager.AddBehaviourToPredefinedNode, allowing construction before node indexing/import. NonTransparentRedundancyState exists; ServerConfiguration has no RedundancySupport setting. The local SDK source checkout is 1.5.377-preview and was not treated as matching implementation proof.
This is preliminary static feasibility, not a proven wire fix. Final IDs, HasTypeDefinition/HasProperty graph, registration and Browse/Read/TranslateBrowsePaths require a bounded hermetic probe; design approval is pending. Warm/Hot mode policy, config-source authority and static versus dynamic peer advertisement remain undecided. ApplicationConfigPath can change the effective self URI; a fix must use the effective configuration, not an options default. Election, actor readiness, write/ack/history gates and G8/G9 live acceptance stay separate.
Approved hermetic spike completed and independently rerun by root.
Pinned SDK 1.5.378.106 supports the correct startup-static non-transparent metadata graph through StandardServer.CreateMainNodeManagerFactory and ConfigurationNodeManager.AddBehaviourToPredefinedNode, without an upgrade. Real loopback client Read/Browse/Translate passed for synthetic Warm and standalone None. Root rerun exited 0 in 2.08 seconds; actual listeners were exclusively 127.0.0.1 and both ports were absent after cleanup.
Warm: object 2296 has type 2039, support 3709 = 2, and read-only String[] ServerUriArray 11314 with effective self URI then synthetic partner. None: support 0, type 2034, no manufactured ServerUriArray. SDK later visits a passive predefined 11314; that visit must resolve to the same constructed property instance before Core import to avoid a null wire value.
Recommend the supported pre-registration seam for a separately approved production design. The fixture is throwaway only; no production code/configuration, actor behavior, external endpoints or dependency changes. Advertised mode policy remains undecided. CTT, real production-client acceptance, election/readiness and dynamic membership remain unproven. #717 and #615 remain open; this resolves only SDK feasibility.
Read-only policy audit after the successful spike identified remaining design decisions, without choosing or implementing a production mode.
DriverHostActor initializes every local driver and sends its desired subscriptions regardless of redundancy role (lines 3658–3693). Secondary read/monitoring and alarm evaluation remain active; existing device-write, native-ack and HistoryUpdate Primary gates deliberately differ by startup/operation. ServiceLevelCalculator uses health basis 240 plus 10 Primary bonus; publication inputs do not directly establish driver-subscription readiness or configuration revision.
Current docs/Redundancy.md claims both Warm/Hot, defers true ServerUriArray and incorrectly describes an SDK upgrade as necessary. Its generic standby/no-device-contact, ungated-ack and exactly-once wording needs reconciliation with actual acquisition, native-ack gating and documented at-least-once failover. These belong in #717's consistent discovery/operational-contract documentation; #716 remains the distinct obsolete runbook repair.
Official OPC Foundation Part 4 §6.6.2.4 describes Hot parallel acquisition and client requests to the highest-ServiceLevel server; Warm backup functionality and ServiceLevel differ. Continuous acquisition alone does not establish full Hot acceptance. https://reference.opcfoundation.org/specs/OPC-10000-4/6.6.2.4
Owner clarification requested for intended advertised contract. Configuration authority, XML precedence, static/dynamic peer identity and startup readiness also need an approved production design. Preserve current mutation/election policies; the spike does not authorize their modification or close CTT/production-client gates.
Owner selected Warm with an explicit degraded standby contract, with standalone None.
This establishes the intended contract, not approval of an implementation design. Official Part 4 §6.6.2.4.5.3 requires inactive-server MonitoredItems to publish appropriate errors instead of actual data; standby ServiceLevel belongs in the degraded 2–199 range (§6.6.2.4.2). Existing live secondary monitoring and healthy 240 cannot simply be relabeled Warm.
https://reference.opcfoundation.org/specs/OPC-10000-4/6.6.2.4.5.3
https://reference.opcfoundation.org/specs/OPC-10000-4/6.6.2.4.2
Scope therefore moves beyond metadata graph to an architectural design for SDK graph, explicit configuration, role/readiness authority, ServiceLevel and client notification behavior while preserving internal acquisition and existing mutation gates. Read-only seam investigation is underway; no production change, retained spike code, new tests, deployment or live call authorized by this contract choice. Written design and implementation-plan review remain required. CTT and production-client cutover gates remain separate.
Owner further clarified Warm standby: ordinary tag Reads and subscriptions should degrade; internal acquisition continues, with discovery/ServiceLevel diagnostics available. These are contract decisions, not implementation approval.
Exact pinned-SDK read-only event audit found event-only MonitoredItems cannot receive DataValue error notifications via QueueValue, and SetSamplingError only affects create/modify results. Existing built-in StatusChangeNotification paths are timeout/transfer, not a generic per-item standby status. A supported custom event-queue factory exists, but item flushing does not remove notification messages already constructed in the subscription's Publish/Republish buffers. Promotion condition refresh has public seams but needs wire verification.
Consequently event standby indication, pending-message boundaries and promotion refresh require their own explicit contract and approved hermetic proof before the architectural design can claim Warm event behavior. No event probe, product change or new dependency was performed. Keep CTT/client acceptance gates open.
Owner selected the event contract: keep standby event subscriptions established and suppress newly generated alarm events, with ServiceLevel indicating standby and condition refresh on promotion. Existing prepared/in-flight messages and Republish retention remain explicit boundary questions.
A separate throwaway pinned-SDK event probe is approved and running: supported event-queue extension, one loopback server/client, synthetic conditions only, no product actors/configuration/dependencies or external calls. It will observe queue/delivery boundaries and refresh behavior. Root will independently inspect and rerun final stable evidence before reporting feasibility. This contract/probe approval does not approve retaining prototype code or implementing the architectural change; #717 remains open.
#717 event spike results
The approved throwaway probe shows that pinned SDK 1.5.378.106 can suppress new standby event queue arrivals while preserving subscriptions and their Reporting mode. Promotion delivered RefreshStart, the current retained synthetic alarm condition, and RefreshEnd on both subscriptions, followed by new alarms.
I independently reviewed and reran the fixture: exit 0 in 5.379 seconds, 28 assertions passed, 25 admissions and 6 drops. Only a loopback server and local client were used; the listener was closed after cleanup.
The boundary is queue admission. Previously admitted active alarms still arrived after demotion, and Republish retained its original message and sequence. The outstanding client Publish task does not prove an exact server-side pending-Publish/demotion race. Concurrent generation, overflow, durable queues, client refresh during standby, and production alarm versus audit/system event categories remain unverified.
This is feasibility evidence only. Production tag quality, role transitions, readiness, actor integration, CTT, and production client acceptance remain open under #717/#615. No product changes or external calls were made.
Evidence: scratch
717-event-spike/RESULTS.md,result.json,run.log,dependency-evidence.json, and independentroot-cleanup-evidence.json. Architectural design remains unapproved; #717 stays open.Read-only follow-up after the approved spikes:
Architectural design/spec/plan remain unapproved. Internal acquisition must continue, ordinary tag Reads/subscriptions degrade on standby, diagnostics remain available, and event subscriptions remain established. A client-initiated standby ConditionRefresh policy question is pending with the owner. #717 stays open; no product code changed.
Read-only follow-up on the selected Warm contract, using the exact cached SDK 1.5.378.106. No product changes, builds, tests, live calls or dependency changes.
Sampling BadNoCommunication does not flush earlier Good data values. The SDK keeps a separate backlog for queue sizes above one; both discard policies can retain older Good entries. Size-one and empty-queue ResendData use the last-value slot instead of the custom queue factory.
A supported MonitoredItem Publish override could project values as they leave an item, but the current production node manager creates standard MonitoredItem instances. Installing a subclass requires preserving node-manager creation, registration, initial reads and restore lifecycle; replacing only the returned item would leave registrations pointing to the original. Root independently checked the exact default manager creation/restore and Publish dispatch source.
Constructed-but-unsent NotificationMessages and Republish are another boundary: Subscription returns its cached message before invoking monitored-item Publish, and Republish returns the cached message directly. No selective ordinary-tag message-cache rewrite/clear extension was established. These are source findings, not demonstrated wire transitions.
#717 remains open. The architecture must explicitly define treatment of pre-demotion queued data, prebuilt first-delivery messages and retransmission before selecting a transition mechanism. The pending client-initiated standby ConditionRefresh choice, readiness policy, approved architectural design and subsequent verification are still required. The earlier metadata/event spikes do not establish these missing guarantees.
Owner selected the strict first-delivery contract for #717: after demotion, subsequent first-delivery Publish responses must not contain Good ordinary tag values, including previously queued samples and prebuilt batches. Republish history and responses already in flight remain preserved. This is a requirement decision, not approval of an implementation mechanism or a new spike.
The Warm service description supports returning an appropriate monitored-item error from standby Publish responses: https://reference.opcfoundation.org/specs/OPC-10000-4/6.6.2.4.5.3 . Its broad wording does not by itself resolve our precise transition/cache boundaries. The owner decision above does; no CTT/client acceptance is claimed.
Further read-only source exploration found a broader supported service seam: StandardServer.PublishAsync can be overridden around the manager response, and CreateSubscriptionManager exposes a public-interface replacement seam. These may reach prebuilt messages that an item-level projection misses. They are candidates requiring feasibility verification, not proven solutions: returned messages alias SDK cached objects, and rewriting only the wire response can disagree with Republish. Ordinary-tag origin also must survive client-handle changes/deletion/reuse and mixed subscriptions; the current live mapping alone is insufficient proof.
The source audit is continuing. No product code, new runtime probes, SDK upgrades or external calls were introduced. The pending standby client-initiated ConditionRefresh choice and architectural readiness/design/spec/plan reviews remain separate requirements.
Completed the exact-SDK read-only Publish-response audit. The production StandardServer subclass can override PublishAsync and RepublishAsync, reaching successful responses including prebuilt batches. This broadens the feasible integration seams beyond item-level sampling; it does not prove a complete mechanism.
The response aliases SDK retransmission-cache objects, potentially including retained last DataValues. Publication state is released before an outer override resumes, so in-place mutation can race concurrent requests and persistence. Detached projection needs coordinated first-delivery history so later Republish returns the payload actually chosen for that message. Prebuilt messages can be advertised for Republish before ordinary Publish exposes them; first delivery through Republish must therefore be considered, rather than assuming every Republish is already delivered history.
Final notifications contain ClientHandle and Value, not immutable NodeId/server-item provenance. Current item maps alone cannot classify old notifications safely after handle modification, deletion, reuse, or ambiguous handles. A supported custom-item creation/publication path can potentially capture identity before batching, but its real lifecycle integration remains unverified. The broader subscription-manager factory is also not a trivial wrapper: the SDK has a concrete SubscriptionManager cast on expiry.
The owner-selected strict first-delivery contract remains the target. No new-samples-only substitute, product implementation, new runtime probe or CTT acceptance is claimed. A separately approved throwaway provenance spike has been proposed and is pending; it would test the identity prerequisite, not the entire history/concurrency mechanism. Architectural design/spec/plan and readiness decisions remain required.
Approved throwaway provenance spike completed on pinned SDK1.5.378.106. Agent run and root independent unchanged-source run both exited0 and passed66 assertions. Root confirmed the listener on127.0.0.1:63206 was gone afterward (lsof1,empty).
A supported explicit creation-interface override on a subclass of the default data-item manager installs the original custom items while retaining SDK registries, factory validation, initial reads and lifecycle callbacks. A protected per-value Publish override calls base and stamps the newly created original notification objects with immutable node/attribute/category identity. The outer PublishAsync override then resolves those same objects without touching the cache or rewriting responses.
Actual client evidence: identical initial values and shared ClientHandle77 still classify as two different fixture categories; earlier queued samples publish after handle modification to88. The public whole-cache snapshot showed prebuilt sequences4–8 before both items changed to99 and were deleted. Later wire delivery of those sequences retained prior object stamps/categories and historical handle88, even though live item/node registrations were empty. Status-only filtering, queue-size-one, Sampling→Reporting and Disabled controls passed.
Scope limits: diagnostic is a synthetic variable in the custom namespace, not complete SDK namespace-zero routing. Reuse was tested after old batches drained. Normal PublishAvailable lists through the sent index; the public whole-cache snapshot was the prebuilt proof, not a TransferSubscriptions run. No strict demotion projection/history, concurrent Publish/Republish, transfer/restore, aggregate/overflow matrix or CTT proof. Fixture build had14 warnings and0errors; no product files or dependencies changed.
Recommendation: retain this supported provenance route as a candidate for the architectural design, with production taxonomy/lifecycle review. The strict first-delivery and exact Republish-history mechanism still needs separate feasibility verification/approval. #717 remains open; the fixture remains throwaway.
Approved throwaway delivery-history spike completed on cached SDK 1.5.378.106. Independent root source review and one cleared-environment loopback run passed: 53 assertions, 16 successful wire responses, 20 provenance stamps and 16 delivery decisions, with endpoint cleanup confirmed. No product edits or external historian calls.
Public Publish/Republish overrides call SDK base first, then weakly associate each original cached message with one detached chosen payload. A short lock synchronizes that first choice with fixture role changes; no lock spans service awaits. Earlier delivered Good history survives standby Republish; queued/prebuilt ordinary values first delivered on standby become BadNoCommunication/null while synthetic diagnostics remain Good. Republish can establish the first choice for an unsent prebuilt message; promotion and concurrent responses preserve the same choice. The pending case observes an incomplete SDK base Publish operation before demotion. Successful acknowledgement still yields SDK BadMessageNotAvailable on later Republish. Original cached payloads remain unchanged, including explicit sacrificial-clone checks.
Limits: scalar synthetic custom-namespace diagnostics only; decoded payload fields, not encoded bytes. The first-choice lock is a fixture boundary after base completion, not proof of transport completion or product-wide demotion. No full namespace-zero taxonomy, canceled/failed-response policy, cache-eviction runtime case, transfer, durable restore, sequence wrap, GC/resource bounds or CTT evidence. Production Warm design, role/readiness coordination and implementation still require review/approval; #717 remains open.
Evidence: outputs/717-delivery-history-spike-results.md; work/717-delivery-history-spike/{Program.cs,agent-result.json,result.json,root-run.log,root-cleanup-evidence.json}. Final source SHA256 fe491d26d71446ef9a8e33e247da07d7f9d69bab702079b937b70972a3247316.
Read-only production notification-taxonomy audit completed and root-reviewed against current head
8828e999and pinned SDK 1.5.378.106.Ordinary raw/UNS/equipment/virtual Value items use the product CustomNodeManager2 route. That same manager also owns condition children, metadata attributes and legacy alarm bool[2] fallback;
_variablesmembership, manager identity, numeric namespace or ClientHandle alone cannot define ordinary-tag data. Capture declared/creator intent plus actual node identity and AttributeId at item creation. Namespace-zero ServiceLevel/server/discovery diagnostics belong to ConfigurationNodeManager -> DiagnosticsNodeManager -> CustomNodeManager2. A supported main-node-manager factory seam exists, but replacing it is not established as necessary merely to degrade product tags.Aggregate processed values and size-one/resend use the item's protected per-value Publish path; raw input DataValue identity or a FIFO queue marker alone is incomplete. Restore has a separate factory and prebuilt stored messages require their own provenance treatment. Events/status/keepalive are distinct bodies. No implicit unknown-origin rewrite policy is approved.
This supplements the already passed synthetic provenance/history spikes. No production edits or new runtime probes; #717 remains open for architectural design, complete category/lifecycle coverage and role/readiness coordination. Evidence: work/lmx717-notification-taxonomy-audit.md and work/lmx717-requirements-evidence-matrix.md.
Read-only exact SDK1.5.378.106/net10 clone-chain follow-up: generated standard notification fields recurse through Utils/CoreUtils.Clone; arrays/matrices/byte strings have recursive paths, but arbitrary custom encodeables depend on their own clone implementations and can shallow-alias or lack a supported path. MonitoredItemNotification.Message/DiagnosticInfo and EventFieldList.Handle/Message helper references are not reset/cloned; SDK GetDataChanges/GetEvents can set Message to the original graph. DiagnosticInfo copy recursion truncates nested InnerDiagnosticInfo beyond MaxInnerDepth5. These are source-derived limits, not new observed runtime failures. The approved53-assertion scalar history spike remains valid for its exercised graph; it does not establish full payload/encoded-history fidelity for these cases. Requirements matrix now calls out mutable arrays/extension bodies/diagnostics/events and helper references for bounded verification before choosing a product clone policy. No runtime runs, product edits, dependency changes or architecture approval; #717 stays open.