Files
lmxopcua/docs/plans/2026-11-15-grpc-data-connections-design.md
T
Joseph Doherty 5dbdee1ea3
v2-ci / unit-tests (push) Failing after 2h40m48s
docs: plan grpc data connections and record Astra reviews
2026-11-15 20:59:53 -05:00

69 KiB
Raw Blame History

gRPC data connections between nodes and clusters

Date: 2026-11-15 Status: Astra reviewed, including scaling architecture; approved for user design review after revision. Awaiting user approval; no implementation authorized.

Review: Astra findings and resolutions. Implementation has not started.

1. Decision and scope

Add an explicit data-plane connection over gRPC for cached reads, acknowledged writes, subscriptions, browsing, and metadata. Use the same protocol for a connection to a named node within an application cluster and a connection to another application cluster. Preserve independent Akka clusters and their existing elections; gRPC is a data transport, not a replacement membership or replication protocol.

User-directed namespace model: expose both current Raw and UNS hierarchies through mappings to build expandable naming structures spanning configured nodes and clusters, with cross-node and cross-cluster browse included in the first release. Map remote Raw branches into the local Raw view and remote UNS branches into the local UNS view rather than maintaining an unrelated flat import list. A mapping may expose an authorized subtree, so operators can expand its hierarchy and discover descendants without authoring one import per tag. Reads, writes and subscriptions resolve the same mapping to the owning cluster. Also expose an internal connection API usable by runtime consumers without OPC UA loopback. Connections are directional; reverse access requires its own grants and configuration. A remote cluster retains ownership of its drivers, values, permissions, and write decisions.

The user clarification resolves the initial imports-versus-routing question: routing is driven by the composed Raw/UNS namespaces and their explicit mappings. The same gRPC service handles browse, metadata, read, write and subscribe for nodes within one cluster and for different clusters; intracluster access is a required first-release topology, not only a diagnostic case. It does not enable arbitrary destinations supplied by callers or transitive peer forwarding.

Alternatives

Approach Benefit Cost / decision
Dedicated gRPC data service and runtime connection adapter (recommended) One protocol for node and cluster access; explicit quality, lifecycle, and write semantics; fits existing hosting Requires a typed codec, value hub, and lifecycle management
Reuse the OPC UA client driver between servers Existing browse/read/write/subscribe interoperability Does not deliver the requested gRPC transport; retains OPC UA session and encoding overhead
Extend the existing mesh-command and telemetry messages Reuses existing listeners and dialers Command receipt is not a write outcome; telemetry is lossy and lacks a tag-value contract. Keep their semantics intact

Included in the first release

  • Explicit configured node or cluster connections, batched cached reads, value writes, browse/metadata, and data-change subscriptions, including actual gRPC calls between different nodes of the same cluster.
  • Raw and UNS branch mappings with cross-node and cross-cluster browse/expand, stable origin identities, demand-driven value imports, local quality updates, and optional explicitly authorized write-through.
  • Peer authentication, target-side operation authorization, bounded resource use, reconnect, pair failover, diagnostics, and compatibility negotiation.
  • Shared backing samples and authorized demand aggregation, partitioned processing, compact subscription handles/batches, cached authorization, client filtering, and controlled recovery as specified in §6a. No fixed throughput or tag/client capacity is promised.

History reads and alarm/event subscriptions and methods are subsequent capability increments. Reserve their extension boundary now, but do not advertise them until implemented and tested. History updates, arbitrary method execution, automatic discovery of untrusted peers, configuration replication, and replacing Akka are excluded. The initial service is an application data API, not a complete OPC UA service tunnel.

2. Repository evidence

Paths below are relative to the repository root and describe the inspected checkout at 6dd2985b8.

Existing surface Consequence for this design
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto ExecuteAck means received/re-emitted, not applied. Do not tunnel writes through NodeCommandService.Execute
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/telemetry.proto and src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/TelemetryLocalHub.cs Existing stream is alarms/logs/health/resilience; its DropOldest behavior cannot silently become the data subscription contract
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs, ForwardBatchToMux Driver batches fan out to realm-qualified Raw and UNS values. Capture authoritative samples and source-removal/comms-loss updates without coupling to a network writer
Same file, HandleRouteNodeWrite, and Drivers/ActorNodeWriteGateway.cs Existing async write route checks primary and resolves (realm, identifier) to the driver. Preserve its bounded driver execution; enrich its result before promising precise remote outcomes
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs Publication also includes virtual values, alarm state, status-only changes, deferred apply and deletions. A driver-only tap would miss part of the address space
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs and Security/DataPlaneAuthorizer.cs Current ACL enforcement covers request seams and live-subscription revalidation; extract/reuse policy and subject resolution, never bypass it by calling a driver gateway
src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/DataPlaneAuthority.cs Immutable authority generations and expiry/revocation inputs already exist
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs Endpoint mapping has regression guards for cookie fallback and interceptor authentication. New service needs equally explicit middleware/auth metadata and listener isolation
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs and Entities/ServerCluster.cs Node identity differs from cluster identity. Existing GrpcPort describes telemetry; do not silently redefine it as a secure data endpoint
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/UnsArea.cs, Entities/UnsLine.cs, Entities/Equipment.cs and Entities/UnsTagReference.cs Existing UNS is Area → Line → Equipment → effective signal; references are local raw-tag FKs. Add a distinct remote branch mapping instead of putting a remote identifier into a local TagId
src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs Current UNS uses URI-qualified, segment-validated paths; retain local identities and centralize projected identity construction
docs/Redundancy.md Application clusters are independent singleton/pair meshes; central is not a required relay

docs/ReadWriteOperations.md contains older claims that ACLs are not evaluated and reads have no override. Current source contradicts these claims. Treat current source and its tests as authoritative; correct those portions as part of implementation documentation.

3. Architecture and component ownership

flowchart LR
    C[Raw and UNS browse / mapped data / runtime consumers] --> N[Realm-qualified mapping resolver]
    N --> R[DataConnectionManager]
    R --> L[Local adapter]
    R --> G[gRPC connection adapter]
    G --> S[Peer data service in Host]
    S --> A[Authenticated principal + operation policy]
    A --> V[Local data access ports]
    V --> H[Partitioned versioned sample hub]
    H --> F[Bounded delivery scheduler]
    F --> S
    V --> W[DriverHost write route + primary gate]
    H --> P[Existing address-space publication sources]

Proposed components, not existing classes:

  • Core.Abstractions: transport-neutral IDataConnection, IDataReadService, IDataWriteService, IDataSubscriptionService, catalog contracts and typed results. No gRPC, Akka, ASP.NET or OPC UA SDK dependencies.
  • Commons: data_connection.v1 protobuf, codec and contract tests, following current shared-contract placement. Keep generated DTOs out of domain interfaces.
  • Runtime: partitioned data/catalog hub, delivery scheduler and authorization caches, actor adapters, connection manager, realm-qualified mapping resolver, federated browse/catalog projection and shared remote-demand subscriptions. Local calls and remote ingress use the same authorization and execution services.
  • Host: gRPC ingress, authentication, listener configuration, endpoint admission, metrics and connection registration. It converts DTOs and invokes ports; it does not resolve actors itself.
  • Configuration/ControlPlane: connection/export/namespace-mapping entities, validation and deployment-artifact projection. DB-less driver nodes receive resolved connection configuration through artifacts, with secrets delivered through the existing secret mechanism, never plaintext in artifacts.
  • Driver integration: a gRPC data connection adapter shaped to the existing capability model (IReadable, ISubscribable, write capability). It supplies imported values through ordinary runtime publication and dependency evaluation. Registering this adapter must not create a second remote route for the same backing point.

The local hub is a versioned projection of the active local catalog and authoritative value/status feeds, independent of whether the OPC UA endpoint is running. Build one shared desired catalog/ACL-subject projection from the deployment composition for both OPC UA and the hub; do not derive alias paths or UNS ownership independently. Each adapter reports activation readiness against that shared projection. Extract a shared publication boundary used by driver, virtual-tag and status/removal paths; attach before any optional OPC UA sink early return. Stage the SDK-independent catalog and value projection for each deployment, and serve only addresses whose catalog subjects match the active authority generation. Authority revocation publishes immediately, including authority-only deployments; never roll it back because topology materialization fails. If the new catalog cannot activate, mark the data endpoint unavailable for mismatched subjects until repaired. A coherent served generation means fail-closed compatibility, not an atomic rollback of authority and topology. This explicitly accounts for OpcUaPublishActor.ApplyPlanCore feeding authority before a potentially degraded address-space apply. Do not read SDK NodeState internals or optimistic operator-write values as authoritative device samples.

Before building this seam, inventory every publication source and verify parity for driver values, virtual tags, comms quality, status-only updates, removal, rename and apply buffering. A node is unavailable until its initial catalog generation is ready. A known node with no sample returns BadWaitingForInitialData, never Good/null by default.

4. Identity, catalog and value representation

A public data address is (originClusterId, namespaceUri, identifierKind, identifier), with realm and resolved ACL subject included in catalog metadata. Only supported identifier kinds are advertised. Numeric OPC UA namespace indexes are process-local and never cross the wire as identity. Do not confuse the Commons machine NodeId with a data address.

Each connection binds an expected cluster identity; a node connection also binds an expected machine identity. Renaming a path invalidates that address and produces catalog removal/addition unless a configured stable backing ID explicitly survives it. An imported node stores its original address and maps to a distinct local identity; it does not acquire ownership of the origin.

The initial codec uses explicit typed protobuf values: null, Boolean, signed/unsigned integers with declared widths, Float/Double, string, bytes, UTC timestamps, GUID, supported typed arrays and dimensions. Preserve UInt64 precision, empty versus null, NaN/infinity, and array shape. Node references carry namespace URIs. Carry the complete uint32 OPC UA status, optional source/server timestamps, origin identity, and sample version. Unsupported structured/ExtensionObject types return BadDataEncodingUnsupported; never stringify or deserialize arbitrary CLR types. Advertise the supported type set and reject unsupported imports at validation time.

Current DriverInstanceActor.AttributeValueItem collapses source/server timestamps into one TimestampUtc. To retain both when available, extend the shared publication envelope and its consumers; do not relabel the collapsed timestamp as two original timestamps. When a source supplies only one timestamp, the other remains absent. Origin acquisition timestamps and local import-receive time are distinct.

Catalog metadata includes data type, rank/dimensions, writable capability, origin, export permissions, revision and optional engineering units. Browse is paged and authorization-filtered. Opaque continuations bind principal, connection, filter and catalog revision, expire after 60 seconds, and are invalidated on revision change. Peer export catalogs contain origin-local nodes only in v1. The local client-facing Raw and UNS views can combine mapped branches from multiple directly connected peers, but a peer data-service call never recursively re-exports another peer’s mapped branches. Each remote operation uses one direct connection to its owning cluster, eliminating forwarding loops and ambiguous ownership.

Raw and UNS mappings and cross-node/cross-cluster browse

Use the existing Raw and UNS structures as naming authorities. Generalize the earlier UnsRemoteMapping proposal into a versioned RemoteNamespaceMapping containing an explicit realm, local parent identity and exposed branch name, connection, remote root (stable entity identity plus current canonical address), allowed operation subset and projection limits. Local and remote realms must match in v1. Bind local parents through existing Raw-folder or UNS entity identities and compile paths through the shared deployed composition; do not introduce a parallel manually maintained naming tree. A Raw mapping exposes Folder → Driver → Device → TagGroup → Tag, including nested folders and tag groups as currently presented. A UNS mapping exposes Area → Line → Equipment → signal. Either supports an explicitly mapped leaf as a constrained special case. Browse returns the origin’s deployed/addressable catalog, not uncommitted device-discovery results; browsing a remote driver does not connect a second driver to its device. Multiple disjoint mappings let the namespace grow across nodes and clusters without placing cluster names in every tag path. Preserve current local paths; optional enterprise/site grouping is authored through mappings and must not silently change existing NodeIds.

For example, a local assembly area may retain its local line-1 and expose assembly/remote-line-2 mapped to Cluster B’s packaging/line-2. Expanding that branch reveals its authorized equipment and signals; selecting assembly/remote-line-2/filler/speed routes to the Cluster B origin address. Likewise, a Raw branch Plant/remote-b can map to Cluster B’s Plant folder, exposing Plant/remote-b/Modbus/dev1/Speed with the same public driver/device/tag metadata and ordering (browse name, node kind, datatype and access capabilities; never driver credentials or private connection configuration). The mapping prefix distinguishes the view; the origin RawPath remains Plant/Modbus/dev1/Speed. A same-cluster node mapping uses this identical mechanism with a Node-target connection.

The mapped branch name is configuration, not a hostname or a guess from a server’s display name. Paths name views; origin cluster/entity identity names ownership.

Browse proceeds through the following boundaries:

  1. The requested realm’s local catalog returns local children and configured remote branch anchors under the requested parent. A branch is shown only when the local caller has browse access; remote child names require upstream Browse permission as well. Local Browse grants never grant remote Read/Write/Subscribe implicitly.
  2. Expanding a remote anchor resolves its mapping and issues a paged Browse to the configured origin’s local export root in that same realm, using the connection service principal. Project returned children under the local parent with origin identity, mapping ID/revision and opaque child handle. Further expansion continues on that origin through the same mapping. Browse does not subscribe to values or poll devices.
  3. Each returned handle is bound to the mapping, origin and catalog revision. Read/write/subscribe resolve it through the same canonical resolver; never accept a caller-supplied replacement cluster/endpoint or derive a driver route from display text. Local permission, remote grant and mapping operation restrictions all apply.
  4. Merge sibling results deterministically using the existing ordinal identity/segment rules. Collisions between local children, mapped branches or projected children reject the mapping/catalog activation; never use last-writer-wins or silently merge different owners. Several views of one origin may share a value subscription but retain distinct local permissions and handles. Shared sample caching uses explicit canonical backing-source identity supplied by the origin, plus serving node/incarnation, Node-versus-Cluster target mode, consistency policy, authenticated upstream principal/export scope and active authority/catalog epochs; two pinned replica caches are never coalesced merely because they report the same origin address; authorization, browse names and write admission remain keyed to the actual requested Raw or UNS view. A local primary sample must never satisfy a read/subscription pinned to another node’s cache. Role/incarnation changes create a new sharing/stream epoch and invalidate old cache provenance. Reauthorize each local view independently on delivery, even when its upstream demand is shared. If canonical equivalence is unavailable, retain separate subscriptions rather than guessing equivalence from paths.
  5. A remote continuation stays bound to its origin endpoint incarnation, service principal, caller authorization context, mapping revision, root/filter and remote catalog revision. The gateway wraps it rather than exposing a reusable raw peer token. Local merged pages additionally bind the local catalog revision and origin cursor positions. Enforce aggregate page, RPC-fan-out, depth, node-cache and timeout budgets; callers expand one level at a time, not an unbounded recursive crawl. Pin required metadata/provenance for active value items within the admission budget and evict idle browse pages first; exceeding the sibling/projection budget returns an explicit limit/incomplete result rather than silently truncating a folder. On peer failover or relevant revision change, invalidate the continuation and require a new browse.
  6. An unavailable remote branch remains identifiable as a configured branch with explicit unavailable/stale metadata for authorized callers. Its expansion returns a retryable unavailable result, not a successful empty list. Paginated responses include per-branch completeness/failure so a failed remote page cannot masquerade as a complete namespace. Cached metadata is explicitly revision-stamped and stale; metadata availability never marks old values fresh.
  7. Invalidate cached catalog pages/handles on mapping change, remote catalog revision change, local or remote authority change, expiry and deletion. Remove inaccessible child names as well as value payloads. A remote rename/removal yields model/catalog change and invalidates affected paths; preserve stable origin entity identity only where the origin explicitly supplies it. Remote browse responses include a catalog epoch, authority epoch and metadata-authorization lease (maximum 15 seconds by default, capped by credential expiry). A control-only catalog watch over the subscription control stream invalidates leases on remote authority/catalog change; it requires Browse/export access and no value-monitored items. Background authenticated probes renew leases when no watch is available. Anchor the local monotonic lease deadline to the renewal request start plus the returned bounded duration, with a conservative transport margin, rather than to reply receipt; delayed responses cannot extend authorization validity; a new epoch invalidates all old pages before reuse. A partition or failed renewal hides remote descendant names and denies cached remote reads/metadata when the lease expires, retaining only the locally configured authorized branch anchor with unavailable state. Online notifications accelerate revocation; partitioned revocation is bounded by the maximum remaining lease, not claimed instantaneous. Explicitly test that bound. Refresh/reconcile on expansion and background revision probes: new authorized descendants appear within an already deployed subtree mapping without creating a new authored tag row. Changing roots or permissions still requires deploying the mapping/grants.

The asynchronous gRPC/internal browse path supports direct remote expansion. For OPC UA, project bounded remote metadata into the local address space through the shared catalog/materialization pipeline on background tasks, before SDK Browse consumes it; never make network calls under node-manager locks. Track projection readiness per parent, not for the entire remote subtree. Mapping activation schedules the root’s first child page; a browse of a not-yet-ready parent schedules/coalesces one bounded background expansion and returns BadWaitingForInitialData. Once that page is materialized, return its references and an opaque gateway continuation; BrowseNext may schedule the next page and return BadWaitingForInitialData without consuming the continuation until the page is ready. Page requests are idempotent and readiness/model-change notifications allow retry. Remote model changes reconcile bounded pages and emit authorization-filtered model-change notifications. An unreachable peer or expired metadata lease returns BadNoCommunication for that remote parent; a revoked/changed mapping or origin incarnation returns BadContinuationPointInvalid for its old continuation. Browse on the local parent can still return other valid children and the authorized branch anchor; status on expansion exposes the branch failure. These behaviors require explicit SDK Browse/BrowseNext adapter handling and tests; neither an unpopulated page nor an outage returns a successful empty list. This is an explicit extension to today’s deployed-only served tree, not implicit remote database configuration replication.

Keep remote mapping separate from UnsTagReference: that entity’s TagId remains a same-cluster raw-tag FK. Materialized remote leaves are runtime projection records with mapping provenance, not synthetic locally owned raw tags. Catalog subject resolution must accept mapped nodes with their explicit local realm/scope and origin realm/scope without assuming every UNS reference has exactly four segments. Introduce an explicit MappedNamespace subject kind containing realm, mapping ID/revision, local parent ancestry, projected relative segments and immutable origin provenance. Resolve MappedNamespace subjects ahead of both the Raw path-based BuildScope/decision-cache logic and the UNS custom subject resolver; current custom resolution is UNS-only, so merely adding a new UNS subject kind is insufficient. Include realm, mapping revision, serving provenance and authority generation in cached authorization decisions. A Raw mapping uses the Raw ACL hierarchy; a UNS mapping uses the UNS ACL hierarchy. Identical bare paths across realms or mappings never alias implicitly. Extend NodeAclScopeKind with a deployable remote-mapping scope and compile grants into the shared authority snapshot; mapped descendant scope derives from that mapping/ancestry, never from a local TagId or a guessed four-segment path. Existing ancestor grants may flow to the mapping under the same evaluator inheritance/deny rules, but every mapped operation must pass the explicit mapping gate and remote export policy. Unknown/mismatched provenance denies. Extend subject resolver and trie tests for mapped descendants and deeper folders while retaining current local identity rules.

5. RPC contract and operation behavior

The following is a semantic contract; exact protobuf tags are assigned during implementation and then locked by tests. Additive evolution only; reserve removed tags; enum zero is UNSPECIFIED and rejected for required decisions.

RPC Shape and required semantics
GetCapabilities Unary. Protocol major/minor, authenticated server node/cluster, role/readiness, catalog revision, codec types, limits and supported operations
Browse / GetMetadata Unary pages/batches. Peer ingress serves authorized origin-local Raw and UNS exports; client-facing browse composes local and mapped remote branches, with bounded pages, opaque origin handles and explicit partial/unavailable results. Capabilities are intersected with the caller’s grants
Read Unary batch. Correlation, target addresses, cached-only mode and optional maximum acceptable age. Ordered per-item value/status results; no implicit device read
Write Unary batch. Per-item operation ID, address, typed value, expected catalog revision and remaining execution budget. Ordered per-item outcomes; partial success allowed, no batch atomicity
Subscribe Bidirectional stream. Open/add/modify/remove commands with request IDs, acknowledged revised settings and per-item statuses; server sends snapshot chunks, deltas, catalog invalidation, heartbeat and reset controls

Unknown unsupported RPCs/capabilities return Unimplemented/explicit unsupported status. Transport authentication, malformed envelopes and global overload use gRPC status. Valid batched requests use per-item statuses for authorization, unknown node, type mismatch and execution failures. Do not reveal a denied node's existence through diagnostics.

Reads

Reads return authoritative cached values and original status/timestamps; maximum age is a freshness requirement, not a driver refresh request. For an over-age sample return a stale-data result with the last sample and its original quality separately, so consumers cannot mistake it for fresh Good. Missing and denied nodes produce per-item failures. No cross-node or cross-item atomic snapshot is promised.

Writes

Authorize at the target before role disclosure and routing; validate the current catalog revision, type, writability, export scope and deadline. The primary gate is checked at admission and again immediately before device dispatch. Carry a validated admission ticket (principal/key ID, canonical subject, authority/catalog revision, node/driver incarnation, operation ID and monotonic expiry) into DriverInstanceActor; both immediate launch and queue drain validate it against current role, permissions, mapping and budget. A generation change requires reauthorization against the current subject or rejection. The current child queue has no such checks: adding them is an implementation prerequisite, not a property of the existing gateway. Initial support is value writes only; attribute writes are unsupported.

A result distinguishes RejectedBeforeDispatch, DriverConfirmedSuccess, DriverConfirmedFailure, AcceptedUnconfirmed, and OutcomeUnknown. Acceptance by an actor or gRPC service is not success. A driver with fire-and-forget semantics returns AcceptedUnconfirmed. The existing bool/string write result is insufficient: extend the internal route to preserve dispatch stage and actual driver outcome before exposing this service. The gRPC import adapter also needs this distinction end to end: current IWritable.WriteResult contains only a status code. Add a backward-compatible richer write-outcome capability/result and make the host, actor, import adapter and OPC UA self-correction path consume it; do not collapse Unknown or Unconfirmed into an ordinary success/failure Boolean. Existing drivers can be adapted conservatively according to their actual completion guarantees.

Mutating calls have no configured automatic retry, hedging, offline queue or automatic failover replay. If a connection drops after submission, the caller reports OutcomeUnknown and reconciles by a later read/application procedure. Cancellation after dispatch cannot undo a device write. Readback alone is not proof that a non-idempotent action did or did not execute.

Maintain a bounded node-incarnation request registry keyed by stable authenticated service principal and operation ID across reconnects, with logical payload hash and item identity. Hash address, value and semantic preconditions, excluding transport correlation, remaining budget and retry-varying headers. Every mutation carries the serving-incarnation precondition obtained during discovery; a restarted node rejects an old-incarnation request before dispatch. An actor/driver incarnation change invalidates queued admission tickets as well. Duplicate in-flight requests join the same task; cancellation of one duplicate caller detaches only that waiter and never cancels a shared dispatched operation; completed duplicates reuse the result; conflicting payloads reject. Retain entries for a negotiated retry window and reject new mutations at capacity instead of evicting in-flight entries. After restart, failover, or retention expiry, the registry offers no duplicate protection. Clients must never replay an ambiguous operation across those boundaries. This is not exactly-once delivery.

Require enough remaining deadline to cover the configured queue/write/abandon bounds plus response margin; otherwise reject before dispatch. Carry remaining budget into the actor request and use its monotonic write clock for the final pre-dispatch decision. Do not reuse a fixed eight-second Ask; preserve the existing derived DriverWriteBounds ordering. Once dispatched, record any eventual result even if the caller disconnects. Replace the current abandoned-write path that discards eventual outcomes with a bounded late-completion observer linked to the request registry. Until a definitive result arrives the outcome stays Unknown; if the observer/retention window ends, preserve Unknown and expose a diagnostic. A late result may refine an audit/registry entry, never automatically launch a replacement write. Admission is bounded so outstanding token-ignoring driver tasks cannot create unlimited observers.

For an imported OPC UA node, preserve the existing nonblocking write hook and explicitly document its immediate acceptance semantics. The remote outcome is async; rejection restores the last authoritative sample, and an unknown outcome marks quality uncertain pending a fresh source sample. A network failure must not become a fabricated definitive device failure or a new authoritative optimistic Good sample.

6. Subscription lifecycle, backpressure and recovery

Each stream is authenticated and pinned to one serving node incarnation and catalog/authority generation. Stream controls carry subscription ID, stream epoch, command ID and sequence. Sequence ordering is per stream, not fleet-wide.

  1. Authorize each item and apply export limits. Atomically register interest and capture the owning partition’s version watermark in the hub; a multi-partition subscription captures a vector of watermarks as specified in §6a.
  2. Freeze the item values/versions at the captured watermark (an immutable snapshot view), then send SnapshotBegin, bounded snapshot chunks and SnapshotEnd, followed by deltas newer than that watermark. The snapshot has a negotiated memory/item budget. Each item also has a monotonic version within the catalog/incarnation so consumers discard regressions. Snapshot transmission does not hold an actor or SDK lock. The bounded snapshot catch-up queue must reset if it overflows; SnapshotEnd alone cannot conceal a gap.
  3. Serialize stream commands through a bounded inbound command queue with per-principal command rate limits; reject/close abusive streams before queue growth. Open/add acks establish an item revision before its snapshot/deltas; modify acknowledges the exact boundary after which revised settings apply and preserves the latest baseline for interval-only changes; changes to the selected address or filter require a new item revision and snapshot; remove cancels queued old-revision frames and its ack is the boundary after which that item revision can never appear. A remove followed by add creates a new revision. Reserve control-message capacity so value pressure cannot drop acknowledgments, resets or revocations. Values use latest-state semantics, with per-item sampling/publishing intervals and revised settings in the ack. There is no promise to deliver every intermediate device transition. Multiple local subscribers share upstream demand; sampling honors the fastest admitted request and each downstream delivery applies its own filter/rate.
  4. Use bounded per-stream byte and item budgets. Coalesce latest values per item. Explicitly signal coalescing/overflow; sequence numbers are assigned to emitted frames, so sequence alone cannot detect dropped source transitions. If a coherent latest-state stream cannot be maintained, terminate with ResourceExhausted, mark local imports stale, and require a new snapshot. Never block a driver actor on a socket write.
  5. Remove last interest on cancel, shutdown, principal expiry or timeout; check credential expiry on a timer even when no values or authority changes arrive; disposal releases upstream demand and buffers. Permission changes remove denied monitored items before further delivery under the new authority generation. Every outbound batch checks authority generation; queued unauthorized payloads are discarded.
  6. On interruption, mark affected imports uncertain immediately, then BadNoCommunication after the configured stale timeout. Preserve the last payload and source timestamp, with local connection quality represented separately and composed for OPC UA display. Recovery returns Good only after a fresh authorized snapshot/sample.
  7. Reconnect with jittered exponential backoff and a new stream epoch. V1 always takes a fresh snapshot: no durable replay, Republish or seamless TransferSubscriptions promise. Clients discard old-epoch frames. After a role switch, current-value state can recover, but transitions during the gap can be lost.

Unary calls have explicit per-operation deadlines. Long-lived subscriptions use stream cancellation, heartbeat/silence detection and a renewable bounded session lifetime instead of inheriting a short unary deadline; each stream command has its own acknowledgment timeout. An expired command timeout cannot silently imply that a state change failed: reconnect with a fresh snapshot/item revision when its outcome is uncertain.

Illustrative, tunable commissioning quotas (not capacity targets or required performance numbers): 1,000 items per read/write batch, 500 catalog rows per page, 10,000 monitored items per principal, 100 streams per node, 4 MiB per message and per-stream pending-byte budget, 5-second heartbeat, 15-second stream silence timeout, and 30-second import stale timeout. Reject requests exceeding configured limits; chunk snapshots below message/byte limits. Aggregate principal limits across all its concurrent connections. Add an aggregate hub/subscription memory budget and connection admission cap so per-stream limits cannot multiply into unbounded memory. These are tunable starting limits, not measured capacity claims.

6a. Scaling architecture

The design aims to raise the number of unique tags, downstream clients, monitored-item registrations and delivered updates independently. They are different resource dimensions: source sharing reduces acquisition work; it does not eliminate per-client permissions, filter state, encoding or network deliveries. There are no required hard capacity numbers at this stage. Measure workload envelopes on the intended hardware before setting production quotas.

Shared acquisition and backing samples

Maintain one immutable authoritative sample per canonical backing source and serving provenance within a hub, with lightweight Raw/UNS view descriptors pointing to it. At the first driver callback/adapter boundary, before an asynchronous actor/queue handoff, validate shape/type/size and copy mutable driver payloads (including byte arrays, arrays and nested supported elements) once into an owned immutable sample, or accept an explicit ownership-transfer/immutable lease contract. Current object-valued callback/envelope fields do not establish immutability by themselves. Writers cannot mutate or reuse transferred storage. Snapshots, pending readers and filter baselines retain the same immutable ownership lease until released, including asynchronous encoding; pool return requires all references to be gone. Unsupported mutable object graphs are rejected rather than shared. This is a logical ownership model: current driver, actor and sink stages may retain necessary envelopes, but mappings must not create independent acquisition loops or payload copies solely because a source has multiple paths. Resolve Raw/UNS equivalence only from explicit origin catalog identity. Distinct driver acquisitions or pinned replica caches remain distinct, even for matching tag names.

Aggregate upstream interest using the complete sharing key from §4: source identity, serving node/incarnation, target mode/consistency and upstream principal/export context plus active catalog/authority epochs. Maintain reference-counted local interests and per-view authorization/filter state. One upstream interest can satisfy many compatible downstream registrations; removing one consumer cannot cancel another consumer’s demand. Update upstream settings when the aggregate interval, trigger or filter requirement changes; bound subscription churn while preserving the strongest admitted demand.

Aggregate upstream filtering as a superset of all admitted consumers’ needs: v1 shared upstream demand uses no value deadband by default. A later upstream deadband optimization requires proof of superset behavior across independent consumer baselines; taking the minimum numeric threshold alone does not establish that proof. Union trigger requirements for value/status/timestamp changes. Local consumers apply their own narrower filter; downstream filtering cannot recover transitions already suppressed upstream. Faster local demand cannot force the origin beyond its advertised acquisition capability. An add/modify that strengthens upstream demand remains pending until the origin acknowledges the revised settings and supplies a fresh versioned snapshot/boundary; only then acknowledge the new downstream item revision effective. If the origin cannot supply it, reject or explicitly revise that registration. Delay or debounce only reductions/relaxations while retaining a safe superset; never acknowledge a stronger demand while it is still being debounced.

Partition cache updates and fan-out

Route normalized source batches to a configurable bounded set of workers by a stable hash of canonical backing-source identity plus serving provenance. Aliases of one source share an owner. Retain original batching through the actor boundary; do not add one actor message, task or lock acquisition per alias/client/value. Partition owners serialize each source’s sample/status/version/removal updates. Their ingress has node-wide and per-partition byte/item limits and a nonblocking producer contract. Apply those bounds before actor-mailbox admission; an unbounded mailbox or retry queue must not become a hidden substitute for bounded ingress. Control paths and encoded-segment queues are bounded separately as well. A source update advances one source-level dirty generation and replaces its latest sample; it does not touch every downstream registration.

When ingress is full, coalesce by source within a bounded dirty-source table, retaining version/gap information; if distinct-source or lifecycle capacity is exhausted, invalidate affected stream epochs and cached-source readiness with an explicit reset/overload result rather than dropping updates invisibly or blocking a driver actor. If affected source identities cannot be retained within the budget, invalidate the partition’s readiness epoch conservatively. Reads and reconnect snapshots must then return unavailable/uncertain state, never the old cached sample as fresh Good. Restore readiness only after a fresh authoritative retained/source sample or an explicitly tracked resynchronization establishes current state; stream recreation alone is insufficient. Keep a reserved ordered control path for status/removal/catalog/authority invalidation so a coalesced value cannot resurrect a deleted source. Network encoding and socket waits occur outside driver and partition owners, so no central actor or shared lock must receive and fan out every tag update.

This partitions new hub/delivery work; it does not by itself parallelize current DriverHostActor acquisition routing or legacy Raw/UNS SDK sink materialization. Inventory those existing single-owner stages, preserve batch handling, and measure them alongside the new service. Where the common publication boundary can emit normalized source batches before alias fan-out, let the new hub consume that source-level feed without adding another per-alias hot-path pass. Any further refactor of existing actor/sink ownership requires separate ordering/lifecycle tests, not an assumption that worker count removes that bottleneck.

Catalog/mapping lifecycle, admission and authority publication form a separate low-rate control path. Publish immutable routing/authority snapshots to workers. Every queued item carries its view/mapping, authority, catalog and stream/item revisions; delivery gates discard invalid items regardless of partition progress. A demotion, removal or revocation invalidates admission immediately through the control path; it does not wait behind a source-data backlog. Preserve existing revocation and metadata-lease semantics.

Snapshot registration is atomic within each owning partition. A multi-partition stream uses a vector of partition watermarks and immutable captured samples; it does not promise a cross-partition point-in-time snapshot. Emit SnapshotEnd only after every admitted partition’s snapshot completes, then drain its valid newer deltas without regressing per-item versions. Snapshot/catch-up budgets cover all participating workers and retained immutable samples. Repartitioning drains/resets affected stream epochs before changing ownership; do not silently reorder samples while moving a source.

Compact wire format and stream handles

Negotiate compact-batch capability and codec version in GetCapabilities. Subscription add/open acknowledgments bind an opaque numeric item handle to the canonical requested view and its metadata; steady-state updates carry the handle, item revision, sample version and typed sample fields instead of repeating full paths and mapping metadata. Handles are valid only in their authenticated stream epoch and serving incarnation, are not reused within that epoch, and are invalidated on removal/rebind. They confer no access by themselves: admission and delivery still validate the actual requested view and current authority. Unknown/stale handles reject explicitly. Writes retain their full resolved identity, admission ticket and operation ID; a numeric handle must never weaken mutation preconditions.

A batch header may hold common origin/epoch, quality and timestamp defaults, with explicit per-item presence/override markers. Preserve null versus absent value, absent timestamps, complete status subcodes, timestamp precision and each item’s original sample version. Group only compatible provenance/revision contexts; mixed groups use separate headers. Frame limits apply to decoded size and item count as well as encoded bytes; invalid counts, dimensions and handle references fail before allocation grows. Use a typed legacy envelope when compact capability is not negotiated; both representations must produce equivalent domain samples.

Use pooled/ref-counted immutable payload storage where profiling justifies it. A buffer cannot return to its pool while any snapshot, pending delivery or async writer retains it. Count retained bytes across owners and subscribers in global admission accounting; zero-copy optimization cannot bypass memory limits. Compression is optional and negotiated per connection/workload, not a default requirement: compare CPU cost against bandwidth savings for numeric batches versus strings/arrays.

Delivery scheduling, filtering and authorization caches

Keep source acquisition cadence separate from downstream publishing. Each downstream registration has revised publishing interval, change trigger, optional numeric deadband, last delivered baseline and latest pending sample. Deadbands apply only to supported numeric types with negotiated units/range semantics; reject unsupported requests. Status, connection-quality and deletion changes are never suppressed by a value deadband. Timestamp-only delivery follows an explicit trigger setting. Reconnect/rebind creates a new baseline from the authorized snapshot. Advance a registration’s filter baseline only when its sample is committed by the serialized stream writer (successful bounded send), never when a candidate is encoded, coalesced, dropped or suppressed. Filter decisions use the committed baseline/version; if it changes while a candidate is prepared, re-evaluate the latest pending source sample before suppressing or sending it. A failed/ambiguous send resets that stream epoch rather than treating the candidate as delivered.

Do not perform an unbounded full-client walk on each source update: maintain per-source interest indexes on interest changes, and update only the latest source sample plus dirty generation on data arrival. A timed scheduler traverses dirty/due interests using resumable, fair bounded slices; each registration records its last considered/delivered source version and reads the latest immutable sample when due. If a source changes during traversal, its later dirty generation schedules another slice without restarting or starving later consumers. Bound scheduled work, dirty tables, baseline storage and due-index/timer entries as well as payload queues; a high-fan-out tag cannot allocate a new task/pending payload for every client on every source change. Distribute scheduling work across workers. Each gRPC stream has one serialized writer and ordered command/removal barriers; encode bounded ready batches outside partition locks and let the writer merge only revision-valid segments. Prevent a hot tag or slow client from monopolizing workers through per-principal and per-stream byte/item/work budgets. Isolate browse/snapshot work from steady-state delivery and reserve bounded control/admission resources. Overload produces explicit throttling/coalescing/reset, not unbounded queues; control priority must not starve ordinary delivery indefinitely.

Cache local authorization decisions by authenticated key/principal, operation, exact realm-qualified view/subject, mapping revision, serving provenance and authority generation; entries also honor credential expiry and remote metadata/export leases. Use the existing generation-based permission cache pattern rather than walking grants per update. Published generation changes and expiry invalidate admission and queued delivery even when a cache entry was Allow. Bound cache entries and negative-cache lifetime, and recheck the generation/lease at the final send gate. That gate defines emission for revocation tests: bytes already handed to the transport cannot be recalled, while unsent segments must pass the current authority gate. Do not share a cached Allow across Raw/UNS aliases, principals, node-pinned contexts or independently restricted mappings.

Demand-driven catalogs and controlled recovery

Retain the per-parent paged browse/materialization model from §4. Browse loads metadata only; acquisition begins when an explicit read/subscription/runtime dependency requires it. Cached reads do not create driver refresh loops. Pin metadata needed by active interests within their admitted budget, and evict idle pages/sample payloads only where no consumer or existing local-runtime requirement retains them. Coalesce equivalent concurrent browse requests within the same authorization/mapping context, with bounded waiters and shared-operation cancellation isolated per waiter.

Use a node-wide snapshot admission scheduler with global and per-principal limits for concurrent snapshots, bytes in flight and snapshot bandwidth. Jitter reconnects, deduplicate compatible upstream reconnects and pace downstream snapshots. Retain capacity for live traffic and control messages; a recovering client must not stop local publication or monopolize the active server. Each admitted snapshot has a bounded lifetime and catch-up budget; reject/defer overload with explicit retry guidance instead of a reset/retry storm. Failover still requires new epochs and fresh snapshots, and never replays ambiguous writes. Write admission and outcome tracking have reserved bounded resources independent of subscription bursts.

Horizontal growth and release boundary

The first release includes the in-process partitioning, shared samples/demand with immutable ingestion and filter-superset admission, bounded partition ingress/fair traversal, compact negotiated batches, filters, authorization caches, demand-driven catalogs and recovery scheduler above. Partition count, budgets and compression are implementation/commissioning choices subject to measurement, not new promised capacity levels.

Scale authoritative acquisition horizontally by assigning independent driver/source ownership to additional application clusters and presenting them through Raw/UNS mappings. This does not distribute one field-device write across competing owners. A redundant pair is sized so its eligible active member can carry the admitted workload alone; failover admission cannot assume both members contribute aggregate capacity. Node-pinned replica reads remain opt-in with explicit freshness/readiness semantics.

Consumer-side aggregation services or read-distribution nodes are a future extension to reduce source fan-out across many consumers. They are not implicitly enabled by subscription sharing: v1 peer exports remain origin-local and one hop. A later relay design must separately specify provenance, end-to-end authorization/revocation leases, bounded staleness, loop prevention, recovery and capacity ownership before allowing projected values to be re-exported. Writes continue to the authoritative active owner. No automatic data-plane load balancing across primary and standby is introduced here.

7. Node and cluster routing, redundancy and failure boundaries

The protocol supports the following required topologies with identical authorization, codec, paging and outcome semantics:

Topology Browse/metadata Read/subscribe Write
Node A → node B in the same cluster gRPC against B’s authorized local Raw/UNS catalog gRPC against B’s ready local cache/streams gRPC with B’s role/readiness and final device-dispatch gate
Node A → a node in another cluster The same node-pinned gRPC service The same cache/stream semantics The same explicit per-item outcomes
Node A → a logical cluster endpoint set Browse current eligible primary’s catalog Read/subscribe on selected primary; new epoch after failover Select primary before submission; ambiguous operations never replay

Node connection: dial the configured node and verify its machine and cluster identities; the target cluster may equal the caller’s cluster. It remains pinned; no silent substitution of its partner. All enabled operations are normal supported API use, not a diagnostics-only endpoint. A ready secondary may serve its own cached reads and subscriptions only under an explicit connection consistency policy and the existing Warm/Hot rules; stamp results with serving node, incarnation, role and freshness so the caller knows which cache it used. An uninitialized Warm standby returns unavailable/initial-data status rather than pretending its cache is current. Writes require a ready primary: a pinned secondary returns RejectedBeforeDispatch, and the caller must explicitly choose a cluster-target connection if it wants primary selection.

Different-process intracluster access uses gRPC, not actor selection, DPS, OPC UA loopback or shared-memory assumptions. Akka remains responsible for the current election and actor work inside the receiving process. Same-process local consumers may use the local adapter with the same policy/outcome contracts. Being in the same cluster conveys no implicit authentication or grant, and does not bypass TLS or mapping scopes. The existing device primary gate is unchanged by adding this connection.

Origin ownership remains (cluster, realm, stable entity/address) for redundant replicas; serving machine/incarnation is separate provenance and a required Node-target routing/precondition. Configured mapped views receive distinct local paths even when originClusterId equals localClusterId. Reject a mapping that routes back into its own serving process/view or into its own descendants; a different node in the same cluster is explicitly allowed. Reject origin records that are themselves projected mappings. Neither endpoint forwards a rejected pinned write to its partner, and mapping a peer’s data never starts a second field-device driver.

Consumer-node execution scope: connections and mappings are authored within a cluster but are not implicitly active on every member. Add explicit ConsumerNodeIds to DataConnection and RemoteNamespaceMapping; a mapping’s set must be a subset of its connection’s. Resolve each consumer against deployed machine identities, and project only its effective connections/mappings/secrets into that node’s execution graph. The source-node set controls local execution, while target-side authorization still uses authenticated principals and grants, never a caller-supplied source-node header.

For a pinned A→B mapping in one cluster, consumers are {A} and the target is B. A receives the mount and starts the gRPC connection; B receives no active projection of that mapping, creates no corresponding anchor/subscription/driver, and continues serving its origin-local catalog. Validate every intended consumer/target combination at deployment time: explicitly selecting B as a consumer of its own B mount is a configuration error, not a reason to reject the valid A mapping. If one authored mapping is intended for both replicas, author separate per-node targets/mappings with disjoint consumer scopes or use a validated cluster-target policy that excludes the consumer itself; do not silently retarget a Node connection. Last-known artifacts preserve this scope offline. Removing a node from the consumer set drains its scoped connections, removes its projected anchors/items and invalidates associated handles; queued writes are rejected before dispatch and dispatched writes retain their existing outcome tracking. Failover never implicitly expands the consumer set or retargets a pinned connection; a newly active consumer uses its own validated effective scope.

A direct IDataConnection request does not require mounting a namespace. It uses an allowed connection in the caller node’s execution scope and the same remote service, permissions, readiness and outcomes. Namespace mappings add browse/materialization on top of that connection. Nonconsumer nodes omit those projected nodes intentionally; node-specific browse catalogs and readiness are observable, not assumed identical because their cluster IDs match.

Cluster connection: configure ordered eligible endpoints for the expected application cluster; client probes capabilities/readiness and uses the current primary. Discovery uses deployed endpoint records, not a fleet gossip ring. Try another eligible endpoint on a failed read or new subscription within the original deadline. Failover is per write item: an item may change endpoint only following an explicit authenticated RejectedBeforeDispatch result. Never resend an entire partly successful batch. DriverConfirmedSuccess/Failure, AcceptedUnconfirmed and OutcomeUnknown items are not replayed. A lost batch response makes every item without a known definitive result ambiguous and prevents replay. Rediscover the new serving incarnation and construct a new attempt only for items proven never dispatched.

Role changes stop accepting new writes, close primary-only streams with a retryable status, invalidate routing hints and trigger discovery. After reconnect, the client requires a fresh snapshot. Do not merge samples from two nodes into one subscription epoch. A stale role hint is never authorization or a fence.

This feature inherits the existing cluster's election and device split-brain limitations. A role recheck reduces stale admissions but is not a distributed fencing lease. If two partitioned nodes both believe they are primary, gRPC cannot prove exclusive device ownership. Do not claim new split-brain safety. Tests must cover known-secondary rejection, startup without a role snapshot, partition behavior and in-flight demotion; stronger external fencing is a separate architecture decision.

Connections operate without a central relay or live ConfigDb dependency. A local cluster continues local work when a remote cluster or central is unavailable. The import adapter follows the existing local Warm/Hot role policy: a Warm standby cannot independently activate an import driver or publish a second active feed; a Hot standby may keep its own cache warm only where that policy permits it, without enabling writes. Verify role transitions and demand ownership in the import lifecycle tests. No subscription carries Akka references or forwards into remote DPS. Maintenance/disabled endpoints are excluded from new connection selection through the deployed connection revision.

8. Authentication, authorization and deployment configuration

Use a dedicated HTTP/2 TLS data listener with explicit advertised HTTPS URI and server identity, separate from existing h2c telemetry/command endpoints. Production requires TLS and certificate verification. Development cleartext requires an explicit loopback-only override. Register the service only when enabled and enforce listener/host admission; ASP.NET endpoint mapping alone must not expose it through every Kestrel listener.

Authenticate a connection using a target-issued data-plane API key over TLS, resolved through the existing authority machinery; separate credentials per directional connection permit revocation. Bind the authenticated principal to expected source connection/cluster policy. Claimed source IDs in payloads are audit context, not identity. Keep mesh PSKs scoped to the existing command plane. Optional mTLS can add peer authentication later; it must not replace operation grants.

V1 uses a named service principal for upstream imports, not user impersonation. Local end-user authorization gates access to imported nodes, including every browse result/continuation and cached metadata response; target authorization independently gates the service principal's remote scope. Effective access is the intersection of local permission, imported capability, target export allowlist and remote service-principal grants. Remote audits identify the service principal and separately record untrusted originating-user context. Delegated per-user remote identity is a future feature, not implicitly trusted headers.

Extract SDK-independent policy and catalog subject resolution from the existing authorizer, including UNS reference variables/folders, Raw aliases and scripted-condition ownership. Reuse authority generations, API-key expiry/revocation and operation mapping. The new data endpoint always enforces its export/grant policy even when legacy OPC UA ACL enforcement is disabled. Mapped Raw and UNS nodes always enforce their local mapping ACL gate, even if Security:Acl:Enabled is false for ordinary local OPC UA nodes. Insert the gate ahead of current AclEnforced bypasses, including Browse/BrowseNext, read and monitored-item delivery; a configured anchor alone grants no descendant access. Remote grant leases complement rather than replace these local per-caller checks. No granted/ready authority snapshot means deny. Authorization of one Raw alias must not accidentally confer access via another UNS alias, or vice versa.

Proposed configuration entities:

  • DataConnection: stable ID, owner cluster, explicit consumer node IDs, Node/Cluster target kind, expected target identities, ordered HTTPS endpoints, secret reference, enabled flag, timeout/backoff/quotas and permitted operation subset.
  • DataExport: origin-local selected roots/nodes, peer principal, operation grants and per-principal limits. Remote imports are excluded from selection in v1.
  • RemoteNamespaceMapping: stable mapping ID, owning local cluster, explicit consumer node IDs, explicit Raw/UNS realm, local parent identity/branch name, connection ID, remote same-realm root/entity identity, enabled flag, mapping ACL scope/grants, permitted operations, subtree projection/cache limits and metadata-lease policy, sample/publish defaults, staleness policy and write-through flag (default false). Optional leaf selections narrow a subtree; they are not required for browse discovery.
  • Runtime projection records: mapping revision, realm-qualified local path/subject, origin entity/address, remote catalog revision and value-demand state; derived from browse/catalog, not separate authored per-tag import rows.

Deploy resolved catalog references/configuration atomically; validate collisions, ownership, consumer scopes, per-node self-route rejection, endpoint identity, TLS scheme, unsupported types, prohibited re-export and missing secret references before apply. Endpoint changes drain old subscriptions and start a new epoch. Cross-node/cross-cluster schema permits explicit same-realm namespace mapping edges, rejects a local descendant mounted beneath itself and prohibits peer re-export; do not relax the existing same-cluster driver-bound equipment invariant. Runtime nodes hold last successfully deployed configuration, never wait on central SQL to serve data.

9. Observability, rollout and verification

Expose connection state (Disabled, Connecting, Ready, Degraded, Backoff, AuthFailed), selected endpoint, last successful snapshot, last sample age, role, protocol version and diagnostic reason. Count unique backing sources, upstream interests, downstream registrations, streams/clients, source updates and delivered updates separately. Measure sharing ratio, bytes per delivered sample, per-partition queue/CPU time, scheduling delay, snapshot backlog, authorization-cache hit/miss/invalidation, retained payload bytes and slow-consumer throttles, as well as reconnects, coalescing, overflow/resets, denials and write outcomes. Trace correlation across ingress and actor route. Keep tag IDs/operation IDs out of metric labels; audit writes with principal, canonical address, operation ID and outcome, without secrets or value payloads by default.

Roll out disabled by default. First add contracts/policy/hub parity tests; then local service and TLS ingress; then read/browse/subscription canaries with compact/fallback equivalence and partition/recovery gates; then imports; finally enable explicitly granted writes. Negotiate capabilities before enabling a connection. Mixed versions fail individual unsupported capabilities without disabling local OPC UA. Rollback disables connection/import execution, marks imported values unavailable and keeps local driver ownership and existing mesh traffic intact; it never replays pending writes.

Required verification before release:

Area Evidence / acceptance gate
Wire/codec Golden protobuf contracts, additive-version peer tests, full status fidelity, UInt64 boundaries, null/arrays/timestamps and malformed/oversized payload rejection
Hub Driver + virtual + status/removal parity; snapshot/delta race; catalog swap; no optimistic value leakage; bounded buffering under stalled clients
Authorization Empty authority fail-closed; wrong/expired/revoked key; TLS/cluster mismatch; export boundaries; Raw/UNS alias cases; no cookie challenge or exposure on unintended listener; stream revocation with queued frames
Writes Real adapter success versus fire-and-forget acceptance; pre-dispatch timeout; dropped response after device success; duplicate concurrent ID/conflicting payload; registry capacity; demotion; cancellation; no replay after failover
Lifecycle Add/modify/remove acknowledgments, revised sampling, snapshot overflow, idle timeout, reconnect epoch, gap visibility, last-interest cleanup and endpoint revision change
Topology Actual same-cluster A→B gRPC browse/read/write/subscribe tests, cluster-wide artifact with A-only consumer projection and B serving-only behavior, direct API without mount, no pinned-cache substitution or cross-principal sharing, pinned-primary and secondary/Warm readiness behavior, identical Raw/UNS paths kept separate; two independent redundant clusters (four nodes), no shared Akka membership; node-pinned access, cluster failover, warm standby, maintenance, central outage and partition scenarios
Scaling Compare one versus multiple workers and compact versus fallback encoding; Raw/UNS many-view source sharing; principal/pinned-node isolation; last-interest cleanup and interval churn; deadband/status/timestamp trigger superset and stronger-demand admission boundary; mutable array/bytes reuse after callbacks and baseline lifetime; cross-partition snapshot/command ordering; revocation while batches encode; payload retention/pool lifetime, nonblocking ingress overflow/source-dirty traversal fairness, cache-readiness invalidation after lost ingress and decoded frame bounds
Capacity Vary unique sources, registrations/client overlap, clients/streams, change rates, payload shapes and slow consumers independently; measure source versus delivered throughput, CPU, retained memory, wire bytes and p95/p99 latency under steady state, bursts, concurrent browse, reconnect storms and single-member failover. Verify global/principal admission, control/write reservations and bounded recovery; record the measured workload envelope before accepting commissioning quotas. No hard capacity threshold is required by this design
Cross-node/cross-cluster browse Expand Raw Folder/Driver/Device/TagGroup/Tag and UNS Area/Line/Equipment/signal in each topology; newly added descendants; deep projected subject permissions; collisions/aliases; deterministic merged paging; partial peer outage versus empty folder; bounded metadata revocation during partition; explicit mapped ACL scope with legacy ACL toggle off; per-parent projection readiness and BrowseNext retries; continuation replay/revocation/failover; bounded browse fan-out; no subscriptions from browse; asynchronous OPC UA projection and model-change notifications
Imports Collision/loop prevention; correct dependencies/fan-out; stale quality and recovery; write-through local+remote authorization; removal and rename behavior

Use software driver fixtures for protocol/fault tests; use explicit hardware commissioning only after software gates. A documentation-only draft requires link/path verification, not a runtime test run.

10. Review decisions for the user

  1. Confirm the Raw and UNS mapping model and cross-node/cross-cluster browse/read/write/subscribe semantics above; these implement the user’s requested expandable naming structure. The earlier flat selected-import assumption is superseded.
  2. Approve cached reads and latest-state subscriptions, with visible gaps and fresh snapshots on failover?
  3. Approve service-principal identity for imports, dedicated TLS listener and target-side enforced export grants?
  4. Approve value writes in v1, with no automatic ambiguous retry and honest unconfirmed/unknown outcomes?
  5. Approve deferring history and alarm/event operations to negotiated extensions?
  6. Review the §6a scaling architecture: partitioned processing and compact/shared delivery in v1; read-distribution relays remain a separately designed extension, with capacity established through workload measurements.

After user approval, create a detailed implementation plan and execution batches. This document authorizes no code changes or deployment.

References

  • gRPC .NET implementation, checked through Context7 for channel reuse and deadline/cancellation propagation.
  • Microsoft: gRPC performance: reuse channels, distinguish HTTP/2 stream concurrency from tag subscriptions, and account for flow control and long-lived calls.
  • Microsoft: retries: built-in retries do not resume a stream after responses have begun; application recovery is necessary.
  • Microsoft: deadlines and cancellation: a caller deadline can expire after server work completes, and cancellation must be propagated explicitly.