docs: plan grpc data connections and record Astra reviews
v2-ci / unit-tests (push) Failing after 2h40m48s

This commit is contained in:
Joseph Doherty
2026-11-15 20:59:53 -05:00
parent 6dd2985b8b
commit 5dbdee1ea3
4 changed files with 1165 additions and 0 deletions
@@ -0,0 +1,204 @@
# Astra review: gRPC data connections
Date: 2026-11-15
Reviewer: Astra design-review agent
Scope: design only; repository evidence inspected, no runtime changes or tests performed.
Reviewed document: [gRPC data connections design](2026-11-15-grpc-data-connections-design.md), initial draft and same-session revision.
## Disposition
**Approved for user design review, including the scaling revision.** The proposed dedicated data service, independent clusters, target-side authorization, explicit write uncertainty, and latest-state subscription contract fit the repository. Substantive findings from the original review and all scope/scaling follow-ups are resolved in the revised design. Implementation is not authorized, and capacity remains subject to workload measurement.
The findings below preserve the initial review and required corrections for implementation traceability. User approval remains required; this review does not authorize implementation.
### Follow-up resolution
| Finding | Revision assessment |
|---|---|
| H1 | Resolved: item-level retry exclusions, serving-incarnation precondition and logical payload hashing now explicit. |
| H2 | Resolved: child launch/queue admission ticket, rich outcomes across capabilities and bounded late-completion observer now required. |
| H3 | Resolved: immediate non-rollback authority revocations and fail-closed subject/catalog compatibility now explicit. |
| M1 | Resolved: immutable watermark snapshot, item versions, command serialization, removal barriers and reserved control capacity now explicit. |
| M2 | Resolved: richer timestamp envelope and distinct original/receive timestamps now required. |
The final revision also explicitly closes the smaller follow-up details: one shared catalog/subject projection, bounded inbound command backlog, separate stream lifetime and command acknowledgment timeouts, stable principal-based deduplication across reconnects, and cancellation isolation for duplicate waiters. These requirements should carry into the implementation plan.
## Findings
### H1 — Make mixed-batch failover and deduplication boundaries explicit
**Severity: High.** Sections 5 and 7 allow partial-success write batches and permit endpoint changes after `RejectedBeforeDispatch`, but do not explicitly prohibit replaying the entire batch when only some items received that outcome. That would duplicate already confirmed or uncertain writes. The node-incarnation registry is also not represented in the proposed capabilities/write fields, so the server cannot reject a stale duplicate arriving at the same endpoint after process restart.
**Required correction:** Endpoint retry applies only to individual items with an authenticated `RejectedBeforeDispatch` outcome. Never replay successful, accepted-unconfirmed, unknown, or missing-response items. Include the serving incarnation and deduplication window in negotiation and bind mutations to that incarnation; an incarnation mismatch rejects before execution and does not authorize replay of an earlier ambiguous operation. Define the deduplication payload fingerprint over immutable operation content (address, value, relevant preconditions), excluding a changing remaining deadline. A duplicate request may time out waiting without canceling the shared execution. State precisely whether operation IDs are scoped to the service principal or configured connection and make that identity stable across reconnects.
**Evidence:** `DriverHostActor.cs:2334` forwards independent child writes; `DriverInstanceActor.cs:1212` explicitly protects irreversible field actions from retry. The draft itself specifies per-item outcomes and node-incarnation-only protection. There is no transactional batch primitive to make a whole-batch retry safe.
### H2 — Specify the final dispatch gate and abandoned-write outcome path
**Severity: High.** The draft correctly requires a second primary check and richer outcomes, but calling this an extension of the internal route understates the necessary change inside the driver child. A write can sit in the child's queue after the host has checked role and catalog. Existing timeout/abandon paths flatten uncertainty into failure and discard the eventual outcome, contrary to the proposed registry behavior.
**Required correction:** Carry a resolved write admission ticket through the queue: canonical target, catalog/authority generation, authenticated operation scope, role/admission epoch, operation ID and monotonic deadline. Revalidate permission, mapping, role and deadline at the actual child dispatch boundary; a revoked, remapped or demoted queued operation must not execute. Preserve explicit pre-dispatch rejection versus post-dispatch uncertainty through `IWritable`, driver result, child result, host result and imported OPC UA completion handling. Name the policy for late completions after abandonment: bounded observation/audit and registry update where retained, never a second caller completion or silent downgrade from unknown to definitive failure. Do not promise device-confirmed success from an adapter that only acknowledges enqueueing.
**Evidence:** `DriverHostActor.cs:2286` checks role before routing, then sends `WriteAttribute` at approximately line 2336. `DriverInstanceActor.cs:1163-1195` queues and launches without authority/role preconditions; `DrainWriteQueue` around line 1279 launches later. `RunWriteAsync` around lines 1230-1258 discards an abandoned eventual outcome and returns bool/string failure. `ActorNodeWriteGateway.cs` similarly maps cancellation and Ask timeout to `NodeWriteOutcome(false, ...)`.
### H3 — Define deployment failure semantics without rolling back revocation
**Severity: High.** A shared hub is feasible, but “swap catalog/authority/value generations coherently” needs a concrete failure rule. Existing authority publication deliberately precedes address-space application, and partial topology failure can leave the old live catalog under new grants. A naive atomic-swap implementation that retains the previous authority when topology fails would restore revoked access. A second independently derived catalog could also disagree with the SDK's Raw/UNS alias and owner resolution.
**Required correction:** Build one SDK-independent desired catalog/subject projection from the existing deployment composition and reuse it for both publication and authorization. Stage its application and tag samples/mappings with their catalog generation. Apply revocations regardless of topology success. If active topology/value provenance cannot be proven compatible with current authority, reject affected reads/writes and reset affected subscriptions until reconciled; do not roll back authority to preserve availability. Distinguish a hub/catalog commit failure from an optional OPC UA sink failure so a healthy independent data service need not inherit an unrelated SDK rendering failure. Include no-op ACL deployment, partial apply, failed apply, rename/rebind and recovery tests.
**Evidence:** `OpcUaPublishActor.cs:1201-1206` installs authority before `plan.IsEmpty` and before applying the topology; lines approximately 1243-1270 describe partial materialization failures and retaining the previous composition. `DataPlaneAuthority.cs` explicitly documents the scripted-alarm live-owner versus current-owner mismatch and denies it. `DriverHostActor.cs:1765` currently returns before value fan-out when no OPC UA publish actor exists, confirming that this boundary must be extracted, not merely observed by a new network service.
### M1 — Complete snapshot and subscription-command ordering semantics
**Severity: Medium.** Registering interest and capturing a watermark does not alone produce a snapshot at that watermark. If a chunk reads a newer value and the catch-up queue later emits an older delta, a consumer can regress. Add/modify/remove acknowledgments also have no defined barrier semantics, allowing an implementation to deliver already queued removed-item values after acknowledging removal.
**Required correction:** Use an immutable snapshot view captured at the watermark, or explicitly guarantee per-item versions and reject regressions across snapshot/delta delivery. Include item generation along with stream epoch so remove/re-add of the same address cannot accept old frames. Serialize commands per stream, acknowledge the exact effective settings/version, and define removal acknowledgment as a barrier after which no older item-generation payload is emitted. Add operations need their own gap-free initial snapshot boundary; modification must define whether a fresh snapshot is required. Bound inbound command backlog and prioritize control/reset delivery so a full value queue cannot hide invalidation. A stream lives under cancellation/lifetime policy; individual commands use bounded acknowledgments, not the unary read/write deadline applied to the stream lifetime.
**Evidence:** Section 6 already recognizes catch-up overflow, emitted-frame sequences and per-item coalescing; the missing rule is consistency between these mechanisms. `DataPlaneAuthority.cs` exposes synchronous generation-change notification specifically because live subscriptions do not naturally reauthorize via a new request. Correct command barriers and authority checks must cover queued snapshot chunks as well as deltas.
### M2 — Make timestamp fidelity an explicit publication prerequisite
**Severity: Medium.** The wire can carry source and server timestamps, but the current driver publication envelope cannot preserve both. Advertising original timestamp fidelity without changing that boundary would manufacture or silently lose metadata.
**Required correction:** Enrich the shared authoritative sample envelope to retain independently optional source/server timestamps before publication, or declare unavailable timestamp fields absent. Specify timestamp precision and UTC conversion tests. Preserve provenance separately from a locally generated receive time; never label receive time as original source time. Include representative driver, virtual-value and imported-value paths in parity tests.
**Evidence:** `DriverInstanceActor.AttributeValueItem` has one `TimestampUtc`; `HandleDataChange`/batch handling near line 1900 select `SourceTimestampUtc ?? ServerTimestampUtc`. `DriverHostActor.ForwardBatchToMux` passes this single timestamp to `ValueWrite`. The draft's primitive/array codec, URI-based identities and complete uint32 quality approach are otherwise appropriate. Unsupported structured values are honestly scoped out.
## Areas reviewed without a blocking finding
- Dedicated TLS listener and explicit endpoint admission are appropriate. Existing `MeshGrpcEndpoints.cs` documents the cookie fallback/interceptor ordering trap; retaining equivalent regression coverage is necessary. API-key authentication must be checked for expiry during a quiet long-lived stream as well as on generation changes, as the draft's expiry cleanup requirement implies.
- Reuse of authority generations and exact Raw/UNS subject resolution is preferable to bypassing permissions via the driver gateway. Service-principal imports with separate local user authorization are an honest initial identity model.
- Versioned latest-state delivery, bounded queues, visible coalescing, snapshot-on-reconnect and no seamless transition-history promise are appropriate. Per-principal limits should aggregate all concurrent connections for that principal, not reset with each stream.
- Cluster-pinned versus node-pinned routing and independent Akka membership fit the current redundancy model. The explicit refusal to claim fencing or exactly-once writes is correct. Local import activation/standby ownership should be tied to the existing runtime role policy during implementation to avoid duplicate local publication.
- History/events/methods are clearly deferred and capability-negotiated. Import-only first consumption remains a user decision; the internal API preserves an extension path without pretending to provide a complete OPC UA tunnel.
## Approval gate
The follow-up review is complete and the design is approved for **user design review**, with runtime implementation still unauthorized. No runtime tests are warranted for this documentation-only review; implementation must satisfy the design's protocol, authorization, fault, redundancy and capacity gates.
## 2026-11-15 follow-up: expandable UNS and cross-cluster browse
User clarification: “It should rely on the UNS mapping to build an expandable naming structure. (note: add support for browse cross cluster)”. The revised design makes remote subtree mappings the namespace authority, includes cross-cluster expansion, and keeps authored mappings separate from local `UnsTagReference` rows. This is a suitable direction. One-hop origin exports, opaque routing handles, collision rejection and nonblocking SDK materialization are particularly important and correctly retained.
**Final follow-up disposition: approved for user design review after correction.** Initial findings are retained below. I reread the revised UNS section, mapped authorization gates and release verification criteria after the author addressed them.
| Finding | Follow-up resolution |
|---|---|
| H4 | Resolved: explicit `MappedUns` provenance, deployable mapping ACL scope, current ancestry validation and mandatory mapped-node checks ahead of legacy ACL bypasses. |
| H5 | Resolved: independent catalog/authority epochs, control-only invalidation watch, authenticated renewal probes and bounded metadata authorization lease; expired leases hide remote descendants while preserving only locally authorized anchors. |
| M3 | Resolved at design level: per-parent first-page readiness, coalesced background loading, explicit pending/unavailable statuses and generation-bound gateway continuations. SDK continuation behavior remains an explicit implementation acceptance gate. |
The implementation plan must validate SDK/client handling of pending BrowseNext without accidentally consuming the cursor; define pinned metadata behavior for active monitored items and explicit over-limit sibling-set results; and calculate leases conservatively using request timing so delayed replies cannot extend remote authorization validity. Those details refine the selected architecture and must be tested before shipping.
### H4 — Mapped nodes need explicit authorization subjects and authored grant semantics
**Severity: High.** Extending a four-segment check alone cannot make runtime mapped folders/leaves safe or usable. The existing resolver validates folder ancestry against deployed local entity IDs; remote runtime nodes are not those entities. Existing Tag-scoped grants resolve a local raw `TagId`, while the proposal correctly avoids synthetic local raw tags. Thus the design needs a concrete local grant scope for a mapping, or an explicit inherited-only policy, and a distinct mapped-node subject bound to the current mapping placement. Falling back to parsing a remote path could bypass the generation/placement safety that existing code intentionally enforces.
**Required correction:** Introduce an explicit mapped-node ACL subject containing mapping ID/revision, locally validated ancestry, local projected scope and origin provenance. Resolve it through the active shared mapping/catalog projection, fail closed on mismatch or missing bindings, and never classify it as a legacy unknown/path/reference variable. Choose and document grant authoring: for example an appended mapping scope keyed by mapping ID with descendant inheritance, while retaining existing ancestor grants and deferring per-leaf remote grants. Define how mapping moves affect permissions and prevent grants at one local view from authorizing another view. Apply these gates at every mapped OPC UA access seam even if legacy local-node ACL enforcement is disabled; imported remote capability/export restrictions cannot disappear behind that switch.
**Evidence:** `DataPlaneAuthorizer.TryPlaceUnsSubject` around lines 440-508 requires current `UnsFolderScopes` ancestry and special-cases four-segment unknown IDs. `OtOpcUaNodeManager.ResolveUnsAclSubject` around lines 5090-5180 distinguishes local opaque folder IDs, reference variables and folder-owned values. `NodeAclScopeKind.Tag` requires a local raw tag and deliberately fans that grant out to its local UNS references; no remote mapping scope exists. `AclScopePathResolver` is constructed from deployed local area/line/equipment/reference maps. `OtOpcUaNodeManager.Browse` around lines 5296-5305 bypasses its authorization checks entirely when `AclEnforced` is false.
### H5 — Bound remote metadata authority freshness during disconnection
**Severity: High.** Removing cached child names on remote authority change requires a way to observe that change. The new design has background revision probes but does not specify whether they include authority-only revisions, how quickly they run, or how cached SDK metadata becomes inaccessible during a partition. A quiet value-free browse mount otherwise can expose names indefinitely after remote Browse permission is revoked. Ordinary catalog revision changes do not necessarily capture ACL-only deployments.
**Required correction:** Negotiate a remote catalog epoch and authority/export epoch for metadata, with a bounded authorization freshness lease or equivalent conservative policy. Specify an invalidation signal and/or bounded probes that run even without value subscriptions. On observed revocation or expired freshness lease, invalidate remote continuations/handles and hide remote-derived child metadata from gRPC and SDK Browse/Translate/Read seams; the configured local anchor can remain visible under local permission. Distinguish freshness of public configured anchor metadata from freshness of remote-derived child names. State the maximum connected and partitioned revocation visibility delay honestly; no instantaneous cross-cluster revocation claim. Do not resurrect cached names merely because a value stream reconnects without renewed metadata authorization.
**Evidence:** `OpcUaPublishActor.ApplyPlanCore` around lines 1201-1206 explicitly supports authority-only changes with empty topology plans. `IDataPlaneAuthoritySource.Changed` in `DataPlaneAuthority.cs` is process-local notification, not a cross-cluster metadata invalidation protocol. The draft's new background catalog projection can live independently of value subscriptions, so value-stream revocation alone cannot cover it.
### M3 — Define SDK browse readiness and continuation behavior per parent
**Severity: Medium.** “Initial authorized metadata projection is complete” could mean fully crawling an arbitrarily large subtree before making a mount usable. That conflicts with bounded caches and one-level expansion. The asynchronous internal path has partial-result metadata, while OPC UA Browse returns standard reference pages and service statuses; a degraded folder property alone does not prevent a client interpreting an empty successful Browse as a complete empty folder.
**Required correction:** Define readiness at an immediate-child-page/parent generation boundary, not the entire remote subtree. Specify the background population trigger for an unready parent without blocking SDK locks. Return an explicit standard retryable bad status while a parent's coherent child set is unavailable; invalidate/reject BrowseNext when mapping/catalog/authority generation changes. Specify the chosen status mapping and a metadata-cache eviction policy that preserves nodes with active monitored items or fails/removes them explicitly. State limits for large individual sibling sets and how clients can retry/continue without treating a cap as successful exhaustion. Test empty versus not-yet-loaded, outage, over-limit parent, concurrent refresh and mapping removal during BrowseNext.
**Evidence:** `OtOpcUaNodeManager.Browse` around lines 5296-5355 delegates to the SDK's materialized tree and filters only references appended by the current page. It makes no remote call and has no current unavailable-projection state. The design rightly prohibits network I/O under SDK locks; the missing step is a concrete client-visible loading and pagination contract.
No runtime implementation or tests were performed for this follow-up. The revised design is ready for the user's review, with the listed implementation and verification gates retained.
## 2026-11-15 follow-up: Raw exposure and intracluster operations
User clarification requests the existing Raw mappings cross-cluster as well as gRPC browse/read/write/subscribe between nodes inside a cluster. The realm-qualified `RemoteNamespaceMapping` and topology matrix address those requests directly. Keeping Raw and UNS views separate, preserving origin addresses, disallowing peer re-export and retaining primary-only device writes are consistent with current runtime behavior. A ready secondary's node-pinned cache is a distinct observation of the same cluster-owned source, not an interchangeable primary result.
**Final disposition: approved for user design review after revision.** The initial findings below are retained for traceability. I reread the new consumer execution scope, full cache-sharing identity, Raw authorization dispatch and verification criteria.
| Finding | Follow-up resolution |
|---|---|
| H6 | Resolved: explicit consumer node sets, mapping-set subset validation, per-node effective execution graphs and the A-only/B-serving-only example. Direct API usage requires an allowed connection but no namespace mount. |
| H7 | Resolved: serving incarnation, target mode, consistency, upstream principal/export scope and authority/catalog epochs qualify sharing; independent local-view checks and no local-cache substitution for pinned peers are explicit. |
| Raw subject gate | Resolved: `MappedNamespace` dispatch precedes both Raw path-based decisions and UNS custom resolution; mapping/provenance-aware authorization cache identity is required. |
Implementation must test consumer-scope removal drains subscriptions and removes projected nodes, and that an A-only mount does not silently migrate to B on failover. The design now makes those node-specific identities and execution scopes observable; this is a deliberate difference from replicated origin ownership.
### H6 — Define which nodes execute cluster-owned node-target mappings
**Severity: High.** The connection and mapping entities are owned by a cluster, and existing deployment composition is selected by cluster. Consequently, a configured same-cluster A→B mapping may also be installed on B. The new rule rejecting a mapping that targets its own serving process would then either invalidate B's deployment or make the valid A→B mapping undeployable. Distinguishing cluster identity from machine identity is necessary but does not settle this per-node execution question.
**Required correction:** Define explicit consumer-node execution scope for node-target connections/mounts and project that scope through deployment artifacts. An A→B mount activates on A, while B receives either no executable mount or an explicitly inactive projection; validate self-routes only for selected consumer nodes. State how inactive/nonconsumer anchors behave, how scope edits drain work, and how failover interacts with an A-only mount. Direct internal API access to B should not require mounting a branch. Do not silently substitute local samples when the mapping is inactive or its target is the current process.
**Evidence:** `DeploymentArtifact.ResolveClusterScope` and `ParsedDeploymentArtifact.ParseDriverSpecs` around lines 887-942 select composition/driver specifications by cluster rather than a general per-source-node connection scope. The proposed entities currently name an owner cluster; the topology specifically permits that cluster to equal the target cluster. Existing replicated local ownership must not be accidentally repurposed into per-node mounting rules.
### H7 — Preserve node-pinned cache and authorization context when sharing demand
**Severity: High.** Canonical backing-source identity is sufficient to identify ownership, but insufficient to deduplicate requests to two replicas. A local primary cache and a pinned secondary cache can contain different values/status/timestamps under the same origin address. Sharing one upstream subscription across them would erase the newly promised serving-node and consistency behavior. Sharing across service principals can also bypass a narrower upstream export policy.
**Required correction:** Include serving node/incarnation, connection target mode, consistency policy and upstream authorization/export context in cache/subscription sharing identity, in addition to canonical source. Share Raw/UNS delivery only after authorizing each actual requested view and establishing equivalence within that context. A role/incarnation or selected-endpoint change creates a fresh epoch and cannot preserve an old sharing identity. Never satisfy pinned B demand from local A's samples merely because both serve the same cluster-owned source.
**Evidence:** The design's node-pinned policy explicitly permits a secondary's own cache. `DriverHostActor.ForwardBatchToMux` receives local driver samples and fans them out to realm-qualified views, while `HandleRouteNodeWrite` checks the current local role. `DataPlaneAuthorizer` checks the actual requested view, and `NodeAclScopeKind.Tag`'s legacy alias fan-out does not imply arbitrary cross-principal or cross-node equivalence.
### Raw implementation gates
The current custom subject resolver in `DataPlaneAuthorizer.Evaluate` runs only when `realm == AddressSpaceRealm.Uns` (around line 608); Raw otherwise reaches path-based `BuildScope` (around line 632). The planned explicit `MappedNamespace` subject must therefore be resolved ahead of both realm paths and their decision caches. A Raw mount cannot inherit correct provenance validation merely by renaming the UNS subject type. Preserve existing native Raw IDs and scope resolution. Public Raw driver/device metadata must exclude private driver configuration and credentials. These are implementation gates for the chosen extension, not reasons to redesign it.
No runtime code was changed or tested. The complete revised design is ready for the user's review, with implementation and verification gates retained.
## 2026-11-15 follow-up: scaling architecture
The user requested scaling improvements followed by another Astra review. The new §6a correctly separates unique source count from registrations, delivered updates and clients. Partition ownership, vector-watermark snapshots, compact epoch-scoped handles, authorization-aware sharing and bounded recovery fit the earlier safety contract. A redundant pair must carry admitted load on one eligible member, and the explicit future-only relay boundary preserves the v1 one-hop export rule. There is no justified numerical capacity claim yet, and none is needed for this design review.
**Final disposition: approved for user design review after revision.** Initial findings are retained below. I reread the corrected immutable ingestion, aggregate filter admission, baseline commit, bounded ingress/source-dirty scheduling and release verification requirements.
| Finding | Follow-up resolution |
|---|---|
| H8 | Resolved: validate/copy mutable payloads once or accept explicit immutable ownership transfer; snapshots, filters and asynchronous encoding retain storage ownership. |
| H9 | Resolved: aggregate intervals and trigger/filter requirements; v1 shared upstream demand defaults to no deadband; stronger requirements wait for origin acknowledgment and a fresh boundary. Filter baselines advance only at successful serialized writer commit. |
| M4 | Resolved: bounded nonblocking ingress, source-level dirty generations and fair resumable interest traversal; overflow invalidates source/partition cache readiness as well as stream epochs. Legacy actor/SDK serialization limits are explicitly retained and measured. |
No architectural blocker remains. Implementation must still prove mutable-buffer isolation, filter equivalence, nonstarvation, overflow recovery and final-send revocation behavior under adversarial concurrency. Approval is of the design and its gates, not a claim of measured capacity or validated runtime performance.
### H8 — Establish immutable payload ownership before shared publication
**Severity: High.** Calling a sample or payload immutable does not make its array/byte content immutable. Existing publication messages retain arbitrary `object` references from driver snapshots. A reused producer array could mutate a supposedly frozen snapshot, change a deadband baseline or corrupt concurrently encoded frames even if the sample record itself is immutable. Ref-counting protects pool lifetime only after the producer/consumer ownership boundary is established.
**Required correction:** At the first shared ingestion boundary, normalize/copy supported mutable arrays and bytes once into owned immutable storage, or require an explicit immutable/leased ownership transfer from the producer. Borrowed objects may not enter snapshot/cache state. Retain ownership through snapshots, deadband baselines, pending delivery and actual serialization completion; cancellation/disposal must release every reference exactly once. Apply allocation/shape limits before expensive copies. Test producer mutation immediately after publishing and pool reuse while a snapshot/writer is stalled.
**Evidence:** `DriverInstanceActor.AttributeValueItem` at approximately line 145 contains `object? Value`. `OnDataChangeForward` and `OnDataChangeBatchForward` around lines 1898-1919 pass `Snapshot.Value` directly. `DriverHostActor.ForwardBatchToMux` forwards those same values into `ValueWrite` per alias. No immutable ownership contract is established by these envelope constructors.
### H9 — Aggregate upstream filters without losing downstream changes
**Severity: High.** Fastest sampling/publishing intervals alone cannot make shared acquisition satisfy independently configured downstream triggers/deadbands. For example, applying one subscriber's large deadband upstream can hide changes required by a second subscriber with no deadband, even at the same interval. Debouncing an upstream increase while acknowledging a newly faster subscriber can also promise settings not yet effective.
**Required correction:** Aggregate upstream trigger/filter settings as a conservative superset of every admitted downstream requirement. The simplest safe v1 rule is no upstream value deadband for shared acquisition and downstream filtering per registration; otherwise prove the filter union for supported types. Upstream status/timestamp trigger selection must include every required downstream trigger. A new or faster registration becomes effective only after the origin acknowledges revised settings and a suitable initial snapshot/version boundary; report revised achievable settings when the source cannot satisfy the request. Debounce reductions freely within resource limits, but never claim an unactivated increase. Define when a deadband baseline advances (successful ordered writer acceptance, not merely encoding or a coalesced/dropped candidate), and retain full status/quality-trigger behavior.
**Evidence:** §6a introduces independent downstream deadbands/triggers but its upstream aggregation paragraph originally updates only fastest intervals. §6 already promises acknowledged revised settings and gap-free admission. Existing `DriverInstanceActor` maintains shared `_desiredRefs`/`_desiredInterval`; this does not supply the richer filter-composition semantics automatically.
### M4 — Bound producer ingress and avoid per-update registration walks
**Severity: Medium.** Partitioning alone does not bound ingress, and updating a latest-pending slot for every interested registration on each source change remains O(registrations) per source update. “Bounded fan-out per turn” prevents one long turn but can still create an unbounded work backlog if each new update starts another traversal. The existing host actor also receives every driver batch and constructs alias fan-out, so a claim that no central actor receives all updates must be scoped to the new delivery service rather than assumed true of the entire runtime.
**Required correction:** Define bounded, nonblocking partition ingress, source-level dirty generations and a deduplicated dirty queue. Coalesce repeated updates to the same source without scheduling duplicate traversals; fan out latest generation in fair resumable slices, retaining work for a source changed mid-traversal. Preserve ordering/version/removal barriers and signal/reset when ingress cannot retain coherent latest state. Charge traversal/encoding work as well as bytes to fairness budgets. Explicitly identify `DriverHostActor` and existing SDK publication as legacy serialization points: route the new hub before expanded alias fan-out, avoid per-client work there, and measure remaining host/sink limits without promising they disappeared. Ensure unbounded actor mailboxes cannot become the hidden substitute for the bounded ingress queue.
**Evidence:** `DriverHostActor.ForwardBatchToMux` around lines 1741-1788 processes all local driver batches and loops over every bound alias. `TelemetryLocalHub.Publish` around lines 72-96 uses a shared lock and per-subscriber walk; it is an unsuitable template for the new high-cardinality fan-out. The draft already rejects socket waits in source workers, but queue ownership and deduplicated work scheduling need the same precision.
**Follow-up precision:** If ingress exhaustion loses the latest sample, reset the affected cache readiness/quality as well as streams. Otherwise a new snapshot can return the old cached value as Good after the lost update. Restore readiness only from a coherent retained/new source sample. Also change upstream settings on trigger/filter changes even when intervals are unchanged. A minimum upstream deadband threshold alone is not a superset proof when subscribers have independent baselines; use no shared upstream deadband in v1 unless equivalence is demonstrated.
### Other scaling checks
- Vector watermarks plus immutable per-partition snapshots are compatible with the explicit absence of cross-item atomic snapshots. Snapshot admission and retained bytes must be charged globally, including a stream waiting for its slowest partition.
- Handle non-reuse within an epoch and final authority/lease validation preserve security across compact encoding; malformed dimensions and decoded sizes must be bounded before pool rental/allocation.
- Fair service needs bounded work per principal/stream and explicit recovery admission, as proposed. Test a hot source, many quiet sources, abusive control commands and a reconnecting principal simultaneously; mean throughput alone does not establish fairness.
- Control changes must invalidate queued segments before the final writer gate. Bytes already handed to the transport cannot be recalled, so acceptance tests should define the emission boundary rather than imply revocation erases already transmitted data.
- Existing origin-local exports and the future-only relay paragraph are consistent. This revision does not authorize relay re-export, automatic primary/standby load balancing or competing field-device ownership.
No runtime code or tests were changed. The scaling revision is ready for the user's review, with the implementation gates and future-only relay boundary retained.
@@ -0,0 +1,309 @@
# gRPC data connections between nodes and clusters
Date: 2026-11-15
Status: Astra reviewed, including scaling architecture; approved for user design review after revision. Awaiting user approval; no implementation authorized.
Review: [Astra findings and resolutions](2026-11-15-grpc-data-connections-astra-review.md). Implementation has not started.
## 1. Decision and scope
Add an explicit data-plane connection over gRPC for cached reads, acknowledged writes, subscriptions, browsing, and metadata. Use the same protocol for a connection to a named node within an application cluster and a connection to another application cluster. Preserve independent Akka clusters and their existing elections; gRPC is a data transport, not a replacement membership or replication protocol.
User-directed namespace model: expose both current Raw and UNS hierarchies through mappings to build expandable naming structures spanning configured nodes and clusters, with cross-node and cross-cluster browse included in the first release. Map remote Raw branches into the local Raw view and remote UNS branches into the local UNS view rather than maintaining an unrelated flat import list. A mapping may expose an authorized subtree, so operators can expand its hierarchy and discover descendants without authoring one import per tag. Reads, writes and subscriptions resolve the same mapping to the owning cluster. Also expose an internal connection API usable by runtime consumers without OPC UA loopback. Connections are directional; reverse access requires its own grants and configuration. A remote cluster retains ownership of its drivers, values, permissions, and write decisions.
The user clarification resolves the initial imports-versus-routing question: routing is driven by the composed Raw/UNS namespaces and their explicit mappings. The same gRPC service handles browse, metadata, read, write and subscribe for nodes within one cluster and for different clusters; intracluster access is a required first-release topology, not only a diagnostic case. It does not enable arbitrary destinations supplied by callers or transitive peer forwarding.
### Alternatives
| Approach | Benefit | Cost / decision |
|---|---|---|
| Dedicated gRPC data service and runtime connection adapter (recommended) | One protocol for node and cluster access; explicit quality, lifecycle, and write semantics; fits existing hosting | Requires a typed codec, value hub, and lifecycle management |
| Reuse the OPC UA client driver between servers | Existing browse/read/write/subscribe interoperability | Does not deliver the requested gRPC transport; retains OPC UA session and encoding overhead |
| Extend the existing mesh-command and telemetry messages | Reuses existing listeners and dialers | Command receipt is not a write outcome; telemetry is lossy and lacks a tag-value contract. Keep their semantics intact |
### Included in the first release
- Explicit configured node or cluster connections, batched cached reads, value writes, browse/metadata, and data-change subscriptions, including actual gRPC calls between different nodes of the same cluster.
- Raw and UNS branch mappings with cross-node and cross-cluster browse/expand, stable origin identities, demand-driven value imports, local quality updates, and optional explicitly authorized write-through.
- Peer authentication, target-side operation authorization, bounded resource use, reconnect, pair failover, diagnostics, and compatibility negotiation.
- Shared backing samples and authorized demand aggregation, partitioned processing, compact subscription handles/batches, cached authorization, client filtering, and controlled recovery as specified in §6a. No fixed throughput or tag/client capacity is promised.
History reads and alarm/event subscriptions and methods are subsequent capability increments. Reserve their extension boundary now, but do not advertise them until implemented and tested. History updates, arbitrary method execution, automatic discovery of untrusted peers, configuration replication, and replacing Akka are excluded. The initial service is an application data API, not a complete OPC UA service tunnel.
## 2. Repository evidence
Paths below are relative to the repository root and describe the inspected checkout at `6dd2985b8`.
| Existing surface | Consequence for this design |
|---|---|
| `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto` | `ExecuteAck` means received/re-emitted, not applied. Do not tunnel writes through `NodeCommandService.Execute` |
| `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/telemetry.proto` and `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/TelemetryLocalHub.cs` | Existing stream is alarms/logs/health/resilience; its DropOldest behavior cannot silently become the data subscription contract |
| `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`, `ForwardBatchToMux` | Driver batches fan out to realm-qualified Raw and UNS values. Capture authoritative samples and source-removal/comms-loss updates without coupling to a network writer |
| Same file, `HandleRouteNodeWrite`, and `Drivers/ActorNodeWriteGateway.cs` | Existing async write route checks primary and resolves `(realm, identifier)` to the driver. Preserve its bounded driver execution; enrich its result before promising precise remote outcomes |
| `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs` | Publication also includes virtual values, alarm state, status-only changes, deferred apply and deletions. A driver-only tap would miss part of the address space |
| `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` and `Security/DataPlaneAuthorizer.cs` | Current ACL enforcement covers request seams and live-subscription revalidation; extract/reuse policy and subject resolution, never bypass it by calling a driver gateway |
| `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/DataPlaneAuthority.cs` | Immutable authority generations and expiry/revocation inputs already exist |
| `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs` | Endpoint mapping has regression guards for cookie fallback and interceptor authentication. New service needs equally explicit middleware/auth metadata and listener isolation |
| `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs` and `Entities/ServerCluster.cs` | Node identity differs from cluster identity. Existing `GrpcPort` describes telemetry; do not silently redefine it as a secure data endpoint |
| `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/UnsArea.cs`, `Entities/UnsLine.cs`, `Entities/Equipment.cs` and `Entities/UnsTagReference.cs` | Existing UNS is Area → Line → Equipment → effective signal; references are local raw-tag FKs. Add a distinct remote branch mapping instead of putting a remote identifier into a local `TagId` |
| `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs` | Current UNS uses URI-qualified, segment-validated paths; retain local identities and centralize projected identity construction |
| `docs/Redundancy.md` | Application clusters are independent singleton/pair meshes; central is not a required relay |
`docs/ReadWriteOperations.md` contains older claims that ACLs are not evaluated and reads have no override. Current source contradicts these claims. Treat current source and its tests as authoritative; correct those portions as part of implementation documentation.
## 3. Architecture and component ownership
```mermaid
flowchart LR
C[Raw and UNS browse / mapped data / runtime consumers] --> N[Realm-qualified mapping resolver]
N --> R[DataConnectionManager]
R --> L[Local adapter]
R --> G[gRPC connection adapter]
G --> S[Peer data service in Host]
S --> A[Authenticated principal + operation policy]
A --> V[Local data access ports]
V --> H[Partitioned versioned sample hub]
H --> F[Bounded delivery scheduler]
F --> S
V --> W[DriverHost write route + primary gate]
H --> P[Existing address-space publication sources]
```
Proposed components, not existing classes:
- **Core.Abstractions:** transport-neutral `IDataConnection`, `IDataReadService`, `IDataWriteService`, `IDataSubscriptionService`, catalog contracts and typed results. No gRPC, Akka, ASP.NET or OPC UA SDK dependencies.
- **Commons:** `data_connection.v1` protobuf, codec and contract tests, following current shared-contract placement. Keep generated DTOs out of domain interfaces.
- **Runtime:** partitioned data/catalog hub, delivery scheduler and authorization caches, actor adapters, connection manager, realm-qualified mapping resolver, federated browse/catalog projection and shared remote-demand subscriptions. Local calls and remote ingress use the same authorization and execution services.
- **Host:** gRPC ingress, authentication, listener configuration, endpoint admission, metrics and connection registration. It converts DTOs and invokes ports; it does not resolve actors itself.
- **Configuration/ControlPlane:** connection/export/namespace-mapping entities, validation and deployment-artifact projection. DB-less driver nodes receive resolved connection configuration through artifacts, with secrets delivered through the existing secret mechanism, never plaintext in artifacts.
- **Driver integration:** a gRPC data connection adapter shaped to the existing capability model (`IReadable`, `ISubscribable`, write capability). It supplies imported values through ordinary runtime publication and dependency evaluation. Registering this adapter must not create a second remote route for the same backing point.
The local hub is a versioned projection of the active local catalog and authoritative value/status feeds, independent of whether the OPC UA endpoint is running. Build one shared desired catalog/ACL-subject projection from the deployment composition for both OPC UA and the hub; do not derive alias paths or UNS ownership independently. Each adapter reports activation readiness against that shared projection. Extract a shared publication boundary used by driver, virtual-tag and status/removal paths; attach before any optional OPC UA sink early return. Stage the SDK-independent catalog and value projection for each deployment, and serve only addresses whose catalog subjects match the active authority generation. Authority revocation publishes immediately, including authority-only deployments; never roll it back because topology materialization fails. If the new catalog cannot activate, mark the data endpoint unavailable for mismatched subjects until repaired. A coherent served generation means fail-closed compatibility, not an atomic rollback of authority and topology. This explicitly accounts for `OpcUaPublishActor.ApplyPlanCore` feeding authority before a potentially degraded address-space apply. Do not read SDK NodeState internals or optimistic operator-write values as authoritative device samples.
Before building this seam, inventory every publication source and verify parity for driver values, virtual tags, comms quality, status-only updates, removal, rename and apply buffering. A node is unavailable until its initial catalog generation is ready. A known node with no sample returns BadWaitingForInitialData, never Good/null by default.
## 4. Identity, catalog and value representation
A public data address is `(originClusterId, namespaceUri, identifierKind, identifier)`, with realm and resolved ACL subject included in catalog metadata. Only supported identifier kinds are advertised. Numeric OPC UA namespace indexes are process-local and never cross the wire as identity. Do not confuse the Commons machine `NodeId` with a data address.
Each connection binds an expected cluster identity; a node connection also binds an expected machine identity. Renaming a path invalidates that address and produces catalog removal/addition unless a configured stable backing ID explicitly survives it. An imported node stores its original address and maps to a distinct local identity; it does not acquire ownership of the origin.
The initial codec uses explicit typed protobuf values: null, Boolean, signed/unsigned integers with declared widths, Float/Double, string, bytes, UTC timestamps, GUID, supported typed arrays and dimensions. Preserve UInt64 precision, empty versus null, NaN/infinity, and array shape. Node references carry namespace URIs. Carry the complete uint32 OPC UA status, optional source/server timestamps, origin identity, and sample version. Unsupported structured/ExtensionObject types return BadDataEncodingUnsupported; never stringify or deserialize arbitrary CLR types. Advertise the supported type set and reject unsupported imports at validation time.
Current `DriverInstanceActor.AttributeValueItem` collapses source/server timestamps into one `TimestampUtc`. To retain both when available, extend the shared publication envelope and its consumers; do not relabel the collapsed timestamp as two original timestamps. When a source supplies only one timestamp, the other remains absent. Origin acquisition timestamps and local import-receive time are distinct.
Catalog metadata includes data type, rank/dimensions, writable capability, origin, export permissions, revision and optional engineering units. Browse is paged and authorization-filtered. Opaque continuations bind principal, connection, filter and catalog revision, expire after 60 seconds, and are invalidated on revision change. Peer export catalogs contain origin-local nodes only in v1. The local client-facing Raw and UNS views can combine mapped branches from multiple directly connected peers, but a peer data-service call never recursively re-exports another peer’s mapped branches. Each remote operation uses one direct connection to its owning cluster, eliminating forwarding loops and ambiguous ownership.
### Raw and UNS mappings and cross-node/cross-cluster browse
Use the existing Raw and UNS structures as naming authorities. Generalize the earlier `UnsRemoteMapping` proposal into a versioned `RemoteNamespaceMapping` containing an explicit realm, local parent identity and exposed branch name, connection, remote root (stable entity identity plus current canonical address), allowed operation subset and projection limits. Local and remote realms must match in v1. Bind local parents through existing Raw-folder or UNS entity identities and compile paths through the shared deployed composition; do not introduce a parallel manually maintained naming tree. A Raw mapping exposes Folder → Driver → Device → TagGroup → Tag, including nested folders and tag groups as currently presented. A UNS mapping exposes Area → Line → Equipment → signal. Either supports an explicitly mapped leaf as a constrained special case. Browse returns the origin’s deployed/addressable catalog, not uncommitted device-discovery results; browsing a remote driver does not connect a second driver to its device. Multiple disjoint mappings let the namespace grow across nodes and clusters without placing cluster names in every tag path. Preserve current local paths; optional enterprise/site grouping is authored through mappings and must not silently change existing NodeIds.
For example, a local `assembly` area may retain its local `line-1` and expose `assembly/remote-line-2` mapped to Cluster B’s `packaging/line-2`. Expanding that branch reveals its authorized equipment and signals; selecting `assembly/remote-line-2/filler/speed` routes to the Cluster B origin address. Likewise, a Raw branch `Plant/remote-b` can map to Cluster B’s `Plant` folder, exposing `Plant/remote-b/Modbus/dev1/Speed` with the same public driver/device/tag metadata and ordering (browse name, node kind, datatype and access capabilities; never driver credentials or private connection configuration). The mapping prefix distinguishes the view; the origin RawPath remains `Plant/Modbus/dev1/Speed`. A same-cluster node mapping uses this identical mechanism with a Node-target connection.
The mapped branch name is configuration, not a hostname or a guess from a server’s display name. Paths name views; origin cluster/entity identity names ownership.
Browse proceeds through the following boundaries:
1. The requested realm’s local catalog returns local children and configured remote branch anchors under the requested parent. A branch is shown only when the local caller has browse access; remote child names require upstream Browse permission as well. Local Browse grants never grant remote Read/Write/Subscribe implicitly.
2. Expanding a remote anchor resolves its mapping and issues a paged `Browse` to the configured origin’s local export root in that same realm, using the connection service principal. Project returned children under the local parent with origin identity, mapping ID/revision and opaque child handle. Further expansion continues on that origin through the same mapping. Browse does not subscribe to values or poll devices.
3. Each returned handle is bound to the mapping, origin and catalog revision. Read/write/subscribe resolve it through the same canonical resolver; never accept a caller-supplied replacement cluster/endpoint or derive a driver route from display text. Local permission, remote grant and mapping operation restrictions all apply.
4. Merge sibling results deterministically using the existing ordinal identity/segment rules. Collisions between local children, mapped branches or projected children reject the mapping/catalog activation; never use last-writer-wins or silently merge different owners. Several views of one origin may share a value subscription but retain distinct local permissions and handles. Shared sample caching uses explicit canonical backing-source identity supplied by the origin, plus serving node/incarnation, Node-versus-Cluster target mode, consistency policy, authenticated upstream principal/export scope and active authority/catalog epochs; two pinned replica caches are never coalesced merely because they report the same origin address; authorization, browse names and write admission remain keyed to the actual requested Raw or UNS view. A local primary sample must never satisfy a read/subscription pinned to another node’s cache. Role/incarnation changes create a new sharing/stream epoch and invalidate old cache provenance. Reauthorize each local view independently on delivery, even when its upstream demand is shared. If canonical equivalence is unavailable, retain separate subscriptions rather than guessing equivalence from paths.
5. A remote continuation stays bound to its origin endpoint incarnation, service principal, caller authorization context, mapping revision, root/filter and remote catalog revision. The gateway wraps it rather than exposing a reusable raw peer token. Local merged pages additionally bind the local catalog revision and origin cursor positions. Enforce aggregate page, RPC-fan-out, depth, node-cache and timeout budgets; callers expand one level at a time, not an unbounded recursive crawl. Pin required metadata/provenance for active value items within the admission budget and evict idle browse pages first; exceeding the sibling/projection budget returns an explicit limit/incomplete result rather than silently truncating a folder. On peer failover or relevant revision change, invalidate the continuation and require a new browse.
6. An unavailable remote branch remains identifiable as a configured branch with explicit unavailable/stale metadata for authorized callers. Its expansion returns a retryable unavailable result, not a successful empty list. Paginated responses include per-branch completeness/failure so a failed remote page cannot masquerade as a complete namespace. Cached metadata is explicitly revision-stamped and stale; metadata availability never marks old values fresh.
7. Invalidate cached catalog pages/handles on mapping change, remote catalog revision change, local or remote authority change, expiry and deletion. Remove inaccessible child names as well as value payloads. A remote rename/removal yields model/catalog change and invalidates affected paths; preserve stable origin entity identity only where the origin explicitly supplies it. Remote browse responses include a catalog epoch, authority epoch and metadata-authorization lease (maximum 15 seconds by default, capped by credential expiry). A control-only catalog watch over the subscription control stream invalidates leases on remote authority/catalog change; it requires Browse/export access and no value-monitored items. Background authenticated probes renew leases when no watch is available. Anchor the local monotonic lease deadline to the renewal request start plus the returned bounded duration, with a conservative transport margin, rather than to reply receipt; delayed responses cannot extend authorization validity; a new epoch invalidates all old pages before reuse. A partition or failed renewal hides remote descendant names and denies cached remote reads/metadata when the lease expires, retaining only the locally configured authorized branch anchor with unavailable state. Online notifications accelerate revocation; partitioned revocation is bounded by the maximum remaining lease, not claimed instantaneous. Explicitly test that bound. Refresh/reconcile on expansion and background revision probes: new authorized descendants appear within an already deployed subtree mapping without creating a new authored tag row. Changing roots or permissions still requires deploying the mapping/grants.
The asynchronous gRPC/internal browse path supports direct remote expansion. For OPC UA, project bounded remote metadata into the local address space through the shared catalog/materialization pipeline on background tasks, before SDK Browse consumes it; never make network calls under node-manager locks. Track projection readiness per parent, not for the entire remote subtree. Mapping activation schedules the root’s first child page; a browse of a not-yet-ready parent schedules/coalesces one bounded background expansion and returns BadWaitingForInitialData. Once that page is materialized, return its references and an opaque gateway continuation; BrowseNext may schedule the next page and return BadWaitingForInitialData without consuming the continuation until the page is ready. Page requests are idempotent and readiness/model-change notifications allow retry. Remote model changes reconcile bounded pages and emit authorization-filtered model-change notifications. An unreachable peer or expired metadata lease returns BadNoCommunication for that remote parent; a revoked/changed mapping or origin incarnation returns BadContinuationPointInvalid for its old continuation. Browse on the local parent can still return other valid children and the authorized branch anchor; status on expansion exposes the branch failure. These behaviors require explicit SDK Browse/BrowseNext adapter handling and tests; neither an unpopulated page nor an outage returns a successful empty list. This is an explicit extension to today’s deployed-only served tree, not implicit remote database configuration replication.
Keep remote mapping separate from `UnsTagReference`: that entity’s `TagId` remains a same-cluster raw-tag FK. Materialized remote leaves are runtime projection records with mapping provenance, not synthetic locally owned raw tags. Catalog subject resolution must accept mapped nodes with their explicit local realm/scope and origin realm/scope without assuming every UNS reference has exactly four segments. Introduce an explicit MappedNamespace subject kind containing realm, mapping ID/revision, local parent ancestry, projected relative segments and immutable origin provenance. Resolve MappedNamespace subjects ahead of both the Raw path-based BuildScope/decision-cache logic and the UNS custom subject resolver; current custom resolution is UNS-only, so merely adding a new UNS subject kind is insufficient. Include realm, mapping revision, serving provenance and authority generation in cached authorization decisions. A Raw mapping uses the Raw ACL hierarchy; a UNS mapping uses the UNS ACL hierarchy. Identical bare paths across realms or mappings never alias implicitly. Extend `NodeAclScopeKind` with a deployable remote-mapping scope and compile grants into the shared authority snapshot; mapped descendant scope derives from that mapping/ancestry, never from a local `TagId` or a guessed four-segment path. Existing ancestor grants may flow to the mapping under the same evaluator inheritance/deny rules, but every mapped operation must pass the explicit mapping gate and remote export policy. Unknown/mismatched provenance denies. Extend subject resolver and trie tests for mapped descendants and deeper folders while retaining current local identity rules.
## 5. RPC contract and operation behavior
The following is a semantic contract; exact protobuf tags are assigned during implementation and then locked by tests. Additive evolution only; reserve removed tags; enum zero is UNSPECIFIED and rejected for required decisions.
| RPC | Shape and required semantics |
|---|---|
| `GetCapabilities` | Unary. Protocol major/minor, authenticated server node/cluster, role/readiness, catalog revision, codec types, limits and supported operations |
| `Browse` / `GetMetadata` | Unary pages/batches. Peer ingress serves authorized origin-local Raw and UNS exports; client-facing browse composes local and mapped remote branches, with bounded pages, opaque origin handles and explicit partial/unavailable results. Capabilities are intersected with the caller’s grants |
| `Read` | Unary batch. Correlation, target addresses, cached-only mode and optional maximum acceptable age. Ordered per-item value/status results; no implicit device read |
| `Write` | Unary batch. Per-item operation ID, address, typed value, expected catalog revision and remaining execution budget. Ordered per-item outcomes; partial success allowed, no batch atomicity |
| `Subscribe` | Bidirectional stream. Open/add/modify/remove commands with request IDs, acknowledged revised settings and per-item statuses; server sends snapshot chunks, deltas, catalog invalidation, heartbeat and reset controls |
Unknown unsupported RPCs/capabilities return Unimplemented/explicit unsupported status. Transport authentication, malformed envelopes and global overload use gRPC status. Valid batched requests use per-item statuses for authorization, unknown node, type mismatch and execution failures. Do not reveal a denied node's existence through diagnostics.
### Reads
Reads return authoritative cached values and original status/timestamps; maximum age is a freshness requirement, not a driver refresh request. For an over-age sample return a stale-data result with the last sample and its original quality separately, so consumers cannot mistake it for fresh Good. Missing and denied nodes produce per-item failures. No cross-node or cross-item atomic snapshot is promised.
### Writes
Authorize at the target before role disclosure and routing; validate the current catalog revision, type, writability, export scope and deadline. The primary gate is checked at admission and again immediately before device dispatch. Carry a validated admission ticket (principal/key ID, canonical subject, authority/catalog revision, node/driver incarnation, operation ID and monotonic expiry) into `DriverInstanceActor`; both immediate launch and queue drain validate it against current role, permissions, mapping and budget. A generation change requires reauthorization against the current subject or rejection. The current child queue has no such checks: adding them is an implementation prerequisite, not a property of the existing gateway. Initial support is value writes only; attribute writes are unsupported.
A result distinguishes **RejectedBeforeDispatch**, **DriverConfirmedSuccess**, **DriverConfirmedFailure**, **AcceptedUnconfirmed**, and **OutcomeUnknown**. Acceptance by an actor or gRPC service is not success. A driver with fire-and-forget semantics returns AcceptedUnconfirmed. The existing bool/string write result is insufficient: extend the internal route to preserve dispatch stage and actual driver outcome before exposing this service. The gRPC import adapter also needs this distinction end to end: current `IWritable.WriteResult` contains only a status code. Add a backward-compatible richer write-outcome capability/result and make the host, actor, import adapter and OPC UA self-correction path consume it; do not collapse Unknown or Unconfirmed into an ordinary success/failure Boolean. Existing drivers can be adapted conservatively according to their actual completion guarantees.
Mutating calls have no configured automatic retry, hedging, offline queue or automatic failover replay. If a connection drops after submission, the caller reports OutcomeUnknown and reconciles by a later read/application procedure. Cancellation after dispatch cannot undo a device write. Readback alone is not proof that a non-idempotent action did or did not execute.
Maintain a bounded node-incarnation request registry keyed by stable authenticated service principal and operation ID across reconnects, with logical payload hash and item identity. Hash address, value and semantic preconditions, excluding transport correlation, remaining budget and retry-varying headers. Every mutation carries the serving-incarnation precondition obtained during discovery; a restarted node rejects an old-incarnation request before dispatch. An actor/driver incarnation change invalidates queued admission tickets as well. Duplicate in-flight requests join the same task; cancellation of one duplicate caller detaches only that waiter and never cancels a shared dispatched operation; completed duplicates reuse the result; conflicting payloads reject. Retain entries for a negotiated retry window and reject new mutations at capacity instead of evicting in-flight entries. After restart, failover, or retention expiry, the registry offers no duplicate protection. Clients must never replay an ambiguous operation across those boundaries. This is not exactly-once delivery.
Require enough remaining deadline to cover the configured queue/write/abandon bounds plus response margin; otherwise reject before dispatch. Carry remaining budget into the actor request and use its monotonic write clock for the final pre-dispatch decision. Do not reuse a fixed eight-second Ask; preserve the existing derived `DriverWriteBounds` ordering. Once dispatched, record any eventual result even if the caller disconnects. Replace the current abandoned-write path that discards eventual outcomes with a bounded late-completion observer linked to the request registry. Until a definitive result arrives the outcome stays Unknown; if the observer/retention window ends, preserve Unknown and expose a diagnostic. A late result may refine an audit/registry entry, never automatically launch a replacement write. Admission is bounded so outstanding token-ignoring driver tasks cannot create unlimited observers.
For an imported OPC UA node, preserve the existing nonblocking write hook and explicitly document its immediate acceptance semantics. The remote outcome is async; rejection restores the last authoritative sample, and an unknown outcome marks quality uncertain pending a fresh source sample. A network failure must not become a fabricated definitive device failure or a new authoritative optimistic Good sample.
## 6. Subscription lifecycle, backpressure and recovery
Each stream is authenticated and pinned to one serving node incarnation and catalog/authority generation. Stream controls carry subscription ID, stream epoch, command ID and sequence. Sequence ordering is per stream, not fleet-wide.
1. Authorize each item and apply export limits. Atomically register interest and capture the owning partition’s version watermark in the hub; a multi-partition subscription captures a vector of watermarks as specified in §6a.
2. Freeze the item values/versions at the captured watermark (an immutable snapshot view), then send SnapshotBegin, bounded snapshot chunks and SnapshotEnd, followed by deltas newer than that watermark. The snapshot has a negotiated memory/item budget. Each item also has a monotonic version within the catalog/incarnation so consumers discard regressions. Snapshot transmission does not hold an actor or SDK lock. The bounded snapshot catch-up queue must reset if it overflows; SnapshotEnd alone cannot conceal a gap.
3. Serialize stream commands through a bounded inbound command queue with per-principal command rate limits; reject/close abusive streams before queue growth. Open/add acks establish an item revision before its snapshot/deltas; modify acknowledges the exact boundary after which revised settings apply and preserves the latest baseline for interval-only changes; changes to the selected address or filter require a new item revision and snapshot; remove cancels queued old-revision frames and its ack is the boundary after which that item revision can never appear. A remove followed by add creates a new revision. Reserve control-message capacity so value pressure cannot drop acknowledgments, resets or revocations. Values use latest-state semantics, with per-item sampling/publishing intervals and revised settings in the ack. There is no promise to deliver every intermediate device transition. Multiple local subscribers share upstream demand; sampling honors the fastest admitted request and each downstream delivery applies its own filter/rate.
4. Use bounded per-stream byte and item budgets. Coalesce latest values per item. Explicitly signal coalescing/overflow; sequence numbers are assigned to emitted frames, so sequence alone cannot detect dropped source transitions. If a coherent latest-state stream cannot be maintained, terminate with ResourceExhausted, mark local imports stale, and require a new snapshot. Never block a driver actor on a socket write.
5. Remove last interest on cancel, shutdown, principal expiry or timeout; check credential expiry on a timer even when no values or authority changes arrive; disposal releases upstream demand and buffers. Permission changes remove denied monitored items before further delivery under the new authority generation. Every outbound batch checks authority generation; queued unauthorized payloads are discarded.
6. On interruption, mark affected imports uncertain immediately, then BadNoCommunication after the configured stale timeout. Preserve the last payload and source timestamp, with local connection quality represented separately and composed for OPC UA display. Recovery returns Good only after a fresh authorized snapshot/sample.
7. Reconnect with jittered exponential backoff and a new stream epoch. V1 always takes a fresh snapshot: no durable replay, Republish or seamless TransferSubscriptions promise. Clients discard old-epoch frames. After a role switch, current-value state can recover, but transitions during the gap can be lost.
Unary calls have explicit per-operation deadlines. Long-lived subscriptions use stream cancellation, heartbeat/silence detection and a renewable bounded session lifetime instead of inheriting a short unary deadline; each stream command has its own acknowledgment timeout. An expired command timeout cannot silently imply that a state change failed: reconnect with a fresh snapshot/item revision when its outcome is uncertain.
Illustrative, tunable commissioning quotas (not capacity targets or required performance numbers): 1,000 items per read/write batch, 500 catalog rows per page, 10,000 monitored items per principal, 100 streams per node, 4 MiB per message and per-stream pending-byte budget, 5-second heartbeat, 15-second stream silence timeout, and 30-second import stale timeout. Reject requests exceeding configured limits; chunk snapshots below message/byte limits. Aggregate principal limits across all its concurrent connections. Add an aggregate hub/subscription memory budget and connection admission cap so per-stream limits cannot multiply into unbounded memory. These are tunable starting limits, not measured capacity claims.
## 6a. Scaling architecture
The design aims to raise the number of unique tags, downstream clients, monitored-item registrations and delivered updates independently. They are different resource dimensions: source sharing reduces acquisition work; it does not eliminate per-client permissions, filter state, encoding or network deliveries. There are no required hard capacity numbers at this stage. Measure workload envelopes on the intended hardware before setting production quotas.
### Shared acquisition and backing samples
Maintain one immutable authoritative sample per canonical backing source and serving provenance within a hub, with lightweight Raw/UNS view descriptors pointing to it. At the first driver callback/adapter boundary, before an asynchronous actor/queue handoff, validate shape/type/size and copy mutable driver payloads (including byte arrays, arrays and nested supported elements) once into an owned immutable sample, or accept an explicit ownership-transfer/immutable lease contract. Current object-valued callback/envelope fields do not establish immutability by themselves. Writers cannot mutate or reuse transferred storage. Snapshots, pending readers and filter baselines retain the same immutable ownership lease until released, including asynchronous encoding; pool return requires all references to be gone. Unsupported mutable object graphs are rejected rather than shared. This is a logical ownership model: current driver, actor and sink stages may retain necessary envelopes, but mappings must not create independent acquisition loops or payload copies solely because a source has multiple paths. Resolve Raw/UNS equivalence only from explicit origin catalog identity. Distinct driver acquisitions or pinned replica caches remain distinct, even for matching tag names.
Aggregate upstream interest using the complete sharing key from §4: source identity, serving node/incarnation, target mode/consistency and upstream principal/export context plus active catalog/authority epochs. Maintain reference-counted local interests and per-view authorization/filter state. One upstream interest can satisfy many compatible downstream registrations; removing one consumer cannot cancel another consumer’s demand. Update upstream settings when the aggregate interval, trigger or filter requirement changes; bound subscription churn while preserving the strongest admitted demand.
Aggregate upstream filtering as a superset of all admitted consumers’ needs: v1 shared upstream demand uses no value deadband by default. A later upstream deadband optimization requires proof of superset behavior across independent consumer baselines; taking the minimum numeric threshold alone does not establish that proof. Union trigger requirements for value/status/timestamp changes. Local consumers apply their own narrower filter; downstream filtering cannot recover transitions already suppressed upstream. Faster local demand cannot force the origin beyond its advertised acquisition capability. An add/modify that strengthens upstream demand remains pending until the origin acknowledges the revised settings and supplies a fresh versioned snapshot/boundary; only then acknowledge the new downstream item revision effective. If the origin cannot supply it, reject or explicitly revise that registration. Delay or debounce only reductions/relaxations while retaining a safe superset; never acknowledge a stronger demand while it is still being debounced.
### Partition cache updates and fan-out
Route normalized source batches to a configurable bounded set of workers by a stable hash of canonical backing-source identity plus serving provenance. Aliases of one source share an owner. Retain original batching through the actor boundary; do not add one actor message, task or lock acquisition per alias/client/value. Partition owners serialize each source’s sample/status/version/removal updates. Their ingress has node-wide and per-partition byte/item limits and a nonblocking producer contract. Apply those bounds before actor-mailbox admission; an unbounded mailbox or retry queue must not become a hidden substitute for bounded ingress. Control paths and encoded-segment queues are bounded separately as well. A source update advances one source-level dirty generation and replaces its latest sample; it does not touch every downstream registration.
When ingress is full, coalesce by source within a bounded dirty-source table, retaining version/gap information; if distinct-source or lifecycle capacity is exhausted, invalidate affected stream epochs and cached-source readiness with an explicit reset/overload result rather than dropping updates invisibly or blocking a driver actor. If affected source identities cannot be retained within the budget, invalidate the partition’s readiness epoch conservatively. Reads and reconnect snapshots must then return unavailable/uncertain state, never the old cached sample as fresh Good. Restore readiness only after a fresh authoritative retained/source sample or an explicitly tracked resynchronization establishes current state; stream recreation alone is insufficient. Keep a reserved ordered control path for status/removal/catalog/authority invalidation so a coalesced value cannot resurrect a deleted source. Network encoding and socket waits occur outside driver and partition owners, so no central actor or shared lock must receive and fan out every tag update.
This partitions new hub/delivery work; it does not by itself parallelize current `DriverHostActor` acquisition routing or legacy Raw/UNS SDK sink materialization. Inventory those existing single-owner stages, preserve batch handling, and measure them alongside the new service. Where the common publication boundary can emit normalized source batches before alias fan-out, let the new hub consume that source-level feed without adding another per-alias hot-path pass. Any further refactor of existing actor/sink ownership requires separate ordering/lifecycle tests, not an assumption that worker count removes that bottleneck.
Catalog/mapping lifecycle, admission and authority publication form a separate low-rate control path. Publish immutable routing/authority snapshots to workers. Every queued item carries its view/mapping, authority, catalog and stream/item revisions; delivery gates discard invalid items regardless of partition progress. A demotion, removal or revocation invalidates admission immediately through the control path; it does not wait behind a source-data backlog. Preserve existing revocation and metadata-lease semantics.
Snapshot registration is atomic within each owning partition. A multi-partition stream uses a vector of partition watermarks and immutable captured samples; it does not promise a cross-partition point-in-time snapshot. Emit SnapshotEnd only after every admitted partition’s snapshot completes, then drain its valid newer deltas without regressing per-item versions. Snapshot/catch-up budgets cover all participating workers and retained immutable samples. Repartitioning drains/resets affected stream epochs before changing ownership; do not silently reorder samples while moving a source.
### Compact wire format and stream handles
Negotiate compact-batch capability and codec version in `GetCapabilities`. Subscription add/open acknowledgments bind an opaque numeric item handle to the canonical requested view and its metadata; steady-state updates carry the handle, item revision, sample version and typed sample fields instead of repeating full paths and mapping metadata. Handles are valid only in their authenticated stream epoch and serving incarnation, are not reused within that epoch, and are invalidated on removal/rebind. They confer no access by themselves: admission and delivery still validate the actual requested view and current authority. Unknown/stale handles reject explicitly. Writes retain their full resolved identity, admission ticket and operation ID; a numeric handle must never weaken mutation preconditions.
A batch header may hold common origin/epoch, quality and timestamp defaults, with explicit per-item presence/override markers. Preserve null versus absent value, absent timestamps, complete status subcodes, timestamp precision and each item’s original sample version. Group only compatible provenance/revision contexts; mixed groups use separate headers. Frame limits apply to decoded size and item count as well as encoded bytes; invalid counts, dimensions and handle references fail before allocation grows. Use a typed legacy envelope when compact capability is not negotiated; both representations must produce equivalent domain samples.
Use pooled/ref-counted immutable payload storage where profiling justifies it. A buffer cannot return to its pool while any snapshot, pending delivery or async writer retains it. Count retained bytes across owners and subscribers in global admission accounting; zero-copy optimization cannot bypass memory limits. Compression is optional and negotiated per connection/workload, not a default requirement: compare CPU cost against bandwidth savings for numeric batches versus strings/arrays.
### Delivery scheduling, filtering and authorization caches
Keep source acquisition cadence separate from downstream publishing. Each downstream registration has revised publishing interval, change trigger, optional numeric deadband, last delivered baseline and latest pending sample. Deadbands apply only to supported numeric types with negotiated units/range semantics; reject unsupported requests. Status, connection-quality and deletion changes are never suppressed by a value deadband. Timestamp-only delivery follows an explicit trigger setting. Reconnect/rebind creates a new baseline from the authorized snapshot. Advance a registration’s filter baseline only when its sample is committed by the serialized stream writer (successful bounded send), never when a candidate is encoded, coalesced, dropped or suppressed. Filter decisions use the committed baseline/version; if it changes while a candidate is prepared, re-evaluate the latest pending source sample before suppressing or sending it. A failed/ambiguous send resets that stream epoch rather than treating the candidate as delivered.
Do not perform an unbounded full-client walk on each source update: maintain per-source interest indexes on interest changes, and update only the latest source sample plus dirty generation on data arrival. A timed scheduler traverses dirty/due interests using resumable, fair bounded slices; each registration records its last considered/delivered source version and reads the latest immutable sample when due. If a source changes during traversal, its later dirty generation schedules another slice without restarting or starving later consumers. Bound scheduled work, dirty tables, baseline storage and due-index/timer entries as well as payload queues; a high-fan-out tag cannot allocate a new task/pending payload for every client on every source change. Distribute scheduling work across workers. Each gRPC stream has one serialized writer and ordered command/removal barriers; encode bounded ready batches outside partition locks and let the writer merge only revision-valid segments. Prevent a hot tag or slow client from monopolizing workers through per-principal and per-stream byte/item/work budgets. Isolate browse/snapshot work from steady-state delivery and reserve bounded control/admission resources. Overload produces explicit throttling/coalescing/reset, not unbounded queues; control priority must not starve ordinary delivery indefinitely.
Cache local authorization decisions by authenticated key/principal, operation, exact realm-qualified view/subject, mapping revision, serving provenance and authority generation; entries also honor credential expiry and remote metadata/export leases. Use the existing generation-based permission cache pattern rather than walking grants per update. Published generation changes and expiry invalidate admission and queued delivery even when a cache entry was Allow. Bound cache entries and negative-cache lifetime, and recheck the generation/lease at the final send gate. That gate defines emission for revocation tests: bytes already handed to the transport cannot be recalled, while unsent segments must pass the current authority gate. Do not share a cached Allow across Raw/UNS aliases, principals, node-pinned contexts or independently restricted mappings.
### Demand-driven catalogs and controlled recovery
Retain the per-parent paged browse/materialization model from §4. Browse loads metadata only; acquisition begins when an explicit read/subscription/runtime dependency requires it. Cached reads do not create driver refresh loops. Pin metadata needed by active interests within their admitted budget, and evict idle pages/sample payloads only where no consumer or existing local-runtime requirement retains them. Coalesce equivalent concurrent browse requests within the same authorization/mapping context, with bounded waiters and shared-operation cancellation isolated per waiter.
Use a node-wide snapshot admission scheduler with global and per-principal limits for concurrent snapshots, bytes in flight and snapshot bandwidth. Jitter reconnects, deduplicate compatible upstream reconnects and pace downstream snapshots. Retain capacity for live traffic and control messages; a recovering client must not stop local publication or monopolize the active server. Each admitted snapshot has a bounded lifetime and catch-up budget; reject/defer overload with explicit retry guidance instead of a reset/retry storm. Failover still requires new epochs and fresh snapshots, and never replays ambiguous writes. Write admission and outcome tracking have reserved bounded resources independent of subscription bursts.
### Horizontal growth and release boundary
The first release includes the in-process partitioning, shared samples/demand with immutable ingestion and filter-superset admission, bounded partition ingress/fair traversal, compact negotiated batches, filters, authorization caches, demand-driven catalogs and recovery scheduler above. Partition count, budgets and compression are implementation/commissioning choices subject to measurement, not new promised capacity levels.
Scale authoritative acquisition horizontally by assigning independent driver/source ownership to additional application clusters and presenting them through Raw/UNS mappings. This does not distribute one field-device write across competing owners. A redundant pair is sized so its eligible active member can carry the admitted workload alone; failover admission cannot assume both members contribute aggregate capacity. Node-pinned replica reads remain opt-in with explicit freshness/readiness semantics.
Consumer-side aggregation services or read-distribution nodes are a future extension to reduce source fan-out across many consumers. They are not implicitly enabled by subscription sharing: v1 peer exports remain origin-local and one hop. A later relay design must separately specify provenance, end-to-end authorization/revocation leases, bounded staleness, loop prevention, recovery and capacity ownership before allowing projected values to be re-exported. Writes continue to the authoritative active owner. No automatic data-plane load balancing across primary and standby is introduced here.
## 7. Node and cluster routing, redundancy and failure boundaries
The protocol supports the following required topologies with identical authorization, codec, paging and outcome semantics:
| Topology | Browse/metadata | Read/subscribe | Write |
|---|---|---|---|
| Node A → node B in the same cluster | gRPC against B’s authorized local Raw/UNS catalog | gRPC against B’s ready local cache/streams | gRPC with B’s role/readiness and final device-dispatch gate |
| Node A → a node in another cluster | The same node-pinned gRPC service | The same cache/stream semantics | The same explicit per-item outcomes |
| Node A → a logical cluster endpoint set | Browse current eligible primary’s catalog | Read/subscribe on selected primary; new epoch after failover | Select primary before submission; ambiguous operations never replay |
**Node connection:** dial the configured node and verify its machine and cluster identities; the target cluster may equal the caller’s cluster. It remains pinned; no silent substitution of its partner. All enabled operations are normal supported API use, not a diagnostics-only endpoint. A ready secondary may serve its own cached reads and subscriptions only under an explicit connection consistency policy and the existing Warm/Hot rules; stamp results with serving node, incarnation, role and freshness so the caller knows which cache it used. An uninitialized Warm standby returns unavailable/initial-data status rather than pretending its cache is current. Writes require a ready primary: a pinned secondary returns RejectedBeforeDispatch, and the caller must explicitly choose a cluster-target connection if it wants primary selection.
Different-process intracluster access uses gRPC, not actor selection, DPS, OPC UA loopback or shared-memory assumptions. Akka remains responsible for the current election and actor work inside the receiving process. Same-process local consumers may use the local adapter with the same policy/outcome contracts. Being in the same cluster conveys no implicit authentication or grant, and does not bypass TLS or mapping scopes. The existing device primary gate is unchanged by adding this connection.
Origin ownership remains `(cluster, realm, stable entity/address)` for redundant replicas; serving machine/incarnation is separate provenance and a required Node-target routing/precondition. Configured mapped views receive distinct local paths even when originClusterId equals localClusterId. Reject a mapping that routes back into its own serving process/view or into its own descendants; a different node in the same cluster is explicitly allowed. Reject origin records that are themselves projected mappings. Neither endpoint forwards a rejected pinned write to its partner, and mapping a peer’s data never starts a second field-device driver.
**Consumer-node execution scope:** connections and mappings are authored within a cluster but are not implicitly active on every member. Add explicit `ConsumerNodeIds` to `DataConnection` and `RemoteNamespaceMapping`; a mapping’s set must be a subset of its connection’s. Resolve each consumer against deployed machine identities, and project only its effective connections/mappings/secrets into that node’s execution graph. The source-node set controls local execution, while target-side authorization still uses authenticated principals and grants, never a caller-supplied source-node header.
For a pinned A→B mapping in one cluster, consumers are `{A}` and the target is `B`. A receives the mount and starts the gRPC connection; B receives no active projection of that mapping, creates no corresponding anchor/subscription/driver, and continues serving its origin-local catalog. Validate every intended consumer/target combination at deployment time: explicitly selecting B as a consumer of its own B mount is a configuration error, not a reason to reject the valid A mapping. If one authored mapping is intended for both replicas, author separate per-node targets/mappings with disjoint consumer scopes or use a validated cluster-target policy that excludes the consumer itself; do not silently retarget a Node connection. Last-known artifacts preserve this scope offline. Removing a node from the consumer set drains its scoped connections, removes its projected anchors/items and invalidates associated handles; queued writes are rejected before dispatch and dispatched writes retain their existing outcome tracking. Failover never implicitly expands the consumer set or retargets a pinned connection; a newly active consumer uses its own validated effective scope.
A direct `IDataConnection` request does not require mounting a namespace. It uses an allowed connection in the caller node’s execution scope and the same remote service, permissions, readiness and outcomes. Namespace mappings add browse/materialization on top of that connection. Nonconsumer nodes omit those projected nodes intentionally; node-specific browse catalogs and readiness are observable, not assumed identical because their cluster IDs match.
**Cluster connection:** configure ordered eligible endpoints for the expected application cluster; client probes capabilities/readiness and uses the current primary. Discovery uses deployed endpoint records, not a fleet gossip ring. Try another eligible endpoint on a failed read or new subscription within the original deadline. Failover is per write item: an item may change endpoint only following an explicit authenticated RejectedBeforeDispatch result. Never resend an entire partly successful batch. DriverConfirmedSuccess/Failure, AcceptedUnconfirmed and OutcomeUnknown items are not replayed. A lost batch response makes every item without a known definitive result ambiguous and prevents replay. Rediscover the new serving incarnation and construct a new attempt only for items proven never dispatched.
Role changes stop accepting new writes, close primary-only streams with a retryable status, invalidate routing hints and trigger discovery. After reconnect, the client requires a fresh snapshot. Do not merge samples from two nodes into one subscription epoch. A stale role hint is never authorization or a fence.
This feature inherits the existing cluster's election and device split-brain limitations. A role recheck reduces stale admissions but is not a distributed fencing lease. If two partitioned nodes both believe they are primary, gRPC cannot prove exclusive device ownership. Do not claim new split-brain safety. Tests must cover known-secondary rejection, startup without a role snapshot, partition behavior and in-flight demotion; stronger external fencing is a separate architecture decision.
Connections operate without a central relay or live ConfigDb dependency. A local cluster continues local work when a remote cluster or central is unavailable. The import adapter follows the existing local Warm/Hot role policy: a Warm standby cannot independently activate an import driver or publish a second active feed; a Hot standby may keep its own cache warm only where that policy permits it, without enabling writes. Verify role transitions and demand ownership in the import lifecycle tests. No subscription carries Akka references or forwards into remote DPS. Maintenance/disabled endpoints are excluded from new connection selection through the deployed connection revision.
## 8. Authentication, authorization and deployment configuration
Use a dedicated HTTP/2 TLS data listener with explicit advertised HTTPS URI and server identity, separate from existing h2c telemetry/command endpoints. Production requires TLS and certificate verification. Development cleartext requires an explicit loopback-only override. Register the service only when enabled and enforce listener/host admission; ASP.NET endpoint mapping alone must not expose it through every Kestrel listener.
Authenticate a connection using a target-issued data-plane API key over TLS, resolved through the existing authority machinery; separate credentials per directional connection permit revocation. Bind the authenticated principal to expected source connection/cluster policy. Claimed source IDs in payloads are audit context, not identity. Keep mesh PSKs scoped to the existing command plane. Optional mTLS can add peer authentication later; it must not replace operation grants.
V1 uses a named service principal for upstream imports, not user impersonation. Local end-user authorization gates access to imported nodes, including every browse result/continuation and cached metadata response; target authorization independently gates the service principal's remote scope. Effective access is the intersection of local permission, imported capability, target export allowlist and remote service-principal grants. Remote audits identify the service principal and separately record untrusted originating-user context. Delegated per-user remote identity is a future feature, not implicitly trusted headers.
Extract SDK-independent policy and catalog subject resolution from the existing authorizer, including UNS reference variables/folders, Raw aliases and scripted-condition ownership. Reuse authority generations, API-key expiry/revocation and operation mapping. The new data endpoint always enforces its export/grant policy even when legacy OPC UA ACL enforcement is disabled. Mapped Raw and UNS nodes always enforce their local mapping ACL gate, even if `Security:Acl:Enabled` is false for ordinary local OPC UA nodes. Insert the gate ahead of current `AclEnforced` bypasses, including Browse/BrowseNext, read and monitored-item delivery; a configured anchor alone grants no descendant access. Remote grant leases complement rather than replace these local per-caller checks. No granted/ready authority snapshot means deny. Authorization of one Raw alias must not accidentally confer access via another UNS alias, or vice versa.
Proposed configuration entities:
- `DataConnection`: stable ID, owner cluster, explicit consumer node IDs, Node/Cluster target kind, expected target identities, ordered HTTPS endpoints, secret reference, enabled flag, timeout/backoff/quotas and permitted operation subset.
- `DataExport`: origin-local selected roots/nodes, peer principal, operation grants and per-principal limits. Remote imports are excluded from selection in v1.
- `RemoteNamespaceMapping`: stable mapping ID, owning local cluster, explicit consumer node IDs, explicit Raw/UNS realm, local parent identity/branch name, connection ID, remote same-realm root/entity identity, enabled flag, mapping ACL scope/grants, permitted operations, subtree projection/cache limits and metadata-lease policy, sample/publish defaults, staleness policy and write-through flag (default false). Optional leaf selections narrow a subtree; they are not required for browse discovery.
- Runtime projection records: mapping revision, realm-qualified local path/subject, origin entity/address, remote catalog revision and value-demand state; derived from browse/catalog, not separate authored per-tag import rows.
Deploy resolved catalog references/configuration atomically; validate collisions, ownership, consumer scopes, per-node self-route rejection, endpoint identity, TLS scheme, unsupported types, prohibited re-export and missing secret references before apply. Endpoint changes drain old subscriptions and start a new epoch. Cross-node/cross-cluster schema permits explicit same-realm namespace mapping edges, rejects a local descendant mounted beneath itself and prohibits peer re-export; do not relax the existing same-cluster driver-bound equipment invariant. Runtime nodes hold last successfully deployed configuration, never wait on central SQL to serve data.
## 9. Observability, rollout and verification
Expose connection state (Disabled, Connecting, Ready, Degraded, Backoff, AuthFailed), selected endpoint, last successful snapshot, last sample age, role, protocol version and diagnostic reason. Count unique backing sources, upstream interests, downstream registrations, streams/clients, source updates and delivered updates separately. Measure sharing ratio, bytes per delivered sample, per-partition queue/CPU time, scheduling delay, snapshot backlog, authorization-cache hit/miss/invalidation, retained payload bytes and slow-consumer throttles, as well as reconnects, coalescing, overflow/resets, denials and write outcomes. Trace correlation across ingress and actor route. Keep tag IDs/operation IDs out of metric labels; audit writes with principal, canonical address, operation ID and outcome, without secrets or value payloads by default.
Roll out disabled by default. First add contracts/policy/hub parity tests; then local service and TLS ingress; then read/browse/subscription canaries with compact/fallback equivalence and partition/recovery gates; then imports; finally enable explicitly granted writes. Negotiate capabilities before enabling a connection. Mixed versions fail individual unsupported capabilities without disabling local OPC UA. Rollback disables connection/import execution, marks imported values unavailable and keeps local driver ownership and existing mesh traffic intact; it never replays pending writes.
Required verification before release:
| Area | Evidence / acceptance gate |
|---|---|
| Wire/codec | Golden protobuf contracts, additive-version peer tests, full status fidelity, UInt64 boundaries, null/arrays/timestamps and malformed/oversized payload rejection |
| Hub | Driver + virtual + status/removal parity; snapshot/delta race; catalog swap; no optimistic value leakage; bounded buffering under stalled clients |
| Authorization | Empty authority fail-closed; wrong/expired/revoked key; TLS/cluster mismatch; export boundaries; Raw/UNS alias cases; no cookie challenge or exposure on unintended listener; stream revocation with queued frames |
| Writes | Real adapter success versus fire-and-forget acceptance; pre-dispatch timeout; dropped response after device success; duplicate concurrent ID/conflicting payload; registry capacity; demotion; cancellation; no replay after failover |
| Lifecycle | Add/modify/remove acknowledgments, revised sampling, snapshot overflow, idle timeout, reconnect epoch, gap visibility, last-interest cleanup and endpoint revision change |
| Topology | Actual same-cluster A→B gRPC browse/read/write/subscribe tests, cluster-wide artifact with A-only consumer projection and B serving-only behavior, direct API without mount, no pinned-cache substitution or cross-principal sharing, pinned-primary and secondary/Warm readiness behavior, identical Raw/UNS paths kept separate; two independent redundant clusters (four nodes), no shared Akka membership; node-pinned access, cluster failover, warm standby, maintenance, central outage and partition scenarios |
| Scaling | Compare one versus multiple workers and compact versus fallback encoding; Raw/UNS many-view source sharing; principal/pinned-node isolation; last-interest cleanup and interval churn; deadband/status/timestamp trigger superset and stronger-demand admission boundary; mutable array/bytes reuse after callbacks and baseline lifetime; cross-partition snapshot/command ordering; revocation while batches encode; payload retention/pool lifetime, nonblocking ingress overflow/source-dirty traversal fairness, cache-readiness invalidation after lost ingress and decoded frame bounds |
| Capacity | Vary unique sources, registrations/client overlap, clients/streams, change rates, payload shapes and slow consumers independently; measure source versus delivered throughput, CPU, retained memory, wire bytes and p95/p99 latency under steady state, bursts, concurrent browse, reconnect storms and single-member failover. Verify global/principal admission, control/write reservations and bounded recovery; record the measured workload envelope before accepting commissioning quotas. No hard capacity threshold is required by this design |
| Cross-node/cross-cluster browse | Expand Raw Folder/Driver/Device/TagGroup/Tag and UNS Area/Line/Equipment/signal in each topology; newly added descendants; deep projected subject permissions; collisions/aliases; deterministic merged paging; partial peer outage versus empty folder; bounded metadata revocation during partition; explicit mapped ACL scope with legacy ACL toggle off; per-parent projection readiness and BrowseNext retries; continuation replay/revocation/failover; bounded browse fan-out; no subscriptions from browse; asynchronous OPC UA projection and model-change notifications |
| Imports | Collision/loop prevention; correct dependencies/fan-out; stale quality and recovery; write-through local+remote authorization; removal and rename behavior |
Use software driver fixtures for protocol/fault tests; use explicit hardware commissioning only after software gates. A documentation-only draft requires link/path verification, not a runtime test run.
## 10. Review decisions for the user
1. Confirm the Raw and UNS mapping model and cross-node/cross-cluster browse/read/write/subscribe semantics above; these implement the user’s requested expandable naming structure. The earlier flat selected-import assumption is superseded.
2. Approve cached reads and latest-state subscriptions, with visible gaps and fresh snapshots on failover?
3. Approve service-principal identity for imports, dedicated TLS listener and target-side enforced export grants?
4. Approve value writes in v1, with no automatic ambiguous retry and honest unconfirmed/unknown outcomes?
5. Approve deferring history and alarm/event operations to negotiated extensions?
6. Review the §6a scaling architecture: partitioned processing and compact/shared delivery in v1; read-distribution relays remain a separately designed extension, with capacity established through workload measurements.
After user approval, create a detailed implementation plan and execution batches. This document authorizes no code changes or deployment.
## References
- [gRPC .NET implementation](https://github.com/grpc/grpc-dotnet), checked through Context7 for channel reuse and deadline/cancellation propagation.
- [Microsoft: gRPC performance](https://learn.microsoft.com/en-us/aspnet/core/grpc/performance?view=aspnetcore-10.0): reuse channels, distinguish HTTP/2 stream concurrency from tag subscriptions, and account for flow control and long-lived calls.
- [Microsoft: retries](https://learn.microsoft.com/en-us/aspnet/core/grpc/retries?view=aspnetcore-10.0): built-in retries do not resume a stream after responses have begun; application recovery is necessary.
- [Microsoft: deadlines and cancellation](https://learn.microsoft.com/en-us/aspnet/core/grpc/deadlines-cancellation?view=aspnetcore-10.0): a caller deadline can expire after server work completes, and cancellation must be propagated explicitly.
@@ -0,0 +1,568 @@
# gRPC Data Connections Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Implement scalable, authorized Raw/UNS browse, cached read, value write and subscription connections between processes in the same cluster and across independent redundant clusters, verified with focused unit tests and local Docker/OPC UA CLI integration.
**Architecture:** Keep Akka membership/election and existing command/telemetry contracts intact. Build SDK-independent data/mapping policy and a partitioned authoritative sample hub behind transport-neutral ports; Host supplies TLS gRPC ingress, Runtime supplies local/remote adapters, and the existing OPC UA server materializes bounded mapped views. Preserve primary-only mutation, node-pinned provenance, explicit uncertain outcomes, bounded resource use and authority revocation across every path.
**Tech Stack:** Existing .NET 10/C#, protobuf/grpc-dotnet, Akka.NET, EF Core/SQL Server, xUnit/VSTest, OPC UA SDK and `otopcua-cli`; Python 3 orchestration and local Docker Compose for reproducible gates. Retain repository-pinned package versions unless a demonstrated requirement warrants a separate change.
---
Status: Astra reviewed; approved for user implementation-plan review after revision. Awaiting user review. This task prepares a plan only; it does not authorize implementation, builds, running fixtures or deployment.
Design: [reviewed design](2026-11-15-grpc-data-connections-design.md). Design review: [Astra design findings](2026-11-15-grpc-data-connections-astra-review.md). Plan review: [Astra implementation-plan review](2026-11-15-grpc-data-connections-plan-astra-review.md).
All commands and fully qualified task paths below are relative to the repository root `/Users/dohertj2/Desktop/OtOpcUa`. New names are planned files/classes, not claims that they already exist. Existing source paths were inspected; implementation must reread nearby instructions and source before editing. The checkout contains unrelated AB CIP fixture changes; leave them out of this work and its commits.
Task path prefixes are explicit shorthand for these assembly directories: `Runtime/` = `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/`; `OpcUaServer/` = `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`; `AdminUI/` = `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/`; `Core/` = `src/Core/ZB.MOM.WW.OtOpcUa.Core/`. Bare `Drivers/` and `OpcUa/` paths in Runtime task Modify lists are relative to the Runtime directory; bare Configuration snapshot/migration filenames are relative to `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/`. New harness subpaths (`seed/`, `probe/`, `device/`, `tests/`) in P10–P12 are relative to `scripts/validation/grpc-data/`. Expand them into exact owned paths in every worker packet; never create top-level alias directories.
## Execution ownership, models and checkpoints
The coordinator owns the dependency graph, interface decisions, integration, test manifest and final evidence. Use at most four active agents total: coordinator, two disjoint implementers, and one reviewer. Reduce concurrency when shared-file ownership, compilation or Docker contention requires it. Only the coordinator starts tests, EF migration generation, Docker gates and final commits; agents may prepare test commands but cannot launch competing test/build processes. The coordinator can explicitly lease one focused red/green run to a worker while all other runs remain paused.
| Role | Model / effort | Assignment and reason |
|---|---|---|
| Coordinator/integrator | `gpt-6.1-sol`, high | Track requirements, own shared wiring and commands, integrate results, assess evidence and report blockers |
| Implementation worker | `gpt-6.1-terra`, high | Bounded production-code task plus its behavioral tests; explicit file ownership |
| Protocol/security/concurrency implementation | `gpt-6.1-terra`, xhigh | Authority extraction, dispatch-ticket/unknown-write semantics, partition/snapshot ordering; use only for these higher-risk tasks |
| Test/fixture worker | `gpt-6.1-sol`, high | Independent test harness, Docker/CLI orchestration, fixtures and evidence parsing |
| Mechanical documentation/manifest work | `gpt-6.1-sol`, medium | Exact path/selector updates and runbook changes after contracts are frozen |
| Astra reviewer | `gpt-6-astra`, xhigh | Review frozen contracts and security/concurrency boundaries, phase evidence and final integrated implementation; no implementation edits |
When selecting an explicit model/effort in `spawn_agent`, use `fork_turns="none"` and provide a self-contained packet. Do not silently substitute a missing model: report the unavailable choice and use the coordinator to do independent work until the user can resolve a required reviewer dependency. No reviewer approves their own implementation. Do not use more agents than the current concurrency budget.
Example worker dispatch after plan approval:
```text
spawn_agent(task_name="p04_hub", model="gpt-6.1-terra", reasoning_effort="xhigh", fork_turns="none",
message="Implement only P04 task 9 against <commit>. Read design §§3,4,6a and plan P04.
Own Runtime/DataConnections/PartitionedSampleHub.cs and named tests only.
Frozen interfaces are in Core.Abstractions/DataConnections. Do not edit DriverHostActor,
Host wiring, schema, test manifest or other workers' files. Return diff summary,
invariants and the exact P04 focused commands. Test execution is coordinator-owned.")
```
Every handoff includes the baseline commit, design sections, exact owned paths, interface version, prerequisites, negative cases and allowed test scope. Workers return changed paths, test cases added, commands/results (if explicitly leased), risks and open decisions. No worker stages another worker's changes or makes repository-wide formatting changes.
Use the suitable current checkout or an already attached worktree. If isolation requires a new one, discover/use `create_worktree` and a `codex/grpc-data-connections` branch; do not create a worktree merely because this document exists. Reuse a current suitable worktree when execution starts. Do not discard user edits.
### Dependency and concurrency graph
```mermaid
flowchart TD
P00[Preflight and test manifest] --> P01[Contracts and codec]
P01 --> P02[Schema and deployment projection]
P01 --> P03[Policy and mapped subjects]
P02 --> P03
P01 --> P04[Immutable samples and partitioned hub]
P03 --> P04
P03 --> P05[Write tickets and outcomes]
P04 --> P06[Stream engine and scheduler]
P03 --> P06
P04 --> P07[TLS ingress and remote connection client]
P05 --> P07
P06 --> P07
P07 --> P08[Raw and UNS projection / adapter]
P02 --> P08
P08 --> P09[Authoring and diagnostics]
P01 --> RIG[Local rig / CLI harness preparation]
P09 --> P10[Docker CLI and fault integration]
RIG --> P10
P10 --> P11[Scaling and recovery evidence]
P11 --> P12[Final build / full tests / review]
```
Parallelism is by file ownership, not by diagram alone. Safe examples: P02 entity files versus P04 new hub files after P01 contracts freeze; rig/probe scaffolding versus production worker. Serialize changes to Commons/Core.Abstractions contracts, `DriverInstanceActor`, `DriverHostActor`, `OpcUaPublishActor`, Host `Program.cs`, `ServiceCollectionExtensions`, solution/project registration, `Directory.Packages.props`, model snapshot, and test manifest. The diagram's rig preparation is read/scaffold work until contracts/wiring are usable; no early whole-suite run.
Astra checkpoints: P01 contract freeze; P03/P05 security and mutation evidence; P06 stream/scaling evidence; P08 mapped SDK behavior; P10–P12 integrated/final evidence. Intermediate review can run while an independent next task is prepared, but dependent work does not proceed past an unresolved high-severity boundary. Fixes receive focused regression tests and review. User review of this plan is the next step; do not start implementation in the plan-writing turn.
## Test policy: applicable units by phase, full tests only at the end
Create `scripts/validation/grpc-data/unit-phases.json` and `run-unit-phase.py` in task 1. The manifest lists exact test projects, phase traits for new tests and selected existing class filters below. All new unit test classes have `[Trait("Feature", "GrpcData")]` and `[Trait("Phase", "Pnn")]`. All tests for one class use one phase; a shared helper has no traits of its own. Every selector is a manifest entry, not an inferred changed-files heuristic.
Representative manifest record:
```json
{
"P05": [{
"project": "tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj",
"filter": "(Feature=GrpcData&Phase=P05)|FullyQualifiedName~DriverHostActorWriteRoutingTests|FullyQualifiedName~DriverInstanceActorConcurrentWriteTests|FullyQualifiedName~ActorNodeWriteGatewayTests|FullyQualifiedName~PrimaryGatePolicyTests|FullyQualifiedName~DriverWriteBoundsTests"
}]
}
```
The runner invokes subprocess argv, never `eval`, with:
```bash
AKKA_TEST_TIMEFACTOR=3 dotnet test <exact-project.csproj> -c Release -m:1 \
-p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
--filter '<manifest expression>' --logger trx \
--results-directory 'artifacts/grpc-data/<run>/unit/<phase>/<project>'
```
Build the selected project and its dependencies normally; do not use stale `--no-build` binaries during phase runs. `--no-restore` is allowed only after restore for the current dependency graph. List/discover tests for each selector first and fail on zero matches. Parse phase TRX for executed>0, zero failures and zero unexpected skips, per project; a broad solution floor is not enough for a phase. Include a per-new-class discovery manifest so a typo cannot be masked by matching an old class. Record exact filter, SHA/dirty diff identity and durations. Phases run only their manifest units, not every prior phase or all solution tests. If a later change modifies a shared earlier contract, add only the affected earlier classes to the current phase and explain why.
Red/green task loop, required for each task below:
1. Add the stated failure/edge-case tests before the corresponding behavior; select one new test/class with `run-unit-phase.py Pnn --case <ClassName>` and verify an assertion/compile failure attributable to the missing behavior.
2. Implement only that task's behavior. Compile errors caused by new interfaces are resolved within the owned dependency boundary; do not fabricate passing behavior or disable existing tests.
3. Run that task's selected classes to green. Repeat this small loop for the task's scenario groups rather than writing all production code first.
4. After all tasks in the phase, coordinator runs `python3 scripts/validation/grpc-data/run-unit-phase.py Pnn` once as the phase gate, then inspects TRX/discovery and reviewer feedback.
5. Commit only explicit owned production/test/docs files after the gate: `git add -- <explicit paths>` followed by a concise phase commit. Do not use `git add .` or commit generated test artifacts, credentials or unrelated edits.
Existing focused unit tests may internally start in-process actor systems/SDK fixtures; the phase gate still excludes fixture-backed integration test projects. SQL migration checks, TLS wire tests, CLI subprocess tests and Docker scenarios are named integration gates in their specified phases, not hidden whole-suite repeats. No timing retries, timeout widening or skip changes to mask failures.
### Exact existing test projects
| Key | Project path |
|---|---|
| A | `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` |
| C | `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ZB.MOM.WW.OtOpcUa.Commons.Tests.csproj` |
| F | `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/ZB.MOM.WW.OtOpcUa.Configuration.Tests.csproj` |
| K | `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/ZB.MOM.WW.OtOpcUa.Core.Tests.csproj` |
| R | `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj` |
| O | `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj` |
| H | `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj` |
| P | `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj` |
| U | `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj` |
| L | `tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj` |
New tests named below live in `<project directory>/DataConnections/<ClassName>.cs` unless a different exact path is given. Existing classes keep their existing paths. Expand project keys into full paths in the runner manifest; keys are document shorthand only.
## P00 — Freeze scope and establish execution controls
**Owner/model:** coordinator, Sol/high; mechanical runner worker Sol/medium. **Prerequisites:** user accepts plan. **Gate:** runner Python unit tests and focused U discovery-isolation units only, no solution tests.
### Task 1: Preflight, discovery isolation, phase runner and requirement ledger
**Modify:** `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/AdminUiSqlServerFixture.cs` to add an explicit unit-only discovery switch before any SQL probe/reaper and an injected probe/reaper seam.
**Create test:** U `DataConnections/GrpcDataUnitDiscoveryIsolationTests.cs`, tagged P00; exercise the actual attribute/probe path with a counting fake connector/reaper, not only the Python environment mock.
**Create:** `scripts/validation/grpc-data/run-unit-phase.py`, `scripts/validation/grpc-data/unit-phases.json`, `scripts/validation/grpc-data/test_unit_phase_runner.py`, `docs/plans/evidence/grpc-data/README.md`.
**Read:** design and all Astra design findings; `scripts/ci/ci.runsettings`; `scripts/ci/assert-not-all-skipped.sh`; `scripts/ci/README.md`; source/test project references; execution skill. Record current SHA, dirty files, SDK, Docker context metadata and host architecture without starting containers or printing secrets. Discover current test project/class names and replace every conditional path in this draft with verified entries in the execution manifest.
**Behavior/tests:** fake subprocess results prove runner rejects zero discovery, missing new classes, skipped-only TRX, stale/missing result files, nonzero exit and unknown phases; supports exact class narrowing; uses unique phase/project output directories and preserves exit codes. No blanket retries. Set a dedicated `OTOPCUA_UNIT_ONLY_DISCOVERY=1` in BOTH discovery and execution children for unit phases; `SqlServerFactAttribute`/probe returns a unit-mode unavailable marker before opening SQL. This does not count SQL tests as exercised: they are unselected and their required final integration remains enabled/audited. Sanitize inherited fixture/connection variables using an allowlist, replace unselected default endpoints with loopback unavailable sentinels, and audit discovery-time attributes/fixtures in every touched test assembly. Verify with a fake network/probe factory that unit-mode discovery makes zero fixture connections, including to the current AdminUI default shared SQL host. No unit runner invokes reapers. Unset this switch for integration/final runs.
After this guard is in place, perform one focused baseline compile/discovery (not a full test run) of the selected phase projects on the starting candidate. Core.Abstractions.Tests transitively references Core and many concrete drivers; record unrelated baseline compile failures/dirty AB CIP effects without editing them silently. Selectors limit executed tests, not transitive compilation. Snapshot design invariants in a requirement-to-task-to-test ledger, including H1–H9/M1–M4 and their resolved corrections.
**Run:** `python3 -m unittest discover -s scripts/validation/grpc-data -p 'test_unit_phase_runner.py'`. Expected: all runner tests execute/pass. Then run `python3 scripts/validation/grpc-data/run-unit-phase.py P00` with manifest entry U `(Feature=GrpcData&Phase=P00)`; prove the actual attribute/discovery guard makes zero connector/reaper calls and final-mode permits the injected connector. Add dry-run invocation displaying selected project/filter without executing dotnet. Commit `test: add focused grpc data phase gates`.
## P01 — Transport-neutral contracts and lossless protobuf
**Owner/model:** Terra/high; Astra/xhigh contract review. **Depends:** P00. **Unit gate:** C new P01 plus `TelemetryProtoContractTests`, `MeshTransportProtoContractTests` and `DeploymentArtifactProtoTests`; A new P01 plus existing `DataChangeBatchContractTests`.
### Task 2: Define addresses, views, capabilities and rich outcomes
**Create:** `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DataConnections/DataAddress.cs`, `DataContracts.cs`, `IDataConnection.cs`, `IDataServices.cs`, `IWriteOutcomeCapability.cs`; A `GrpcDataContractsTests.cs`.
Freeze these contract decisions in code before parallel work:
```csharp
public enum DataWriteState {
Unspecified = 0, RejectedBeforeDispatch, DriverConfirmedSuccess,
DriverConfirmedFailure, AcceptedUnconfirmed, OutcomeUnknown
}
public sealed record DataAddress(string OriginClusterId, string NamespaceUri,
string IdentifierKind, string Identifier);
public sealed record ServingIdentity(string NodeId, string Incarnation,
string Role, long CatalogEpoch, long AuthorityEpoch);
```
Actual sample/lease contracts have explicit ownership/disposal and optional original timestamps; do not place SDK/actor/generated types into these ports. Define connection target mode, explicit realm, versioned mapping/item/stream identity, capability/limit negotiation, per-item statuses, cached age/readiness and snapshot control records. `IDataConnection` composes browse/metadata/read/write and async subscription lifetime without exposing channels/actor refs. Define monotonic-time abstraction through `TimeProvider` and bounded budget inputs.
**Red/green:** addresses distinguish same bare Raw/UNS paths and serving-node provenance; invalid/zero enum values reject; Unknown and Unconfirmed cannot become success Booleans; cancellation/lifetime and null timestamp semantics are explicit. Preserve existing `IWritable` drivers through an additive rich-outcome capability, not a breaking rewrite of every driver.
### Task 3: Protobuf/codec and negotiation
**Create:** `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/data_connection.proto`, `Protos/DataConnectionProtoContract.cs`, `DataConnections/DataProtoMapper.cs`, `DataConnections/TypedValueCodec.cs`; C `GrpcDataProtoContractTests.cs`, `GrpcDataCodecTests.cs`, `GrpcDataCompactCodecTests.cs`.
**Modify:** Commons `.csproj` to generate both endpoints for this proto only.
Define GetCapabilities/Browse/GetMetadata/Read/Write/Subscribe as in design, with per-branch browse completeness, epoch-bound continuations and control-only metadata watch. Add negotiated compact handles, explicit batch-default override/presence, immutable sample provenance, write operation/serving-incarnation preconditions and exact domain status codes. Bind operation-ID payload hashes to immutable semantic fields, excluding retry-varying budgets. Assign/freeze field numbers and versioned capability bits; existing mesh/telemetry semantics do not change.
**Red/green:** full status subcodes, signed/unsigned boundaries, UInt64, NaN/infinity, null/empty arrays/bytes, ranks/dimensions, optional timestamps and unsupported graphs; compact/fallback domain equivalence; absent timestamp override versus common default; unknown handle, removed item revision, major-version mismatch, malformed dimension/count and decoded/decompressed budget failure before large allocation. Golden payload and additive old/new peer tests. No compression by default until measured.
**Phase completion:** `run-unit-phase.py P01`; Astra reviews frozen contract/provenance/ownership and rich-result signatures. Commit `feat: define grpc data contracts and codecs`.
## P02 — Configuration, validation and per-node deployment
**Owner/model:** Terra/high for entities; Sol/high for artifact tests, serialized integration by coordinator. **Depends:** P01. **Unit gate:** F new P02 plus `DraftValidatorNodeAclScopeTests`; P new P02 plus `DataPlaneAuthorityArtifactTests`; R new P02 plus `DeploymentArtifactDataPlaneAuthorityTests`, `ParsedDeploymentArtifactTests` and `DeploymentArtifactEquipRefParityTests`; P also `ConfigComposerTests`.
### Task 4: Schema and deterministic validation
**Create:** `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/DataConnection.cs`, `DataExport.cs`, `RemoteNamespaceMapping.cs`; `Validation/DataConnectionValidator.cs`; F `GrpcDataConfigurationTests.cs`, `GrpcDataMappingValidationTests.cs`.
**Modify:** `OtOpcUaConfigDbContext.cs`, `Validation/DraftSnapshot.cs`, `DraftSnapshotFactory.cs`, `DraftValidator.cs`, `Enums/NodeAclScopeKind.cs`.
Add owner cluster, explicit consumer-node IDs, Node/Cluster target, expected identities, HTTPS endpoints, secret references, realm-qualified root/parent IDs, branch names, operation subset, scopes, bounded policies and disabled/write-through defaults. Append mapping ACL scope without renumbering existing enum values. Resolve local path from existing Raw/UNS entity identity. Validate subset consumer scope, self-route only for selected consumers, nested/self-mount loops, same-realm origin, duplicate sibling names, current parent ownership, unsupported type/operation and secret reference existence. A→B in the same cluster is allowed on consumer A only. A cluster endpoint set must have a nonself eligible target for every declared consumer; never silently alter node-pinned targets.
**Red/green:** reject invalid topology/collision/credential configuration while preserving existing local Raw/UNS validation and disabled feature behavior. Test mapping moves and inherited/granted scopes; no synthetic remote `TagId`.
### Task 5: Migration and artifact projection
**Create:** EF-generated migration named `AddGrpcDataConnections` and its Designer under Configuration `Migrations/`; exact timestamp is produced once by EF during this task, recorded in ledger. Add F `GrpcDataSchemaModelTests.cs`, P `GrpcDataArtifactProjectionTests.cs`, R `GrpcDataArtifactReadTests.cs`.
**Modify:** `Migrations/OtOpcUaConfigDbContextModelSnapshot.cs`, `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs` for ordered artifact serialization and `Coordinators/ConfigPublishCoordinator.cs` only for coordination changes; Runtime `Drivers/DeploymentArtifact.cs`.
Use `dotnet ef migrations add AddGrpcDataConnections --project src/Core/ZB.MOM.WW.OtOpcUa.Configuration --startup-project src/Core/ZB.MOM.WW.OtOpcUa.Configuration` using the existing migrator’s EF 10.0.7 setup. Set `OTOPCUA_CONFIG_CONNECTION` to a nonconnecting loopback design-only sentinel while scaffolding (the design-time factory requires the variable); `migrations add` must not migrate or contact a shared database. Real apply uses the isolated P10 SQL connection. Do not use the shared-host example embedded in the factory documentation. Freeze generation/serialization ordering and include connection/export/mapping/grant revisions in the artifact hash. Compile node-effective execution graphs: A-only mapping absent/inactive on B; credentials are references delivered through secret machinery, not plaintext blobs/logs. Old artifacts decode feature-disabled; reject partially configured enabled artifacts fail-closed. Projection with unsupported major capability never reports ready.
**Red/green:** stable hashes independent of collection enumeration, source/consumer isolation, authority-only deploy, disabled/maintenance changes, no plaintext secrets, old artifact compatibility and no B self-mount. Run focused migration model tests without SQL during the unit gate; actual apply/reapply/backward-disable compatibility uses disposable SQL during P10, not a live database migration now.
**Phase completion:** P02 gate and artifact diff review; commit `feat: deploy scoped grpc data connection configuration`.
## P03 — Shared policy and mapped authorization
**Owner/model:** Terra/xhigh; Astra/xhigh independent review. **Depends:** P02. **Unit gate:** K new P03 plus `TriePermissionEvaluatorTests`, `PermissionTrieCacheTests`, `DataPlaneAuthorityStateTests`, `PermissionTrieBrowseTransitTests`; O new P03 plus `DataPlaneAuthorizerTests`, `SessionAclDecisionCacheTests`, `NodeManagerAclEnforcementTests`.
### Task 6: Extract SDK-independent subject/policy service
**Create:** `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/DataOperationPolicy.cs`, `MappedNamespaceSubject.cs`, `MappedSubjectResolver.cs`, `DataAuthorizationCache.cs`; K `GrpcDataPolicyTests.cs`, `GrpcDataAuthorityCacheTests.cs`; O `GrpcDataMappedSubjectTests.cs`.
**Modify:** `DataPlaneAuthority.cs`, `AclScopePathResolver.cs`, `PermissionTrieBuilder.cs`; OPC UA `Security/DataPlaneAuthorizer.cs`, `Security/SessionAclDecisionCache.cs` and relevant subject representation.
Delegate exact existing local decisions to shared policy rather than independently deriving paths. Resolve mapped subjects ahead of BOTH Raw BuildScope/cache and UNS custom resolution, including mapping revision/local ancestry/origin proof. Bind decisions to actual view, operation, caller key/principal, serving identity and active authority/catalog; retain inherited/deny semantics. Unknown subjects and empty authority deny. Mapped access remains gated with legacy `Security:Acl:Enabled=false`; ordinary local-node behavior remains unchanged. Cache expiry/key disable and metadata/export lease boundaries; bounded negative cache.
**Red/green:** Raw/UNS identical path, multiple aliases, opaque folders, deleted/rebound mapping, local ancestor versus mapping grant, wrong principal/cluster, expiry and same-key generation change; cached Allow never survives changed provenance or authority. Regression local grant behavior must remain equivalent.
### Task 7: Catalog staging, remote authority leases and revocation gates
**Create:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/DataConnections/CatalogActivation.cs`, `MetadataAuthorityLease.cs`; R `GrpcDataCatalogActivationTests.cs`, `GrpcDataMetadataLeaseTests.cs`.
**Modify:** `Runtime/OpcUa/OpcUaPublishActor.cs` authority/catalog integration only (coordinator-owned file).
Build one desired catalog/subject projection reused by hub and SDK. Publish revocation even if topology apply fails or plan is empty; fail closed on catalog/authority mismatch without rolling back authority. Lease expiry uses request-start monotonic time plus bounded duration/margin; delayed replies cannot extend authorization. Watch/probes include quiet browse-only mounts and authority-only epochs. Last-known local anchor may remain locally visible; derived remote names/continuations/reads expire. Validate final send gate versus already-emitted bytes.
**Red/green:** no-op ACL deploy, partial/failed topology apply, lease expiry under partition, delayed renewal, clock jump, quiet key expiry and cached payload invalidation. P03 manifest includes R new tests plus `OpcUaPublishActorApplyFailureTests` and `OpcUaPublishActorApplyOutcomeTests` for this task only.
**Phase completion:** P03 gate; Astra review grants, generation publication and lease reasoning; commit `feat: share grpc mapped authorization and authority leases`.
## P04 — Immutable source feed and partitioned hub
**Owner/model:** Terra/xhigh; no concurrent driver/publish actor edits. **Depends:** P01/P03 for phase completion; source/hub scaffolding may be prepared after P01 while policy work finishes. **Unit gate:** R new P04 plus `DriverInstanceActorBatchForwardTests`, `DriverHostActorBatchFanOutTests`, `DriverHostActorSourceRemovedTests`, `DriverHostActorCommsLossTests`, `OpcUaPublishActorBatchTests`, `OpcUaPublishActorStatusBatchTests`.
### Task 8: Own payloads before asynchronous handoff and preserve timestamps
**Create:** `Runtime/DataConnections/OwnedDataSample.cs`, `SourceSamplePublisher.cs`; R `GrpcDataSampleOwnershipTests.cs`, `GrpcDataSourceFeedParityTests.cs`.
**Modify:** `Drivers/DriverInstanceActor.cs`, `Drivers/DriverHostActor.cs`, `OpcUa/OpcUaPublishActor.cs`, `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs` and affected sample envelopes if required.
Freeze/copy supported mutable payloads at callback/adapter boundary BEFORE actor enqueue, or take explicit immutable ownership. Preserve independently optional original source/server timestamps rather than manufacturing two from the current collapsed `TimestampUtc`. Emit source-level normalized batches before optional OPC UA sink early return and before extra alias fan-out. Include virtual values, status-only/comms-loss/removal/rebind and deferred apply; never cache optimistic operator writes as authoritative samples. Retain existing downstream contracts through adapters and batch delivery, preserving full uint status.
**Red/green:** producer mutates arrays/bytes immediately after callback, pool reuse while writer stalls, disposal/cancel once, original timestamp absence/fidelity; virtual/status/removal parity and retired-incarnation rejection with no SDK sink. Record remaining legacy actor/sink serialization; no claim that new partitions parallelize existing drivers.
### Task 9: Bounded partitions, snapshot registration and readiness
**Create:** `Runtime/DataConnections/PartitionedSampleHub.cs`, `SamplePartition.cs`, `SnapshotLease.cs`, `DataResourceBudget.cs`; R `GrpcDataPartitionTests.cs`, `GrpcDataSnapshotTests.cs`, `GrpcDataIngressOverflowTests.cs`.
Normalize source ownership and immutable view descriptors. Bounded nonblocking ingress precedes actor mailboxes; source latest/dirty generation advances without touching every registration. Coalesce in bounded dirty tables; overflow invalidates affected source/partition readiness and signals reset. Ordered reserved controls prevent value resurrection after removal. Per-partition interest registration captures immutable snapshot/watermark; multi-partition vector admits no cross-item atomicity promise. Count shared storage, baseline refs, pending frames and all partition snapshots in global budgets. Repartition resets epochs before ownership change.
**Red/green:** concurrent snapshot/delta/update, slow partition, overflow with stale previously Good cache then reconnect/read, concurrent removal, source changes during dirty traversal, memory rejection before allocation, cancellation/ref release and fake-time expiry. Unit assertions test boundedness and invariants, not machine-specific throughput.
### Task 10: Implement local data ports and direct API without a mount
**Create:** `Runtime/DataConnections/LocalDataConnection.cs`, `LocalDataServices.cs`, `LocalExportCatalog.cs`, `DataCapabilitiesProvider.cs`; R `GrpcDataLocalConnectionTests.cs`, `GrpcDataCachedReadTests.cs`, `GrpcDataLocalExportCatalogTests.cs` (P04).
Implement the local `IDataConnection` and `IDataReadService`/metadata/capability facades explicitly. Origin-local browse returns the paged authorized deployed catalog, never projected peer nodes; metadata carries exact type/rank/units/status/timestamps/serving provenance and effective grants/limits. Cached reads distinguish missing, denied, initial-data, over-age, authority mismatch and ingress-invalidated readiness; no driver refresh is implied. Resolve the same source/view descriptors used by the hub and shared policy. Local write/subscription ports delegate to P05/P06 implementations when wired, and advertise only ready implemented capabilities; never return a placeholder Good/success. During this phase test unavailable delegation with fakes. P07 ingress can use these ports before federated mapping arrives in P08; origin-local versus composed browse remains an explicit facade boundary.
**Red/green:** direct API browse/read/metadata on a local origin with no namespace mount; per-item order/partial failure; Raw/UNS aliases with different grants; initial/stale/overflow cache states and timestamps; origin-only exports; capability intersection/readiness; local adapter and gRPC service use the same policy inputs. Add loopback direct-API coverage through the real remote client in P10 rather than requiring a mapping solely to prove connectivity.
**Phase completion:** P04 gate; commit `feat: add immutable partitioned data publication hub`.
## P05 — Mutation admission, uncertainty and late outcomes
**Owner/model:** Terra/xhigh; Astra/xhigh review. **Depends:** P03 and P01 rich outcomes. **Unit gate:** R new P05 plus the exact existing classes in the representative manifest above; O new P05 plus `NodeManagerWriteDispatchTests`, `NodeManagerWriteRevertTests`, `NodeManagerBatchWriteTests` and `EquipmentWriteGateTests`.
### Task 11: Dispatch tickets and additive rich driver outcomes
**Create:** `Runtime/DataConnections/WriteAdmissionTicket.cs`, `DataWriteExecutor.cs`; R `GrpcDataWriteAdmissionTests.cs`, `GrpcDataDriverOutcomeTests.cs`.
**Modify:** `Drivers/DriverInstanceActor.cs`, `Drivers/DriverHostActor.cs`, `Drivers/ActorNodeWriteGateway.cs`, `Drivers/DriverWriteBounds.cs`, Commons `OpcUa/IOpcUaNodeWriteGateway.cs`; additive Core.Abstractions rich-outcome interface from P01.
Ticket carries principal/key, exact view/subject and resolved backing target, catalog/authority, serving/driver incarnation, operation ID and monotonic expiry. Admission rejects unsupported/nonwritable/type-mismatched targets after authorization. Immediate launch AND queue drain revalidate role, mapping, authority and remaining budget. Preserve bounded device execution and derived timeout ordering; no automatic retry/hedge. Adapt legacy confirmed versus fire-and-forget drivers conservatively; status Good alone is not necessarily device confirmation. Demotion/expiry before dispatch returns RejectedBeforeDispatch; post-dispatch cancellation/timeout is Unknown unless a definitive driver outcome exists.
**Red/green:** queued revocation/remap/demotion, changed driver incarnation, insufficient budget, token-ignoring driver and existing per-tag serialization; legacy driver result compatibility. Do not weaken write-primary checks to make intracluster tests pass.
### Task 12: Registry, late completion and OPC UA self-correction
**Create:** `Runtime/DataConnections/WriteOperationRegistry.cs`, `LateWriteObserver.cs`; R `GrpcDataWriteRegistryTests.cs`, `GrpcDataLateWriteTests.cs`; O `GrpcDataMappedWriteCompletionTests.cs`.
**Modify:** node manager completion/self-correction seam for rich results only.
Registry key is stable authenticated principal plus operation ID with immutable semantic payload hash and serving incarnation. In-flight duplicates share task; canceled waiter does not cancel dispatched work. Reject conflicting payload and capacity without evicting in-flight entries. Stale-incarnation retry rejects before dispatch; no ambiguous cross-incarnation replay. Retain bounded late result observation/audit after caller disconnect or driver abandonment; if retention ends preserve Unknown. Imported node hook remains nonblocking: definitive rejection restores authoritative sample; Unknown becomes uncertainty pending real data; AcceptedUnconfirmed is not final success. Old optimistic values must not enter hub snapshots.
**Red/green:** concurrent duplicate task, changed budget with same semantic hash, conflicting payload, partial batch success and endpoint switch of only proven-undispatched items, response loss after simulated device commit, registry capacity and late observer exhaustion. Test exact device-call count with spy backend; no effect can run twice through retry.
**Phase completion:** P05 gate; Astra reviews actual dispatch boundary and uncertainty paths; commit `feat: preserve grpc write admission and uncertain outcomes`.
## P06 — Shared demand, streams and scalable delivery
**Owner/model:** Terra/xhigh; Sol/high may own disjoint codec tests only. **Depends:** P03/P04. **Unit gate:** R new P06 only; C `GrpcDataCompactCodecTests` only if protocol revisions were necessary and recorded.
### Task 13: Authorized shared demand and filter composition
**Create:** `Runtime/DataConnections/UpstreamDemandRegistry.cs`, `SubscriptionSettingsAggregator.cs`, `ClientFilterState.cs`; R `GrpcDataDemandSharingTests.cs`, `GrpcDataFilterAggregationTests.cs`.
Use full source/serving-node/target-mode/consistency/upstream principal/export/epoch key; per-view permissions remain independent. Ref-count interests and pin active metadata. Fastest admitted interval plus union status/timestamp triggers defines upstream demand; shared upstream value deadband defaults off. Stronger add/modify awaits origin acknowledgment plus snapshot/version boundary; only safe relaxation can debounce. Unsupported filters return revised/rejected settings. Last consumer removes demand; a narrow consumer cannot inherit broad permissions.
**Red/green:** many Raw/UNS views one source, different principals/pinned nodes isolated, reference removal, burst settings churn, slow/no-deadband and fast/high-deadband consumers, independent baselines, upstream inability and pending stronger settings. Use fake-time tests, no sleeps to simulate workload.
### Task 14: Subscription state machine, ordered writer and fair scheduler
**Create:** `Runtime/DataConnections/DataSubscriptionSession.cs`, `DeliveryScheduler.cs`, `SubscriptionItemHandleTable.cs`, `SnapshotAdmissionScheduler.cs`; R `GrpcDataStreamOrderingTests.cs`, `GrpcDataDeliveryFairnessTests.cs`, `GrpcDataRecoveryAdmissionTests.cs`.
Implement open/add/modify/remove command acknowledgments, item revisions, epoch-scoped never-reused numeric handles, snapshot begin/chunks/end and control-only watches. Bounded inbound commands, source-dirty deduplication, fair resumable due-interest traversal and one serialized writer per stream. Encoding happens off owner locks; writer checks current authority/leases/revisions. Baseline changes only on successful bounded send; re-evaluate candidates prepared against old baseline. Reserve control/write resources without data starvation. Coalescing/reset visible; slow-client pressure cannot delay actors. Snapshot admission limits lifetime/bytes/concurrency/bandwidth globally/per principal; jittered retry guidance prevents reconnect/reset storms.
**Red/green:** removal ack barrier, remove/re-add with old frame, revocation while encoding, hot tag plus quiet tags and abusive command stream, multi-partition slow snapshot, canceled snapshots release memory, failed writer resets baseline/epoch, heartbeat expiry, overflow control survives full values queue, late old-epoch frame ignored, bounded timers/work indexes. No unit throughput threshold.
**Phase completion:** P06 gate and Astra stream/scaling review; commit `feat: add bounded shared grpc subscriptions and delivery scheduling`.
## P07 — TLS server and node/cluster clients
**Owner/model:** Terra/high; coordinator owns Host wiring. **Depends:** P04/P05/P06. **Unit gate:** H new P07 plus `DriverWriteBoundsBindingTests`, `ApiKeyStoreRegistrationTests` and `ApiKeyAuthenticationTests`; R new P07 plus `WarmRoleAuthorityTests` and `PrimaryGatePolicyTests` only if role adapter changes.
### Task 15: Dedicated TLS ingress and admission
**Create:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/DataConnectionSecretProvisioner.cs`, `DataConnectionCredentialResolver.cs`, `DataConnectionSecretChangeHandler.cs`, `DataConnectionTrustPolicy.cs`; H `GrpcDataSecretProvisioningTests.cs`, `GrpcDataSecretRotationTests.cs`, `GrpcDataCredentialRetirementTests.cs`, `GrpcDataTrustRotationTests.cs`; `src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/DataConnectionGrpcService.cs`, `Configuration/DataConnectionOptions.cs`, `DataConnectionOptionsValidator.cs`, `DataConnectionAuthInterceptor.cs`, `DataConnectionEndpoints.cs`; H `GrpcDataListenerTests.cs`, `GrpcDataIngressAuthTests.cs`, `GrpcDataServiceDispatchTests.cs`.
**Modify:** Host `Program.cs`, `Configuration/SecretsRegistration.cs`, `Configuration/SecretChangeSignallingInvalidator.cs`, `Configuration/MeshGrpcEndpoints.cs` only for explicit separation/admission census, Host `.csproj` if generated-type references require it.
Bind dedicated HTTP/2 HTTPS endpoint with certificate validation and explicit listener/host admission. Feature disabled maps no service. Dev cleartext is explicit loopback only; Docker gate uses TLS. Existing command/telemetry h2c listeners cannot reach data service. Use target-issued API key, current snapshot and mapped operation grants; cookie fallback must not redirect an authenticated gRPC call. Missing key/pepper/authority fails closed. Ingress invokes the P04 local capability/catalog/read services and P05/P06 execution ports, not actor resolution.
Provision outbound credentials independently on EACH declared consumer node (including both replicas when both may consume). The existing store is local SQLite by default; opt-in secret replication is pair-local and never a cross-cluster provisioning mechanism. Add an explicit local provisioning source: an operator/orchestrator mounts a restricted versioned credential bundle per consumer, naming connection/secret reference, expected owner cluster and consumer NodeId; a Host provisioner validates current node execution scope, reference allowlist and monotonic version, then writes through `ISecretStore`/cipher/cache-invalidator using the existing production secret write pattern. Do not place presented credentials in central ConfigDb deployment artifacts or use an unauthenticated secret-upload endpoint. Fixture seeder mints the target key and installs it into every intended consumer’s mounted bundle/local store; populating CONTROL alone is insufficient. Missing/undecryptable/wrong-scope secret leaves that connection AuthFailed/unready. Restart uses the persisted local store; bundle removal alone is not a revocation protocol.
Handle atomic local bundle replacement and existing store invalidation signals; retire cached outbound credentials/channels, cancel affected streams and establish fresh epochs after a version change. Compose invalidation with existing consumers rather than replacing their handler. Rotate in order: deploy/activate new target key and grants, provision each consumer, verify new-key sessions/readiness, then revoke the old target key and deploy authority. An in-flight mutation retains its original operation/outcome observer; rotation cannot replay or relabel it. Target-side key expiry/revocation remains enforced on quiet streams. On scope/connection removal, drain and deny credential use before retiring its local cached reference; delete only after no other allowed connection references that secret. Versioned explicit deletion/tombstones prevent old bundles from resurrecting retired material after restart. Removing a consumer from the artifact allowlist also prevents reimport from a stale mounted bundle. In-flight dispatched writes retain only their bounded outcome observer, never a live route that can launch a replacement.
TLS trust/certificate lifecycle is separately versioned: stage new trust on every eligible consumer, rotate the target server certificate with valid SAN identity, retire affected channels and renew streams/epochs, then remove the old CA/trust entry after verifying migration. Use a rolling server restart if the current Kestrel configuration cannot safely reload its certificate; do not invent unsupported hot-reload behavior. Trust policy fingerprint is part of channel identity. Wrong/removed CA/SAN cannot trigger permissive fallback, and trust removal invalidates old connections. Include unit policy tests plus real old/new-CA and server-cert rotation in D14.
Tests cover offline DB-less restart, consumer missing secret, wrong node/bundle, encrypted-store failure, independent-pair isolation, version downgrade rejection, both consumer replicas provisioned and rotation while reads/streams/write outcome tracking are active. Test invalidation does not break existing Galaxy secret-change behavior. Add a P10 real secret-store/provision/rotate wire case. Unary deadlines/stream command timeouts use distinct policies. Disable configured automatic write retries/hedges.
**Red/green:** wrong listener/cookie policy, missing/revoked/expired key, method-prefix bypass, oversized message, partial authorized batches, caller claims wrong source identity, disabled surface, insecure nonloopback rejection, timer expiry on quiet stream. Create `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DataConnections/GrpcDataTlsWireTests.cs` and `GrpcDataListenerIsolationWireTests.cs` for real TLS/cert/listener tests, tagged Feature=GrpcData, Category=Integration. Run those in P10, alongside existing `MeshGrpcAnonymousMetadataTests`, `MeshGrpcAuthStackTests` and `MeshGrpcBoundaryTests` in that integration project. They are not misclassified as H unit tests.
### Task 16: Remote connection manager and failover
**Create:** `Runtime/DataConnections/GrpcDataConnection.cs`, `DataConnectionManager.cs`, `ClusterEndpointSelector.cs`, `DataConnectionRoleAdapter.cs`; R `GrpcDataConnectionClientTests.cs`, `GrpcDataEndpointSelectionTests.cs`, `GrpcDataConsumerScopeTests.cs`.
**Modify:** Runtime `ServiceCollectionExtensions.cs` and Host registration.
Reuse bounded cached channels per endpoint/trust identity; authenticate service principal and verify target node/cluster/major capability. Node target stays pinned; secondary reads require explicit consistency/readiness and never borrow local cache. Cluster target selects authenticated ready primary; queries/read retries keep original budget and renew identity after endpoint changes. Mutation failover only per explicit RejectedBeforeDispatch item. Source/driver role follows existing Warm/Hot verdict; no new election or stronger fencing claim. Consumer scope removes/drains mappings, connections and handles; same-cluster access between processes still uses gRPC, not actor/DPS selection.
**Red/green:** stale primary hint, no-ready-primary backoff, pinned secondary write rejection, no partner substitution, mixed batch lost reply, demotion/new epoch, channel retirement, TLS identity mismatch surfaced through client seam, consumer A-only scope and unknown-role startup. Reconnect no pending write replay.
**Phase completion:** P07 gate includes new provisioning/rotation tests and `SecretsPreHostContentRootSourceTests`; commit `feat: host tls data service and route node and cluster connections`.
## P08 — Raw/UNS browse, metadata projection and runtime adapter
**Owner/model:** Terra/high, Astra/xhigh review. **Depends:** P07/P02. **Unit gate:** R new P08; O new P08 plus `NodeManagerAclEnforcementTests`, `NodeManagerModelChangeFilterTests`, `NodeManagerModelChangeOnAddTests`, `NodeManagerSurgicalRemoveTests`, `AddressSpaceApplierRawUnsTests`, `AddressSpaceApplierHierarchyTests` and `UnsReferenceBackingParityTests`; K mapped-subject tests if placement logic changes.
### Task 17: Federated catalog resolver and wrapped continuations
**Create:** `Runtime/DataConnections/RemoteNamespaceResolver.cs`, `FederatedBrowseService.cs`, `ProjectedCatalogCache.cs`, `BrowseContinuationRegistry.cs`; R `GrpcDataFederatedBrowseTests.cs`, `GrpcDataContinuationTests.cs`.
Compose origin-local exports with configured realm-specific mapped anchors at local client-facing view. Origin peer ingress never re-exports projections. Authorize every returned name and operation, bind handles to mapping/origin/epochs, preserve deterministic ordering, public metadata and same-path realm separation. Bounded one-parent pages with explicit completeness/failure, shared authenticated browse work and bounded waiter cancellation. Metadata pinning versus idle eviction; active lease expiry hides remote-derived names even without subscriptions. Preserve origin-local address and any explicit source-equivalence proof.
**Red/green:** Raw Folder/Driver/Device/TagGroup/Tag and UNS Area/Line/Equipment/signal; nested folders, renamed roots, collisions, pagination while revision changes, peer unavailable versus empty, revoked queued pages, wrong-principal token reuse, lease loss and unknown handles. Returning a successful empty list for pending/unreachable is a failing test.
### Task 18: SDK per-parent materialization and gRPC import adapter
**Create:** `Runtime/DataConnections/GrpcDataDriverAdapter.cs`, `ProjectedNamespaceApplier.cs`; O `GrpcDataSdkBrowseReadinessTests.cs`, `GrpcDataSdkMappedAccessTests.cs`; R `GrpcDataAdapterLifecycleTests.cs`.
**Modify:** `OpcUaServer/AddressSpaceComposer.cs`, `AddressSpaceApplier.cs`, `OtOpcUaNodeManager.cs`, `Security/DataPlaneAuthorizer.cs`, `Core/Hosting/DriverFactoryRegistry.cs`/Host registration only where the existing factory extension pattern requires it; no concrete driver dependency added to Runtime.
Reuse shared desired subjects/catalog and existing sink interfaces. Background per-parent first-page population, explicit BadWaitingForInitialData while loading; BrowseNext preserves pending page identity or uses an explicitly tested continuation strategy. BadNoCommunication for unreachable/expired branch, BadContinuationPointInvalid on identity/revision loss. No network I/O under SDK Lock. Authorize mapped Browse/Translate/Read/Write/monitored-item creation/delivery even with legacy ACL switch off. Apply filtered model changes. Adapter supplies values through ordinary dependency/publication paths with rich Unknown/Unconfirmed outcomes and distinct connection quality. Same-cluster A-only mount creates no B reciprocal import or duplicate field driver. Deactivation removes nodes and demand.
**Red/green:** pending page never appears empty, continuation retry without cursor consumption, root/child paging, map removal with active monitored items, local+remote authority composition, same-path Raw/UNS reads, mapped unknown outcome self-correction, Warm/Hot consumer role transitions, dependency fan-out and no optimistic-cache pollution.
**Phase completion:** P08 gate; Astra checks SDK lock boundaries and actual continuation behavior before Docker gates. Commit `feat: expose mapped raw and uns data through grpc adapters`.
## P09 — Authoring, diagnostics and operational controls
**Owner/model:** Terra/high for UI/service work; Sol/medium docs; apply applicable UI skill during execution when touching interface design. **Depends:** P08. **Unit gate:** U new P09 and `AclScopeServiceTests`, `RawBrowseCommitMapperTests` and `RawBrowseCommitMapperParityTests`; no unrelated UI/Host project tests.
### Task 19: Author connection/export/mapping configuration and inspect health
**Create:** `AdminUI/DataConnections/DataConnectionAdminService.cs`, `DataConnectionEditorModel.cs`, `RemoteMappingEditorModel.cs`, `DataConnectionDiagnostics.cs`; `AdminUI/Components/Pages/Clusters/ClusterDataConnections.razor`; U `GrpcDataAuthoringTests.cs`, `GrpcDataEditorValidationTests.cs`, `GrpcDataDiagnosticsTests.cs`.
**Modify:** Raw/UNS authoring integration and cluster navigation following current component/service patterns; Configuration snapshot authoring as needed, explicitly coordinator-owned.
Provide disabled-by-default create/edit directional connection, consumer set, realm parent/root picker, export/mapping grants, secret reference and optional write-through settings. Existing schema validator remains authoritative server-side. No raw secrets in rendered summaries/logs. Display disabled/connecting/ready/degraded/auth-failed/backoff, selected endpoint/role/provenance, metadata lease/sample age, source/registration/stream counts, reset/overflow and write outcome metrics with bounded labels. Test draft versus deployed state so saving does not falsely indicate active imports. Local readonly/audit permissions cannot author or enable writes. Add rollout/disable controls through existing deployment flow; do not add a general unauthenticated config-upload endpoint.
**Phase completion:** P09 gate; update `docs/Configuration.md`, `docs/Raw.md`, `docs/Uns.md`, `docs/security.md`, `docs/ReadWriteOperations.md` (correct stale ACL claims), `docs/Redundancy.md`, `docs/README.md`; preserve future-only history/alarms/relays boundary. Commit `feat: author and inspect grpc data connections`.
## P10 — Hermetic local Docker and OPC UA CLI integration
**Owner/model:** Sol/high fixture worker; coordinator executes gates; Terra/high fixes production defects within their phase ownership. **Preparation:** scaffold harness after P01, fill in seeder/probes only after dependent contracts exist. **Units:** harness/process/parser tests only plus applicable production classes for any fix. **Integration:** named local wire/CLI scenarios, not a full solution run.
### Task 20: Standalone local rig and production-path fixture seeder
**Create:** `scripts/validation/grpc-data/compose.yml`, `Dockerfile`, `rig.py`, `seed/GrpcDataSeed.csproj`, `seed/Program.cs`, `device/compose-service.yml` or equivalent build context, `device/server.py`, `tests/test_rig.py`, `README.md`.
**Reuse/read:** `docker-dev/Dockerfile`, `docker-dev/docker-compose.yml`, `docker-dev/seed/entrypoint.sh`, `scripts/validation/pair-gate/Program.cs`, `Host/Configuration` bootstrap guards, `AdminUI/Api/DeployApiEndpoints.cs`.
Use a STANDALONE compose file with a unique project/run label, image, network and volumes, no fixed `container_name`. Do not inherit base rig endpoints: it has LDAP/SQL/Modbus/opc-plc references to `10.100.0.35`. Require local Docker socket/npipe/context and reject SSH/TCP remote daemon/context; validate resolved compose/seed has no shared-host endpoints. Bind host test ports to loopback with collision-safe allocation recorded in a manifest. Use current server image build graph/SourceRevisionId, not a stale historical image. Include local SQL and software source fixtures. Admin login is disabled for the local machine API gate; LDAP options must still satisfy startup validation using isolated unused local settings or a local fixture if code actually needs directory access.
Topology: two independent data pairs `SOURCE` (source-1/2) and `CONSUMER` (consumer-1/2), plus a separate `CONTROL` pair where existing deployment requires admin/controller independence. Six processes preserve controller-outage testing without stopping either data pair. Register self-first seeds/cluster roles exactly per pair, ConfigServe/command/telemetry gRPC and DB-less fetch/cache on data nodes, separate LocalDb volumes/sync for each pair as applicable. Data gRPC has its own generated private CA and per-node SAN certificates, mounted only into this rig; clients must trust this CA and verify service identity. Keep cert/key files restricted and out of Git.
Provision target-issued outbound credentials into each eligible consumer’s local store/bundle through the P07 provisioner before declaring ready; verify the unprovisioned-node negative case and key rotation across the independent pairs.
Use a deterministic local writable Modbus software source (adapt the repository fixture engine and version), with changing counter, scalar types, array/bytes samples where supported, readback and a test-only request/effect ledger exposed inside the isolated rig. Its control API is fixture-only, never production Host. Every source driver points at local service DNS, not another server’s OPC UA endpoint; otherwise a fallback bridge could hide a missing gRPC path. Seed existing Raw and UNS local references plus both remote realm mappings, connection consumer scopes and keys using EF/production authoring validators and `ApiKeyAdminService` patterns from PairGate. Save presented data keys only to `0600` credential files; secrets do not enter artifact manifests or evidence logs. A normal deployment must compose/seal/apply; direct SQL rows alone are not an active runtime.
Manifest contains dynamically allocated host/container endpoints, expected cluster/node/application identity, TLS public trust paths, namespace URIs and resolved NodeIds, Raw/UNS aliases and origin paths, credential-file paths, fixture control endpoints and exact owned container labels. Do not assume ns=2/3 across servers. Seeder/Client.Shared helper reads NamespaceArray and validates each resolved URI.
**Commands to implement and then use:**
```bash
python3 scripts/validation/grpc-data/rig.py preflight
python3 scripts/validation/grpc-data/rig.py up --run-id grpc-data-local
python3 scripts/validation/grpc-data/rig.py seed --run-id grpc-data-local
python3 scripts/validation/grpc-data/rig.py deploy --run-id grpc-data-local
```
`up` builds once, migrates disposable SQL, starts pairs and waits on explicit health/role/catalog readiness with bounded deadlines. `deploy` POSTs only `{ "createdBy": "grpc-data-local-gate" }` to `/api/deployments` with `X-Api-Key`, follows returned Location and waits for every intended node's apply outcome/revision; reject degraded/unexpected states. Handle no-change/in-flight responses explicitly. The endpoint DOES NOT accept artifact.json/configuration uploads. If desired, invoke existing `docker-dev/deploy.sh` with overrides only for its bodyless POST semantics; it does not replace per-node apply polling.
**Units:** subprocess argv/timeout enforcement, remote-context/shared-host rejection, schema/port manifest validation, ownership-only cleanup, redaction, readiness/deploy polling, atomic secret files and interrupted setup cleanup. Actual migrate/reapply model snapshot test runs here against disposable SQL. Remove feature config/disable on rollback, not destructive down migration of active production data.
### Task 21: CLI process harness and direct gRPC probe
**Create:** `scripts/validation/grpc-data/cli_gate.py`, `processes.py`, `tests/test_cli_gate.py`, `probe/GrpcDataProbe.csproj`, `probe/Program.cs`, `scenarios.json`; add narrowly scoped tests to L only if CLI behavior itself changes.
Publish CLI once and execute the built DLL with `dotnet`, portable on macOS/Linux/Windows; no `.exe`-only locator and no `dotnet run` rebuild inside each background process. Process harness passes argv, enforces real timeout, captures stdout/stderr/exit, bounds retained output, cleans up subscribers/containers in `finally` and redacts credentials from command/evidence logs. Existing `--summary-file` is text, not JSON. Read/write bad statuses and individual subscription failures can still return exit 0: assert status/content and notification evidence, not merely successful process exit. Missing summary/output fails gate.
Exact existing CLI command shapes (the harness substitutes from the manifest and never logs the key):
```bash
dotnet publish src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ZB.MOM.WW.OtOpcUa.Client.CLI.csproj \
-c Release -o artifacts/grpc-data/tools/cli
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll connect -u "$gate_url" -U apikey -P "$gate_key"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll browse -u "$gate_url" -U apikey -P "$gate_key" \
-n "$gate_root" -r -d 6
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll read -u "$gate_url" -U apikey -P "$gate_key" -n "$gate_leaf"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll write -u "$gate_url" -U apikey -P "$gate_key" -n "$gate_leaf" -v 42
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll subscribe -u "$gate_url" -U apikey -P "$gate_key" \
-n "$gate_leaf" -i 200 --duration 30 --summary-file "$gate_summary"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll redundancy -u "$gate_url" -U apikey -P "$gate_key"
```
The current csproj has no AssemblyName override, so the DLL above is its project assembly name; verify/record the published output in the manifest. The operator-facing command name `otopcua-cli` in docs is not sufficient evidence to guess an assembly filename. All shown `$gate_*` variables are task-specific, sourced by the harness from the restricted manifest/credential file, not globals or mandatory user-provided values. Integration code calls the process wrapper directly rather than shell-interpolating these examples. Use local OPC UA security mode supported by the seeded test endpoint; gRPC TLS trust must be verified independently because the CLI's automatic UA certificate acceptance is not proof of gRPC trust.
Probe uses generated client and fixture fault controls to inspect semantics CLI cannot expose: capability/compact negotiation, actual serving node/role/epoch, source/delivery metrics, per-item write outcome and device-call count, stream acks/handle revisions, continuation reuse and byte limits. CLI remains the required end-to-end consumer evidence. Do not add nonexistent JSON/BrowseNext CLI flags; test unsupported observability through probe/Client.Shared SDK fixture.
**Units:** parser fixtures include exit-0 Bad read, write failure, zero attached subscriptions/no notifications, absent summary, late process kill and redaction; probe pure scenario-result validation/CLI fake process tests run in P10. TLS and processes are integration. Create `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/DataConnections/GrpcDataSdkWireTests.cs` for wire-level mapped Browse/BrowseNext/ACL and readonly namespace behavior, tagged Feature=GrpcData and Category=Integration; use these cases where the CLI cannot inspect continuation controls directly. P10 runs the new Host/OPC UA integration classes and the named existing mesh regression classes only, with nonzero per-class results.
### Task 22: Required scenario matrix and reproducible evidence
Implement `python3 scripts/validation/grpc-data/cli_gate.py --run-id grpc-data-local --scenario all` and `dotnet run --project scripts/validation/grpc-data/probe/GrpcDataProbe.csproj -c Release -- --manifest <restricted-manifest> --scenario all` (publish probe once inside runner). Each scenario has a nonzero assertion count, prerequisites, hard timeout, restoration and result row. `all` excludes no required feature scenario silently. Software fixture gaps are implementation failures, not successful skips.
| ID | Setup/action | Required assertions |
|---|---|---|
| D01 | Deploy/read identities for CONTROL/SOURCE/CONSUMER | Each pair sees only its two Akka members; source/consumer processes differ; active roles/capabilities ready; all apply revisions match |
| D01a | Direct gRPC client uses a permitted connection with no namespace mount | Capabilities/browse/metadata/read/write/subscribe work against origin-local facade; no mapped anchor is required; policy/role/outcome behavior matches mounted access |
| D02 | Same-cluster source-1→source-2 Node mapping with consumer scope source-1 only | Actual TLS gRPC calls for browse/read/subscribe; source-2 no reciprocal mount; serving source-2 provenance; no local-cache substitution; primary-only write tested when source-2 is primary |
| D03 | Cross-cluster Raw and UNS mounts | CLI recursively expands each hierarchy; read status/value/timestamps match origin; duplicate bare paths remain distinct; new child, rename/removal and public metadata behavior |
| D04 | Subscribe before controlled source changes; write through both realm views | Nonzero changing Good notifications per expected node; CLI write acceptance plus eventual origin/simulator readback; remote effect ledger exactly one expected effect and correlated trace; no duplicate field acquisition |
| D05 | Cold parent/page limit and mapping revision while browsing | Pending versus empty/unavailable distinguishable; bounded paging; BrowseNext cursor/revision invalidation via SDK/probe; no network call under SDK lock instrumentation |
| D06 | Local denial, narrower target export, revoked/expired key with live stream | Denied names absent; explicit read/write denial; no post-emission-gate unauthorized payload; quiet browse lease expires within negotiated bound; anchors only as locally authorized |
| D07 | Fault only remote gRPC path / stop selected SOURCE primary | Hold CONSUMER OPC UA CLI URL fixed, omit `-F`; observe connection quality gap, new selected SOURCE primary/epoch and fresh snapshot before recovery; no ambiguous write replay |
| D08 | Drop response after effect; mixed-success batch; same-ID duplicate/restart | Probe receives Unknown or definitive observed result as appropriate; no double effect; only proven-undispatched items eligible for endpoint switch; stale-incarnation request rejects |
| D09 | Pinned secondary / Warm standby and demotion | Own-node cache only when explicit policy/readiness allows; pinned secondary write rejected; no automatic partner substitution; Warm activation/catch-up honest; new epoch on role/incarnation change |
| D10 | Separate OPC UA client failover | Now use `-F "$partnerUrl"`; distinguish UA reconnection from server-side gRPC failover using serving/connection evidence; consumers configured on both nodes where this test requires it |
| D11 | Stop CONTROL pair only | SOURCE/CONSUMER continue local and cross-cluster data from applied artifacts; controller restart reconciles without replay; no SQL dependency on driver-only nodes |
| D12 | Remove consumer scope, export or mapping; disable feature and redeploy | Projected nodes/demand/handles removed, pinned scope does not migrate; writes rejected; ordinary local Raw/UNS and existing command/telemetry still function |
| D13 | Wrong CA/SAN, bad API key, wrong listener and raw peer re-export | TLS/identity/auth rejects; no data service via h2c/HTTP UI listener; mounted peer data cannot be recursively exported; shared-context access never crosses principals |
| D14 | Consumer secret absent/provisioned/rotated, trust rotation and retirement, offline restart | Missing secret unready; each eligible replica gets scoped material; new-key sessions ready before old-key revocation; new/old CA/server-cert trust behavior proven; retired scope cannot reload stale bundle; restart DB-less consumer with CONTROL pair stopped and verify persisted store; in-flight write tracking persists without replay |
The scenario file must expand coverage over this explicit matrix; a passing cross-cluster Cluster case does not substitute for Node-target or intracluster behavior:
| Target mode / topology | Raw + UNS browse/metadata | Read + subscribe | Write | Additional boundary |
|---|---|---|---|---|
| Node target, same-cluster A→B | D02 in both realms | D02 with pinned ready/secondary policy | D02 direct-runtime positive when B is primary; negative when B secondary | A-only scope; A’s local sample cannot substitute B |
| Node target, cross-cluster CONSUMER→one SOURCE node | D03/D05 configured as Node | D03/D04, no partner substitution under D09 | D04 positive to primary and rejection to secondary | D07 fault leaves pinned target unavailable rather than selecting partner |
| Cluster target, cross-cluster CONSUMER→SOURCE pair | D03/D05 configured as Cluster | D04/D07 primary selection and fresh epoch | D04/D08 per-item outcome rules | Disable/maintenance selected endpoint and prove exclusion; no unintended source selected |
| Direct API, no mounted view | D01a for Node and Cluster target (same-cluster direct Node included) | D01a | D01a with primary enforcement | No namespace projection is a prerequisite |
| Consumer failover | D10 mapping on both eligible consumers, then active consumer changes | Fresh demand/epoch on newly eligible node | Authority/role checks maintained | Repeat with A-only scope: mount must remain absent on nonconsumer, never auto-migrate |
Use a rig-only node probe/hosted command runner to execute `IDataConnection` through the ACTUAL caller node’s DI/connection manager for direct API cases; an external client connecting straight to B is not evidence of A→B execution. Create `scripts/validation/grpc-data/node-probe/GrpcDataNodeProbe.csproj` and register its bounded file-command/result runner only in the isolated fixture composition. No production network/control endpoint is added; normal Host startup tests assert the fixture runner is absent outside the test composition. Commands and output are restricted to the run-owned mounted directory and use redacted connection IDs, not bearer secrets. The CLI still proves mounted browse/read/write/subscribe in D03/D04 and other user-facing cases.
Within one redundant pair, A cannot be local OPC UA Primary while B is the target Primary simultaneously. Preserve the local SDK primary gate: positive same-cluster A→B write is a direct runtime-API test from A to active B; a CLI write to A’s standby mapped node correctly rejects locally. Test both behaviors rather than weakening either role gate to manufacture a positive mapped write.
For D02 role setup, wait for stable selected primary after planned local failover or choose the manifest's roles rather than assuming a host suffix is primary. For D09 use a scoped Warm compose/config variant; restore before subsequent cases. Network faults target only recorded run-owned containers/sidecars and specific gRPC ports; do not disconnect the developer's host or shared fixture network. Device-command ambiguity needs a fixture-controlled response-loss point, not hoping a random kill happens after commit. Restore faults deterministically in `finally`.
Write evidence under `artifacts/grpc-data/<run>/`: redacted resolved compose/manifest, build SHA/image revision, topology/role observations, deploy/apply statuses, CLI stdout/stderr/summaries and process exit/timeouts, gRPC probe assertions/metrics, simulator effect ledger, fault timeline, logs and scenario results JSON. Do not store credential-bearing manifest, private keys or raw values beyond safe fixture values in committed docs. Generate a concise tracked evidence index with hashes/paths, not a copy of large logs. Cleanup command: `python3 scripts/validation/grpc-data/rig.py down --run-id grpc-data-local` removes ONLY that manifest's project/resources; retain logs before cleanup.
**Phase completion:** run `python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_rig.py'` and the same command with `-p 'test_cli_gate.py'`; then the explicit P10 Host/OPC UA integration class selectors plus D01–D14 (including D01a)/direct probe gates. Astra reviews topology/auth/outcome evidence. Fix defects through the owning phase's applicable class filters, then rerun affected integration scenarios only. Commit `test: validate grpc data connections with local docker and opcua cli`.
## P11 — Scaling and controlled recovery evidence
**Owner/model:** Sol/high harness owner; Terra/xhigh for confirmed concurrency defects. **Depends:** P10. **Unit gate:** only new P11 workload/report tests plus affected P04/P06 classes if code changes. No full suite yet.
### Task 23: Parameterized workload and fairness experiments
**Create:** `scripts/validation/grpc-data/workload-profiles.json`, `workload.py`, `tests/test_workload_results.py`; extend existing probe with measured workload commands and source metrics. New R `GrpcDataWorkloadAccountingTests.cs` only if production accounting is added.
Independently vary unique sources, shared Raw/UNS views, client count, registrations/overlap, scalar/string/array payloads, source rates, intervals/deadbands, worker count, compact/fallback encoding and slow consumers. Include burst/concurrent browse, quiet/hot source fairness, same-origin different pinned node/principal isolation, stalled writer/slow snapshot, producer array reuse, ingress saturation, revocation during encoding and reconnect storm after active-node loss. Run source/readiness invalidation and catch-up memory scenarios with deterministic assertions before performance sampling.
Command: `python3 scripts/validation/grpc-data/workload.py --run-id grpc-data-local --profiles scripts/validation/grpc-data/workload-profiles.json`. Profiles contain small smoke, steady, fan-out and recovery settings, not mandatory advertised capacity numbers. Record actual hardware/container limits/architecture, TLS/compression, source versus delivered rate, bytes/sample, CPU per actor/partition, retained/pool/cache bytes, GC, scheduling p95/p99 and snapshot/recovery time. Compare configurations on identical input. Success means correctness, bounded resource/fairness invariants and useful measurements; no fabricated universal throughput threshold. Detect leaks only against agreed steady-state accounting/tolerance for the specific run, not fragile instantaneous GC values. Record current DriverHostActor/SDK serialization ceilings explicitly.
**Phase completion:** `python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_workload_results.py'`, P11 applicable dotnet classes (only if added/changed), and workload evidence; tune quotas only from the measured envelope, preserving single-member failover headroom and user-configurability. Astra reviews any change to snapshot/sharing/ordering contracts before release. Commit `perf: measure grpc fanout and recovery bounds` only if code/harness changes justify it.
## P12 — Final whole-solution verification and integrated Astra review
**Owner/model:** coordinator Sol/high; independent Astra/xhigh final review. **Depends:** all prior phases clean. **This is the only whole-solution test phase.**
### Task 24: Provide local prerequisites and run the full tests
**Modify:** `scripts/ci/integration-fixtures.sh` to support explicit local-owned mode and restricted JSON environment output without printing credentials; preserve the current CI invocation outside this opt-in mode. Local mode requires a nonempty unique `CI_FIXTURE_SUFFIX`, run-owner labels and manifest, local socket/context validation, `127.0.0.1` collision-safe published ports, per-run NAT certificate/output paths and label-verified cleanup. Reject startup when a matching name already exists without the same owner; never call its current name-only `down`/`rm -f` on unverified resources. Retries may remove only the container they just created. Validate resolved target host is loopback in local mode. Add fake Docker subprocess tests for pre-existing foreign names, parallel runs, failed setup/port retries and selective teardown before any real startup. **Create:** `scripts/validation/grpc-data/run-final.py`, `tests/test_final_runner.py`, `tests/test_fixture_ownership.py`, `fixture-environment-map.json`. P00 unit-only discovery mode is explicitly removed from final children and real local fixture variables are required before test discovery.
Inventory EVERY test project and its category/env/fixture gates before the final run. Reuse throwaway local SQL/PostgreSQL/QuestDB/opc-plc/NAT fixture patterns in `scripts/ci/integration-fixtures.sh`, plus required new local rig, optional local Host real-mode deps and any additional runnable software fixture. Pass a mandatory per-run suffix/owner, local mode, loopback host override and isolated certificate directory when invoking this helper; it must share the same preflight/ownership rules as the new rig. Its current empty local suffix and wildcard port bindings are not safe defaults for this gate. This helper emits exports and does not modify its parent environment: `run-final.py` must parse the restricted structured output into an allowlisted child environment, not run `bash ... up` and assume dotnet inherited it. Do not `eval` printed logs.
Explicitly set `OTOPCUA_CONFIG_TEST_SERVER` and `OTOPCUA_CONFIG_TEST_SA_PASSWORD` from disposable SQL as well as script-provided `SQL_TEST_ENDPOINT`, `SQL_TEST_USER/PASSWORD`, `SQL_PG_ENDPOINT/PASSWORD`, `HISTORIAN_SQL_TEST_CONNECTION`, `HISTORIAN_QUESTDB_TEST_CONNECTION`, `OPCUA_SIM_ENDPOINT`, `OPCUA_NAT_ENDPOINT` and `OPCUA_NAT_SERVER_CERT`. Configuration `SchemaComplianceFixture` always migrates real SQL; AdminUI SQL fixture can skip on unreachable/migration errors. Neither may fall back to shared `10.100.0.35`. Bind remaining registered software fixtures locally and isolate their database/table prefixes. No production/QA/shared hardware access is part of this plan.
First run only `python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_final_runner.py'` and the same discovery command with `-p 'test_fixture_ownership.py'` to verify final orchestration/redaction/environment/owned-fixture guards.
Record the remaining known fixture switches in `fixture-environment-map.json`; each maps to a run-owned local service or an explicitly unavailable platform/device, never an inherited shared endpoint:
| Suite family | Variables / prerequisite |
|---|---|
| Modbus / S7 / AB / FOCAS / MTConnect | `MODBUS_SIM_ENDPOINT`, `MODBUS_RTU_SIM_ENDPOINT`, `S7_SIM_ENDPOINT`, `AB_SERVER_ENDPOINT`, `AB_LEGACY_ENDPOINT`, `AB_LEGACY_CIP_PATH`, `OTOPCUA_FOCAS_SIM_ENDPOINT`, `OTOPCUA_FOCAS_SIM_PROFILE`, `MTCONNECT_AGENT_ENDPOINT`; enable AB writes only with `AB_SERVER_ALLOW_WRITES=1` against disposable simulator, never a configured differential hardware endpoint |
| MQTT | `MQTT_FIXTURE_ENDPOINT`, `MQTT_FIXTURE_PLAIN_ENDPOINT`, `MQTT_FIXTURE_USERNAME`, `MQTT_FIXTURE_PASSWORD`, `MQTT_FIXTURE_CA_CERT`, `MQTT_FIXTURE_TOPIC_PREFIX`; local TLS broker/publisher required |
| Host real-mode | `OTOPCUA_HARNESS_USE_SQL`, `OTOPCUA_HARNESS_SQL_HOST`, `OTOPCUA_HARNESS_USE_LDAP`, `OTOPCUA_HARNESS_LDAP_HOST`; choose local software prerequisites explicitly rather than accidentally enabling shared default mode |
| Native/proprietary/live devices | TwinCAT `TWINCAT_TARGET_HOST/NETID/PORT` and native AMS runtime; Heidenhain `HEIDENHAIN_LSV2_HOST`/credentials; gateway `HISTGW_GATEWAY_ENDPOINT`/`HISTGW_GATEWAY_APIKEY` plus suite opt-ins. Treat absent hardware/licensing/platform cases as unavailable coverage with exact test names, not required new-feature substitutes |
| Long-running opt-in tests | `OTOPCUA_SDK2_SOAK` and its duration/mode/output/platform controls, plus folder-event performance controls. Explicitly decide/run software-supported opted-in cases in the final tier and record disabled long-running cases as not executed; a no-category-filter command does not automatically enable them |
Final command, executed once against the final candidate after restore/build and environment setup:
```bash
dotnet restore ZB.MOM.WW.OtOpcUa.slnx --disable-parallel -m:2
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore -m:2
AKKA_TEST_TIMEFACTOR=3 dotnet test ZB.MOM.WW.OtOpcUa.slnx \
-c Release --no-build -m:1 -p:TestTfmsInParallel=false \
--settings scripts/ci/ci.runsettings \
--blame-hang --blame-hang-timeout 20m --blame-hang-dump-type none \
--logger trx --results-directory artifacts/grpc-data/final/trx
MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh artifacts/grpc-data/final/trx/*.trx
```
The final `dotnet test` has NO category/FQN filter: include SlowUnit and all discoverable runnable integration/live categories. The documented CI command excluding E2E/LiveIntegration is useful precedent for serialization but is not a substitute for this user-requested full run. Run final orchestrator via `python3 scripts/validation/grpc-data/run-final.py --run-id grpc-data-local`; it owns environment setup, build, full command, TRX guard and teardown/evidence. `--no-build` is valid here because the same candidate was just fully built. Use the repository's architecture/platform-supported build setup; resolve genuine incompatibilities rather than excluding projects silently.
The 12,000 floor is the current baseline guard, not proof of full execution. Also require a result/discovery inventory per test assembly and every new GrpcData class, zero missing/zero-executed expected projects, zero failed tests, and zero skipped required feature cases. Parse every skipped test name/reason. Hardware/Windows/native/proprietary suites that cannot run on the local host are listed as explicit unavailable coverage with requirements and a next gate; never describe that as all tests passed. Any available software fixture that skipped unexpectedly must be fixed/enabled and its applicable project rerun. A test reaching a forbidden shared host is a gate failure, not permission to use it. Do not lower floors, remove categories, blanket retry, change assertions or mark new feature cases optional to force green.
If final fixes are required: run only the failing/affected unit and integration selectors while iterating. After the candidate changes, rebuild affected binaries and repeat the final whole-solution gate on the new final candidate; record why a repeat was necessary. Do not repeatedly run the suite when there were no changes or unresolved concerns. Docker scenario/workload evidence must reference the same candidate; rerun impacted scenarios after final changes, and run the full required CLI matrix if shared code/wiring changed.
### Task 25: Final review, documentation and completion evidence
Give Astra the final diff, design/plan, phase TRX/discovery manifests, local CLI/probe/topology/fault evidence, full-suite results/skip inventory, workload report and unresolved risks. Ask for concrete correctness/security/concurrency/architecture findings, including bounds before mailboxes, immutable producer ownership, no unsafe upstream filtering, final dispatch policy, cache/provenance isolation, mapped ACL with legacy toggle off, and one-hop export boundary. Astra edits only `docs/plans/evidence/grpc-data/final-astra-review.md`. Coordinator resolves findings with focused tests, then applies the final-candidate rule above.
Update operational docs and `docker-dev/README.md` to distinguish current three-pair topology, API-key data plane and local validation rig; do not copy stale artifact-upload examples. Confirm new options/defaults, TLS trust/rotation, scoped secrets, mapping migration/consumer role behavior, rollback/disable, resource/overload diagnostics and deferred capabilities. Run documentation path checks and relevant script units only; do not invent a new full-suite rerun for prose-only changes.
Deliver commit/diff summary, exact tests/CLI scenarios executed and results, Astra disposition, measured workload configuration/results (without extrapolating capacity), required unavailable tests and any material implementation limitation. Leave a concrete reviewable branch. Pushing/merging/deploying outside the isolated local rig is not included. Remove owned temporary rigs/volumes after evidence unless explicitly needed for the user review; preserve only restricted local artifacts appropriately. No completion claim if a required feature gate failed or skipped.
## Completion checklist and review questions
- Every phase has only its applicable unit filters, nonzero discovery and TRX evidence; whole solution ran at final candidate only (repeated solely for actual final fixes).
- All design invariants and Astra design findings map to named implementation tests; no Raw/UNS or intra/intercluster scope was lost for scaling convenience.
- Local Docker gate actually used separate processes and independent pairs, local-only dependencies, validated TLS and production deployment/auth paths.
- OPC UA CLI demonstrated browse/read/write/subscribe through mapped views, including steady-client remote failover; direct gRPC probes filled CLI observability gaps.
- Write acceptance versus effect/outcome, queued dispatch revalidation, replay exclusions and late results are proven; no benchmark bypasses authorization or role checks.
- Bounded ingress/work/buffers/snapshots and authority/readiness invalidation hold during pressure; no fixed capacity guarantee is inferred from optional measurements.
- Schema, feature-disabled/old artifact compatibility, nonconsumer behavior, authoring and cleanup/rollback are covered.
- Final full-test inventory, environmental omissions and independent Astra review are complete and candid.
After Astra reviews THIS PLAN, the user reviews it. Execute only after the user instructs implementation, using `executeplan` and the agent/test policy above.
## Reference constraints verified during planning
- `scripts/ci/README.md` and `scripts/ci/ci.runsettings`: serialized solution leg, Akka time factor and existing executed floor; no root `ci.runsettings`.
- `docker-dev/docker-compose.yml`: actual independent MAIN/SITE-A/SITE-B pairs; base fixture references are not hermetic local dependencies.
- `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Api/DeployApiEndpoints.cs`: trigger/status API, not config/artifact upload.
- `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/`: actual existing CLI options and nonthrowing per-node failure behavior.
- `scripts/validation/pair-gate/Program.cs`: production API-key mint/grant/revoke and restricted credential-file patterns.
- [gRPC .NET sources](https://github.com/grpc/grpc-dotnet) and [Microsoft performance guidance](https://learn.microsoft.com/en-us/aspnet/core/grpc/performance?view=aspnetcore-10.0): reusable channels, bounded streams/flow control, serialized stream writes and cancellation/deadline propagation must be respected; use current official docs again when implementing API details.
@@ -0,0 +1,84 @@
# Astra review of the gRPC data connections implementation plan
Date: 2026-11-15
Reviewer: `gpt-6-astra`, xhigh
Scope: [implementation plan](2026-11-15-grpc-data-connections-implementation-plan.md), [reviewed design](2026-11-15-grpc-data-connections-design.md), [design findings](2026-11-15-grpc-data-connections-astra-review.md), and referenced repository source at `6dd2985b8b4d42033f32ef588a0a063d5411ae0c`.
**Final disposition: approved for user implementation-plan review after revision.** I reread the corrected dependencies, local services, discovery isolation, secret/trust lifecycle, topology matrix and final fixture gate. All six findings below are resolved in the plan. No implementation, build, test discovery, tests, Docker commands, migration generation or deployment was run during this review.
The initial disposition was revisions required. The initial findings and original task references are retained below for traceability; the revised plan now contains 25 tasks. This approval concerns the implementation plan and its required evidence, not a claim that the feature works or permission to begin implementation.
## Findings
### H1 — Assign the local connection and cached data services before ingress
The plan defines `IDataConnection`/`IDataServices` in P01, the hub in P04, writes in P05, subscriptions in P06 and a remote client/ingress in P07. It never assigns a concrete local connection, batched cached-read service, origin-local catalog/metadata service or capability provider. P07 says ingress calls ports, but the explicitly named browse implementation first appears as the federated resolver in P08. A hub is storage/publication infrastructure; it does not itself establish authorized per-item reads, age/readiness semantics, or origin-local export filtering. The direct internal API without a namespace mount is also a required design gate with no named plan test.
**Correction:** Add an owned task before P07 for the local `IDataConnection` facade and the required origin-local capability/catalog/metadata/read implementations, using the shared policy, active catalog and hub. Place the interfaces early enough for focused compilation, compose existing write/subscription services after they exist, and state the P07 dependency. Add focused cases for partial authorized batches, no-sample versus null value, stale/overflow-unready cache, complete status/timestamps, no device poll, no OPC UA sink, export filtering, and the same policy/outcome behavior through local and remote adapters. Add a mount-free scoped connection integration case; absence of a mapping must not become an accidental prerequisite or authorization bypass.
**Evidence:** Design §§3 and 7 explicitly require a local adapter, common local/remote execution services and a direct `IDataConnection` independent of mounts. Design §3 requires service without the OPC UA endpoint; §4 requires initial-data status instead of Good/null. Plan tasks 2, 9 and 14–17 name the interfaces, storage and callers without assigning this intervening service implementation.
### H2 — Make phase discovery hermetic as well as filtered execution
The unit policy excludes integration test projects, but some selected unit projects contain fixture-backed tests whose attributes execute during discovery. `AdminUI.Tests` has a SQL reachability probe inside `SqlServerFactAttribute`; its default target is the shared host. Therefore P09's mandatory `--list-tests` and filtered test launch can contact shared SQL before the selected unit tests execute. A test filter is not an isolation boundary. The final fixture environment setup is too late to protect these phases.
**Correction:** P00 must inventory discovery side effects in every selected project and construct a sanitized child environment for both discovery and execution. Introduce an explicit fixture-disabled mode before any external probe, or another demonstrably isolated mechanism, for mixed unit assemblies. Ignore inherited shared endpoint/credential settings in phase mode. Do not start SQL/Docker merely to discover a unit selector. Add runner/fixture guard tests showing discovery cannot contact shared infrastructure; selected required tests still may not skip. Final mode explicitly enables and binds the software fixtures locally before its discovery and execution inventory.
**Evidence:** `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/AdminUiSqlServerFixture.cs:133–199` documents discovery-time probing and calls `SqlConnection.Open()` with a shared-host fallback. Its attribute constructor at lines 389–395 invokes that probe. The selected P09 U project contains these attributes even when the requested FQN is unrelated. The unfiltered final suite has analogous reachability attributes, for example `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests/AbServerFixture.cs:68–110`.
### H3 — Carry local resource ownership through the final fixture helper
P10 requires run-owned Compose resources and loopback bindings. P12 introduces reuse of `integration-fixtures.sh` with only structured environment output described as a possible modification. On a desktop without a GitHub run ID or explicit suffix, that helper names fixed containers, invokes `down` at the start of `up`, and removes those names forcibly. It also publishes ports on all host interfaces and writes the NAT public certificate to a shared temporary path. The final gate can therefore violate the earlier ownership and isolation rules even if the feature rig is safe.
**Correction:** Require an explicit unique `CI_FIXTURE_SUFFIX` or equivalent run identity for every final helper invocation, resource labels/ownership verification before removal, local daemon validation, collision-safe loopback bindings and a run-specific certificate/output directory. Preserve existing CI behavior through explicit options if needed. Tests must reject pre-existing unowned resources instead of deleting them and prove interrupted setup/teardown affects only the recorded run. Apply the same rule to every additional final software fixture, not just the gRPC Compose project.
**Evidence:** `scripts/ci/integration-fixtures.sh:31–38` allows an empty suffix; `down` removes every computed name; `up` invokes `down` before starting fixtures. The `-p` arguments in `up` do not specify loopback. The NAT certificate path is `${RUNNER_TEMP:-${TMPDIR:-/tmp}}/otopcua-ci-fixtures/opc-plc-nat-server.der`. These are read-only observations of the helper, not actions taken during review.
### H4 — Make secret provisioning and change propagation concrete
P02/P10 refer to existing secret machinery and a central production-path seeder, but the feature rig deliberately has independent Akka clusters and DB-less data nodes. Configuration artifacts contain only references. Creating a target-issued key at CONTROL does not make the presented outbound credential available in each authorized consumer's local secret store. Nor does the present cache invalidator notify an as-yet nonexistent gRPC channel/credential cache. Without an explicit lifecycle, initial connection, restart after controller outage and credential rotation can depend on accidental shared storage or indefinite stale credentials.
**Correction:** Assign the production secret resolver/lifecycle and describe the supported provisioning procedure into the selected consumer stores. Reuse encrypted local stores and optional within-pair replication; do not invent cross-cluster secret replication or put plaintext into artifacts. Make seeding populate the correct selected consumers explicitly. Specify missing-secret readiness, reference/scope removal, API-key change/revocation and TLS trust/certificate change behavior, including when channels/streams are retired and how caches are invalidated. Add unit cases and a local integration rotation/restart case with CONTROL unavailable and no shared SQL dependency. State how source and consumer secrets are prevented from leaking into unrelated execution graphs.
**Evidence:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs:20–24,62–85` uses local SQLite by default and opt-in Akka replication. `SecretChangeSignallingInvalidator.cs:6–24` says the secrets library has no change event and currently notifies only `GalaxyAckIdentityDirectory`. The design §§3/7/8 requires selected consumer execution scope, reference-only artifacts and independent clusters. This correction can be met by explicit provisioning and change handling; it does not require a new secret-distribution service.
### M1 — Make every required topology an explicit matrix entry
D02 explicitly exercises a same-cluster Node target, and D07 establishes logical cluster failover. D03's cross-cluster mounts and D09's pinned secondary tests leave the exact connection modes and realms implicit. The design separately requires a cross-cluster Node target and direct internal access without a mount. It also requires maintenance exclusion and independent source/consumer role behavior. A single successful cross-cluster mount does not establish all these cases.
**Correction:** Bind scenario rows to explicit `(consumer cluster/node, target mode, target cluster/node, realm, operation)` records. Require same-cluster Node, cross-cluster Node and logical Cluster browse/metadata/read/write/subscribe across Raw/UNS where mappings apply, plus mount-free API access. Explicitly include maintenance/disabled endpoint exclusion, node-pinned no-substitution and consumer-scope behavior after consumer failover. The harness should report missing matrix cells as failures, not infer them from the scenario name. Shared scenario setup may satisfy several cells without duplicating every run.
**Evidence:** Design §7 topology table distinguishes all three modes, and its verification table explicitly names direct API without mount, maintenance, independent pairs and node-pinned behavior. The plan's D01–D13 rows do not yet provide that explicit coverage mapping.
### M2 — State the actual early build and migration prerequisites
P01's A test project is not an isolated contracts-only build: it references Core and almost every concrete driver. The unrelated dirty AB CIP source can therefore affect the first focused contract gate. P02's exact EF command also omits a mandatory design-time environment variable; a fresh shell fails before producing the migration. Neither warrants running the full suite early or changing unrelated work, but the plan should make the early dependency boundary executable.
**Correction:** Record the transitive compile graph and identify unrelated baseline build blockers during authorized preflight/first focused discovery. Preserve unrelated changes and report them distinctly; do not silently weaken the project references or filters. Supply `OTOPCUA_CONFIG_CONNECTION` explicitly for migration generation using a noncontacted local design-time connection, or the disposable local SQL when the apply gate runs. Ensure migration generation does not start Host services/connect to shared infrastructure. Verify the installed EF tool version against the repository's package version before generation and record the actual command.
**Evidence:** `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj:29–50` references Core and concrete driver projects. `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs:23–37` throws when `OTOPCUA_CONFIG_CONNECTION` is absent. The documented default/example uses shared infrastructure and must not be copied into this local-only gate.
## Checks that passed this source review
- Existing project paths and named existing test classes were found; the Commons proto contract regression names, `ConfigComposer` location and Host integration mesh test placement match source. New files/classes are clearly labeled as planned.
- Phase-local selectors, new-class discovery, per-project TRX accounting, no stale `--no-build`, no skip/floor weakening and final unfiltered solution execution match the requested testing policy. The final result explicitly distinguishes unavailable hardware/native coverage from a full pass.
- Agent roles specify model/effort, disjoint ownership, coordinator-controlled tests and migration generation, a four-agent limit and independent Astra reviews. Contract and shared-file changes are serialized.
- The CLI assembly filename and connect/browse/read/write/subscribe/redundancy option shapes match source. The plan correctly treats exit-zero Bad values and empty notification summaries as failures, separates OPC UA failover from server-side gRPC failover, and uses direct probes only for missing CLI observability.
- The six-process CONTROL/SOURCE/CONSUMER rig preserves independent pairs and makes controller-outage testing meaningful. Real deployment trigger/status semantics, namespace URI resolution, TLS identity checks and deterministic effect-ledger faults are included.
- Design H1–H9/M1–M4 are substantially retained: late/unknown outcomes, no replay, mapped ACL with legacy switch off, leases and final send gates, immutable payloads, conservative shared filters, bounded ingress/snapshots, vector watermarks and source-dirty fair scheduling have named tasks/tests.
- Scaling evidence correctly avoids unsupported fixed capacity promises. Tests distinguish correctness/boundedness from hardware-dependent throughput and retain the legacy actor/SDK serialization limits.
## Revision disposition
| Finding | Verified resolution in the revised plan |
|---|---|
| H1 | P04 task 10 explicitly owns `LocalDataConnection`, `LocalDataServices`, `LocalExportCatalog` and `DataCapabilitiesProvider`, with focused cached-read/catalog tests. Its P03 dependency and P07 prerequisite are explicit; absent later write/subscription implementations report unavailable rather than success. D01a exercises the real caller node's connection manager without requiring a mount. |
| H2 | P00 owns the early unit-only discovery guard and actual C# `GrpcDataUnitDiscoveryIsolationTests`, with a counting connector/reaper seam and an exact focused U gate. Both discovery and execution child environments are sanitized; integration/final mode removes the switch and supplies local fixtures before discovery. This closes the discovery-time SQL exposure without counting unselected SQL tests as executed. |
| H3 | P12 requires opt-in local-owned fixture mode, mandatory unique suffix/labels, local context validation, loopback ports, run-specific certificate paths and manifest/label-verified cleanup. Foreign-name refusal, partial startup, port collision and parallel-run behavior have named fake-process unit tests. |
| H4 | P07 task 15 owns per-consumer provisioning, local encrypted-store resolution, change handling, scoped reference retirement/tombstones and independent TLS trust policy. It explicitly rejects CONTROL-only provisioning and cross-cluster replication assumptions. D14 covers credential/CA/server-certificate rotation, stale-bundle prevention and DB-less consumer restart while CONTROL is stopped. Dispatched write observation survives retirement without replay. |
| M1 | P10 task 22 now expands a target-mode/topology × realm × operation matrix for same-cluster Node, cross-cluster Node, logical Cluster, mount-free direct API and consumer failover. It names maintenance exclusion and pinned no-substitution. A fixture-only node runner exercises the actual caller's DI/manager and is absent from normal Host startup. The same-pair positive direct-API write and correct local standby OPC UA rejection are distinguished explicitly, preserving both primary gates. |
| M2 | P00 records transitive compile/discovery effects, including unrelated dirty driver work, without broad early test execution. P02 uses Configuration's design-time factory/startup project, explicit noncontacting local `OTOPCUA_CONFIG_CONNECTION` for scaffolding and the repository's EF 10.0.7 setup; real migration applies only to P10 disposable SQL. |
The remaining uncertainties are appropriately implementation gates: actual SDK Browse/BrowseNext behavior, adversarial ordering/ownership tests, architecture-supported local fixtures, final per-assembly/skip accounting and measured capacity. The plan requires concrete evidence for them and forbids treating skipped required feature cases or missing software fixtures as success. No unresolved plan-level blocker remains.
Only this review document was edited by the reviewer. The coordinator revised the implementation plan. No runtime files or tests were changed by the reviewer.