83 KiB
gRPC Data Connections Implementation Plan
For Codex: REQUIRED SUB-SKILL: Use
executeplanto implement this plan task-by-task.
Goal: Implement scalable, authorized Raw/UNS browse, cached read, value write and subscription connections between processes in the same cluster and across independent redundant clusters, verified with focused unit tests and local Docker/OPC UA CLI integration.
Architecture: Keep Akka membership/election and existing command/telemetry contracts intact. Build SDK-independent data/mapping policy and a partitioned authoritative sample hub behind transport-neutral ports; Host supplies TLS gRPC ingress, Runtime supplies local/remote adapters, and the existing OPC UA server materializes bounded mapped views. Preserve primary-only mutation, node-pinned provenance, explicit uncertain outcomes, bounded resource use and authority revocation across every path.
Tech Stack: Existing .NET 10/C#, protobuf/grpc-dotnet, Akka.NET, EF Core/SQL Server, xUnit/VSTest, OPC UA SDK and otopcua-cli; Python 3 orchestration and local Docker Compose for reproducible gates. Retain repository-pinned package versions unless a demonstrated requirement warrants a separate change.
Status: Astra reviewed; approved for user implementation-plan review after revision. Awaiting user review. This task prepares a plan only; it does not authorize implementation, builds, running fixtures or deployment.
Design: reviewed design. Design review: Astra design findings. Plan review: Astra implementation-plan review.
All commands and fully qualified task paths below are relative to the repository root /Users/dohertj2/Desktop/OtOpcUa. New names are planned files/classes, not claims that they already exist. Existing source paths were inspected; implementation must reread nearby instructions and source before editing. The checkout contains unrelated AB CIP fixture changes; leave them out of this work and its commits.
Task path prefixes are explicit shorthand for these assembly directories: Runtime/ = src/Server/ZB.MOM.WW.OtOpcUa.Runtime/; OpcUaServer/ = src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/; AdminUI/ = src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/; Core/ = src/Core/ZB.MOM.WW.OtOpcUa.Core/. Bare Drivers/ and OpcUa/ paths in Runtime task Modify lists are relative to the Runtime directory; bare Configuration snapshot/migration filenames are relative to src/Core/ZB.MOM.WW.OtOpcUa.Configuration/. New harness subpaths (seed/, probe/, device/, tests/) in P10–P12 are relative to scripts/validation/grpc-data/. Expand them into exact owned paths in every worker packet; never create top-level alias directories.
Execution ownership, models and checkpoints
The coordinator owns the dependency graph, interface decisions, integration, test manifest and final evidence. Use at most four active agents total: coordinator, two disjoint implementers, and one reviewer. Reduce concurrency when shared-file ownership, compilation or Docker contention requires it. Only the coordinator starts tests, EF migration generation, Docker gates and final commits; agents may prepare test commands but cannot launch competing test/build processes. The coordinator can explicitly lease one focused red/green run to a worker while all other runs remain paused.
| Role | Model / effort | Assignment and reason |
|---|---|---|
| Coordinator/integrator | gpt-6.1-sol, high |
Track requirements, own shared wiring and commands, integrate results, assess evidence and report blockers |
| Implementation worker | gpt-6.1-terra, high |
Bounded production-code task plus its behavioral tests; explicit file ownership |
| Protocol/security/concurrency implementation | gpt-6.1-terra, xhigh |
Authority extraction, dispatch-ticket/unknown-write semantics, partition/snapshot ordering; use only for these higher-risk tasks |
| Test/fixture worker | gpt-6.1-sol, high |
Independent test harness, Docker/CLI orchestration, fixtures and evidence parsing |
| Mechanical documentation/manifest work | gpt-6.1-sol, medium |
Exact path/selector updates and runbook changes after contracts are frozen |
| Astra reviewer | gpt-6-astra, xhigh |
Review frozen contracts and security/concurrency boundaries, phase evidence and final integrated implementation; no implementation edits |
When selecting an explicit model/effort in spawn_agent, use fork_turns="none" and provide a self-contained packet. Do not silently substitute a missing model: report the unavailable choice and use the coordinator to do independent work until the user can resolve a required reviewer dependency. No reviewer approves their own implementation. Do not use more agents than the current concurrency budget.
Example worker dispatch after plan approval:
spawn_agent(task_name="p04_hub", model="gpt-6.1-terra", reasoning_effort="xhigh", fork_turns="none",
message="Implement only P04 task 9 against <commit>. Read design §§3,4,6a and plan P04.
Own Runtime/DataConnections/PartitionedSampleHub.cs and named tests only.
Frozen interfaces are in Core.Abstractions/DataConnections. Do not edit DriverHostActor,
Host wiring, schema, test manifest or other workers' files. Return diff summary,
invariants and the exact P04 focused commands. Test execution is coordinator-owned.")
Every handoff includes the baseline commit, design sections, exact owned paths, interface version, prerequisites, negative cases and allowed test scope. Workers return changed paths, test cases added, commands/results (if explicitly leased), risks and open decisions. No worker stages another worker's changes or makes repository-wide formatting changes.
Use the suitable current checkout or an already attached worktree. If isolation requires a new one, discover/use create_worktree and a codex/grpc-data-connections branch; do not create a worktree merely because this document exists. Reuse a current suitable worktree when execution starts. Do not discard user edits.
Dependency and concurrency graph
flowchart TD
P00[Preflight and test manifest] --> P01[Contracts and codec]
P01 --> P02[Schema and deployment projection]
P01 --> P03[Policy and mapped subjects]
P02 --> P03
P01 --> P04[Immutable samples and partitioned hub]
P03 --> P04
P03 --> P05[Write tickets and outcomes]
P04 --> P06[Stream engine and scheduler]
P03 --> P06
P04 --> P07[TLS ingress and remote connection client]
P05 --> P07
P06 --> P07
P07 --> P08[Raw and UNS projection / adapter]
P02 --> P08
P08 --> P09[Authoring and diagnostics]
P01 --> RIG[Local rig / CLI harness preparation]
P09 --> P10[Docker CLI and fault integration]
RIG --> P10
P10 --> P11[Scaling and recovery evidence]
P11 --> P12[Final build / full tests / review]
Parallelism is by file ownership, not by diagram alone. Safe examples: P02 entity files versus P04 new hub files after P01 contracts freeze; rig/probe scaffolding versus production worker. Serialize changes to Commons/Core.Abstractions contracts, DriverInstanceActor, DriverHostActor, OpcUaPublishActor, Host Program.cs, ServiceCollectionExtensions, solution/project registration, Directory.Packages.props, model snapshot, and test manifest. The diagram's rig preparation is read/scaffold work until contracts/wiring are usable; no early whole-suite run.
Astra checkpoints: P01 contract freeze; P03/P05 security and mutation evidence; P06 stream/scaling evidence; P08 mapped SDK behavior; P10–P12 integrated/final evidence. Intermediate review can run while an independent next task is prepared, but dependent work does not proceed past an unresolved high-severity boundary. Fixes receive focused regression tests and review. User review of this plan is the next step; do not start implementation in the plan-writing turn.
Test policy: applicable units by phase, full tests only at the end
Create scripts/validation/grpc-data/unit-phases.json and run-unit-phase.py in task 1. The manifest lists exact test projects, phase traits for new tests and selected existing class filters below. All new unit test classes have [Trait("Feature", "GrpcData")] and [Trait("Phase", "Pnn")]. All tests for one class use one phase; a shared helper has no traits of its own. Every selector is a manifest entry, not an inferred changed-files heuristic.
Representative manifest record:
{
"P05": [{
"project": "tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj",
"filter": "(Feature=GrpcData&Phase=P05)|FullyQualifiedName~DriverHostActorWriteRoutingTests|FullyQualifiedName~DriverInstanceActorConcurrentWriteTests|FullyQualifiedName~ActorNodeWriteGatewayTests|FullyQualifiedName~PrimaryGatePolicyTests|FullyQualifiedName~DriverWriteBoundsTests"
}]
}
The runner invokes subprocess argv, never eval, with:
AKKA_TEST_TIMEFACTOR=3 dotnet test <exact-project.csproj> -c Release -m:1 \
-p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
--filter '<manifest expression>' --logger trx \
--results-directory 'artifacts/grpc-data/<run>/unit/<phase>/<project>'
Build the selected project and its dependencies normally; do not use stale --no-build binaries during phase runs. --no-restore is allowed only after restore for the current dependency graph. List/discover tests for each selector first and fail on zero matches. Parse phase TRX for executed>0, zero failures and zero unexpected skips, per project; a broad solution floor is not enough for a phase. Include a per-new-class discovery manifest so a typo cannot be masked by matching an old class. Record exact filter, SHA/dirty diff identity and durations. Phases run only their manifest units, not every prior phase or all solution tests. If a later change modifies a shared earlier contract, add only the affected earlier classes to the current phase and explain why.
Red/green task loop, required for each task below:
- Add the stated failure/edge-case tests before the corresponding behavior; select one new test/class with
run-unit-phase.py Pnn --case <ClassName>and verify an assertion/compile failure attributable to the missing behavior. - Implement only that task's behavior. Compile errors caused by new interfaces are resolved within the owned dependency boundary; do not fabricate passing behavior or disable existing tests.
- Run that task's selected classes to green. Repeat this small loop for the task's scenario groups rather than writing all production code first.
- After all tasks in the phase, coordinator runs
python3 scripts/validation/grpc-data/run-unit-phase.py Pnnonce as the phase gate, then inspects TRX/discovery and reviewer feedback. - Commit only explicit owned production/test/docs files after the gate:
git add -- <explicit paths>followed by a concise phase commit. Do not usegit add .or commit generated test artifacts, credentials or unrelated edits.
Existing focused unit tests may internally start in-process actor systems/SDK fixtures; the phase gate still excludes fixture-backed integration test projects. SQL migration checks, TLS wire tests, CLI subprocess tests and Docker scenarios are named integration gates in their specified phases, not hidden whole-suite repeats. No timing retries, timeout widening or skip changes to mask failures.
Exact existing test projects
| Key | Project path |
|---|---|
| A | tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj |
| C | tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ZB.MOM.WW.OtOpcUa.Commons.Tests.csproj |
| F | tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/ZB.MOM.WW.OtOpcUa.Configuration.Tests.csproj |
| K | tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/ZB.MOM.WW.OtOpcUa.Core.Tests.csproj |
| R | tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj |
| O | tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj |
| H | tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj |
| P | tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj |
| U | tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj |
| L | tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj |
New tests named below live in <project directory>/DataConnections/<ClassName>.cs unless a different exact path is given. Existing classes keep their existing paths. Expand project keys into full paths in the runner manifest; keys are document shorthand only.
P00 — Freeze scope and establish execution controls
Owner/model: coordinator, Sol/high; mechanical runner worker Sol/medium. Prerequisites: user accepts plan. Gate: runner Python unit tests and focused U discovery-isolation units only, no solution tests.
Task 1: Preflight, discovery isolation, phase runner and requirement ledger
Modify: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/AdminUiSqlServerFixture.cs to add an explicit unit-only discovery switch before any SQL probe/reaper and an injected probe/reaper seam.
Create test: U DataConnections/GrpcDataUnitDiscoveryIsolationTests.cs, tagged P00; exercise the actual attribute/probe path with a counting fake connector/reaper, not only the Python environment mock.
Create: scripts/validation/grpc-data/run-unit-phase.py, scripts/validation/grpc-data/unit-phases.json, scripts/validation/grpc-data/test_unit_phase_runner.py, docs/plans/evidence/grpc-data/README.md.
Read: design and all Astra design findings; scripts/ci/ci.runsettings; scripts/ci/assert-not-all-skipped.sh; scripts/ci/README.md; source/test project references; execution skill. Record current SHA, dirty files, SDK, Docker context metadata and host architecture without starting containers or printing secrets. Discover current test project/class names and replace every conditional path in this draft with verified entries in the execution manifest.
Behavior/tests: fake subprocess results prove runner rejects zero discovery, missing new classes, skipped-only TRX, stale/missing result files, nonzero exit and unknown phases; supports exact class narrowing; uses unique phase/project output directories and preserves exit codes. No blanket retries. Set a dedicated OTOPCUA_UNIT_ONLY_DISCOVERY=1 in BOTH discovery and execution children for unit phases; SqlServerFactAttribute/probe returns a unit-mode unavailable marker before opening SQL. This does not count SQL tests as exercised: they are unselected and their required final integration remains enabled/audited. Sanitize inherited fixture/connection variables using an allowlist, replace unselected default endpoints with loopback unavailable sentinels, and audit discovery-time attributes/fixtures in every touched test assembly. Verify with a fake network/probe factory that unit-mode discovery makes zero fixture connections, including to the current AdminUI default shared SQL host. No unit runner invokes reapers. Unset this switch for integration/final runs.
After this guard is in place, perform one focused baseline compile/discovery (not a full test run) of the selected phase projects on the starting candidate. Core.Abstractions.Tests transitively references Core and many concrete drivers; record unrelated baseline compile failures/dirty AB CIP effects without editing them silently. Selectors limit executed tests, not transitive compilation. Snapshot design invariants in a requirement-to-task-to-test ledger, including H1–H9/M1–M4 and their resolved corrections.
Run: python3 -m unittest discover -s scripts/validation/grpc-data -p 'test_unit_phase_runner.py'. Expected: all runner tests execute/pass. Then run python3 scripts/validation/grpc-data/run-unit-phase.py P00 with manifest entry U (Feature=GrpcData&Phase=P00); prove the actual attribute/discovery guard makes zero connector/reaper calls and final-mode permits the injected connector. Add dry-run invocation displaying selected project/filter without executing dotnet. Commit test: add focused grpc data phase gates.
P01 — Transport-neutral contracts and lossless protobuf
Owner/model: Terra/high; Astra/xhigh contract review. Depends: P00. Unit gate: C new P01 plus TelemetryProtoContractTests, MeshTransportProtoContractTests and DeploymentArtifactProtoTests; A new P01 plus existing DataChangeBatchContractTests.
Task 2: Define addresses, views, capabilities and rich outcomes
Create: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DataConnections/DataAddress.cs, DataContracts.cs, IDataConnection.cs, IDataServices.cs, IWriteOutcomeCapability.cs; A GrpcDataContractsTests.cs.
Freeze these contract decisions in code before parallel work:
public enum DataWriteState {
Unspecified = 0, RejectedBeforeDispatch, DriverConfirmedSuccess,
DriverConfirmedFailure, AcceptedUnconfirmed, OutcomeUnknown
}
public sealed record DataAddress(string OriginClusterId, string NamespaceUri,
string IdentifierKind, string Identifier);
public sealed record ServingIdentity(string NodeId, string Incarnation,
string Role, long CatalogEpoch, long AuthorityEpoch);
Actual sample/lease contracts have explicit ownership/disposal and optional original timestamps; do not place SDK/actor/generated types into these ports. Define connection target mode, explicit realm, versioned mapping/item/stream identity, capability/limit negotiation, per-item statuses, cached age/readiness and snapshot control records. IDataConnection composes browse/metadata/read/write and async subscription lifetime without exposing channels/actor refs. Define monotonic-time abstraction through TimeProvider and bounded budget inputs.
Red/green: addresses distinguish same bare Raw/UNS paths and serving-node provenance; invalid/zero enum values reject; Unknown and Unconfirmed cannot become success Booleans; cancellation/lifetime and null timestamp semantics are explicit. Preserve existing IWritable drivers through an additive rich-outcome capability, not a breaking rewrite of every driver.
Task 3: Protobuf/codec and negotiation
Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/data_connection.proto, Protos/DataConnectionProtoContract.cs, DataConnections/DataProtoMapper.cs, DataConnections/TypedValueCodec.cs; C GrpcDataProtoContractTests.cs, GrpcDataCodecTests.cs, GrpcDataCompactCodecTests.cs.
Modify: Commons .csproj to generate both endpoints for this proto only.
Define GetCapabilities/Browse/GetMetadata/Read/Write/Subscribe as in design, with per-branch browse completeness, epoch-bound continuations and control-only metadata watch. Add negotiated compact handles, explicit batch-default override/presence, immutable sample provenance, write operation/serving-incarnation preconditions and exact domain status codes. Bind operation-ID payload hashes to immutable semantic fields, excluding retry-varying budgets. Assign/freeze field numbers and versioned capability bits; existing mesh/telemetry semantics do not change.
Red/green: full status subcodes, signed/unsigned boundaries, UInt64, NaN/infinity, null/empty arrays/bytes, ranks/dimensions, optional timestamps and unsupported graphs; compact/fallback domain equivalence; absent timestamp override versus common default; unknown handle, removed item revision, major-version mismatch, malformed dimension/count and decoded/decompressed budget failure before large allocation. Golden payload and additive old/new peer tests. No compression by default until measured.
Phase completion: run-unit-phase.py P01; Astra reviews frozen contract/provenance/ownership and rich-result signatures. Commit feat: define grpc data contracts and codecs.
P02 — Configuration, validation and per-node deployment
Owner/model: Terra/high for entities; Sol/high for artifact tests, serialized integration by coordinator. Depends: P01. Unit gate: F new P02 plus DraftValidatorNodeAclScopeTests; P new P02 plus DataPlaneAuthorityArtifactTests; R new P02 plus DeploymentArtifactDataPlaneAuthorityTests, ParsedDeploymentArtifactTests and DeploymentArtifactEquipRefParityTests; P also ConfigComposerTests.
Task 4: Schema and deterministic validation
Create: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/DataConnection.cs, DataExport.cs, RemoteNamespaceMapping.cs; Validation/DataConnectionValidator.cs; F GrpcDataConfigurationTests.cs, GrpcDataMappingValidationTests.cs.
Modify: OtOpcUaConfigDbContext.cs, Validation/DraftSnapshot.cs, DraftSnapshotFactory.cs, DraftValidator.cs, Enums/NodeAclScopeKind.cs.
Add owner cluster, explicit consumer-node IDs, Node/Cluster target, expected identities, HTTPS endpoints, secret references, realm-qualified root/parent IDs, branch names, operation subset, scopes, bounded policies and disabled/write-through defaults. Append mapping ACL scope without renumbering existing enum values. Resolve local path from existing Raw/UNS entity identity. Validate subset consumer scope, self-route only for selected consumers, nested/self-mount loops, same-realm origin, duplicate sibling names, current parent ownership, unsupported type/operation and secret reference existence. A→B in the same cluster is allowed on consumer A only. A cluster endpoint set must have a nonself eligible target for every declared consumer; never silently alter node-pinned targets.
Red/green: reject invalid topology/collision/credential configuration while preserving existing local Raw/UNS validation and disabled feature behavior. Test mapping moves and inherited/granted scopes; no synthetic remote TagId.
Task 5: Migration and artifact projection
Create: EF-generated migration named AddGrpcDataConnections and its Designer under Configuration Migrations/; exact timestamp is produced once by EF during this task, recorded in ledger. Add F GrpcDataSchemaModelTests.cs, P GrpcDataArtifactProjectionTests.cs, R GrpcDataArtifactReadTests.cs.
Modify: Migrations/OtOpcUaConfigDbContextModelSnapshot.cs, src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs for ordered artifact serialization and Coordinators/ConfigPublishCoordinator.cs only for coordination changes; Runtime Drivers/DeploymentArtifact.cs.
Use dotnet ef migrations add AddGrpcDataConnections --project src/Core/ZB.MOM.WW.OtOpcUa.Configuration --startup-project src/Core/ZB.MOM.WW.OtOpcUa.Configuration using the existing migrator’s EF 10.0.7 setup. Set OTOPCUA_CONFIG_CONNECTION to a nonconnecting loopback design-only sentinel while scaffolding (the design-time factory requires the variable); migrations add must not migrate or contact a shared database. Real apply uses the isolated P10 SQL connection. Do not use the shared-host example embedded in the factory documentation. Freeze generation/serialization ordering and include connection/export/mapping/grant revisions in the artifact hash. Compile node-effective execution graphs: A-only mapping absent/inactive on B; credentials are references delivered through secret machinery, not plaintext blobs/logs. Old artifacts decode feature-disabled; reject partially configured enabled artifacts fail-closed. Projection with unsupported major capability never reports ready.
Red/green: stable hashes independent of collection enumeration, source/consumer isolation, authority-only deploy, disabled/maintenance changes, no plaintext secrets, old artifact compatibility and no B self-mount. Run focused migration model tests without SQL during the unit gate; actual apply/reapply/backward-disable compatibility uses disposable SQL during P10, not a live database migration now.
Phase completion: P02 gate and artifact diff review; commit feat: deploy scoped grpc data connection configuration.
P03 — Shared policy and mapped authorization
Owner/model: Terra/xhigh; Astra/xhigh independent review. Depends: P02. Unit gate: K new P03 plus TriePermissionEvaluatorTests, PermissionTrieCacheTests, DataPlaneAuthorityStateTests, PermissionTrieBrowseTransitTests; O new P03 plus DataPlaneAuthorizerTests, SessionAclDecisionCacheTests, NodeManagerAclEnforcementTests.
Task 6: Extract SDK-independent subject/policy service
Create: src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/DataOperationPolicy.cs, MappedNamespaceSubject.cs, MappedSubjectResolver.cs, DataAuthorizationCache.cs; K GrpcDataPolicyTests.cs, GrpcDataAuthorityCacheTests.cs; O GrpcDataMappedSubjectTests.cs.
Modify: DataPlaneAuthority.cs, AclScopePathResolver.cs, PermissionTrieBuilder.cs; OPC UA Security/DataPlaneAuthorizer.cs, Security/SessionAclDecisionCache.cs and relevant subject representation.
Delegate exact existing local decisions to shared policy rather than independently deriving paths. Resolve mapped subjects ahead of BOTH Raw BuildScope/cache and UNS custom resolution, including mapping revision/local ancestry/origin proof. Bind decisions to actual view, operation, caller key/principal, serving identity and active authority/catalog; retain inherited/deny semantics. Unknown subjects and empty authority deny. Mapped access remains gated with legacy Security:Acl:Enabled=false; ordinary local-node behavior remains unchanged. Cache expiry/key disable and metadata/export lease boundaries; bounded negative cache.
Red/green: Raw/UNS identical path, multiple aliases, opaque folders, deleted/rebound mapping, local ancestor versus mapping grant, wrong principal/cluster, expiry and same-key generation change; cached Allow never survives changed provenance or authority. Regression local grant behavior must remain equivalent.
Task 7: Catalog staging, remote authority leases and revocation gates
Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/DataConnections/CatalogActivation.cs, MetadataAuthorityLease.cs; R GrpcDataCatalogActivationTests.cs, GrpcDataMetadataLeaseTests.cs.
Modify: Runtime/OpcUa/OpcUaPublishActor.cs authority/catalog integration only (coordinator-owned file).
Build one desired catalog/subject projection reused by hub and SDK. Publish revocation even if topology apply fails or plan is empty; fail closed on catalog/authority mismatch without rolling back authority. Lease expiry uses request-start monotonic time plus bounded duration/margin; delayed replies cannot extend authorization. Watch/probes include quiet browse-only mounts and authority-only epochs. Last-known local anchor may remain locally visible; derived remote names/continuations/reads expire. Validate final send gate versus already-emitted bytes.
Red/green: no-op ACL deploy, partial/failed topology apply, lease expiry under partition, delayed renewal, clock jump, quiet key expiry and cached payload invalidation. P03 manifest includes R new tests plus OpcUaPublishActorApplyFailureTests and OpcUaPublishActorApplyOutcomeTests for this task only.
Phase completion: P03 gate; Astra review grants, generation publication and lease reasoning; commit feat: share grpc mapped authorization and authority leases.
P04 — Immutable source feed and partitioned hub
Owner/model: Terra/xhigh; no concurrent driver/publish actor edits. Depends: P01/P03 for phase completion; source/hub scaffolding may be prepared after P01 while policy work finishes. Unit gate: R new P04 plus DriverInstanceActorBatchForwardTests, DriverHostActorBatchFanOutTests, DriverHostActorSourceRemovedTests, DriverHostActorCommsLossTests, OpcUaPublishActorBatchTests, OpcUaPublishActorStatusBatchTests.
Task 8: Own payloads before asynchronous handoff and preserve timestamps
Create: Runtime/DataConnections/OwnedDataSample.cs, SourceSamplePublisher.cs; R GrpcDataSampleOwnershipTests.cs, GrpcDataSourceFeedParityTests.cs.
Modify: Drivers/DriverInstanceActor.cs, Drivers/DriverHostActor.cs, OpcUa/OpcUaPublishActor.cs, src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs and affected sample envelopes if required.
Freeze/copy supported mutable payloads at callback/adapter boundary BEFORE actor enqueue, or take explicit immutable ownership. Preserve independently optional original source/server timestamps rather than manufacturing two from the current collapsed TimestampUtc. Emit source-level normalized batches before optional OPC UA sink early return and before extra alias fan-out. Include virtual values, status-only/comms-loss/removal/rebind and deferred apply; never cache optimistic operator writes as authoritative samples. Retain existing downstream contracts through adapters and batch delivery, preserving full uint status.
Red/green: producer mutates arrays/bytes immediately after callback, pool reuse while writer stalls, disposal/cancel once, original timestamp absence/fidelity; virtual/status/removal parity and retired-incarnation rejection with no SDK sink. Record remaining legacy actor/sink serialization; no claim that new partitions parallelize existing drivers.
Task 9: Bounded partitions, snapshot registration and readiness
Create: Runtime/DataConnections/PartitionedSampleHub.cs, SamplePartition.cs, SnapshotLease.cs, DataResourceBudget.cs; R GrpcDataPartitionTests.cs, GrpcDataSnapshotTests.cs, GrpcDataIngressOverflowTests.cs.
Normalize source ownership and immutable view descriptors. Bounded nonblocking ingress precedes actor mailboxes; source latest/dirty generation advances without touching every registration. Coalesce in bounded dirty tables; overflow invalidates affected source/partition readiness and signals reset. Ordered reserved controls prevent value resurrection after removal. Per-partition interest registration captures immutable snapshot/watermark; multi-partition vector admits no cross-item atomicity promise. Count shared storage, baseline refs, pending frames and all partition snapshots in global budgets. Repartition resets epochs before ownership change.
Red/green: concurrent snapshot/delta/update, slow partition, overflow with stale previously Good cache then reconnect/read, concurrent removal, source changes during dirty traversal, memory rejection before allocation, cancellation/ref release and fake-time expiry. Unit assertions test boundedness and invariants, not machine-specific throughput.
Task 10: Implement local data ports and direct API without a mount
Create: Runtime/DataConnections/LocalDataConnection.cs, LocalDataServices.cs, LocalExportCatalog.cs, DataCapabilitiesProvider.cs; R GrpcDataLocalConnectionTests.cs, GrpcDataCachedReadTests.cs, GrpcDataLocalExportCatalogTests.cs (P04).
Implement the local IDataConnection and IDataReadService/metadata/capability facades explicitly. Origin-local browse returns the paged authorized deployed catalog, never projected peer nodes; metadata carries exact type/rank/units/status/timestamps/serving provenance and effective grants/limits. Cached reads distinguish missing, denied, initial-data, over-age, authority mismatch and ingress-invalidated readiness; no driver refresh is implied. Resolve the same source/view descriptors used by the hub and shared policy. Local write/subscription ports delegate to P05/P06 implementations when wired, and advertise only ready implemented capabilities; never return a placeholder Good/success. During this phase test unavailable delegation with fakes. P07 ingress can use these ports before federated mapping arrives in P08; origin-local versus composed browse remains an explicit facade boundary.
Red/green: direct API browse/read/metadata on a local origin with no namespace mount; per-item order/partial failure; Raw/UNS aliases with different grants; initial/stale/overflow cache states and timestamps; origin-only exports; capability intersection/readiness; local adapter and gRPC service use the same policy inputs. Add loopback direct-API coverage through the real remote client in P10 rather than requiring a mapping solely to prove connectivity.
Phase completion: P04 gate; commit feat: add immutable partitioned data publication hub.
P05 — Mutation admission, uncertainty and late outcomes
Owner/model: Terra/xhigh; Astra/xhigh review. Depends: P03 and P01 rich outcomes. Unit gate: R new P05 plus the exact existing classes in the representative manifest above; O new P05 plus NodeManagerWriteDispatchTests, NodeManagerWriteRevertTests, NodeManagerBatchWriteTests and EquipmentWriteGateTests.
Task 11: Dispatch tickets and additive rich driver outcomes
Create: Runtime/DataConnections/WriteAdmissionTicket.cs, DataWriteExecutor.cs; R GrpcDataWriteAdmissionTests.cs, GrpcDataDriverOutcomeTests.cs.
Modify: Drivers/DriverInstanceActor.cs, Drivers/DriverHostActor.cs, Drivers/ActorNodeWriteGateway.cs, Drivers/DriverWriteBounds.cs, Commons OpcUa/IOpcUaNodeWriteGateway.cs; additive Core.Abstractions rich-outcome interface from P01.
Ticket carries principal/key, exact view/subject and resolved backing target, catalog/authority, serving/driver incarnation, operation ID and monotonic expiry. Admission rejects unsupported/nonwritable/type-mismatched targets after authorization. Immediate launch AND queue drain revalidate role, mapping, authority and remaining budget. Preserve bounded device execution and derived timeout ordering; no automatic retry/hedge. Adapt legacy confirmed versus fire-and-forget drivers conservatively; status Good alone is not necessarily device confirmation. Demotion/expiry before dispatch returns RejectedBeforeDispatch; post-dispatch cancellation/timeout is Unknown unless a definitive driver outcome exists.
Red/green: queued revocation/remap/demotion, changed driver incarnation, insufficient budget, token-ignoring driver and existing per-tag serialization; legacy driver result compatibility. Do not weaken write-primary checks to make intracluster tests pass.
Task 12: Registry, late completion and OPC UA self-correction
Create: Runtime/DataConnections/WriteOperationRegistry.cs, LateWriteObserver.cs; R GrpcDataWriteRegistryTests.cs, GrpcDataLateWriteTests.cs; O GrpcDataMappedWriteCompletionTests.cs.
Modify: node manager completion/self-correction seam for rich results only.
Registry key is stable authenticated principal plus operation ID with immutable semantic payload hash and serving incarnation. In-flight duplicates share task; canceled waiter does not cancel dispatched work. Reject conflicting payload and capacity without evicting in-flight entries. Stale-incarnation retry rejects before dispatch; no ambiguous cross-incarnation replay. Retain bounded late result observation/audit after caller disconnect or driver abandonment; if retention ends preserve Unknown. Imported node hook remains nonblocking: definitive rejection restores authoritative sample; Unknown becomes uncertainty pending real data; AcceptedUnconfirmed is not final success. Old optimistic values must not enter hub snapshots.
Red/green: concurrent duplicate task, changed budget with same semantic hash, conflicting payload, partial batch success and endpoint switch of only proven-undispatched items, response loss after simulated device commit, registry capacity and late observer exhaustion. Test exact device-call count with spy backend; no effect can run twice through retry.
Phase completion: P05 gate; Astra reviews actual dispatch boundary and uncertainty paths; commit feat: preserve grpc write admission and uncertain outcomes.
P06 — Shared demand, streams and scalable delivery
Owner/model: Terra/xhigh; Sol/high may own disjoint codec tests only. Depends: P03/P04. Unit gate: R new P06 only; C GrpcDataCompactCodecTests only if protocol revisions were necessary and recorded.
Task 13: Authorized shared demand and filter composition
Create: Runtime/DataConnections/UpstreamDemandRegistry.cs, SubscriptionSettingsAggregator.cs, ClientFilterState.cs; R GrpcDataDemandSharingTests.cs, GrpcDataFilterAggregationTests.cs.
Use full source/serving-node/target-mode/consistency/upstream principal/export/epoch key; per-view permissions remain independent. Ref-count interests and pin active metadata. Fastest admitted interval plus union status/timestamp triggers defines upstream demand; shared upstream value deadband defaults off. Stronger add/modify awaits origin acknowledgment plus snapshot/version boundary; only safe relaxation can debounce. Unsupported filters return revised/rejected settings. Last consumer removes demand; a narrow consumer cannot inherit broad permissions.
Red/green: many Raw/UNS views one source, different principals/pinned nodes isolated, reference removal, burst settings churn, slow/no-deadband and fast/high-deadband consumers, independent baselines, upstream inability and pending stronger settings. Use fake-time tests, no sleeps to simulate workload.
Task 14: Subscription state machine, ordered writer and fair scheduler
Create: Runtime/DataConnections/DataSubscriptionSession.cs, DeliveryScheduler.cs, SubscriptionItemHandleTable.cs, SnapshotAdmissionScheduler.cs; R GrpcDataStreamOrderingTests.cs, GrpcDataDeliveryFairnessTests.cs, GrpcDataRecoveryAdmissionTests.cs.
Implement open/add/modify/remove command acknowledgments, item revisions, epoch-scoped never-reused numeric handles, snapshot begin/chunks/end and control-only watches. Bounded inbound commands, source-dirty deduplication, fair resumable due-interest traversal and one serialized writer per stream. Encoding happens off owner locks; writer checks current authority/leases/revisions. Baseline changes only on successful bounded send; re-evaluate candidates prepared against old baseline. Reserve control/write resources without data starvation. Coalescing/reset visible; slow-client pressure cannot delay actors. Snapshot admission limits lifetime/bytes/concurrency/bandwidth globally/per principal; jittered retry guidance prevents reconnect/reset storms.
Red/green: removal ack barrier, remove/re-add with old frame, revocation while encoding, hot tag plus quiet tags and abusive command stream, multi-partition slow snapshot, canceled snapshots release memory, failed writer resets baseline/epoch, heartbeat expiry, overflow control survives full values queue, late old-epoch frame ignored, bounded timers/work indexes. No unit throughput threshold.
Phase completion: P06 gate and Astra stream/scaling review; commit feat: add bounded shared grpc subscriptions and delivery scheduling.
P07 — TLS server and node/cluster clients
Owner/model: Terra/high; coordinator owns Host wiring. Depends: P04/P05/P06. Unit gate: H new P07 plus DriverWriteBoundsBindingTests, ApiKeyStoreRegistrationTests and ApiKeyAuthenticationTests; R new P07 plus WarmRoleAuthorityTests and PrimaryGatePolicyTests only if role adapter changes.
Task 15: Dedicated TLS ingress and admission
Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/DataConnectionSecretProvisioner.cs, DataConnectionCredentialResolver.cs, DataConnectionSecretChangeHandler.cs, DataConnectionTrustPolicy.cs; H GrpcDataSecretProvisioningTests.cs, GrpcDataSecretRotationTests.cs, GrpcDataCredentialRetirementTests.cs, GrpcDataTrustRotationTests.cs; src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/DataConnectionGrpcService.cs, Configuration/DataConnectionOptions.cs, DataConnectionOptionsValidator.cs, DataConnectionAuthInterceptor.cs, DataConnectionEndpoints.cs; H GrpcDataListenerTests.cs, GrpcDataIngressAuthTests.cs, GrpcDataServiceDispatchTests.cs.
Modify: Host Program.cs, Configuration/SecretsRegistration.cs, Configuration/SecretChangeSignallingInvalidator.cs, Configuration/MeshGrpcEndpoints.cs only for explicit separation/admission census, Host .csproj if generated-type references require it.
Bind dedicated HTTP/2 HTTPS endpoint with certificate validation and explicit listener/host admission. Feature disabled maps no service. Dev cleartext is explicit loopback only; Docker gate uses TLS. Existing command/telemetry h2c listeners cannot reach data service. Use target-issued API key, current snapshot and mapped operation grants; cookie fallback must not redirect an authenticated gRPC call. Missing key/pepper/authority fails closed. Ingress invokes the P04 local capability/catalog/read services and P05/P06 execution ports, not actor resolution.
Provision outbound credentials independently on EACH declared consumer node (including both replicas when both may consume). The existing store is local SQLite by default; opt-in secret replication is pair-local and never a cross-cluster provisioning mechanism. Add an explicit local provisioning source: an operator/orchestrator mounts a restricted versioned credential bundle per consumer, naming connection/secret reference, expected owner cluster and consumer NodeId; a Host provisioner validates current node execution scope, reference allowlist and monotonic version, then writes through ISecretStore/cipher/cache-invalidator using the existing production secret write pattern. Do not place presented credentials in central ConfigDb deployment artifacts or use an unauthenticated secret-upload endpoint. Fixture seeder mints the target key and installs it into every intended consumer’s mounted bundle/local store; populating CONTROL alone is insufficient. Missing/undecryptable/wrong-scope secret leaves that connection AuthFailed/unready. Restart uses the persisted local store; bundle removal alone is not a revocation protocol.
Handle atomic local bundle replacement and existing store invalidation signals; retire cached outbound credentials/channels, cancel affected streams and establish fresh epochs after a version change. Compose invalidation with existing consumers rather than replacing their handler. Rotate in order: deploy/activate new target key and grants, provision each consumer, verify new-key sessions/readiness, then revoke the old target key and deploy authority. An in-flight mutation retains its original operation/outcome observer; rotation cannot replay or relabel it. Target-side key expiry/revocation remains enforced on quiet streams. On scope/connection removal, drain and deny credential use before retiring its local cached reference; delete only after no other allowed connection references that secret. Versioned explicit deletion/tombstones prevent old bundles from resurrecting retired material after restart. Removing a consumer from the artifact allowlist also prevents reimport from a stale mounted bundle. In-flight dispatched writes retain only their bounded outcome observer, never a live route that can launch a replacement.
TLS trust/certificate lifecycle is separately versioned: stage new trust on every eligible consumer, rotate the target server certificate with valid SAN identity, retire affected channels and renew streams/epochs, then remove the old CA/trust entry after verifying migration. Use a rolling server restart if the current Kestrel configuration cannot safely reload its certificate; do not invent unsupported hot-reload behavior. Trust policy fingerprint is part of channel identity. Wrong/removed CA/SAN cannot trigger permissive fallback, and trust removal invalidates old connections. Include unit policy tests plus real old/new-CA and server-cert rotation in D14.
Tests cover offline DB-less restart, consumer missing secret, wrong node/bundle, encrypted-store failure, independent-pair isolation, version downgrade rejection, both consumer replicas provisioned and rotation while reads/streams/write outcome tracking are active. Test invalidation does not break existing Galaxy secret-change behavior. Add a P10 real secret-store/provision/rotate wire case. Unary deadlines/stream command timeouts use distinct policies. Disable configured automatic write retries/hedges.
Red/green: wrong listener/cookie policy, missing/revoked/expired key, method-prefix bypass, oversized message, partial authorized batches, caller claims wrong source identity, disabled surface, insecure nonloopback rejection, timer expiry on quiet stream. Create tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DataConnections/GrpcDataTlsWireTests.cs and GrpcDataListenerIsolationWireTests.cs for real TLS/cert/listener tests, tagged Feature=GrpcData, Category=Integration. Run those in P10, alongside existing MeshGrpcAnonymousMetadataTests, MeshGrpcAuthStackTests and MeshGrpcBoundaryTests in that integration project. They are not misclassified as H unit tests.
Task 16: Remote connection manager and failover
Create: Runtime/DataConnections/GrpcDataConnection.cs, DataConnectionManager.cs, ClusterEndpointSelector.cs, DataConnectionRoleAdapter.cs; R GrpcDataConnectionClientTests.cs, GrpcDataEndpointSelectionTests.cs, GrpcDataConsumerScopeTests.cs.
Modify: Runtime ServiceCollectionExtensions.cs and Host registration.
Reuse bounded cached channels per endpoint/trust identity; authenticate service principal and verify target node/cluster/major capability. Node target stays pinned; secondary reads require explicit consistency/readiness and never borrow local cache. Cluster target selects authenticated ready primary; queries/read retries keep original budget and renew identity after endpoint changes. Mutation failover only per explicit RejectedBeforeDispatch item. Source/driver role follows existing Warm/Hot verdict; no new election or stronger fencing claim. Consumer scope removes/drains mappings, connections and handles; same-cluster access between processes still uses gRPC, not actor/DPS selection.
Red/green: stale primary hint, no-ready-primary backoff, pinned secondary write rejection, no partner substitution, mixed batch lost reply, demotion/new epoch, channel retirement, TLS identity mismatch surfaced through client seam, consumer A-only scope and unknown-role startup. Reconnect no pending write replay.
Phase completion: P07 gate includes new provisioning/rotation tests and SecretsPreHostContentRootSourceTests; commit feat: host tls data service and route node and cluster connections.
P08 — Raw/UNS browse, metadata projection and runtime adapter
Owner/model: Terra/high, Astra/xhigh review. Depends: P07/P02. Unit gate: R new P08; O new P08 plus NodeManagerAclEnforcementTests, NodeManagerModelChangeFilterTests, NodeManagerModelChangeOnAddTests, NodeManagerSurgicalRemoveTests, AddressSpaceApplierRawUnsTests, AddressSpaceApplierHierarchyTests and UnsReferenceBackingParityTests; K mapped-subject tests if placement logic changes.
Task 17: Federated catalog resolver and wrapped continuations
Create: Runtime/DataConnections/RemoteNamespaceResolver.cs, FederatedBrowseService.cs, ProjectedCatalogCache.cs, BrowseContinuationRegistry.cs; R GrpcDataFederatedBrowseTests.cs, GrpcDataContinuationTests.cs.
Compose origin-local exports with configured realm-specific mapped anchors at local client-facing view. Origin peer ingress never re-exports projections. Authorize every returned name and operation, bind handles to mapping/origin/epochs, preserve deterministic ordering, public metadata and same-path realm separation. Bounded one-parent pages with explicit completeness/failure, shared authenticated browse work and bounded waiter cancellation. Metadata pinning versus idle eviction; active lease expiry hides remote-derived names even without subscriptions. Preserve origin-local address and any explicit source-equivalence proof.
Red/green: Raw Folder/Driver/Device/TagGroup/Tag and UNS Area/Line/Equipment/signal; nested folders, renamed roots, collisions, pagination while revision changes, peer unavailable versus empty, revoked queued pages, wrong-principal token reuse, lease loss and unknown handles. Returning a successful empty list for pending/unreachable is a failing test.
Task 18: SDK per-parent materialization and gRPC import adapter
Create: Runtime/DataConnections/GrpcDataDriverAdapter.cs, ProjectedNamespaceApplier.cs; O GrpcDataSdkBrowseReadinessTests.cs, GrpcDataSdkMappedAccessTests.cs; R GrpcDataAdapterLifecycleTests.cs.
Modify: OpcUaServer/AddressSpaceComposer.cs, AddressSpaceApplier.cs, OtOpcUaNodeManager.cs, Security/DataPlaneAuthorizer.cs, Core/Hosting/DriverFactoryRegistry.cs/Host registration only where the existing factory extension pattern requires it; no concrete driver dependency added to Runtime.
Reuse shared desired subjects/catalog and existing sink interfaces. Background per-parent first-page population, explicit BadWaitingForInitialData while loading; BrowseNext preserves pending page identity or uses an explicitly tested continuation strategy. BadNoCommunication for unreachable/expired branch, BadContinuationPointInvalid on identity/revision loss. No network I/O under SDK Lock. Authorize mapped Browse/Translate/Read/Write/monitored-item creation/delivery even with legacy ACL switch off. Apply filtered model changes. Adapter supplies values through ordinary dependency/publication paths with rich Unknown/Unconfirmed outcomes and distinct connection quality. Same-cluster A-only mount creates no B reciprocal import or duplicate field driver. Deactivation removes nodes and demand.
Red/green: pending page never appears empty, continuation retry without cursor consumption, root/child paging, map removal with active monitored items, local+remote authority composition, same-path Raw/UNS reads, mapped unknown outcome self-correction, Warm/Hot consumer role transitions, dependency fan-out and no optimistic-cache pollution.
Phase completion: P08 gate; Astra checks SDK lock boundaries and actual continuation behavior before Docker gates. Commit feat: expose mapped raw and uns data through grpc adapters.
P09 — Authoring, diagnostics and operational controls
Owner/model: Terra/high for UI/service work; Sol/medium docs; apply applicable UI skill during execution when touching interface design. Depends: P08. Unit gate: U new P09 and AclScopeServiceTests, RawBrowseCommitMapperTests and RawBrowseCommitMapperParityTests; no unrelated UI/Host project tests.
Task 19: Author connection/export/mapping configuration and inspect health
Create: AdminUI/DataConnections/DataConnectionAdminService.cs, DataConnectionEditorModel.cs, RemoteMappingEditorModel.cs, DataConnectionDiagnostics.cs; AdminUI/Components/Pages/Clusters/ClusterDataConnections.razor; U GrpcDataAuthoringTests.cs, GrpcDataEditorValidationTests.cs, GrpcDataDiagnosticsTests.cs.
Modify: Raw/UNS authoring integration and cluster navigation following current component/service patterns; Configuration snapshot authoring as needed, explicitly coordinator-owned.
Provide disabled-by-default create/edit directional connection, consumer set, realm parent/root picker, export/mapping grants, secret reference and optional write-through settings. Existing schema validator remains authoritative server-side. No raw secrets in rendered summaries/logs. Display disabled/connecting/ready/degraded/auth-failed/backoff, selected endpoint/role/provenance, metadata lease/sample age, source/registration/stream counts, reset/overflow and write outcome metrics with bounded labels. Test draft versus deployed state so saving does not falsely indicate active imports. Local readonly/audit permissions cannot author or enable writes. Add rollout/disable controls through existing deployment flow; do not add a general unauthenticated config-upload endpoint.
Phase completion: P09 gate; update docs/Configuration.md, docs/Raw.md, docs/Uns.md, docs/security.md, docs/ReadWriteOperations.md (correct stale ACL claims), docs/Redundancy.md, docs/README.md; preserve future-only history/alarms/relays boundary. Commit feat: author and inspect grpc data connections.
P10 — Hermetic local Docker and OPC UA CLI integration
Owner/model: Sol/high fixture worker; coordinator executes gates; Terra/high fixes production defects within their phase ownership. Preparation: scaffold harness after P01, fill in seeder/probes only after dependent contracts exist. Units: harness/process/parser tests only plus applicable production classes for any fix. Integration: named local wire/CLI scenarios, not a full solution run.
Task 20: Standalone local rig and production-path fixture seeder
Create: scripts/validation/grpc-data/compose.yml, Dockerfile, rig.py, seed/GrpcDataSeed.csproj, seed/Program.cs, device/compose-service.yml or equivalent build context, device/server.py, tests/test_rig.py, README.md.
Reuse/read: docker-dev/Dockerfile, docker-dev/docker-compose.yml, docker-dev/seed/entrypoint.sh, scripts/validation/pair-gate/Program.cs, Host/Configuration bootstrap guards, AdminUI/Api/DeployApiEndpoints.cs.
Use a STANDALONE compose file with a unique project/run label, image, network and volumes, no fixed container_name. Do not inherit base rig endpoints: it has LDAP/SQL/Modbus/opc-plc references to 10.100.0.35. Require local Docker socket/npipe/context and reject SSH/TCP remote daemon/context; validate resolved compose/seed has no shared-host endpoints. Bind host test ports to loopback with collision-safe allocation recorded in a manifest. Use current server image build graph/SourceRevisionId, not a stale historical image. Include local SQL and software source fixtures. Admin login is disabled for the local machine API gate; LDAP options must still satisfy startup validation using isolated unused local settings or a local fixture if code actually needs directory access.
Topology: two independent data pairs SOURCE (source-1/2) and CONSUMER (consumer-1/2), plus a separate CONTROL pair where existing deployment requires admin/controller independence. Six processes preserve controller-outage testing without stopping either data pair. Register self-first seeds/cluster roles exactly per pair, ConfigServe/command/telemetry gRPC and DB-less fetch/cache on data nodes, separate LocalDb volumes/sync for each pair as applicable. Data gRPC has its own generated private CA and per-node SAN certificates, mounted only into this rig; clients must trust this CA and verify service identity. Keep cert/key files restricted and out of Git.
Provision target-issued outbound credentials into each eligible consumer’s local store/bundle through the P07 provisioner before declaring ready; verify the unprovisioned-node negative case and key rotation across the independent pairs.
Use a deterministic local writable Modbus software source (adapt the repository fixture engine and version), with changing counter, scalar types, array/bytes samples where supported, readback and a test-only request/effect ledger exposed inside the isolated rig. Its control API is fixture-only, never production Host. Every source driver points at local service DNS, not another server’s OPC UA endpoint; otherwise a fallback bridge could hide a missing gRPC path. Seed existing Raw and UNS local references plus both remote realm mappings, connection consumer scopes and keys using EF/production authoring validators and ApiKeyAdminService patterns from PairGate. Save presented data keys only to 0600 credential files; secrets do not enter artifact manifests or evidence logs. A normal deployment must compose/seal/apply; direct SQL rows alone are not an active runtime.
Manifest contains dynamically allocated host/container endpoints, expected cluster/node/application identity, TLS public trust paths, namespace URIs and resolved NodeIds, Raw/UNS aliases and origin paths, credential-file paths, fixture control endpoints and exact owned container labels. Do not assume ns=2/3 across servers. Seeder/Client.Shared helper reads NamespaceArray and validates each resolved URI.
Commands to implement and then use:
python3 scripts/validation/grpc-data/rig.py preflight
python3 scripts/validation/grpc-data/rig.py up --run-id grpc-data-local
python3 scripts/validation/grpc-data/rig.py seed --run-id grpc-data-local
python3 scripts/validation/grpc-data/rig.py deploy --run-id grpc-data-local
up builds once, migrates disposable SQL, starts pairs and waits on explicit health/role/catalog readiness with bounded deadlines. deploy POSTs only { "createdBy": "grpc-data-local-gate" } to /api/deployments with X-Api-Key, follows returned Location and waits for every intended node's apply outcome/revision; reject degraded/unexpected states. Handle no-change/in-flight responses explicitly. The endpoint DOES NOT accept artifact.json/configuration uploads. If desired, invoke existing docker-dev/deploy.sh with overrides only for its bodyless POST semantics; it does not replace per-node apply polling.
Units: subprocess argv/timeout enforcement, remote-context/shared-host rejection, schema/port manifest validation, ownership-only cleanup, redaction, readiness/deploy polling, atomic secret files and interrupted setup cleanup. Actual migrate/reapply model snapshot test runs here against disposable SQL. Remove feature config/disable on rollback, not destructive down migration of active production data.
Task 21: CLI process harness and direct gRPC probe
Create: scripts/validation/grpc-data/cli_gate.py, processes.py, tests/test_cli_gate.py, probe/GrpcDataProbe.csproj, probe/Program.cs, scenarios.json; add narrowly scoped tests to L only if CLI behavior itself changes.
Publish CLI once and execute the built DLL with dotnet, portable on macOS/Linux/Windows; no .exe-only locator and no dotnet run rebuild inside each background process. Process harness passes argv, enforces real timeout, captures stdout/stderr/exit, bounds retained output, cleans up subscribers/containers in finally and redacts credentials from command/evidence logs. Existing --summary-file is text, not JSON. Read/write bad statuses and individual subscription failures can still return exit 0: assert status/content and notification evidence, not merely successful process exit. Missing summary/output fails gate.
Exact existing CLI command shapes (the harness substitutes from the manifest and never logs the key):
dotnet publish src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ZB.MOM.WW.OtOpcUa.Client.CLI.csproj \
-c Release -o artifacts/grpc-data/tools/cli
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll connect -u "$gate_url" -U apikey -P "$gate_key"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll browse -u "$gate_url" -U apikey -P "$gate_key" \
-n "$gate_root" -r -d 6
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll read -u "$gate_url" -U apikey -P "$gate_key" -n "$gate_leaf"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll write -u "$gate_url" -U apikey -P "$gate_key" -n "$gate_leaf" -v 42
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll subscribe -u "$gate_url" -U apikey -P "$gate_key" \
-n "$gate_leaf" -i 200 --duration 30 --summary-file "$gate_summary"
dotnet artifacts/grpc-data/tools/cli/ZB.MOM.WW.OtOpcUa.Client.CLI.dll redundancy -u "$gate_url" -U apikey -P "$gate_key"
The current csproj has no AssemblyName override, so the DLL above is its project assembly name; verify/record the published output in the manifest. The operator-facing command name otopcua-cli in docs is not sufficient evidence to guess an assembly filename. All shown $gate_* variables are task-specific, sourced by the harness from the restricted manifest/credential file, not globals or mandatory user-provided values. Integration code calls the process wrapper directly rather than shell-interpolating these examples. Use local OPC UA security mode supported by the seeded test endpoint; gRPC TLS trust must be verified independently because the CLI's automatic UA certificate acceptance is not proof of gRPC trust.
Probe uses generated client and fixture fault controls to inspect semantics CLI cannot expose: capability/compact negotiation, actual serving node/role/epoch, source/delivery metrics, per-item write outcome and device-call count, stream acks/handle revisions, continuation reuse and byte limits. CLI remains the required end-to-end consumer evidence. Do not add nonexistent JSON/BrowseNext CLI flags; test unsupported observability through probe/Client.Shared SDK fixture.
Units: parser fixtures include exit-0 Bad read, write failure, zero attached subscriptions/no notifications, absent summary, late process kill and redaction; probe pure scenario-result validation/CLI fake process tests run in P10. TLS and processes are integration. Create tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/DataConnections/GrpcDataSdkWireTests.cs for wire-level mapped Browse/BrowseNext/ACL and readonly namespace behavior, tagged Feature=GrpcData and Category=Integration; use these cases where the CLI cannot inspect continuation controls directly. P10 runs the new Host/OPC UA integration classes and the named existing mesh regression classes only, with nonzero per-class results.
Task 22: Required scenario matrix and reproducible evidence
Implement python3 scripts/validation/grpc-data/cli_gate.py --run-id grpc-data-local --scenario all and dotnet run --project scripts/validation/grpc-data/probe/GrpcDataProbe.csproj -c Release -- --manifest <restricted-manifest> --scenario all (publish probe once inside runner). Each scenario has a nonzero assertion count, prerequisites, hard timeout, restoration and result row. all excludes no required feature scenario silently. Software fixture gaps are implementation failures, not successful skips.
| ID | Setup/action | Required assertions |
|---|---|---|
| D01 | Deploy/read identities for CONTROL/SOURCE/CONSUMER | Each pair sees only its two Akka members; source/consumer processes differ; active roles/capabilities ready; all apply revisions match |
| D01a | Direct gRPC client uses a permitted connection with no namespace mount | Capabilities/browse/metadata/read/write/subscribe work against origin-local facade; no mapped anchor is required; policy/role/outcome behavior matches mounted access |
| D02 | Same-cluster source-1→source-2 Node mapping with consumer scope source-1 only | Actual TLS gRPC calls for browse/read/subscribe; source-2 no reciprocal mount; serving source-2 provenance; no local-cache substitution; primary-only write tested when source-2 is primary |
| D03 | Cross-cluster Raw and UNS mounts | CLI recursively expands each hierarchy; read status/value/timestamps match origin; duplicate bare paths remain distinct; new child, rename/removal and public metadata behavior |
| D04 | Subscribe before controlled source changes; write through both realm views | Nonzero changing Good notifications per expected node; CLI write acceptance plus eventual origin/simulator readback; remote effect ledger exactly one expected effect and correlated trace; no duplicate field acquisition |
| D05 | Cold parent/page limit and mapping revision while browsing | Pending versus empty/unavailable distinguishable; bounded paging; BrowseNext cursor/revision invalidation via SDK/probe; no network call under SDK lock instrumentation |
| D06 | Local denial, narrower target export, revoked/expired key with live stream | Denied names absent; explicit read/write denial; no post-emission-gate unauthorized payload; quiet browse lease expires within negotiated bound; anchors only as locally authorized |
| D07 | Fault only remote gRPC path / stop selected SOURCE primary | Hold CONSUMER OPC UA CLI URL fixed, omit -F; observe connection quality gap, new selected SOURCE primary/epoch and fresh snapshot before recovery; no ambiguous write replay |
| D08 | Drop response after effect; mixed-success batch; same-ID duplicate/restart | Probe receives Unknown or definitive observed result as appropriate; no double effect; only proven-undispatched items eligible for endpoint switch; stale-incarnation request rejects |
| D09 | Pinned secondary / Warm standby and demotion | Own-node cache only when explicit policy/readiness allows; pinned secondary write rejected; no automatic partner substitution; Warm activation/catch-up honest; new epoch on role/incarnation change |
| D10 | Separate OPC UA client failover | Now use -F "$partnerUrl"; distinguish UA reconnection from server-side gRPC failover using serving/connection evidence; consumers configured on both nodes where this test requires it |
| D11 | Stop CONTROL pair only | SOURCE/CONSUMER continue local and cross-cluster data from applied artifacts; controller restart reconciles without replay; no SQL dependency on driver-only nodes |
| D12 | Remove consumer scope, export or mapping; disable feature and redeploy | Projected nodes/demand/handles removed, pinned scope does not migrate; writes rejected; ordinary local Raw/UNS and existing command/telemetry still function |
| D13 | Wrong CA/SAN, bad API key, wrong listener and raw peer re-export | TLS/identity/auth rejects; no data service via h2c/HTTP UI listener; mounted peer data cannot be recursively exported; shared-context access never crosses principals |
| D14 | Consumer secret absent/provisioned/rotated, trust rotation and retirement, offline restart | Missing secret unready; each eligible replica gets scoped material; new-key sessions ready before old-key revocation; new/old CA/server-cert trust behavior proven; retired scope cannot reload stale bundle; restart DB-less consumer with CONTROL pair stopped and verify persisted store; in-flight write tracking persists without replay |
The scenario file must expand coverage over this explicit matrix; a passing cross-cluster Cluster case does not substitute for Node-target or intracluster behavior:
| Target mode / topology | Raw + UNS browse/metadata | Read + subscribe | Write | Additional boundary |
|---|---|---|---|---|
| Node target, same-cluster A→B | D02 in both realms | D02 with pinned ready/secondary policy | D02 direct-runtime positive when B is primary; negative when B secondary | A-only scope; A’s local sample cannot substitute B |
| Node target, cross-cluster CONSUMER→one SOURCE node | D03/D05 configured as Node | D03/D04, no partner substitution under D09 | D04 positive to primary and rejection to secondary | D07 fault leaves pinned target unavailable rather than selecting partner |
| Cluster target, cross-cluster CONSUMER→SOURCE pair | D03/D05 configured as Cluster | D04/D07 primary selection and fresh epoch | D04/D08 per-item outcome rules | Disable/maintenance selected endpoint and prove exclusion; no unintended source selected |
| Direct API, no mounted view | D01a for Node and Cluster target (same-cluster direct Node included) | D01a | D01a with primary enforcement | No namespace projection is a prerequisite |
| Consumer failover | D10 mapping on both eligible consumers, then active consumer changes | Fresh demand/epoch on newly eligible node | Authority/role checks maintained | Repeat with A-only scope: mount must remain absent on nonconsumer, never auto-migrate |
Use a rig-only node probe/hosted command runner to execute IDataConnection through the ACTUAL caller node’s DI/connection manager for direct API cases; an external client connecting straight to B is not evidence of A→B execution. Create scripts/validation/grpc-data/node-probe/GrpcDataNodeProbe.csproj and register its bounded file-command/result runner only in the isolated fixture composition. No production network/control endpoint is added; normal Host startup tests assert the fixture runner is absent outside the test composition. Commands and output are restricted to the run-owned mounted directory and use redacted connection IDs, not bearer secrets. The CLI still proves mounted browse/read/write/subscribe in D03/D04 and other user-facing cases.
Within one redundant pair, A cannot be local OPC UA Primary while B is the target Primary simultaneously. Preserve the local SDK primary gate: positive same-cluster A→B write is a direct runtime-API test from A to active B; a CLI write to A’s standby mapped node correctly rejects locally. Test both behaviors rather than weakening either role gate to manufacture a positive mapped write.
For D02 role setup, wait for stable selected primary after planned local failover or choose the manifest's roles rather than assuming a host suffix is primary. For D09 use a scoped Warm compose/config variant; restore before subsequent cases. Network faults target only recorded run-owned containers/sidecars and specific gRPC ports; do not disconnect the developer's host or shared fixture network. Device-command ambiguity needs a fixture-controlled response-loss point, not hoping a random kill happens after commit. Restore faults deterministically in finally.
Write evidence under artifacts/grpc-data/<run>/: redacted resolved compose/manifest, build SHA/image revision, topology/role observations, deploy/apply statuses, CLI stdout/stderr/summaries and process exit/timeouts, gRPC probe assertions/metrics, simulator effect ledger, fault timeline, logs and scenario results JSON. Do not store credential-bearing manifest, private keys or raw values beyond safe fixture values in committed docs. Generate a concise tracked evidence index with hashes/paths, not a copy of large logs. Cleanup command: python3 scripts/validation/grpc-data/rig.py down --run-id grpc-data-local removes ONLY that manifest's project/resources; retain logs before cleanup.
Phase completion: run python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_rig.py' and the same command with -p 'test_cli_gate.py'; then the explicit P10 Host/OPC UA integration class selectors plus D01–D14 (including D01a)/direct probe gates. Astra reviews topology/auth/outcome evidence. Fix defects through the owning phase's applicable class filters, then rerun affected integration scenarios only. Commit test: validate grpc data connections with local docker and opcua cli.
P11 — Scaling and controlled recovery evidence
Owner/model: Sol/high harness owner; Terra/xhigh for confirmed concurrency defects. Depends: P10. Unit gate: only new P11 workload/report tests plus affected P04/P06 classes if code changes. No full suite yet.
Task 23: Parameterized workload and fairness experiments
Create: scripts/validation/grpc-data/workload-profiles.json, workload.py, tests/test_workload_results.py; extend existing probe with measured workload commands and source metrics. New R GrpcDataWorkloadAccountingTests.cs only if production accounting is added.
Independently vary unique sources, shared Raw/UNS views, client count, registrations/overlap, scalar/string/array payloads, source rates, intervals/deadbands, worker count, compact/fallback encoding and slow consumers. Include burst/concurrent browse, quiet/hot source fairness, same-origin different pinned node/principal isolation, stalled writer/slow snapshot, producer array reuse, ingress saturation, revocation during encoding and reconnect storm after active-node loss. Run source/readiness invalidation and catch-up memory scenarios with deterministic assertions before performance sampling.
Command: python3 scripts/validation/grpc-data/workload.py --run-id grpc-data-local --profiles scripts/validation/grpc-data/workload-profiles.json. Profiles contain small smoke, steady, fan-out and recovery settings, not mandatory advertised capacity numbers. Record actual hardware/container limits/architecture, TLS/compression, source versus delivered rate, bytes/sample, CPU per actor/partition, retained/pool/cache bytes, GC, scheduling p95/p99 and snapshot/recovery time. Compare configurations on identical input. Success means correctness, bounded resource/fairness invariants and useful measurements; no fabricated universal throughput threshold. Detect leaks only against agreed steady-state accounting/tolerance for the specific run, not fragile instantaneous GC values. Record current DriverHostActor/SDK serialization ceilings explicitly.
Phase completion: python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_workload_results.py', P11 applicable dotnet classes (only if added/changed), and workload evidence; tune quotas only from the measured envelope, preserving single-member failover headroom and user-configurability. Astra reviews any change to snapshot/sharing/ordering contracts before release. Commit perf: measure grpc fanout and recovery bounds only if code/harness changes justify it.
P12 — Final whole-solution verification and integrated Astra review
Owner/model: coordinator Sol/high; independent Astra/xhigh final review. Depends: all prior phases clean. This is the only whole-solution test phase.
Task 24: Provide local prerequisites and run the full tests
Modify: scripts/ci/integration-fixtures.sh to support explicit local-owned mode and restricted JSON environment output without printing credentials; preserve the current CI invocation outside this opt-in mode. Local mode requires a nonempty unique CI_FIXTURE_SUFFIX, run-owner labels and manifest, local socket/context validation, 127.0.0.1 collision-safe published ports, per-run NAT certificate/output paths and label-verified cleanup. Reject startup when a matching name already exists without the same owner; never call its current name-only down/rm -f on unverified resources. Retries may remove only the container they just created. Validate resolved target host is loopback in local mode. Add fake Docker subprocess tests for pre-existing foreign names, parallel runs, failed setup/port retries and selective teardown before any real startup. Create: scripts/validation/grpc-data/run-final.py, tests/test_final_runner.py, tests/test_fixture_ownership.py, fixture-environment-map.json. P00 unit-only discovery mode is explicitly removed from final children and real local fixture variables are required before test discovery.
Inventory EVERY test project and its category/env/fixture gates before the final run. Reuse throwaway local SQL/PostgreSQL/QuestDB/opc-plc/NAT fixture patterns in scripts/ci/integration-fixtures.sh, plus required new local rig, optional local Host real-mode deps and any additional runnable software fixture. Pass a mandatory per-run suffix/owner, local mode, loopback host override and isolated certificate directory when invoking this helper; it must share the same preflight/ownership rules as the new rig. Its current empty local suffix and wildcard port bindings are not safe defaults for this gate. This helper emits exports and does not modify its parent environment: run-final.py must parse the restricted structured output into an allowlisted child environment, not run bash ... up and assume dotnet inherited it. Do not eval printed logs.
Explicitly set OTOPCUA_CONFIG_TEST_SERVER and OTOPCUA_CONFIG_TEST_SA_PASSWORD from disposable SQL as well as script-provided SQL_TEST_ENDPOINT, SQL_TEST_USER/PASSWORD, SQL_PG_ENDPOINT/PASSWORD, HISTORIAN_SQL_TEST_CONNECTION, HISTORIAN_QUESTDB_TEST_CONNECTION, OPCUA_SIM_ENDPOINT, OPCUA_NAT_ENDPOINT and OPCUA_NAT_SERVER_CERT. Configuration SchemaComplianceFixture always migrates real SQL; AdminUI SQL fixture can skip on unreachable/migration errors. Neither may fall back to shared 10.100.0.35. Bind remaining registered software fixtures locally and isolate their database/table prefixes. No production/QA/shared hardware access is part of this plan.
First run only python3 -m unittest discover -s scripts/validation/grpc-data/tests -p 'test_final_runner.py' and the same discovery command with -p 'test_fixture_ownership.py' to verify final orchestration/redaction/environment/owned-fixture guards.
Record the remaining known fixture switches in fixture-environment-map.json; each maps to a run-owned local service or an explicitly unavailable platform/device, never an inherited shared endpoint:
| Suite family | Variables / prerequisite |
|---|---|
| Modbus / S7 / AB / FOCAS / MTConnect | MODBUS_SIM_ENDPOINT, MODBUS_RTU_SIM_ENDPOINT, S7_SIM_ENDPOINT, AB_SERVER_ENDPOINT, AB_LEGACY_ENDPOINT, AB_LEGACY_CIP_PATH, OTOPCUA_FOCAS_SIM_ENDPOINT, OTOPCUA_FOCAS_SIM_PROFILE, MTCONNECT_AGENT_ENDPOINT; enable AB writes only with AB_SERVER_ALLOW_WRITES=1 against disposable simulator, never a configured differential hardware endpoint |
| MQTT | MQTT_FIXTURE_ENDPOINT, MQTT_FIXTURE_PLAIN_ENDPOINT, MQTT_FIXTURE_USERNAME, MQTT_FIXTURE_PASSWORD, MQTT_FIXTURE_CA_CERT, MQTT_FIXTURE_TOPIC_PREFIX; local TLS broker/publisher required |
| Host real-mode | OTOPCUA_HARNESS_USE_SQL, OTOPCUA_HARNESS_SQL_HOST, OTOPCUA_HARNESS_USE_LDAP, OTOPCUA_HARNESS_LDAP_HOST; choose local software prerequisites explicitly rather than accidentally enabling shared default mode |
| Native/proprietary/live devices | TwinCAT TWINCAT_TARGET_HOST/NETID/PORT and native AMS runtime; Heidenhain HEIDENHAIN_LSV2_HOST/credentials; gateway HISTGW_GATEWAY_ENDPOINT/HISTGW_GATEWAY_APIKEY plus suite opt-ins. Treat absent hardware/licensing/platform cases as unavailable coverage with exact test names, not required new-feature substitutes |
| Long-running opt-in tests | OTOPCUA_SDK2_SOAK and its duration/mode/output/platform controls, plus folder-event performance controls. Explicitly decide/run software-supported opted-in cases in the final tier and record disabled long-running cases as not executed; a no-category-filter command does not automatically enable them |
Final command, executed once against the final candidate after restore/build and environment setup:
dotnet restore ZB.MOM.WW.OtOpcUa.slnx --disable-parallel -m:2
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore -m:2
AKKA_TEST_TIMEFACTOR=3 dotnet test ZB.MOM.WW.OtOpcUa.slnx \
-c Release --no-build -m:1 -p:TestTfmsInParallel=false \
--settings scripts/ci/ci.runsettings \
--blame-hang --blame-hang-timeout 20m --blame-hang-dump-type none \
--logger trx --results-directory artifacts/grpc-data/final/trx
MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh artifacts/grpc-data/final/trx/*.trx
The final dotnet test has NO category/FQN filter: include SlowUnit and all discoverable runnable integration/live categories. The documented CI command excluding E2E/LiveIntegration is useful precedent for serialization but is not a substitute for this user-requested full run. Run final orchestrator via python3 scripts/validation/grpc-data/run-final.py --run-id grpc-data-local; it owns environment setup, build, full command, TRX guard and teardown/evidence. --no-build is valid here because the same candidate was just fully built. Use the repository's architecture/platform-supported build setup; resolve genuine incompatibilities rather than excluding projects silently.
The 12,000 floor is the current baseline guard, not proof of full execution. Also require a result/discovery inventory per test assembly and every new GrpcData class, zero missing/zero-executed expected projects, zero failed tests, and zero skipped required feature cases. Parse every skipped test name/reason. Hardware/Windows/native/proprietary suites that cannot run on the local host are listed as explicit unavailable coverage with requirements and a next gate; never describe that as all tests passed. Any available software fixture that skipped unexpectedly must be fixed/enabled and its applicable project rerun. A test reaching a forbidden shared host is a gate failure, not permission to use it. Do not lower floors, remove categories, blanket retry, change assertions or mark new feature cases optional to force green.
If final fixes are required: run only the failing/affected unit and integration selectors while iterating. After the candidate changes, rebuild affected binaries and repeat the final whole-solution gate on the new final candidate; record why a repeat was necessary. Do not repeatedly run the suite when there were no changes or unresolved concerns. Docker scenario/workload evidence must reference the same candidate; rerun impacted scenarios after final changes, and run the full required CLI matrix if shared code/wiring changed.
Task 25: Final review, documentation and completion evidence
Give Astra the final diff, design/plan, phase TRX/discovery manifests, local CLI/probe/topology/fault evidence, full-suite results/skip inventory, workload report and unresolved risks. Ask for concrete correctness/security/concurrency/architecture findings, including bounds before mailboxes, immutable producer ownership, no unsafe upstream filtering, final dispatch policy, cache/provenance isolation, mapped ACL with legacy toggle off, and one-hop export boundary. Astra edits only docs/plans/evidence/grpc-data/final-astra-review.md. Coordinator resolves findings with focused tests, then applies the final-candidate rule above.
Update operational docs and docker-dev/README.md to distinguish current three-pair topology, API-key data plane and local validation rig; do not copy stale artifact-upload examples. Confirm new options/defaults, TLS trust/rotation, scoped secrets, mapping migration/consumer role behavior, rollback/disable, resource/overload diagnostics and deferred capabilities. Run documentation path checks and relevant script units only; do not invent a new full-suite rerun for prose-only changes.
Deliver commit/diff summary, exact tests/CLI scenarios executed and results, Astra disposition, measured workload configuration/results (without extrapolating capacity), required unavailable tests and any material implementation limitation. Leave a concrete reviewable branch. Pushing/merging/deploying outside the isolated local rig is not included. Remove owned temporary rigs/volumes after evidence unless explicitly needed for the user review; preserve only restricted local artifacts appropriately. No completion claim if a required feature gate failed or skipped.
Completion checklist and review questions
- Every phase has only its applicable unit filters, nonzero discovery and TRX evidence; whole solution ran at final candidate only (repeated solely for actual final fixes).
- All design invariants and Astra design findings map to named implementation tests; no Raw/UNS or intra/intercluster scope was lost for scaling convenience.
- Local Docker gate actually used separate processes and independent pairs, local-only dependencies, validated TLS and production deployment/auth paths.
- OPC UA CLI demonstrated browse/read/write/subscribe through mapped views, including steady-client remote failover; direct gRPC probes filled CLI observability gaps.
- Write acceptance versus effect/outcome, queued dispatch revalidation, replay exclusions and late results are proven; no benchmark bypasses authorization or role checks.
- Bounded ingress/work/buffers/snapshots and authority/readiness invalidation hold during pressure; no fixed capacity guarantee is inferred from optional measurements.
- Schema, feature-disabled/old artifact compatibility, nonconsumer behavior, authoring and cleanup/rollback are covered.
- Final full-test inventory, environmental omissions and independent Astra review are complete and candid.
After Astra reviews THIS PLAN, the user reviews it. Execute only after the user instructs implementation, using executeplan and the agent/test policy above.
Reference constraints verified during planning
scripts/ci/README.mdandscripts/ci/ci.runsettings: serialized solution leg, Akka time factor and existing executed floor; no rootci.runsettings.docker-dev/docker-compose.yml: actual independent MAIN/SITE-A/SITE-B pairs; base fixture references are not hermetic local dependencies.src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Api/DeployApiEndpoints.cs: trigger/status API, not config/artifact upload.src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/: actual existing CLI options and nonthrowing per-node failure behavior.scripts/validation/pair-gate/Program.cs: production API-key mint/grant/revoke and restricted credential-file patterns.- gRPC .NET sources and Microsoft performance guidance: reusable channels, bounded streams/flow control, serialized stream writes and cancellation/deadline propagation must be respected; use current official docs again when implementing API details.