40 KiB
Surgical Deploys Implementation Plan
For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
Goal: Make routine config edits non-events instead of fleet-wide outages: a raw-tag attribute edit or a UNS-reference add/remove/re-point no longer tears down the whole address space (killing every client subscription), the artifact is read from SQL once per apply instead of three times, fully JSON-parsed once per actor instead of ~8–11 times, only the delta is materialised instead of the whole composition, blocking apply I/O leaves the hot actor mailboxes, and the artifact blob is deflate-compressed inside the hashed bytes.
Architecture: Three prongs. (1) Surgical routing: AddressSpaceChangeClassifier learns that a non-alarm RawTagDelta and any UnsReferenceDelta are surgical-eligible; AddressSpaceApplier applies raw-tag attribute edits in place via the existing ISurgicalAddressSpaceSink.UpdateTagAttributes (Raw realm) and handles UNS-reference deltas as remove(previous-node)+recreate(current) — the recreate rides the delta materialise pass. Alarm-bearing raw tags keep the full-rebuild path (the WP4 notifier-wiring guard). (2) Parse/read-once: Direct mode threads the already-read blob into the rebuild + subscription pushes (the FetchAndCache path is the template, DriverHostActor.cs:1650-1654); a new DeploymentArtifact.Open() returns a ParsedDeploymentArtifact that parses the JSON once and serves all consumers; the materialise passes gain subset overloads so non-rebuild applies materialise only the plan's added/changed nodes. (3) Mechanics: the LocalDb cache write leaves the mailbox via PipeTo, the Applying-state dispatch deferral stops busy-spinning, and ConfigComposer emits a deflate-compressed blob (magic-sniffed on the read side, hash computed over the compressed bytes so RevisionHash == SHA-256(stored/wire bytes) stays true).
Tech Stack: C# / .NET 10, System.Text.Json (JsonDocument), Akka.NET, System.IO.Compression, xUnit + Shouldly.
Branch: perf/surgical-deploys (branched off perf/value-pipeline-batching — plans 1 and 2 touch the same actors; do NOT branch off master).
Context for implementers (read before any task)
Verified facts — do not re-derive:
- The rebuild cliff:
AddressSpaceChangeClassifier.Classify(src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceChangeClassifier.cs:69-78) routes ANY non-emptyChangedRawTags/ChangedUnsReferenceVariables(among others) toRebuild.RawTagPlan.ReferencingEquipmentPathsparticipates in record equality (AddressSpaceComposer.cs,RawTagPlan.Equals), so adding/removing aUnsTagReferenceflips the backing raw tag to Changed ⇒ full rebuild. The WP4 comment block (AddressSpaceChangeClassifier.cs:62-68) is the binding guard: any surgicalChangedRawTagspath for ALARM tags must re-wire notifiers — we sidestep it by keeping alarm-bearing deltas on the Rebuild path. - The applier already has every surgical primitive:
ISurgicalAddressSpaceSink(src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/ISurgicalAddressSpaceSink.cs) —UpdateTagAttributes(nodeId, writable, historianTagname, dataType, isArray, arrayLength, realm)(realm-aware, returns false ⇒ caller rebuilds),RemoveVariableNode,RemoveAlarmConditionNode,RemoveEquipmentSubtree,UpdateFolderDisplayName. The applier's surgical branch isAddressSpaceApplier.Apply(AddressSpaceApplier.cs:151-224); its false/throw ⇒ full-rebuild ratchet must be preserved for every new surgical operation. - Apply flow (publish actor):
OpcUaPublishActor.HandleRebuild(OpcUaPublishActor.cs:361-512): load blob (msg.Artifact wins; else ConfigDb) →FeedDataPlaneAuthority(artifact)(MUST stay ahead of theplan.IsEmptyreturn — #520 stage 4) →ParseComposition→AddressSpacePlanner.Compute(_lastApplied, composition)→plan.IsEmptyreturn →_applier.Apply(plan)→ six full-composition materialise passes (:456-480) →AnnounceAddedNodeswhen!outcome.RebuildCalled._lastAppliedadvances only after Apply. - Apply flow (host actor): Direct mode
ApplyAndAck(DriverHostActor.cs:1680-1754):ReconcileDrivers(deploymentId)reads the blob from SQL and RETURNS it (appliedBlob), thenRebuildAddressSpace(correlation, deploymentId)(NO bytes ⇒ publish actor re-reads SQL atOpcUaPublishActor.cs:578-593) andPushDesiredSubscriptions(deploymentId)(re-reads SQL again). FetchAndCache modeHandleFetchedForApply(:1609-1674) already does it right:ReconcileDriversFromBlob(id, bytes)→RebuildAddressSpace(corr, id, appliedBlob)→PushDesiredSubscriptionsFromArtifact(id, appliedBlob)→CacheAppliedArtifact. - Parse entry points (
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs):ParseDriverInstances(ReadOnlySpan<byte>):89,ResolveClusterScope(ReadOnlySpan<byte>, string):513,ParseDriverInstances(blob, nodeId):555(scope+parse),ParseDataPlaneAuthority(blob, nodeId):591,ParseComposition(blob):893,ParseComposition(blob, nodeId, onInconsistency):955(scope+parse+BuildClusterSets:1018). EVERY one doesJsonDocument.Parse(blob.ToArray())— a full copy per parse. The privateBuild*helpers already takeJsonElement root, which is what makes the parse-once refactor mechanical. - Per-apply parse count today: host:
ReconcileDrivers*(scope+instances = 2) +PushDesiredSubscriptions*(scope+composition+clustersets = 3) +CacheAppliedArtifact(scope = 1); publish actor:FeedDataPlaneAuthority(authority+scope = 2) +ParseComposition(3). Target: ONEJsonDocumentper actor per apply. - Blocking I/O on mailboxes:
CacheAppliedArtifactdoes.GetAwaiter().GetResult()(DriverHostActor.cs:1901-1904, comment at:1870-1875acknowledges it); the Applying-state deferral re-Forwards to Self in a hot loop (DriverHostActor.cs:944-955);BeginFetchAndCacheApply(:1571-1602) is thePipeTotemplate. - Compression constraint (CLAUDE.md, load-bearing):
RevisionHash == SHA-256(artifact bytes)—ConfigComposercomputes the hash of the exact blob,GrpcDeploymentArtifactFetcherverifies fetched bytes against it, chunked LocalDb cache stores those bytes. Compressing BEFORE hashing preserves the invariant everywhere (hash of the compressed bytes). The dispatch stays payload-free. - Invariants to preserve:
- #485 "no bytes is no answer" — a null/empty artifact fails the apply and keeps last-known-good; never let compression or parse-once break that path.
FeedDataPlaneAuthorityruns ahead ofplan.IsEmpty— a key-revocation-only artifact must still land.- The full-rebuild ratchet — any surgical false/throw falls back to
SafeRebuild(). _lastAppliedadvances only on a successful Apply — the retry diffs against what is actually materialised.- Deterministic composition —
ConfigComposer's deterministic sort is what makes revision-hash no-ops real; compression must not reorder anything (it can't — it wraps the final bytes). - The classifier stays default-closed — any delta shape not explicitly whitelisted still lands on
Rebuild.
- Key records:
RawTagPlan(TagId, NodeId, ParentNodeId, DriverInstanceId, Name, DataType, Writable, Alarm, ReferencingEquipmentPaths, IsHistorized, HistorianTagname, IsArray, ArrayLength, Realm)with customEquals(AddressSpaceComposer.cs:141+);UnsReferenceVariable(UnsTagReferenceId, EquipmentId, NodeId, EffectiveName, BackingRawPath, DataType, Writable, Realm)(:204+);AddressSpacePlan.RawTagDelta(Previous, Current)/UnsReferenceDelta(Previous, Current)(AddressSpacePlan.cs:134-138). - Materialise passes (
AddressSpaceApplier.cs):MaterialiseHierarchy(composition):643,MaterialiseRawSubtree(composition):682(needsBuildEquipmentIdByFolderPath(composition)for alarm notifier wiring),MaterialiseUnsReferences(composition):776(buildsrawByPathfromcomposition.RawTagsfor inherited shape/historian),MaterialiseScriptedAlarms,MaterialiseEquipmentTags,MaterialiseEquipmentVirtualTags(in the:820+region). All idempotent loops over composition lists. - Existing tests to keep green:
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpaceChangeClassifierTests.cs+AddressSpaceChangeClassifierDualNamespaceTests.cs(some assertChangedRawTags ⇒ Rebuild/ChangedUnsReferenceVariables ⇒ Rebuild— those assertions CHANGE deliberately in Task 4; update them to assert the new contract, alarm-delta cases still Rebuild), applier tests,OpcUaPublishActorApplyFailureTests,DriverHostActorFetchAndCacheBootstrapTests,tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/DeploymentArtifactProtoTests.cs, andtests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/SubscriptionSurvivalTests.cs(the live gate shape for "subscriptions survive"). - Commit style:
feat(...)/perf(...)/refactor(...)with scope; one commit per task.
Task 1: Direct mode threads the already-read blob
Classification: small Estimated implement time: ~3 min Parallelizable with: Task 2
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:1726-1730 - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/— extend the existing apply-path test class that exercisesApplyAndAck(grep for it; the FetchAndCache tests show the probe idiom)
Step 1: Failing test — drive a Direct-mode apply with a real blob in the fake ConfigDb; assert the publish-actor probe's RebuildAddressSpace message carries Artifact != null with the exact blob bytes (today it carries null).
Step 3: Implement — in ApplyAndAck, replace lines :1726 and :1729:
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, appliedBlob));
PushDesiredSubscriptionsFromArtifact(deploymentId, appliedBlob);
(appliedBlob is non-null here — the null case returned at :1706.) Leave PushDesiredSubscriptions(deploymentId) in place for any other caller; if it becomes call-site-dead, delete it and its SQL read. Update the comment block: the publish actor no longer reads central SQL on the Direct path either — LoadArtifact/LoadLatestArtifact remain only as the legacy/dev fallback for a bytes-less RebuildAddressSpace.
Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests — PASS.
Step 5: git add -A && git commit -m "perf(runtime): Direct-mode apply threads the in-hand artifact (1 SQL read per apply, not 3)"
Task 2: DeploymentArtifact.Open() — parse once
Classification: high-risk (config-parsing seam; behavior must be byte-identical) Estimated implement time: ~5 min Parallelizable with: Task 1
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/ParsedDeploymentArtifactTests.cs(create)
Step 1: Failing test — build a representative artifact JSON (reuse/borrow the builder from existing DeploymentArtifact tests — grep ParseComposition under tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests for the fixture); assert parity: using var parsed = DeploymentArtifact.Open(blob); then parsed.ResolveClusterScope(node), parsed.ParseDriverInstances(node), parsed.ParseComposition(node, onInconsistency), parsed.ParseDataPlaneAuthority(node) each equal the static counterparts' results (deep equality on the records; compositions compare with their record equality).
Step 3: Implement
- Add:
/// <summary>One-parse view over an artifact blob: a single JsonDocument serves every consumer of an
/// apply (driver reconcile, subscription push, composition, data-plane authority, cluster scope) instead
/// of each entry point re-parsing — at production sizes each static parse was a full LOH copy + document
/// build, and one apply performed ~8-11 of them. Dispose when the apply completes.</summary>
public sealed class ParsedDeploymentArtifact : IDisposable
{
private readonly JsonDocument _doc;
private readonly byte[] _blob; // decoded (post-decompression) bytes
private ClusterScope? _scope; // memoized per node — every consumer resolves the same node
private string? _scopeNodeId;
internal ParsedDeploymentArtifact(JsonDocument doc, byte[] blob) { _doc = doc; _blob = blob; }
public JsonElement Root => _doc.RootElement;
public void Dispose() => _doc.Dispose();
}
public static ParsedDeploymentArtifact Open(byte[] blob)— parse ONCE (JsonDocument.Parse(blob)— no.ToArray()copy; the caller's array is not mutated afterward on any call path).- Move the body of each static entry point into an instance method on
ParsedDeploymentArtifactoperating onRoot(the privateBuild*helpers already takeJsonElement root— passRootstraight in). MemoizeResolveClusterScope(nodeId)per node id (_scope/_scopeNodeId) sinceParseDriverInstances/ParseComposition/ParseDataPlaneAuthorityeach begin with the same scope resolution. - Re-express every existing static as a thin wrapper:
using var parsed = Open(blob.ToArray-if-span-else-array); return parsed.X(...)— public signatures may stayReadOnlySpan<byte>for compatibility (the wrapper pays one copy; the hot path stops using the statics in Task 3). Do not change any output shape.
Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests + dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests — PASS.
Step 5: git commit -m "refactor(runtime): ParsedDeploymentArtifact — one JsonDocument per apply"
Task 3: Actors consume Open()
Classification: standard Estimated implement time: ~4 min Parallelizable with: none (needs 1 + 2)
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs(ReconcileDriversFromBlob,PushDesiredSubscriptionsFromArtifact,CacheAppliedArtifactcall sites within one apply) - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs(HandleRebuild+FeedDataPlaneAuthority)
Implement (behavior-preserving):
- Host: in
ApplyAndAck/HandleFetchedForApply,using var parsed = DeploymentArtifact.Open(appliedBlob);and passparseddown (add internal overloadsReconcileDriversFromParsed,PushDesiredSubscriptionsFromParsed,CacheAppliedArtifact(deploymentId, revision, blob, parsed)— cache still stores the BYTES; it only needsparsedfor the scope lookup it currently re-parses for). The existing blob-based methods stay as wrappers for the boot paths. - Publish actor:
HandleRebuildopens once after the #485 guard;FeedDataPlaneAuthority(parsed, artifactBytes)(the SHA-256 at:535still hashes the raw bytes); the composition parse usesparsed.ParseComposition(...). - Tests: the Task 2 parity tests + the existing apply suites ARE the net; add one assertion-level test only if a seam demands it.
Run both Runtime + OpcUaServer test projects. Commit: perf(runtime): apply path parses the artifact once per actor
Task 4: Classifier — surgical eligibility for raw-tag and UNS-reference deltas
Classification: high-risk (the outage-per-edit fix; default-closed semantics) Estimated implement time: ~5 min Parallelizable with: none
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceChangeClassifier.cs - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpaceChangeClassifierTests.cs+AddressSpaceChangeClassifierDualNamespaceTests.cs(update the deliberately-changed assertions; add the new cases)
Step 1: Write the failing tests (these define the new contract):
Reference_only_raw_tag_delta_is_not_a_rebuild— aRawTagDeltawhose ONLY difference isReferencingEquipmentPaths,Alarm == nullboth sides ⇒ kind is NOTRebuild(with no other deltas ⇒AttributeOnly).Attribute_raw_tag_delta_is_not_a_rebuild— non-alarm delta differing inWritable/IsHistorized/HistorianTagname/DataType/IsArray/ArrayLength⇒ not Rebuild.Alarm_bearing_raw_tag_delta_still_rebuilds—Alarm != nullon either side ⇒Rebuild(the WP4 guard).Identity_field_raw_tag_delta_still_rebuilds— e.g.DriverInstanceIdorNamediffers ⇒Rebuild(default-closed).Uns_reference_delta_is_not_a_rebuild— aUnsReferenceDelta(re-point or display-name case) ⇒ not Rebuild.- Combination: UNS-ref ADD + the induced reference-only raw-tag delta ⇒
PureAdd(the headline case — one added UNS variable, zero teardown).
Step 3: Implement
- Add the predicate (mirror
TagDeltaIsSurgicalEligible's whitelist-via-withidiom, which auto-closes over future fields):
// v3 surgical raw-tag eligibility: a NON-alarm raw tag whose only differences are the in-place-updatable
// attributes (Writable / IsHistorized / HistorianTagname / DataType / IsArray / ArrayLength) and/or its
// ReferencingEquipmentPaths. The referencing set is node-inert for a VALUE tag — it only drives native-alarm
// notifier wiring (the WP4 guard), which is why Alarm != null on EITHER side disqualifies. Identity fields
// (NodeId is the diff key; TagId / DriverInstanceId / Name / ParentNodeId / Realm) still force a rebuild via
// the with-equality default-closed pattern.
internal static bool RawTagDeltaIsSurgicalEligible(AddressSpacePlan.RawTagDelta d) =>
d.Previous.Alarm is null && d.Current.Alarm is null &&
(d.Previous with
{
Writable = d.Current.Writable,
IsHistorized = d.Current.IsHistorized,
HistorianTagname = d.Current.HistorianTagname,
DataType = d.Current.DataType,
IsArray = d.Current.IsArray,
ArrayLength = d.Current.ArrayLength,
ReferencingEquipmentPaths = d.Current.ReferencingEquipmentPaths,
}).Equals(d.Current);
- In
Classifyrule 2 (:69-78): replaceplan.ChangedRawTags.Count > 0withplan.ChangedRawTags.Any(d => !RawTagDeltaIsSurgicalEligible(d)), and DELETEplan.ChangedUnsReferenceVariables.Count > 0from the rebuild condition (everyUnsReferenceDeltais handled surgically in Tasks 5-6: remove-previous + recreate-current). - Rewrite the WP4 comment block to state the NEW contract: alarm-bearing raw-tag deltas still rebuild (notifier wiring); value-tag deltas are surgical; the applier owns
UpdateTagAttributesfor them; UNS-ref deltas are remove+recreate.
Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~AddressSpaceChangeClassifier" — PASS (including the updated legacy assertions).
Step 5: git commit -m "feat(opcuaserver): classifier — non-alarm raw-tag + UNS-reference deltas are surgical, not rebuilds"
Task 5: Applier — apply the new surgical deltas
Classification: high-risk (rebuild ratchet semantics) Estimated implement time: ~5 min Parallelizable with: none (needs 4)
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs(Apply,:151-224) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/— extend the applier test class (find it via grepAddressSpaceApplierin that project; use its recording-fake surgical sink)
Step 1: Failing tests
Surgical_raw_tag_delta_updates_in_place_raw_realm— plan with one eligibleRawTagDelta(historize flip); fake surgical sink recordsUpdateTagAttributes(d.Current.NodeId, writable, historian, dataType, isArray, arrayLength, AddressSpaceRealm.Raw)with historian resolved asMaterialiseRawSubtreeresolves it (override else NodeId,AddressSpaceApplier.cs:748-750) andwritable = Writable && !IsArray; NORebuildAddressSpacecall.Reference_only_raw_tag_delta_is_a_noop_update— onlyReferencingEquipmentPathsdiffers ⇒ either no sink call or one idempotentUpdateTagAttributes(assert no rebuild; prefer SKIPPING the call when the with-comparison excluding the referencing set shows attribute equality — implement that skip).Uns_reference_delta_removes_previous_node— plan with oneUnsReferenceDelta; sink records terminal-BadWriteValue(Previous.NodeId, ..., Uns)thenRemoveVariableNode(Previous.NodeId, Uns); no rebuild. (The recreate happens in the publish actor's delta materialise pass — Task 6 — NOT here.)Surgical_false_falls_back_to_rebuild— surgical sink returns false for the raw-tag update ⇒RebuildAddressSpacecalled (ratchet preserved).
Step 3: Implement in Apply:
- Compute
var surgicalRawTagDeltas = plan.ChangedRawTags.Where(AddressSpaceChangeClassifier.RawTagDeltaIsSurgicalEligible).ToList();(when kind != Rebuild this equals ALL of ChangedRawTags — the classifier guarantees it; still filter, defense-in-depth) andvar unsRefDeltas = plan.ChangedUnsReferenceVariables; - Extend the surgical-branch trigger (
:171):surgicalTagDeltas.Count > 0 || renamedFolders.Count > 0 || hasRemovePass || surgicalRawTagDeltas.Count > 0 || unsRefDeltas.Count > 0. - Inside the
ISurgicalAddressSpaceSinkbranch, after the existing equipment-tag loop:- Raw-tag loop: skip when attribute-equal apart from
ReferencingEquipmentPaths(test 2); elsesurgical.UpdateTagAttributes(d.Current.NodeId, d.Current.Writable && !d.Current.IsArray, d.Current.IsHistorized ? (string.IsNullOrWhiteSpace(d.Current.HistorianTagname) ? d.Current.NodeId : d.Current.HistorianTagname) : null, d.Current.DataType, d.Current.IsArray, d.Current.ArrayLength, AddressSpaceRealm.Raw)with the same try/catch +allAppliedratchet as the equipment-tag loop. - UNS-ref loop:
SafeWriteValue(d.Previous.NodeId, null, OpcUaStatus.Bad, ts, AddressSpaceRealm.Uns);thenSafeRemoveVariable(surgical, d.Previous.NodeId, AddressSpaceRealm.Uns)into the ratchet. Doc-comment WHY remove+recreate (a re-point must move theOrganizesedge and the sink has no RemoveReference; a display-name edit changes the NodeId; remove+recreate covers both, at the cost of that one node's monitored items — vs. every subscription server-wide today).
- Raw-tag loop: skip when attribute-equal apart from
changedCountaccounting already includes both sets (:115-121) — verify, don't double-count.- Also update the
FeedHistorizedRefschanged-raw-tags handling: it already handlesplan.ChangedRawTags(:571-578) — now those deltas reach it on the surgical path too; confirm the logic is delta-shape-agnostic (it is) and leave it.
Step 4: run the OpcUaServer.Tests project — PASS.
Step 5: git commit -m "feat(opcuaserver): applier applies raw-tag + UNS-reference deltas surgically"
Task 6: Delta materialisation (publish actor materialises the plan, not the world)
Classification: high-risk (the O(fleet)-per-apply fix) Estimated implement time: ~5 min Parallelizable with: none (needs 5)
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs(subset overloads on the six passes) - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs:456-480 - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/applier tests +tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/publish-actor tests
Step 1: Failing tests
Pure_add_materialises_only_the_added_nodes— publish-actor-level: apply composition A (N nodes), then A+1 tag; the fake sink'sEnsureVariable/EnsureFoldercalls after the second apply cover ONLY the added tag (+ its added containers), not all N. (This is the headline test.)Rebuild_still_materialises_the_full_composition— force a rebuild-kind delta; full-composition passes run as today.Uns_reference_delta_recreates_current_node— a re-point delta: after Task 5's remove, the UNS pass re-ensuresCurrent.NodeIdwith shape inherited from the FULL composition'srawByPath(backing tag NOT in the delta), plus theAddReferenceOrganizes edge to the NEW backing path.- Applier-level:
MaterialiseRawSubtree(full, subsetTags, subsetContainers)wires alarm notifiers using the FULL composition's equipment map.
Step 3: Implement
- Applier: give the passes subset parameters, full-composition context retained for lookups:
MaterialiseRawSubtree(AddressSpaceComposition composition, IReadOnlyList<RawContainerNode>? onlyContainers = null, IReadOnlyList<RawTagPlan>? onlyTags = null)— iterateonlyContainers ?? composition.RawContainers/onlyTags ?? composition.RawTags;BuildEquipmentIdByFolderPath(composition)stays full.MaterialiseUnsReferences(AddressSpaceComposition composition, IReadOnlyList<UnsReferenceVariable>? only = null)— iterateonly ?? composition.UnsReferenceVariables;rawByPathstays full.MaterialiseScriptedAlarms/MaterialiseEquipmentTags/MaterialiseEquipmentVirtualTags: sameonlypattern over their respective lists.MaterialiseHierarchystays full-composition ALWAYS (folders are O(areas+lines+equipment), cheap, and parents must exist for any add).
- Publish actor (
:456-480): aftervar outcome = _applier.Apply(plan);branch onoutcome.RebuildCalled:- Rebuild ⇒ the six passes exactly as today (full composition).
- Non-rebuild ⇒
MaterialiseHierarchy(composition)full; then subsets: containers =plan.AddedRawContainers, raw tags =plan.AddedRawTags, UNS refs =plan.AddedUnsReferenceVariablesconcatplan.ChangedUnsReferenceVariables.Select(d => d.Current), scripted alarms =plan.AddedAlarms, equipment tags =plan.AddedEquipmentTags, vtags =plan.AddedEquipmentVirtualTags. Skip a pass entirely when its subset is empty. - Comment WHY: the passes are idempotent but not free — each call is a lock acquisition; a one-tag add on a 50k-node server was ~50k sink calls.
AnnounceAddedNodesstays as-is (known pre-existing gap: raw/UNS adds are not announced — out of scope, note it in the comment).
Step 4: OpcUaServer.Tests + Runtime.Tests — PASS (watch OpcUaPublishActorApplyFailureTests + DarkAddressSpaceReasons for full-pass assumptions; update only assertions that pinned the OLD full-pass behavior explicitly, preserving their intent).
Step 5: git commit -m "perf(opcuaserver): non-rebuild applies materialise the delta, not the composition"
Task 7: Blocking I/O off the host mailbox
Classification: high-risk (actor lifecycle) Estimated implement time: ~5 min Parallelizable with: Task 8
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs(CacheAppliedArtifact:1896-1904region; Applying-state deferral:944-955) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/— extend the apply/dispatch test classes
Part A — async cache write. CacheAppliedArtifact's .GetAwaiter().GetResult() becomes fire-and-forget with observed completion:
_artifactCache.StoreAsync(...).PipeTo(Self,
success: () => new ArtifactCached(deploymentId),
failure: ex => new ArtifactCacheFailed(deploymentId, ex.Message));
(private records; handlers just log Info/Warning — the cache is a boot fallback, not apply-critical; the docstring at :1870-1875 already debates exactly this and the apply outcome does not depend on it). CHECK first whether BootstrapFromCache (:788, :859) races a concurrent store — it runs in PreStart before any apply, so no; note that in the comment. Keep the store idempotent-skip (IsAlreadyCachedAsync) inside the async flow.
Failing test: apply completes and acks BEFORE a deliberately-slow fake cache resolves; a follow-up ArtifactCached self-message is observed (TestKit probe or log assertion per the project's idiom).
Part B — stash instead of forward-spin. The Applying-state deferral for a DIFFERENT deployment (:944-955): make DriverHostActor implement IWithUnboundedStash (public IStash Stash { get; set; }), replace Self.Forward(msg) with Stash.Stash(), and call Stash.UnstashAll() at every Become(Steady) transition (both finally blocks + the fetch-failure path). Failing test: send dispatch B while A's fetch is in flight (slow fake fetcher); assert no busy-loop (the log does NOT contain the deferral line more than once) and B applies after A completes.
Step 4: Runtime.Tests — PASS. Step 5: git commit -m "perf(runtime): apply-path cache write off-mailbox; stash deferred dispatches"
Task 8: Compressed artifact blob
Classification: high-risk (wire/storage format; hash invariant) Estimated implement time: ~5 min Parallelizable with: Task 7
Files:
- Modify:
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs(emit compressed; hash unchanged in derivation — SHA-256 of the emitted bytes) - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs(Open()+ static wrappers sniff-and-decompress) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/ParsedDeploymentArtifactTests.cs+ the ConfigComposer test class (grepSnapshotAndFlattenAsyncunder tests/Server)
Format decision (locked): gzip via System.IO.Compression.GZipStream (CompressionLevel.Fastest), detected by the gzip magic bytes 0x1F 0x8B at offset 0 (a JSON artifact can never start with 0x1F). No custom framing. Compression happens INSIDE ConfigComposer immediately after SerializeToUtf8Bytes, BEFORE the revision hash is computed — so RevisionHash == SHA-256(stored bytes) holds and GrpcDeploymentArtifactFetcher's verification (GrpcDeploymentArtifactFetcher.cs:89-90) is untouched. The LocalDb cache + gRPC serve stores/ships the compressed bytes as-is (they never look inside). Decompression lives in exactly one place: DeploymentArtifact.Open() (and thus every static wrapper). OpcUaPublishActor.FeedDataPlaneAuthority's SHA at OpcUaPublishActor.cs:535 hashes the RAW (compressed) bytes it received — verify that call site hashes the bytes as-carried, not a decompressed copy.
Step 1: Failing tests
Open_reads_gzip_compressed_artifacts— compress a fixture blob, parity-assertParseCompositionvs the uncompressed parse.Open_still_reads_uncompressed_artifacts(backward compat — pre-upgrade rows).Composer_emits_gzip_and_hash_of_compressed_bytes—ArtifactBlobstarts with1F 8B; the deployment'sRevisionHashequalsSHA256(ArtifactBlob)lowercase hex.
Step 3: Implement — decompress with a size guard (refuse decompressed output > 512 MB — a corrupt/hostile blob must not OOM the node; treat over-limit exactly like unparseable JSON: the existing catch paths). Note in a comment: gzip output is not guaranteed byte-stable across .NET releases; the hash is computed from the emitted bytes at compose time and never re-derived, so the only consequence of an impl change is one spurious "changed" revision after a runtime upgrade.
Step 4: Runtime.Tests + ControlPlane tests (if the composer test project is DB-backed and the DB is unreachable, run the artifact round-trip tests and note the skip). Step 5: git commit -m "perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)"
Task 9: Full verification + docs
Classification: standard Estimated implement time: ~5 min Parallelizable with: none (last)
Files:
- Modify:
CLAUDE.md— the "v3 OPC UA Address Space" / classifier-adjacent statements that now describe the old behavior (grep CLAUDE.md forChangedRawTagsandRebuild); keep the edits surgical and truthful to the new contract (alarm-bearing deltas still rebuild). - Modify:
docs/Uns.md+docs/Raw.md— one short paragraph each: reference add/remove and attribute edits are now surgical; what still rebuilds (alarm deltas, container renames, identity changes).
Steps: full dotnet build ZB.MOM.WW.OtOpcUa.slnx; run every non-DB test project (same list as plan 1 Task 12 + tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests); record skipped DB-backed suites in the commit message. Commit: docs: record surgical-deploy semantics (plan 2026-08-15).
Explicit non-goals
- No surgical raw-container (folder/device/group) renames — still rebuild (no surgical folder-remove on the sink; pre-existing).
- No alarm-bearing raw-tag surgical path — the WP4 notifier-rewire stays owned by the rebuild.
- No central-side deploy-start dedup (
AdminOperationsActortriple DB read) — separate follow-up. - No
AnnounceAddedNodesextension to raw/UNS adds — pre-existing gap, noted in comments. - No cross-actor shared
ParsedDeploymentArtifact— one parse per actor is the target; sharing a disposable across mailboxes isn't worth the ownership complexity. - No artifact schema changes — the JSON inside the (now compressed) blob is byte-identical.
Follow-ups (recorded, not implemented)
Found during implementation and review of this plan. None is a regression introduced by the branch except where stated; each is recorded here rather than folded in, so the branch stays reviewable.
(a) Raw-realm RaiseNodesAddedModelChange — surgically-added raw nodes are never announced.
AnnounceAddedNodes / ComputeAddAnnouncements only walk the equipment (Uns) add sets, and the
announce hardcodes the Uns realm. The UNS half is now closed — an added UnsReferenceVariable is
announced against its equipment folder. The raw half is not: an added raw container or raw tag is
materialised and then silent. This was neutral before the branch (a plan carrying raw adds went through
a full rebuild, so every client had to re-browse anyway); with surgical adds keeping subscriptions
alive, an unannounced raw add is genuinely invisible until a client happens to browse. The fix is a
realm-aware RaiseNodesAddedModelChange call plus raw-side add sets in ComputeAddAnnouncements. This
is a real follow-up, not an accepted limitation.
(b) RemoveEquipmentSubtree shares the dangling-inverse-edge exposure that RemoveVariableNode now
fixes. I3 on this branch made RemoveVariableNode sever the inverse half of every outgoing
Organizes edge (the variable-node counterpart of UnwireAlarmNotifiers) — before, rebuilds swept
those up, and surgical applies made rebuilds rare enough that they accumulated on the backing raw node
for the life of the process. RemoveEquipmentSubtree removes a whole equipment folder and has the same
shape of exposure for the edges its descendants own; it was not touched here because it is only reached
by a PureRemove/AddRemoveMix plan and needs its own test scaffolding. Same class of leak, same fix.
(c) End-to-end test: a degraded remove-bearing apply followed by a second deploy. The C1 resync
semantic — a degraded non-rebuild apply does not advance _lastApplied, so the next deploy re-diffs the
same composition, re-emits that apply's removes, and the re-remove's false ratchets into a full
rebuild — is reasoned in the code comment and each half is independently pinned (the non-advance, and
the ratchet-on-false). What is not covered is the composed path: drive a degraded apply that carried
removes, then a second deploy, and observe the ratcheted rebuild. Worth building because the semantic is
deliberately self-correcting-by-escalation, and an escalation nobody has watched fire is an assumption.
(d) Delta materialise of scripted alarms uses thin-plan ids against rich rows. The alarm subset handed to the materialise pass is identified by the plan's ids, while the pass reads the full composition's rich alarm rows. An alarm whose predicate script arrives in a later revision than the alarm row itself is therefore invisible to a delta apply: the alarm id is not in any diff set, so the subset never names it, and the pass never re-reads it. This is pre-existing planner blindness (the planner has always diffed on the thin shape), but the delta path widens it — the full passes used to re-read every alarm on every apply and would eventually pick the script up. Until fixed, such an alarm converges only on the next rebuild.
(e) Deploy dispatch carries no monotonic sequence. Under the gRPC command transport, dispatches are fire-and-forget with no retry and no ordering guarantee across nodes, so two dispatches issued close together can be overtaken and "the final revision" is whichever arrived last, not whichever was sealed last. This is pre-existing and independent of this branch — but the stash-instead-of-forward-spin change (Task 7 Part B) makes it the only remaining nondeterminism in the deferral path, so it is now worth naming. A monotonic dispatch sequence that a node refuses to go backwards on would close it.
(f) LocalDb store ordering under concurrent StoreAsync. With the cache write moved off the mailbox
(PipeTo), two applies in quick succession can have their stores in flight simultaneously; the artifact
chunks and the pointer are written separately, so the pointer can briefly name the older artifact. It
self-heals on the next deploy (the newer artifact is stored and the pointer is rewritten), and boot-from-
cache is a fallback rather than a system of record, so the exposure is a stale — never a corrupt — cache
in a narrow window. A store-level sequence guard, or serialising stores per cluster, would remove it.
(g) RestoreApplied could thread its in-hand blob. Task 1 removed the redundant central-SQL reads
from the Direct apply path, but the boot/restore path still reads the artifact and then hands the
publish actor a bytes-less RebuildAddressSpace, which re-reads it. That is one extra SQL read per
node restart — not per apply — which is why it was left out of scope. Threading appliedBlob through
RestoreApplied the same way ApplyAndAck now does would finish the job.
Live gate — PASSED (docker-dev, 2026-08-15)
Run against the local docker-dev rig (6 nodes, image rebuilt from merged master 169ee6a4), with the
value-pipeline-batching branch included. Every leg green:
- Boot back-compat (gzip sniff): all six nodes restarted on the new image over a pre-upgrade
uncompressed
ArtifactBloband restored the full 73-node address space from it (PureAdd, added=73). - First post-compression deploy re-hashes + gzips: deploy
e869a136produced a newRevisionHashfor identical config (the documented one-time re-dispatch) and a 7,173-byte blob starting1F 8B(previous uncompressed blob: 28,742 bytes — ~4×). All six nodes fetched over the config-serve stream (1 chunk), applied (empty plan — no address-space churn), acked, sealed. Data-plane authority still installed (generation 2) on the empty-plan path. - Surgical UNS-reference add mid-subscription: with 17 monitored items open across 4 sessions
(raw pymodbus subtree, two UNS mirrors, one Sql-driver tag), adding a
UnsTagReference(press1→Probe0) and deploying (318f04c8) applied withrebuild=False, added=1, surgicalRawTags=1(the backing tag'sReferencingEquipmentPathsupdated in place) and one-node subset materialisation. Zero items ever went Bad; all ended Good; all received updates. The new nodens=3;s=gate481/line1/press1/Probe0served Good immediately. A DevSql value change written after the surgical apply streamed into the pre-deploy subscription (42.5 → 777.25, both Good) — the batched pipeline stayed live end to end. - Rename contra-test: renaming a raw tag (deploy
e691615d) applied asAddRemoveMix, added=1, removed=1, unsRefDeltas=1, rebuild=False— a rename never pairs as aRawTagDelta(pairing is by RawPath; the delta identity guard is for same-path re-creates), so it is served surgically with a blast radius of exactly the renamed node + its re-pointed referencing mirror. Both new NodeIds served Good. CLAUDE.md's "identity change ⇒ rebuild" wording was corrected to match. - Revert deploy (
5848e289) sealed clean, all six acked, original nodes restored.
Not covered by this gate: site-node FetchAndCache flip (rig stayed in its default modes), alarm-bearing
surgical ratchet (no native-alarm tag on the rig), and load-scale batching throughput (single-tag cadence
only).