docs: record surgical-deploy semantics + artifact compression ops notes (plan 2026-08-15)

CLAUDE.md — a new "How a deploy reaches that address space" block under the v3
Batch 4 address-space section. The blanket ChangedRawTags/ChangedUnsReferenceVariables
=> Rebuild rule is gone, so the doc now names both halves explicitly: what still
rebuilds (alarm-bearing raw-tag deltas, raw-tag identity changes, raw container
renames/removals, changed equipment/alarms) and what is now surgical (non-alarm
attribute deltas, every UnsReferenceDelta). Also records the three semantics that
are easy to break by accident: mirror repair rides UnsReferenceVariable's inherited
backing fields (omitting a newly-inherited field silently reopens the staleness),
a degraded non-rebuild apply deliberately does not advance _lastApplied, and the
raw-realm announce gap is no longer neutral now that surgical adds keep
subscriptions alive.

CLAUDE.md ConfigSource — the RevisionHash == SHA-256(artifact bytes) paragraph is
still true and stays; it gains the clause that the blob is gzip-compressed since
2026-08-15 and the hash covers the COMPRESSED bytes, which is precisely why the
fetcher, cache and serve path were untouched.

docs/Uns.md + docs/Raw.md — one section each on what a deploy now tears down, from
the authoring surface's point of view, including the mirror repair semantics
(identity-preserving backing edits update in place and keep their MonitoredItems;
re-points and effective-name edits remove+recreate that one node).

docs/Configuration.md — "Artifact compression (2026-08-15)" under ConfigSource/
ConfigServe: gzip inside the hashed bytes, magic-sniffed on read, pre-upgrade
uncompressed rows still parse, plus the two operational facts an operator needs.
The first deploy after upgrade re-hashes every revision (same config, new bytes,
new RevisionHash) so it is a one-time full re-dispatch rather than a no-op; and
rollout is ordered — nodes must run artifact-sniffing code BEFORE central starts
emitting compressed blobs, since a pre-upgrade node handed a gzip artifact fails
the apply and keeps last-known-good until it is upgraded.

Plan doc — a "Follow-ups (recorded, not implemented)" section covering the seven
items surfaced during implementation and review: the raw-realm announce gap, the
RemoveEquipmentSubtree dangling-inverse-edge exposure, the missing end-to-end test
for a degraded remove-bearing apply's ratcheted rebuild, the scripted-alarm
thin-plan-ids-against-rich-rows blindness, the absent monotonic dispatch sequence,
LocalDb store ordering under concurrent StoreAsync, and threading the in-hand blob
through RestoreApplied.

tasks.json — all nine tasks completed.

Docs only; no source changes.
This commit is contained in:
Joseph Doherty
2026-08-15 18:02:21 -04:00
parent d787502d54
commit 0b537b5dec
6 changed files with 229 additions and 10 deletions
+57
View File
@@ -130,6 +130,53 @@ the single-namespace `EquipmentTags` materialization path are gone. See
`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.
**How a deploy reaches that address space (surgical-deploys, 2026-08-15).** The old
blanket rule — ANY non-empty `ChangedRawTags` / `ChangedUnsReferenceVariables` routes to
`Rebuild` — is gone, because it made a routine edit a fleet-wide outage:
`RawTagPlan.ReferencingEquipmentPaths` is part of the record's equality, so adding or
dropping **one** `UnsTagReference` flipped its backing raw tag to Changed and tore down
every node on the server, killing every client subscription. `AddressSpaceChangeClassifier`
is still default-closed; what changed is which deltas it whitelists.
- **Still a full rebuild:** an **alarm-bearing** raw-tag delta (`Alarm != null` on either
side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an
**identity** change on a raw tag (`TagId`/`DriverInstanceId`/`Name`/`ParentNodeId`/`Realm`),
a raw **container** (Folder/Driver/Device/TagGroup) rename or removal, `ChangedEquipment` /
`ChangedAlarms`, a non-surgical equipment-tag delta, and anything else the
`with`-equality whitelist does not name (including any FUTURE field).
- **Now surgical:** a **non-alarm** raw-tag delta over the in-place-updatable attributes
(`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / `ArrayLength`)
and/or `ReferencingEquipmentPaths` — applied via `ISurgicalAddressSpaceSink.UpdateTagAttributes`
in the **Raw** realm — and **every** `UnsReferenceDelta` (add / remove / re-point /
backing-edit).
- ⚠️ **UNS mirrors repair through the RECORD, not the classifier.** A raw-side historian or
shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape
(`EnsureVariable` early-returns on an existing node). `UnsReferenceVariable` therefore
carries the backing tag's `Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}`, so the
same edit makes the reference record unequal and the planner emits a **coincident**
`UnsReferenceDelta`. Adding a newly-inherited field needs no classifier change — **omitting
one silently reopens the staleness.** Repair is **in place** (`UpdateTagAttributes`, Uns
realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an
identity change (re-point, effective-name edit) is remove+recreate — and only for that
one node.
- **A non-rebuild apply materialises the DELTA, not the composition.** `MaterialiseHierarchy`
still runs full (parents must exist); the other passes take the plan's added/changed subsets.
Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread.
- ⚠️ **A degraded non-rebuild apply does NOT advance `_lastApplied`.** The full passes used to
re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the
delta path removed that. Holding the composition back is the replacement — the next deploy
re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that
apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct,
bounded resync). A **rebuild** still advances even with failures.
- **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is
what makes every whitelist above safe.
- ⚠️ **Known gap, no longer neutral:** `AnnounceAddedNodes` is Uns-only, so surgically-added
**raw** containers/tags are materialised but never announced — previously invisible (the
rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses.
Added UNS reference variables ARE announced.
See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`.
### Key Concept: Tag Name and FullName
Galaxy objects have a **tag_name** — a globally unique system name used for
@@ -467,6 +514,16 @@ fail-closed). Things worth knowing before touching it:
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
**The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** —
`ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and
hashes the result, so the invariant holds unchanged and nothing between the composer and
`DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever
looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so
pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every
decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already
handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after
upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be
upgraded BEFORE central starts emitting compressed blobs.**
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
+33
View File
@@ -217,6 +217,39 @@ See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-t
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
#### Artifact compression (2026-08-15)
The artifact blob is **gzip-compressed, inside the hashed bytes**. `ConfigComposer.Pack` gzips the
serialised snapshot (`CompressionLevel.Fastest`, ~10–20x on this highly repetitive JSON) immediately
after `SerializeToUtf8Bytes` and **then** takes the SHA-256, so the compressed bytes *are* the artifact
everywhere — `Deployment.ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream, and the
`RevisionHash` a `FetchAndCache` node verifies against. The invariant above is preserved exactly, which
is why the fetcher, the cache and the serve path are untouched: nothing between the composer and
`DeploymentArtifact.Open` looks inside the blob.
Decompression lives in exactly one place, `DeploymentArtifact.Open`, which **sniffs the gzip magic
`1F 8B`** at offset 0 — valid JSON can never start with `0x1F`, so no framing of our own is needed and
**pre-upgrade uncompressed rows keep parsing unchanged**. Inflation runs under a 512 MiB guard so a
corrupt or hostile blob cannot OOM the node, and every decompression failure (corrupt, truncated,
over-limit) is reported as `JsonException` — the same exception family the malformed-JSON path already
owns, so no caller's decision changes.
Two operational facts:
- **The first deploy after the upgrade re-hashes every revision.** The same configuration now serialises
to different (compressed) bytes, so it produces a different `RevisionHash`. Expect a **one-time full
re-dispatch to every node**, not the usual no-op — this is normal, and it happens once.
- **Rollout ordering matters: upgrade the NODES first.** A node must be running artifact-sniffing code
*before* central starts emitting compressed blobs. A pre-upgrade node handed a gzip artifact cannot
parse it and **fails the apply**, which means it keeps serving last-known-good (the #485 contract) —
degraded but not destructive, and it recovers as soon as it is upgraded. Rolling back central is
likewise safe in the other direction, since sniffing code still reads uncompressed blobs.
(gzip output is not guaranteed byte-stable across .NET releases. Harmless here: the hash is computed
from the bytes emitted at compose time and never re-derived from the JSON, so an implementation change
costs at most one spurious "changed" revision after a runtime upgrade. Snapshot determinism is
unaffected — compression wraps the final bytes and reorders nothing.)
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
+33
View File
@@ -70,6 +70,39 @@ serialize the endpoint keys, so `DeviceConfig` is the single source of truth.
tag-groups" toggle mirrors the browse folder nesting. The browser dials a real ephemeral
driver against the **merged** Driver+Device config.
## What a deploy tears down (surgical deploys, 2026-08-15)
Editing a raw **tag's** attributes is no longer a fleet-wide event. A **non-alarm**
raw-tag delta whose only differences are the in-place-updatable attributes —
`Writable`, `IsHistorized`, `HistorianTagname`, `DataType`, `IsArray`, `ArrayLength` —
and/or its set of referencing equipment is applied **in place** on the existing Raw-realm
node (`ISurgicalAddressSpaceSink.UpdateTagAttributes`), so every client subscription on
that tag and on the rest of the server survives. The same is true of **every** UNS
tag-reference change: adding, removing or re-pointing a reference used to flip the
backing raw tag to Changed (the referencing set is part of its record equality) and
rebuild the whole address space for a one-variable edit.
**Still a full rebuild**, deliberately: an **alarm-bearing** raw tag (a delta with
`Alarm != null` on either side — only a rebuild clears and re-wires the native-alarm
notifiers that fan the condition to its referencing equipment folders); an **identity**
change on a tag (`TagId`, `DriverInstanceId`, `Name`, `ParentNodeId`, realm); and any
change to a raw **container** — a Folder / Driver / Device / TagGroup **rename or
removal** — because the surgical sink has no folder-remove. The classifier is
default-closed: any field it does not explicitly whitelist, including fields added
later, still routes to a rebuild, and any surgical call returning `false` or throwing
ratchets back to one.
**Mirror repair.** A raw-side historian or shape edit also reaches every UNS reference
mirroring the tag: the reference record carries the backing tag's historian/shape, so
the edit emits a coincident UNS-reference delta. Identity-preserving repairs update the
mirror **in place** (its MonitoredItems survive); a re-point or effective-name change
removes and recreates **that one mirror node**. See [`Uns.md`](Uns.md) and
[`plans/2026-08-15-surgical-deploys.md`](plans/2026-08-15-surgical-deploys.md).
⚠️ A surgically added raw container/tag is materialised but **not** announced —
`AnnounceAddedNodes` is UNS-only, so model-aware clients see new raw nodes on their next
browse. This mattered less when every add forced a rebuild (and therefore a re-browse).
## The `Calculation` driver
Signal-level calculated tags are ordinary raw tags bound to a `Calculation` driver
+34
View File
@@ -442,6 +442,40 @@ Edits here change the configuration only. As the page header notes,
**changes apply on the next deployment** — run a **Deploy** (Deployments
page) to push them into the running address space.
### What a deploy actually tears down (surgical deploys, 2026-08-15)
Adding, removing or re-pointing a UNS **tag reference** no longer rebuilds the
address space. It used to: `RawTagPlan.ReferencingEquipmentPaths` participates in
the backing raw tag's record equality, so touching one reference flipped that tag to
Changed and the classifier's blanket `ChangedRawTags ⇒ Rebuild` rule tore down and
recreated **every** node on the server — killing every client's subscriptions for a
one-variable edit. Now **every** `UnsReferenceDelta` is applied surgically, and so is
a **non-alarm** raw-tag delta over the in-place-updatable attributes
(`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` /
`ArrayLength`). A **full rebuild** is still forced by an **alarm-bearing** raw-tag
delta (only a rebuild re-wires native-alarm notifiers), an **identity** change on a
raw tag (`TagId` / `DriverInstanceId` / `Name` / `ParentNodeId` / `Realm`), a raw
**container** rename or removal, and any changed equipment / scripted alarm. The
classifier stays default-closed — anything its whitelist does not name still rebuilds,
and any surgical failure ratchets back to a rebuild.
**Mirror repair.** A UNS reference variable mirrors its backing raw tag, so a raw-side
historian or shape edit must reach it too — `UnsReferenceVariable` carries the backing
tag's `IsHistorized` / `HistorianTagname` / `IsArray` / `ArrayLength`, which makes the
reference record unequal on such an edit and emits a **coincident** `UnsReferenceDelta`
alongside the raw one. **Identity-preserving** repairs (the mirror keeps the same
NodeId, equipment and backing RawPath) are applied **in place**, so the mirror's
subscriptions survive an operator historizing or reshaping the tag behind it. Only an
**identity change** — a re-point to a different raw tag, or an effective-name/display-name
edit that moves the NodeId — is a remove + recreate, and it affects **that one node**,
not the tree.
⚠️ Two consequences worth knowing. A surgically **added** UNS reference variable *is*
announced (Part 3 `GeneralModelChangeEvent`), but an added **raw** container/tag is
not — clients pick those up on their next browse. And a **degraded** non-rebuild apply
(some node failed to materialise) deliberately does not record its composition as
applied, so the next deploy re-diffs and repairs it.
## See also
- [Configuration.md](Configuration.md) — the underlying config entities.
+62
View File
@@ -316,3 +316,65 @@ Steps: full `dotnet build ZB.MOM.WW.OtOpcUa.slnx`; run every non-DB test project
- **No `AnnounceAddedNodes` extension to raw/UNS adds** — pre-existing gap, noted in comments.
- **No cross-actor shared `ParsedDeploymentArtifact`** — one parse per actor is the target; sharing a disposable across mailboxes isn't worth the ownership complexity.
- **No artifact schema changes** — the JSON inside the (now compressed) blob is byte-identical.
---
## Follow-ups (recorded, not implemented)
Found during implementation and review of this plan. None is a regression introduced by the branch
except where stated; each is recorded here rather than folded in, so the branch stays reviewable.
**(a) Raw-realm `RaiseNodesAddedModelChange` — surgically-added raw nodes are never announced.**
`AnnounceAddedNodes` / `ComputeAddAnnouncements` only walk the equipment (Uns) add sets, and the
announce hardcodes the Uns realm. The UNS half is now closed — an added `UnsReferenceVariable` is
announced against its equipment folder. The **raw** half is not: an added raw container or raw tag is
materialised and then silent. This was neutral before the branch (a plan carrying raw adds went through
a full rebuild, so every client had to re-browse anyway); with surgical adds keeping subscriptions
alive, an unannounced raw add is genuinely invisible until a client happens to browse. The fix is a
realm-aware `RaiseNodesAddedModelChange` call plus raw-side add sets in `ComputeAddAnnouncements`. This
is a real follow-up, not an accepted limitation.
**(b) `RemoveEquipmentSubtree` shares the dangling-inverse-edge exposure that `RemoveVariableNode` now
fixes.** I3 on this branch made `RemoveVariableNode` sever the inverse half of every outgoing
`Organizes` edge (the variable-node counterpart of `UnwireAlarmNotifiers`) — before, rebuilds swept
those up, and surgical applies made rebuilds rare enough that they accumulated on the backing raw node
for the life of the process. `RemoveEquipmentSubtree` removes a whole equipment folder and has the same
shape of exposure for the edges its descendants own; it was not touched here because it is only reached
by a `PureRemove`/`AddRemoveMix` plan and needs its own test scaffolding. Same class of leak, same fix.
**(c) End-to-end test: a degraded remove-bearing apply followed by a second deploy.** The C1 resync
semantic — a degraded non-rebuild apply does not advance `_lastApplied`, so the next deploy re-diffs the
same composition, re-emits that apply's removes, and the re-remove's `false` ratchets into a full
rebuild — is reasoned in the code comment and each half is independently pinned (the non-advance, and
the ratchet-on-false). What is **not** covered is the composed path: drive a degraded apply that carried
removes, then a second deploy, and observe the ratcheted rebuild. Worth building because the semantic is
deliberately self-correcting-by-escalation, and an escalation nobody has watched fire is an assumption.
**(d) Delta materialise of scripted alarms uses thin-plan ids against rich rows.** The alarm subset
handed to the materialise pass is identified by the plan's ids, while the pass reads the full
composition's rich alarm rows. An alarm whose **predicate script** arrives in a later revision than the
alarm row itself is therefore invisible to a delta apply: the alarm id is not in any diff set, so the
subset never names it, and the pass never re-reads it. This is **pre-existing planner blindness** (the
planner has always diffed on the thin shape), but the delta path widens it — the full passes used to
re-read every alarm on every apply and would eventually pick the script up. Until fixed, such an alarm
converges only on the next rebuild.
**(e) Deploy dispatch carries no monotonic sequence.** Under the gRPC command transport, dispatches are
fire-and-forget with no retry and no ordering guarantee across nodes, so two dispatches issued close
together can be overtaken and "the final revision" is whichever arrived last, not whichever was sealed
last. This is pre-existing and independent of this branch — but the stash-instead-of-forward-spin change
(Task 7 Part B) makes it the **only** remaining nondeterminism in the deferral path, so it is now worth
naming. A monotonic dispatch sequence that a node refuses to go backwards on would close it.
**(f) LocalDb store ordering under concurrent `StoreAsync`.** With the cache write moved off the mailbox
(`PipeTo`), two applies in quick succession can have their stores in flight simultaneously; the artifact
chunks and the pointer are written separately, so the pointer can briefly name the older artifact. It
self-heals on the next deploy (the newer artifact is stored and the pointer is rewritten), and boot-from-
cache is a fallback rather than a system of record, so the exposure is a stale — never a corrupt — cache
in a narrow window. A store-level sequence guard, or serialising stores per cluster, would remove it.
**(g) `RestoreApplied` could thread its in-hand blob.** Task 1 removed the redundant central-SQL reads
from the Direct apply path, but the boot/restore path still reads the artifact and then hands the
publish actor a bytes-less `RebuildAddressSpace`, which re-reads it. That is **one** extra SQL read per
node restart — not per apply — which is why it was left out of scope. Threading `appliedBlob` through
`RestoreApplied` the same way `ApplyAndAck` now does would finish the job.
@@ -1,15 +1,15 @@
{
"planPath": "docs/plans/2026-08-15-surgical-deploys.md",
"tasks": [
{"id": 1, "subject": "Task 1: Direct mode threads the already-read blob", "status": "pending"},
{"id": 2, "subject": "Task 2: DeploymentArtifact.Open() — parse once", "status": "pending"},
{"id": 3, "subject": "Task 3: Actors consume Open()", "status": "pending", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: Classifier — surgical eligibility for raw-tag and UNS-reference deltas", "status": "pending"},
{"id": 5, "subject": "Task 5: Applier — apply the new surgical deltas", "status": "pending", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: Delta materialisation", "status": "pending", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Blocking I/O off the host mailbox", "status": "pending", "blockedBy": [3]},
{"id": 8, "subject": "Task 8: Compressed artifact blob", "status": "pending", "blockedBy": [2]},
{"id": 9, "subject": "Task 9: Full verification + docs", "status": "pending", "blockedBy": [3, 6, 7, 8]}
{"id": 1, "subject": "Task 1: Direct mode threads the already-read blob", "status": "completed"},
{"id": 2, "subject": "Task 2: DeploymentArtifact.Open() — parse once", "status": "completed"},
{"id": 3, "subject": "Task 3: Actors consume Open()", "status": "completed", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: Classifier — surgical eligibility for raw-tag and UNS-reference deltas", "status": "completed"},
{"id": 5, "subject": "Task 5: Applier — apply the new surgical deltas", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: Delta materialisation", "status": "completed", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Blocking I/O off the host mailbox", "status": "completed", "blockedBy": [3]},
{"id": 8, "subject": "Task 8: Compressed artifact blob", "status": "completed", "blockedBy": [2]},
{"id": 9, "subject": "Task 9: Full verification + docs", "status": "completed", "blockedBy": [3, 6, 7, 8]}
],
"lastUpdated": "2026-08-15T00:00:00Z"
"lastUpdated": "2026-08-15T22:30:00Z"
}