diff --git a/CLAUDE.md b/CLAUDE.md index 75b1380f..8a4eb123 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,6 +130,53 @@ the single-namespace `EquipmentTags` materialization path are gone. See `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` + `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`. +**How a deploy reaches that address space (surgical-deploys, 2026-08-15).** The old +blanket rule — ANY non-empty `ChangedRawTags` / `ChangedUnsReferenceVariables` routes to +`Rebuild` — is gone, because it made a routine edit a fleet-wide outage: +`RawTagPlan.ReferencingEquipmentPaths` is part of the record's equality, so adding or +dropping **one** `UnsTagReference` flipped its backing raw tag to Changed and tore down +every node on the server, killing every client subscription. `AddressSpaceChangeClassifier` +is still default-closed; what changed is which deltas it whitelists. + +- **Still a full rebuild:** an **alarm-bearing** raw-tag delta (`Alarm != null` on either + side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an + **identity** change on a raw tag (`TagId`/`DriverInstanceId`/`Name`/`ParentNodeId`/`Realm`), + a raw **container** (Folder/Driver/Device/TagGroup) rename or removal, `ChangedEquipment` / + `ChangedAlarms`, a non-surgical equipment-tag delta, and anything else the + `with`-equality whitelist does not name (including any FUTURE field). +- **Now surgical:** a **non-alarm** raw-tag delta over the in-place-updatable attributes + (`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / `ArrayLength`) + and/or `ReferencingEquipmentPaths` — applied via `ISurgicalAddressSpaceSink.UpdateTagAttributes` + in the **Raw** realm — and **every** `UnsReferenceDelta` (add / remove / re-point / + backing-edit). +- ⚠️ **UNS mirrors repair through the RECORD, not the classifier.** A raw-side historian or + shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape + (`EnsureVariable` early-returns on an existing node). `UnsReferenceVariable` therefore + carries the backing tag's `Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}`, so the + same edit makes the reference record unequal and the planner emits a **coincident** + `UnsReferenceDelta`. Adding a newly-inherited field needs no classifier change — **omitting + one silently reopens the staleness.** Repair is **in place** (`UpdateTagAttributes`, Uns + realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an + identity change (re-point, effective-name edit) is remove+recreate — and only for that + one node. +- **A non-rebuild apply materialises the DELTA, not the composition.** `MaterialiseHierarchy` + still runs full (parents must exist); the other passes take the plan's added/changed subsets. + Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread. +- ⚠️ **A degraded non-rebuild apply does NOT advance `_lastApplied`.** The full passes used to + re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the + delta path removed that. Holding the composition back is the replacement — the next deploy + re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that + apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct, + bounded resync). A **rebuild** still advances even with failures. +- **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is + what makes every whitelist above safe. +- ⚠️ **Known gap, no longer neutral:** `AnnounceAddedNodes` is Uns-only, so surgically-added + **raw** containers/tags are materialised but never announced — previously invisible (the + rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses. + Added UNS reference variables ARE announced. + +See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`. + ### Key Concept: Tag Name and FullName Galaxy objects have a **tag_name** — a globally unique system name used for @@ -467,6 +514,16 @@ fail-closed). Things worth knowing before touching it: hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment` stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch. + **The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** — + `ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and + hashes the result, so the invariant holds unchanged and nothing between the composer and + `DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever + looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so + pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every + decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already + handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after + upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be + upgraded BEFORE central starts emitting compressed blobs.** - **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a diff --git a/docs/Configuration.md b/docs/Configuration.md index 86fa1aeb..b83a8c30 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -217,6 +217,39 @@ See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-t **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification. +#### Artifact compression (2026-08-15) + +The artifact blob is **gzip-compressed, inside the hashed bytes**. `ConfigComposer.Pack` gzips the +serialised snapshot (`CompressionLevel.Fastest`, ~10–20x on this highly repetitive JSON) immediately +after `SerializeToUtf8Bytes` and **then** takes the SHA-256, so the compressed bytes *are* the artifact +everywhere — `Deployment.ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream, and the +`RevisionHash` a `FetchAndCache` node verifies against. The invariant above is preserved exactly, which +is why the fetcher, the cache and the serve path are untouched: nothing between the composer and +`DeploymentArtifact.Open` looks inside the blob. + +Decompression lives in exactly one place, `DeploymentArtifact.Open`, which **sniffs the gzip magic +`1F 8B`** at offset 0 — valid JSON can never start with `0x1F`, so no framing of our own is needed and +**pre-upgrade uncompressed rows keep parsing unchanged**. Inflation runs under a 512 MiB guard so a +corrupt or hostile blob cannot OOM the node, and every decompression failure (corrupt, truncated, +over-limit) is reported as `JsonException` — the same exception family the malformed-JSON path already +owns, so no caller's decision changes. + +Two operational facts: + +- **The first deploy after the upgrade re-hashes every revision.** The same configuration now serialises + to different (compressed) bytes, so it produces a different `RevisionHash`. Expect a **one-time full + re-dispatch to every node**, not the usual no-op — this is normal, and it happens once. +- **Rollout ordering matters: upgrade the NODES first.** A node must be running artifact-sniffing code + *before* central starts emitting compressed blobs. A pre-upgrade node handed a gzip artifact cannot + parse it and **fails the apply**, which means it keeps serving last-known-good (the #485 contract) — + degraded but not destructive, and it recovers as soon as it is upgraded. Rolling back central is + likewise safe in the other direction, since sniffing code still reads uncompressed blobs. + +(gzip output is not guaranteed byte-stable across .NET releases. Harmless here: the hash is computed +from the bytes emitted at compose time and never re-derived from the JSON, so an implementation change +costs at most one spurious "changed" revision after a runtime upgrade. Snapshot determinism is +unaffected — compression wraps the final bytes and reorders nothing.) + On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology. **`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now diff --git a/docs/Raw.md b/docs/Raw.md index 03f7836b..144ba49f 100644 --- a/docs/Raw.md +++ b/docs/Raw.md @@ -70,6 +70,39 @@ serialize the endpoint keys, so `DeviceConfig` is the single source of truth. tag-groups" toggle mirrors the browse folder nesting. The browser dials a real ephemeral driver against the **merged** Driver+Device config. +## What a deploy tears down (surgical deploys, 2026-08-15) + +Editing a raw **tag's** attributes is no longer a fleet-wide event. A **non-alarm** +raw-tag delta whose only differences are the in-place-updatable attributes — +`Writable`, `IsHistorized`, `HistorianTagname`, `DataType`, `IsArray`, `ArrayLength` — +and/or its set of referencing equipment is applied **in place** on the existing Raw-realm +node (`ISurgicalAddressSpaceSink.UpdateTagAttributes`), so every client subscription on +that tag and on the rest of the server survives. The same is true of **every** UNS +tag-reference change: adding, removing or re-pointing a reference used to flip the +backing raw tag to Changed (the referencing set is part of its record equality) and +rebuild the whole address space for a one-variable edit. + +**Still a full rebuild**, deliberately: an **alarm-bearing** raw tag (a delta with +`Alarm != null` on either side — only a rebuild clears and re-wires the native-alarm +notifiers that fan the condition to its referencing equipment folders); an **identity** +change on a tag (`TagId`, `DriverInstanceId`, `Name`, `ParentNodeId`, realm); and any +change to a raw **container** — a Folder / Driver / Device / TagGroup **rename or +removal** — because the surgical sink has no folder-remove. The classifier is +default-closed: any field it does not explicitly whitelist, including fields added +later, still routes to a rebuild, and any surgical call returning `false` or throwing +ratchets back to one. + +**Mirror repair.** A raw-side historian or shape edit also reaches every UNS reference +mirroring the tag: the reference record carries the backing tag's historian/shape, so +the edit emits a coincident UNS-reference delta. Identity-preserving repairs update the +mirror **in place** (its MonitoredItems survive); a re-point or effective-name change +removes and recreates **that one mirror node**. See [`Uns.md`](Uns.md) and +[`plans/2026-08-15-surgical-deploys.md`](plans/2026-08-15-surgical-deploys.md). + +⚠️ A surgically added raw container/tag is materialised but **not** announced — +`AnnounceAddedNodes` is UNS-only, so model-aware clients see new raw nodes on their next +browse. This mattered less when every add forced a rebuild (and therefore a re-browse). + ## The `Calculation` driver Signal-level calculated tags are ordinary raw tags bound to a `Calculation` driver diff --git a/docs/Uns.md b/docs/Uns.md index 43689d43..7ccc5238 100644 --- a/docs/Uns.md +++ b/docs/Uns.md @@ -442,6 +442,40 @@ Edits here change the configuration only. As the page header notes, **changes apply on the next deployment** — run a **Deploy** (Deployments page) to push them into the running address space. +### What a deploy actually tears down (surgical deploys, 2026-08-15) + +Adding, removing or re-pointing a UNS **tag reference** no longer rebuilds the +address space. It used to: `RawTagPlan.ReferencingEquipmentPaths` participates in +the backing raw tag's record equality, so touching one reference flipped that tag to +Changed and the classifier's blanket `ChangedRawTags ⇒ Rebuild` rule tore down and +recreated **every** node on the server — killing every client's subscriptions for a +one-variable edit. Now **every** `UnsReferenceDelta` is applied surgically, and so is +a **non-alarm** raw-tag delta over the in-place-updatable attributes +(`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / +`ArrayLength`). A **full rebuild** is still forced by an **alarm-bearing** raw-tag +delta (only a rebuild re-wires native-alarm notifiers), an **identity** change on a +raw tag (`TagId` / `DriverInstanceId` / `Name` / `ParentNodeId` / `Realm`), a raw +**container** rename or removal, and any changed equipment / scripted alarm. The +classifier stays default-closed — anything its whitelist does not name still rebuilds, +and any surgical failure ratchets back to a rebuild. + +**Mirror repair.** A UNS reference variable mirrors its backing raw tag, so a raw-side +historian or shape edit must reach it too — `UnsReferenceVariable` carries the backing +tag's `IsHistorized` / `HistorianTagname` / `IsArray` / `ArrayLength`, which makes the +reference record unequal on such an edit and emits a **coincident** `UnsReferenceDelta` +alongside the raw one. **Identity-preserving** repairs (the mirror keeps the same +NodeId, equipment and backing RawPath) are applied **in place**, so the mirror's +subscriptions survive an operator historizing or reshaping the tag behind it. Only an +**identity change** — a re-point to a different raw tag, or an effective-name/display-name +edit that moves the NodeId — is a remove + recreate, and it affects **that one node**, +not the tree. + +⚠️ Two consequences worth knowing. A surgically **added** UNS reference variable *is* +announced (Part 3 `GeneralModelChangeEvent`), but an added **raw** container/tag is +not — clients pick those up on their next browse. And a **degraded** non-rebuild apply +(some node failed to materialise) deliberately does not record its composition as +applied, so the next deploy re-diffs and repairs it. + ## See also - [Configuration.md](Configuration.md) — the underlying config entities. diff --git a/docs/plans/2026-08-15-surgical-deploys.md b/docs/plans/2026-08-15-surgical-deploys.md index 24b6ea63..6e3d6899 100644 --- a/docs/plans/2026-08-15-surgical-deploys.md +++ b/docs/plans/2026-08-15-surgical-deploys.md @@ -316,3 +316,65 @@ Steps: full `dotnet build ZB.MOM.WW.OtOpcUa.slnx`; run every non-DB test project - **No `AnnounceAddedNodes` extension to raw/UNS adds** — pre-existing gap, noted in comments. - **No cross-actor shared `ParsedDeploymentArtifact`** — one parse per actor is the target; sharing a disposable across mailboxes isn't worth the ownership complexity. - **No artifact schema changes** — the JSON inside the (now compressed) blob is byte-identical. + +--- + +## Follow-ups (recorded, not implemented) + +Found during implementation and review of this plan. None is a regression introduced by the branch +except where stated; each is recorded here rather than folded in, so the branch stays reviewable. + +**(a) Raw-realm `RaiseNodesAddedModelChange` — surgically-added raw nodes are never announced.** +`AnnounceAddedNodes` / `ComputeAddAnnouncements` only walk the equipment (Uns) add sets, and the +announce hardcodes the Uns realm. The UNS half is now closed — an added `UnsReferenceVariable` is +announced against its equipment folder. The **raw** half is not: an added raw container or raw tag is +materialised and then silent. This was neutral before the branch (a plan carrying raw adds went through +a full rebuild, so every client had to re-browse anyway); with surgical adds keeping subscriptions +alive, an unannounced raw add is genuinely invisible until a client happens to browse. The fix is a +realm-aware `RaiseNodesAddedModelChange` call plus raw-side add sets in `ComputeAddAnnouncements`. This +is a real follow-up, not an accepted limitation. + +**(b) `RemoveEquipmentSubtree` shares the dangling-inverse-edge exposure that `RemoveVariableNode` now +fixes.** I3 on this branch made `RemoveVariableNode` sever the inverse half of every outgoing +`Organizes` edge (the variable-node counterpart of `UnwireAlarmNotifiers`) — before, rebuilds swept +those up, and surgical applies made rebuilds rare enough that they accumulated on the backing raw node +for the life of the process. `RemoveEquipmentSubtree` removes a whole equipment folder and has the same +shape of exposure for the edges its descendants own; it was not touched here because it is only reached +by a `PureRemove`/`AddRemoveMix` plan and needs its own test scaffolding. Same class of leak, same fix. + +**(c) End-to-end test: a degraded remove-bearing apply followed by a second deploy.** The C1 resync +semantic — a degraded non-rebuild apply does not advance `_lastApplied`, so the next deploy re-diffs the +same composition, re-emits that apply's removes, and the re-remove's `false` ratchets into a full +rebuild — is reasoned in the code comment and each half is independently pinned (the non-advance, and +the ratchet-on-false). What is **not** covered is the composed path: drive a degraded apply that carried +removes, then a second deploy, and observe the ratcheted rebuild. Worth building because the semantic is +deliberately self-correcting-by-escalation, and an escalation nobody has watched fire is an assumption. + +**(d) Delta materialise of scripted alarms uses thin-plan ids against rich rows.** The alarm subset +handed to the materialise pass is identified by the plan's ids, while the pass reads the full +composition's rich alarm rows. An alarm whose **predicate script** arrives in a later revision than the +alarm row itself is therefore invisible to a delta apply: the alarm id is not in any diff set, so the +subset never names it, and the pass never re-reads it. This is **pre-existing planner blindness** (the +planner has always diffed on the thin shape), but the delta path widens it — the full passes used to +re-read every alarm on every apply and would eventually pick the script up. Until fixed, such an alarm +converges only on the next rebuild. + +**(e) Deploy dispatch carries no monotonic sequence.** Under the gRPC command transport, dispatches are +fire-and-forget with no retry and no ordering guarantee across nodes, so two dispatches issued close +together can be overtaken and "the final revision" is whichever arrived last, not whichever was sealed +last. This is pre-existing and independent of this branch — but the stash-instead-of-forward-spin change +(Task 7 Part B) makes it the **only** remaining nondeterminism in the deferral path, so it is now worth +naming. A monotonic dispatch sequence that a node refuses to go backwards on would close it. + +**(f) LocalDb store ordering under concurrent `StoreAsync`.** With the cache write moved off the mailbox +(`PipeTo`), two applies in quick succession can have their stores in flight simultaneously; the artifact +chunks and the pointer are written separately, so the pointer can briefly name the older artifact. It +self-heals on the next deploy (the newer artifact is stored and the pointer is rewritten), and boot-from- +cache is a fallback rather than a system of record, so the exposure is a stale — never a corrupt — cache +in a narrow window. A store-level sequence guard, or serialising stores per cluster, would remove it. + +**(g) `RestoreApplied` could thread its in-hand blob.** Task 1 removed the redundant central-SQL reads +from the Direct apply path, but the boot/restore path still reads the artifact and then hands the +publish actor a bytes-less `RebuildAddressSpace`, which re-reads it. That is **one** extra SQL read per +node restart — not per apply — which is why it was left out of scope. Threading `appliedBlob` through +`RestoreApplied` the same way `ApplyAndAck` now does would finish the job. diff --git a/docs/plans/2026-08-15-surgical-deploys.md.tasks.json b/docs/plans/2026-08-15-surgical-deploys.md.tasks.json index 347c712e..74f55a05 100644 --- a/docs/plans/2026-08-15-surgical-deploys.md.tasks.json +++ b/docs/plans/2026-08-15-surgical-deploys.md.tasks.json @@ -1,15 +1,15 @@ { "planPath": "docs/plans/2026-08-15-surgical-deploys.md", "tasks": [ - {"id": 1, "subject": "Task 1: Direct mode threads the already-read blob", "status": "pending"}, - {"id": 2, "subject": "Task 2: DeploymentArtifact.Open() — parse once", "status": "pending"}, - {"id": 3, "subject": "Task 3: Actors consume Open()", "status": "pending", "blockedBy": [1, 2]}, - {"id": 4, "subject": "Task 4: Classifier — surgical eligibility for raw-tag and UNS-reference deltas", "status": "pending"}, - {"id": 5, "subject": "Task 5: Applier — apply the new surgical deltas", "status": "pending", "blockedBy": [4]}, - {"id": 6, "subject": "Task 6: Delta materialisation", "status": "pending", "blockedBy": [5]}, - {"id": 7, "subject": "Task 7: Blocking I/O off the host mailbox", "status": "pending", "blockedBy": [3]}, - {"id": 8, "subject": "Task 8: Compressed artifact blob", "status": "pending", "blockedBy": [2]}, - {"id": 9, "subject": "Task 9: Full verification + docs", "status": "pending", "blockedBy": [3, 6, 7, 8]} + {"id": 1, "subject": "Task 1: Direct mode threads the already-read blob", "status": "completed"}, + {"id": 2, "subject": "Task 2: DeploymentArtifact.Open() — parse once", "status": "completed"}, + {"id": 3, "subject": "Task 3: Actors consume Open()", "status": "completed", "blockedBy": [1, 2]}, + {"id": 4, "subject": "Task 4: Classifier — surgical eligibility for raw-tag and UNS-reference deltas", "status": "completed"}, + {"id": 5, "subject": "Task 5: Applier — apply the new surgical deltas", "status": "completed", "blockedBy": [4]}, + {"id": 6, "subject": "Task 6: Delta materialisation", "status": "completed", "blockedBy": [5]}, + {"id": 7, "subject": "Task 7: Blocking I/O off the host mailbox", "status": "completed", "blockedBy": [3]}, + {"id": 8, "subject": "Task 8: Compressed artifact blob", "status": "completed", "blockedBy": [2]}, + {"id": 9, "subject": "Task 9: Full verification + docs", "status": "completed", "blockedBy": [3, 6, 7, 8]} ], - "lastUpdated": "2026-08-15T00:00:00Z" + "lastUpdated": "2026-08-15T22:30:00Z" }