From 4a6a79d02eb7894370823f8dd52afe9f466085fa Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 18 Jun 2026 02:39:00 -0400 Subject: [PATCH] docs(plans): design for array write ergonomics and default-fill partial writes --- ...026-06-18-array-write-ergonomics-design.md | 193 ++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 docs/plans/2026-06-18-array-write-ergonomics-design.md diff --git a/docs/plans/2026-06-18-array-write-ergonomics-design.md b/docs/plans/2026-06-18-array-write-ergonomics-design.md new file mode 100644 index 0000000..30d3687 --- /dev/null +++ b/docs/plans/2026-06-18-array-write-ergonomics-design.md @@ -0,0 +1,193 @@ +# Array Write Ergonomics & Default-Fill Partial Writes — Design + +Date: 2026-06-18 + +## Problem + +Writing array-typed MXAccess attributes through the gateway has two ergonomic +shortfalls: + +1. **Asymmetric addressing.** An array attribute reads fine by its bare name + (`Obj.Arr`), but writes require the `[]` body suffix (`Obj.Arr[]`). The + handle registered from the bare name is read-capable but not cleanly + write-capable. +2. **Whole-array writes only.** Every write replaces the entire array; to change + a few elements the client must marshal and send the full array. This is a + native MXAccess COM constraint (there is no element-wise write API), but it + pushes avoidable cost onto clients for large arrays. + +This design removes both frictions without breaking MXAccess parity. The worker +is not modified — it continues to perform an honest whole-array COM write. All +new behavior lives in the gateway and the contract. + +## Why MXAccess forces whole-array writes + +The native MXAccess COM `Write` takes a complete VARIANT (`SAFEARRAY` for +arrays). There is no `WriteArrayElement(index, value)`. Confirmed in the worker: +`VariantConverter.ConvertToComArray` marshals the entire CLR array in one shot, +and `MxAccessSession.Write` forwards it verbatim to the COM proxy. Any "partial +write" feature must therefore reconstruct a full array before the COM call. + +We deliberately do **not** reconstruct it from current state (no +read-modify-write merge): that would add latency, cache-staleness, and a race +window against other writers, and would paper over MXAccess semantics. Instead +partial writes are **stateless default-fill** (see below). + +## Goals + +- Writing an array attribute by its bare name works like reading does — the + gateway appends `[]` automatically when it knows the attribute is an array. +- A client can send only the indices it wants plus a total length, instead of + the full array. + +## Non-goals + +- **No preserve-unchanged merge.** Unmentioned indices are written as the + element type's default, **not** kept at their current value. +- No element-wise COM write — MXAccess has no such API; every write is + whole-array and we keep it that way. +- No change to `ReadBulk` string addressing. +- The gateway does **not** infer total length; the client supplies it. + +## Decisions (resolved during brainstorming) + +| Question | Decision | +|---|---| +| Scope | Both: suffix ergonomics **and** partial writes | +| Partial-write semantics | Stateless **default-fill**: unmentioned indices = type default (reset, not preserved) | +| Total length | **Client specifies** `total_length` explicitly | +| Time/timestamp default | **Unix epoch** | +| Suffix fix location/actor | **Gateway**, using in-memory Galaxy `is_array` metadata, at **AddItem** time | +| Suffix fallback when metadata unavailable | **Pass through unchanged** (no regression) | +| Partial-write contract shape | New `MxSparseArray` as a `oneof` arm on `MxValue` | +| Per-client helpers | **Included** in this change | + +## Contract changes (`mxaccess_gateway.proto`) + +A write-only sparse representation, added as a `oneof kind` arm on `MxValue` so +every write command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`, +`WriteBulkEntry`) accepts it without new RPCs: + +```proto +message MxSparseArray { + MxDataType element_data_type = 1; + uint32 total_length = 2; + repeated MxSparseElement elements = 3; +} + +message MxSparseElement { + uint32 index = 1; + MxValue value = 2; // scalar +} + +// added to MxValue oneof kind: +// MxSparseArray sparse_array_value = 19; +``` + +`sparse_array_value` is **write-only**: the worker never produces it, and the +gateway rejects it on any read/event path. Regenerate `Generated/` and commit +the generated `.cs` (the net48 worker build needs the checked-in types — see the +proto-codegen-regen rule). + +## Suffix normalization — at `AddItem`, in the gateway + +The item handle binds to the literal address string at `AddItem` and is reused +for both reads and writes; at write time only the integer handle is available, +which is too late to change the address. So normalization happens at +registration. + +In the gateway's `AddItemCommand` / `AddItem2Command` handling +(`GatewaySession`), before forwarding to the worker: + +1. If `item_definition` already ends with `[]` → leave unchanged. +2. Else look up `item_definition + "[]"` in the in-memory Galaxy hierarchy cache + (`IGalaxyHierarchyCache` → `GalaxyTagLookup.Attribute.IsArray`). The index is + keyed by `FullTagReference`, which already carries the `[]` suffix for + arrays, so the lookup key must include `[]`. If found and `is_array` → + rewrite `item_definition` to the `[]` form. +3. **Fallback:** metadata unavailable or address not found as an array → + forward verbatim (current behavior). + +Store the **normalized** address in `SessionItemRegistration.TagAddress` so +write-time constraint checks (`ConstraintEnforcer`) and readback resolve +consistently against the `[]`-keyed index. + +This is safe for reads: both the bare and `[]` forms return the array on read, +so promoting a registration to the `[]` form does not change read behavior — it +only makes the handle write-capable. + +`AddItem2Command` (with `item_context`) normalizes `item_definition` the same +way. `ReadBulk` is unaffected — it uses raw address strings with its own +ephemeral registration, so bare-name reads continue to work unchanged. + +## Partial-write expansion — at the gateway, worker untouched + +In the gateway write path, before forwarding any write command to the worker, if +`MxValue.KindCase == SparseArrayValue`: + +1. Allocate a full array of `total_length`, element type `element_data_type`. +2. Initialize every slot to the type default: + - `bool` → `false` + - `int32` / `int64` → `0` + - `float` / `double` → `0` + - `string` → `""` + - `time` / `timestamp` → Unix epoch +3. For each `MxSparseElement`, set `array[index]` from the scalar `value`. +4. Replace the `MxValue` with a normal `array_value` (full `MxArray`). + +The worker then receives an ordinary whole-array `MxValue`; +`VariantConverter.ConvertToComArray` and the COM `Write` are unchanged. Parity +preserved — it really is a whole-array write. + +Expansion is applied uniformly to every write variant by normalizing the +`MxValue` of each command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`, +and each `WriteBulkEntry`) before it leaves the gateway. + +## Validation & errors (gateway, `InvalidArgument`) + +- `total_length == 0`, or any `index >= total_length` → reject. +- Duplicate indices → reject (no silent last-wins). +- `element_data_type` must be a supported scalar element type (not `Raw` / + `Unspecified`); each element `value` must match it. +- Empty `elements` with `total_length = N` → valid: writes an all-defaults array + of length N (explicit reset). +- A sparse value arriving on a read/event path → reject (guard; the worker never + produces one). + +## Clients (all five) & docs — same change + +Per the repo rule that docs change with the source: + +- Regenerate proto types for dotnet, go, python, rust, java. Watch the Java + generated-file churn — revert spurious protobuf-version diffs when no `.proto` + semantics changed beyond the new messages; commit the net48-relevant regen. +- Add a thin per-client helper to build a sparse write, e.g. + `WriteArrayElements(handle, totalLength, {index → value})`. +- Update the **"Array writes replace the whole array"** section in all five + client READMEs: document default-fill semantics (unmentioned = reset to + default, not preserved), the `total_length` requirement, and that bare-name + array writes now auto-normalize to the `[]` form. +- Update `gateway.md` (command/value surface) and the value-conversion doc. + +## Testing + +- **Gateway (FakeWorkerHarness):** + - Sparse → full expansion per element type; default-fill sizing; correct + placement of specified indices. + - `total_length == 0`, index-out-of-range, and duplicate-index rejection. + - Empty-elements all-defaults case. + - Suffix normalization: bare array → `[]`; bare scalar → unchanged; + already-`[]` → unchanged; metadata-cold → pass-through. +- **Clients:** helper + round-trip serialization per language. +- **Live MXAccess (opt-in, windev):** one default-fill write and one bare-name + array write against real COM. + +## Affected components + +- Contracts: `mxaccess_gateway.proto` + regenerated `Generated/`. +- Gateway: `GatewaySession` (AddItem normalization, write expansion), + `SessionItemRegistration` (store normalized address), interaction with + `IGalaxyHierarchyCache` / `GalaxyTagLookup`. +- Worker: **unchanged.** +- Clients: dotnet, go, python, rust, java (regenerated types + helper + README). +- Docs: `gateway.md`, value-conversion doc, five client READMEs.