mxaccessgw/docs/plans/2026-06-18-array-write-ergonomics-design.md

Array Write Ergonomics & Default-Fill Partial Writes — Design

Date: 2026-06-18

Problem

Writing array-typed MXAccess attributes through the gateway has two ergonomic shortfalls:

1. Asymmetric addressing. An array attribute reads fine by its bare name (Obj.Arr), but writes require the [] body suffix (Obj.Arr[]). The handle registered from the bare name is read-capable but not cleanly write-capable.
2. Whole-array writes only. Every write replaces the entire array; to change a few elements the client must marshal and send the full array. This is a native MXAccess COM constraint (there is no element-wise write API), but it pushes avoidable cost onto clients for large arrays.

This design removes both frictions without breaking MXAccess parity. The worker is not modified — it continues to perform an honest whole-array COM write. All new behavior lives in the gateway and the contract.

Why MXAccess forces whole-array writes

The native MXAccess COM Write takes a complete VARIANT (SAFEARRAY for arrays). There is no WriteArrayElement(index, value). Confirmed in the worker: VariantConverter.ConvertToComArray marshals the entire CLR array in one shot, and MxAccessSession.Write forwards it verbatim to the COM proxy. Any "partial write" feature must therefore reconstruct a full array before the COM call.

We deliberately do not reconstruct it from current state (no read-modify-write merge): that would add latency, cache-staleness, and a race window against other writers, and would paper over MXAccess semantics. Instead partial writes are stateless default-fill (see below).

Goals

• Writing an array attribute by its bare name works like reading does — the gateway appends [] automatically when it knows the attribute is an array.
• A client can send only the indices it wants plus a total length, instead of the full array.

Non-goals

• No preserve-unchanged merge. Unmentioned indices are written as the element type's default, not kept at their current value.
• No element-wise COM write — MXAccess has no such API; every write is whole-array and we keep it that way.
• No change to ReadBulk string addressing.
• The gateway does not infer total length; the client supplies it.

Decisions (resolved during brainstorming)

Question	Decision
Scope	Both: suffix ergonomics and partial writes
Partial-write semantics	Stateless default-fill: unmentioned indices = type default (reset, not preserved)
Total length	Client specifies total_length explicitly
Time/timestamp default	Unix epoch
Suffix fix location/actor	Gateway, using in-memory Galaxy is_array metadata, at AddItem time
Suffix fallback when metadata unavailable	Pass through unchanged (no regression)
Partial-write contract shape	New MxSparseArray as a oneof arm on MxValue
Per-client helpers	Included in this change

Contract changes (mxaccess_gateway.proto)

A write-only sparse representation, added as a oneof kind arm on MxValue so every write command (Write, Write2, WriteSecured, WriteSecured2, WriteBulkEntry) accepts it without new RPCs:

message MxSparseArray {
  MxDataType element_data_type = 1;
  uint32 total_length = 2;
  repeated MxSparseElement elements = 3;
}

message MxSparseElement {
  uint32 index = 1;
  MxValue value = 2;   // scalar
}

// added to MxValue oneof kind:
//   MxSparseArray sparse_array_value = 19;

sparse_array_value is write-only: the worker never produces it, and the gateway rejects it on any read/event path. Regenerate Generated/ and commit the generated .cs (the net48 worker build needs the checked-in types — see the proto-codegen-regen rule).

Suffix normalization — at AddItem, in the gateway

The item handle binds to the literal address string at AddItem and is reused for both reads and writes; at write time only the integer handle is available, which is too late to change the address. So normalization happens at registration.

In the gateway's AddItemCommand / AddItem2Command handling (GatewaySession), before forwarding to the worker:

1. If item_definition already ends with [] → leave unchanged.
2. Else look up item_definition + "[]" in the in-memory Galaxy hierarchy cache (IGalaxyHierarchyCache → GalaxyTagLookup.Attribute.IsArray). The index is keyed by FullTagReference, which already carries the [] suffix for arrays, so the lookup key must include []. If found and is_array → rewrite item_definition to the [] form.
3. Fallback: metadata unavailable or address not found as an array → forward verbatim (current behavior).

Store the normalized address in SessionItemRegistration.TagAddress so write-time constraint checks (ConstraintEnforcer) and readback resolve consistently against the []-keyed index.

This is safe for reads: both the bare and [] forms return the array on read, so promoting a registration to the [] form does not change read behavior — it only makes the handle write-capable.

AddItem2Command (with item_context) normalizes item_definition the same way. ReadBulk is unaffected — it uses raw address strings with its own ephemeral registration, so bare-name reads continue to work unchanged.

Partial-write expansion — at the gateway, worker untouched

In the gateway write path, before forwarding any write command to the worker, if MxValue.KindCase == SparseArrayValue:

1. Allocate a full array of total_length, element type element_data_type.
2. Initialize every slot to the type default:
   • bool → false
   • int32 / int64 → 0
   • float / double → 0
   • string → ""
   • time / timestamp → Unix epoch
3. For each MxSparseElement, set array[index] from the scalar value.
4. Replace the MxValue with a normal array_value (full MxArray).

The worker then receives an ordinary whole-array MxValue; VariantConverter.ConvertToComArray and the COM Write are unchanged. Parity preserved — it really is a whole-array write.

Expansion is applied uniformly to every write variant by normalizing the MxValue of each command (Write, Write2, WriteSecured, WriteSecured2, and each WriteBulkEntry) before it leaves the gateway.

Validation & errors (gateway, InvalidArgument)

• total_length == 0, or any index >= total_length → reject.
• Duplicate indices → reject (no silent last-wins).
• element_data_type must be a supported scalar element type (not Raw / Unspecified); each element value must match it.
• Empty elements with total_length = N → valid: writes an all-defaults array of length N (explicit reset).
• A sparse value arriving on a read/event path → reject (guard; the worker never produces one).

Clients (all five) & docs — same change

Per the repo rule that docs change with the source:

• Regenerate proto types for dotnet, go, python, rust, java. Watch the Java generated-file churn — revert spurious protobuf-version diffs when no .proto semantics changed beyond the new messages; commit the net48-relevant regen.
• Add a thin per-client helper to build a sparse write, e.g. WriteArrayElements(handle, totalLength, {index → value}).
• Update the "Array writes replace the whole array" section in all five client READMEs: document default-fill semantics (unmentioned = reset to default, not preserved), the total_length requirement, and that bare-name array writes now auto-normalize to the [] form.
• Update gateway.md (command/value surface) and the value-conversion doc.

Testing

• Gateway (FakeWorkerHarness):
  • Sparse → full expansion per element type; default-fill sizing; correct placement of specified indices.
  • total_length == 0, index-out-of-range, and duplicate-index rejection.
  • Empty-elements all-defaults case.
  • Suffix normalization: bare array → []; bare scalar → unchanged; already-[] → unchanged; metadata-cold → pass-through.
• Clients: helper + round-trip serialization per language.
• Live MXAccess (opt-in, windev): one default-fill write and one bare-name array write against real COM.

Affected components

• Contracts: mxaccess_gateway.proto + regenerated Generated/.
• Gateway: GatewaySession (AddItem normalization, write expansion), SessionItemRegistration (store normalized address), interaction with IGalaxyHierarchyCache / GalaxyTagLookup.
• Worker: unchanged.
• Clients: dotnet, go, python, rust, java (regenerated types + helper + README).
• Docs: gateway.md, value-conversion doc, five client READMEs.