mxaccessgw/docs/plans/2026-06-18-array-write-ergonomics-design.md

# Array Write Ergonomics & Default-Fill Partial Writes — Design

Date: 2026-06-18

## Problem

Writing array-typed MXAccess attributes through the gateway has two ergonomic
shortfalls:

1. **Asymmetric addressing.** An array attribute reads fine by its bare name
   (`Obj.Arr`), but writes require the `[]` body suffix (`Obj.Arr[]`). The
   handle registered from the bare name is read-capable but not cleanly
   write-capable.
2. **Whole-array writes only.** Every write replaces the entire array; to change
   a few elements the client must marshal and send the full array. This is a
   native MXAccess COM constraint (there is no element-wise write API), but it
   pushes avoidable cost onto clients for large arrays.

This design removes both frictions without breaking MXAccess parity. The worker
is not modified — it continues to perform an honest whole-array COM write. All
new behavior lives in the gateway and the contract.

## Why MXAccess forces whole-array writes

The native MXAccess COM `Write` takes a complete VARIANT (`SAFEARRAY` for
arrays). There is no `WriteArrayElement(index, value)`. Confirmed in the worker:
`VariantConverter.ConvertToComArray` marshals the entire CLR array in one shot,
and `MxAccessSession.Write` forwards it verbatim to the COM proxy. Any "partial
write" feature must therefore reconstruct a full array before the COM call.

We deliberately do **not** reconstruct it from current state (no
read-modify-write merge): that would add latency, cache-staleness, and a race
window against other writers, and would paper over MXAccess semantics. Instead
partial writes are **stateless default-fill** (see below).

## Goals

- Writing an array attribute by its bare name works like reading does — the
  gateway appends `[]` automatically when it knows the attribute is an array.
- A client can send only the indices it wants plus a total length, instead of
  the full array.

## Non-goals

- **No preserve-unchanged merge.** Unmentioned indices are written as the
  element type's default, **not** kept at their current value.
- No element-wise COM write — MXAccess has no such API; every write is
  whole-array and we keep it that way.
- No change to `ReadBulk` string addressing.
- The gateway does **not** infer total length; the client supplies it.

## Decisions (resolved during brainstorming)

| Question | Decision |
|---|---|
| Scope | Both: suffix ergonomics **and** partial writes |
| Partial-write semantics | Stateless **default-fill**: unmentioned indices = type default (reset, not preserved) |
| Total length | **Client specifies** `total_length` explicitly |
| Time/timestamp default | **Unix epoch** |
| Suffix fix location/actor | **Gateway**, using in-memory Galaxy `is_array` metadata, at **AddItem** time |
| Suffix fallback when metadata unavailable | **Pass through unchanged** (no regression) |
| Partial-write contract shape | New `MxSparseArray` as a `oneof` arm on `MxValue` |
| Per-client helpers | **Included** in this change |

## Contract changes (`mxaccess_gateway.proto`)

A write-only sparse representation, added as a `oneof kind` arm on `MxValue` so
every write command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
`WriteBulkEntry`) accepts it without new RPCs:

```proto
message MxSparseArray {
  MxDataType element_data_type = 1;
  uint32 total_length = 2;
  repeated MxSparseElement elements = 3;
}

message MxSparseElement {
  uint32 index = 1;
  MxValue value = 2;   // scalar
}

// added to MxValue oneof kind:
//   MxSparseArray sparse_array_value = 19;
```

`sparse_array_value` is **write-only**: the worker never produces it, and the
gateway rejects it on any read/event path. Regenerate `Generated/` and commit
the generated `.cs` (the net48 worker build needs the checked-in types — see the
proto-codegen-regen rule).

## Suffix normalization — at `AddItem`, in the gateway

The item handle binds to the literal address string at `AddItem` and is reused
for both reads and writes; at write time only the integer handle is available,
which is too late to change the address. So normalization happens at
registration.

In the gateway's `AddItemCommand` / `AddItem2Command` handling
(`GatewaySession`), before forwarding to the worker:

1. If `item_definition` already ends with `[]` → leave unchanged.
2. Else look up `item_definition + "[]"` in the in-memory Galaxy hierarchy cache
   (`IGalaxyHierarchyCache` → `GalaxyTagLookup.Attribute.IsArray`). The index is
   keyed by `FullTagReference`, which already carries the `[]` suffix for
   arrays, so the lookup key must include `[]`. If found and `is_array` →
   rewrite `item_definition` to the `[]` form.
3. **Fallback:** metadata unavailable or address not found as an array →
   forward verbatim (current behavior).

Store the **normalized** address in `SessionItemRegistration.TagAddress` so
write-time constraint checks (`ConstraintEnforcer`) and readback resolve
consistently against the `[]`-keyed index.

This is safe for reads: both the bare and `[]` forms return the array on read,
so promoting a registration to the `[]` form does not change read behavior — it
only makes the handle write-capable.

`AddItem2Command` (with `item_context`) normalizes `item_definition` the same
way. `ReadBulk` is unaffected — it uses raw address strings with its own
ephemeral registration, so bare-name reads continue to work unchanged.

## Partial-write expansion — at the gateway, worker untouched

In the gateway write path, before forwarding any write command to the worker, if
`MxValue.KindCase == SparseArrayValue`:

1. Allocate a full array of `total_length`, element type `element_data_type`.
2. Initialize every slot to the type default:
   - `bool` → `false`
   - `int32` / `int64` → `0`
   - `float` / `double` → `0`
   - `string` → `""`
   - `time` / `timestamp` → Unix epoch
3. For each `MxSparseElement`, set `array[index]` from the scalar `value`.
4. Replace the `MxValue` with a normal `array_value` (full `MxArray`).

The worker then receives an ordinary whole-array `MxValue`;
`VariantConverter.ConvertToComArray` and the COM `Write` are unchanged. Parity
preserved — it really is a whole-array write.

Expansion is applied uniformly to every write variant by normalizing the
`MxValue` of each command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
and each `WriteBulkEntry`) before it leaves the gateway.

## Validation & errors (gateway, `InvalidArgument`)

- `total_length == 0`, or any `index >= total_length` → reject.
- Duplicate indices → reject (no silent last-wins).
- `element_data_type` must be a supported scalar element type (not `Raw` /
  `Unspecified`); each element `value` must match it.
- Empty `elements` with `total_length = N` → valid: writes an all-defaults array
  of length N (explicit reset).
- A sparse value arriving on a read/event path → reject (guard; the worker never
  produces one).

## Clients (all five) & docs — same change

Per the repo rule that docs change with the source:

- Regenerate proto types for dotnet, go, python, rust, java. Watch the Java
  generated-file churn — revert spurious protobuf-version diffs when no `.proto`
  semantics changed beyond the new messages; commit the net48-relevant regen.
- Add a thin per-client helper to build a sparse write, e.g.
  `WriteArrayElements(handle, totalLength, {index → value})`.
- Update the **"Array writes replace the whole array"** section in all five
  client READMEs: document default-fill semantics (unmentioned = reset to
  default, not preserved), the `total_length` requirement, and that bare-name
  array writes now auto-normalize to the `[]` form.
- Update `gateway.md` (command/value surface) and the value-conversion doc.

## Testing

- **Gateway (FakeWorkerHarness):**
  - Sparse → full expansion per element type; default-fill sizing; correct
    placement of specified indices.
  - `total_length == 0`, index-out-of-range, and duplicate-index rejection.
  - Empty-elements all-defaults case.
  - Suffix normalization: bare array → `[]`; bare scalar → unchanged;
    already-`[]` → unchanged; metadata-cold → pass-through.
- **Clients:** helper + round-trip serialization per language.
- **Live MXAccess (opt-in, windev):** one default-fill write and one bare-name
  array write against real COM.

## Affected components

- Contracts: `mxaccess_gateway.proto` + regenerated `Generated/`.
- Gateway: `GatewaySession` (AddItem normalization, write expansion),
  `SessionItemRegistration` (store normalized address), interaction with
  `IGalaxyHierarchyCache` / `GalaxyTagLookup`.
- Worker: **unchanged.**
- Clients: dotnet, go, python, rust, java (regenerated types + helper + README).
- Docs: `gateway.md`, value-conversion doc, five client READMEs.