docs(plans): design for array write ergonomics and default-fill partial writes
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,193 @@
|
||||
# Array Write Ergonomics & Default-Fill Partial Writes — Design
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
## Problem
|
||||
|
||||
Writing array-typed MXAccess attributes through the gateway has two ergonomic
|
||||
shortfalls:
|
||||
|
||||
1. **Asymmetric addressing.** An array attribute reads fine by its bare name
|
||||
(`Obj.Arr`), but writes require the `[]` body suffix (`Obj.Arr[]`). The
|
||||
handle registered from the bare name is read-capable but not cleanly
|
||||
write-capable.
|
||||
2. **Whole-array writes only.** Every write replaces the entire array; to change
|
||||
a few elements the client must marshal and send the full array. This is a
|
||||
native MXAccess COM constraint (there is no element-wise write API), but it
|
||||
pushes avoidable cost onto clients for large arrays.
|
||||
|
||||
This design removes both frictions without breaking MXAccess parity. The worker
|
||||
is not modified — it continues to perform an honest whole-array COM write. All
|
||||
new behavior lives in the gateway and the contract.
|
||||
|
||||
## Why MXAccess forces whole-array writes
|
||||
|
||||
The native MXAccess COM `Write` takes a complete VARIANT (`SAFEARRAY` for
|
||||
arrays). There is no `WriteArrayElement(index, value)`. Confirmed in the worker:
|
||||
`VariantConverter.ConvertToComArray` marshals the entire CLR array in one shot,
|
||||
and `MxAccessSession.Write` forwards it verbatim to the COM proxy. Any "partial
|
||||
write" feature must therefore reconstruct a full array before the COM call.
|
||||
|
||||
We deliberately do **not** reconstruct it from current state (no
|
||||
read-modify-write merge): that would add latency, cache-staleness, and a race
|
||||
window against other writers, and would paper over MXAccess semantics. Instead
|
||||
partial writes are **stateless default-fill** (see below).
|
||||
|
||||
## Goals
|
||||
|
||||
- Writing an array attribute by its bare name works like reading does — the
|
||||
gateway appends `[]` automatically when it knows the attribute is an array.
|
||||
- A client can send only the indices it wants plus a total length, instead of
|
||||
the full array.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **No preserve-unchanged merge.** Unmentioned indices are written as the
|
||||
element type's default, **not** kept at their current value.
|
||||
- No element-wise COM write — MXAccess has no such API; every write is
|
||||
whole-array and we keep it that way.
|
||||
- No change to `ReadBulk` string addressing.
|
||||
- The gateway does **not** infer total length; the client supplies it.
|
||||
|
||||
## Decisions (resolved during brainstorming)
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Scope | Both: suffix ergonomics **and** partial writes |
|
||||
| Partial-write semantics | Stateless **default-fill**: unmentioned indices = type default (reset, not preserved) |
|
||||
| Total length | **Client specifies** `total_length` explicitly |
|
||||
| Time/timestamp default | **Unix epoch** |
|
||||
| Suffix fix location/actor | **Gateway**, using in-memory Galaxy `is_array` metadata, at **AddItem** time |
|
||||
| Suffix fallback when metadata unavailable | **Pass through unchanged** (no regression) |
|
||||
| Partial-write contract shape | New `MxSparseArray` as a `oneof` arm on `MxValue` |
|
||||
| Per-client helpers | **Included** in this change |
|
||||
|
||||
## Contract changes (`mxaccess_gateway.proto`)
|
||||
|
||||
A write-only sparse representation, added as a `oneof kind` arm on `MxValue` so
|
||||
every write command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
|
||||
`WriteBulkEntry`) accepts it without new RPCs:
|
||||
|
||||
```proto
|
||||
message MxSparseArray {
|
||||
MxDataType element_data_type = 1;
|
||||
uint32 total_length = 2;
|
||||
repeated MxSparseElement elements = 3;
|
||||
}
|
||||
|
||||
message MxSparseElement {
|
||||
uint32 index = 1;
|
||||
MxValue value = 2; // scalar
|
||||
}
|
||||
|
||||
// added to MxValue oneof kind:
|
||||
// MxSparseArray sparse_array_value = 19;
|
||||
```
|
||||
|
||||
`sparse_array_value` is **write-only**: the worker never produces it, and the
|
||||
gateway rejects it on any read/event path. Regenerate `Generated/` and commit
|
||||
the generated `.cs` (the net48 worker build needs the checked-in types — see the
|
||||
proto-codegen-regen rule).
|
||||
|
||||
## Suffix normalization — at `AddItem`, in the gateway
|
||||
|
||||
The item handle binds to the literal address string at `AddItem` and is reused
|
||||
for both reads and writes; at write time only the integer handle is available,
|
||||
which is too late to change the address. So normalization happens at
|
||||
registration.
|
||||
|
||||
In the gateway's `AddItemCommand` / `AddItem2Command` handling
|
||||
(`GatewaySession`), before forwarding to the worker:
|
||||
|
||||
1. If `item_definition` already ends with `[]` → leave unchanged.
|
||||
2. Else look up `item_definition + "[]"` in the in-memory Galaxy hierarchy cache
|
||||
(`IGalaxyHierarchyCache` → `GalaxyTagLookup.Attribute.IsArray`). The index is
|
||||
keyed by `FullTagReference`, which already carries the `[]` suffix for
|
||||
arrays, so the lookup key must include `[]`. If found and `is_array` →
|
||||
rewrite `item_definition` to the `[]` form.
|
||||
3. **Fallback:** metadata unavailable or address not found as an array →
|
||||
forward verbatim (current behavior).
|
||||
|
||||
Store the **normalized** address in `SessionItemRegistration.TagAddress` so
|
||||
write-time constraint checks (`ConstraintEnforcer`) and readback resolve
|
||||
consistently against the `[]`-keyed index.
|
||||
|
||||
This is safe for reads: both the bare and `[]` forms return the array on read,
|
||||
so promoting a registration to the `[]` form does not change read behavior — it
|
||||
only makes the handle write-capable.
|
||||
|
||||
`AddItem2Command` (with `item_context`) normalizes `item_definition` the same
|
||||
way. `ReadBulk` is unaffected — it uses raw address strings with its own
|
||||
ephemeral registration, so bare-name reads continue to work unchanged.
|
||||
|
||||
## Partial-write expansion — at the gateway, worker untouched
|
||||
|
||||
In the gateway write path, before forwarding any write command to the worker, if
|
||||
`MxValue.KindCase == SparseArrayValue`:
|
||||
|
||||
1. Allocate a full array of `total_length`, element type `element_data_type`.
|
||||
2. Initialize every slot to the type default:
|
||||
- `bool` → `false`
|
||||
- `int32` / `int64` → `0`
|
||||
- `float` / `double` → `0`
|
||||
- `string` → `""`
|
||||
- `time` / `timestamp` → Unix epoch
|
||||
3. For each `MxSparseElement`, set `array[index]` from the scalar `value`.
|
||||
4. Replace the `MxValue` with a normal `array_value` (full `MxArray`).
|
||||
|
||||
The worker then receives an ordinary whole-array `MxValue`;
|
||||
`VariantConverter.ConvertToComArray` and the COM `Write` are unchanged. Parity
|
||||
preserved — it really is a whole-array write.
|
||||
|
||||
Expansion is applied uniformly to every write variant by normalizing the
|
||||
`MxValue` of each command (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
|
||||
and each `WriteBulkEntry`) before it leaves the gateway.
|
||||
|
||||
## Validation & errors (gateway, `InvalidArgument`)
|
||||
|
||||
- `total_length == 0`, or any `index >= total_length` → reject.
|
||||
- Duplicate indices → reject (no silent last-wins).
|
||||
- `element_data_type` must be a supported scalar element type (not `Raw` /
|
||||
`Unspecified`); each element `value` must match it.
|
||||
- Empty `elements` with `total_length = N` → valid: writes an all-defaults array
|
||||
of length N (explicit reset).
|
||||
- A sparse value arriving on a read/event path → reject (guard; the worker never
|
||||
produces one).
|
||||
|
||||
## Clients (all five) & docs — same change
|
||||
|
||||
Per the repo rule that docs change with the source:
|
||||
|
||||
- Regenerate proto types for dotnet, go, python, rust, java. Watch the Java
|
||||
generated-file churn — revert spurious protobuf-version diffs when no `.proto`
|
||||
semantics changed beyond the new messages; commit the net48-relevant regen.
|
||||
- Add a thin per-client helper to build a sparse write, e.g.
|
||||
`WriteArrayElements(handle, totalLength, {index → value})`.
|
||||
- Update the **"Array writes replace the whole array"** section in all five
|
||||
client READMEs: document default-fill semantics (unmentioned = reset to
|
||||
default, not preserved), the `total_length` requirement, and that bare-name
|
||||
array writes now auto-normalize to the `[]` form.
|
||||
- Update `gateway.md` (command/value surface) and the value-conversion doc.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Gateway (FakeWorkerHarness):**
|
||||
- Sparse → full expansion per element type; default-fill sizing; correct
|
||||
placement of specified indices.
|
||||
- `total_length == 0`, index-out-of-range, and duplicate-index rejection.
|
||||
- Empty-elements all-defaults case.
|
||||
- Suffix normalization: bare array → `[]`; bare scalar → unchanged;
|
||||
already-`[]` → unchanged; metadata-cold → pass-through.
|
||||
- **Clients:** helper + round-trip serialization per language.
|
||||
- **Live MXAccess (opt-in, windev):** one default-fill write and one bare-name
|
||||
array write against real COM.
|
||||
|
||||
## Affected components
|
||||
|
||||
- Contracts: `mxaccess_gateway.proto` + regenerated `Generated/`.
|
||||
- Gateway: `GatewaySession` (AddItem normalization, write expansion),
|
||||
`SessionItemRegistration` (store normalized address), interaction with
|
||||
`IGalaxyHierarchyCache` / `GalaxyTagLookup`.
|
||||
- Worker: **unchanged.**
|
||||
- Clients: dotnet, go, python, rust, java (regenerated types + helper + README).
|
||||
- Docs: `gateway.md`, value-conversion doc, five client READMEs.
|
||||
Reference in New Issue
Block a user