diff --git a/docs/plans/2026-06-18-array-write-ergonomics.md b/docs/plans/2026-06-18-array-write-ergonomics.md new file mode 100644 index 0000000..dfe8240 --- /dev/null +++ b/docs/plans/2026-06-18-array-write-ergonomics.md @@ -0,0 +1,329 @@ +# Array Write Ergonomics & Default-Fill Partial Writes — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Let clients write array attributes by their bare name (gateway auto-appends `[]` at AddItem), and write a sparse, default-filled array (only the indices they care about + a total length) instead of marshalling the whole array. + +**Architecture:** All new behavior lives in the **contract** and the **gateway**; the worker is untouched and keeps doing an honest whole-array COM write. The gateway intercepts outbound commands at the single choke point `GatewaySession.InvokeAsync(WorkerCommand)`: it (a) normalizes `AddItem`/`AddItem2` `item_definition` to the `[]` form when Galaxy metadata says the attribute is an array, and (b) expands an `MxSparseArray` write value into a full default-filled `MxArray` before it leaves the gateway. Partial writes are **stateless default-fill** — unmentioned indices are the type default (reset), never preserved. + +**Tech Stack:** .NET 10 (gateway) / .NET Framework 4.8 x86 (worker, unchanged), protobuf + Grpc.Tools, xUnit + FakeWorkerHarness; clients in C#, Go, Python, Rust, Java. + +**Design doc:** `docs/plans/2026-06-18-array-write-ergonomics-design.md` + +**Key references for the implementer:** +- Choke point for all outbound commands: `src/ZB.MOM.WW.MxGateway.Server/Sessions/GatewaySession.cs:947-955` (`InvokeAsync(WorkerCommand command, ...)`). `command.Command` is the `MxCommand`. +- Handle→address tracking: `GatewaySession.TrackCommandReply` (lines 975-1014, AddItem at 989, AddItem2 at 992) → `TrackItem` (1826-1837) → `SessionItemRegistration` record (`Sessions/SessionItemRegistration.cs`). Tracking reads the **same** `MxCommand` instance that passed through `InvokeAsync`, so mutating `item_definition` there flows through automatically. +- Galaxy metadata lookup: `IGalaxyHierarchyCache.Current.Index.TagsByAddress.TryGetValue(addr, out GalaxyTagLookup)`, then `lookup.Attribute?.IsArray`. The index is keyed by `FullTagReference`, which **already contains** `[]` for arrays — look up `addr + "[]"`. See `Security/Authorization/ConstraintEnforcer.cs:15-17,197-204` for the injection + lookup pattern. +- Proto: `src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto` — `MxValue` (1026-1044), `MxArray` (1046-1063), `WriteCommand` (244-249), `Write2Command` (251-257), `WriteSecuredCommand`/`WriteSecured2Command`, `WriteBulk*`, `AddItemCommand` (192-195), `AddItem2Command` (197-201). Generated into `Contracts/Generated/MxaccessGateway.cs` (Compile-Removed + regenerated by Grpc.Tools). +- Gateway proto regen + commit rule (memory `project_proto_codegen_regen`): after a `.proto` edit, delete `Generated/*.cs`, rebuild contracts to regenerate, and **commit** `Generated/` or the net48 worker build fails CS0246. +- Java client gotcha (memory `project_java_generated_churn`): gradle regenerates a tracked 64k-line file with spurious protobuf-version churn — revert that churn; build/test Java on **windev** (memory `project_java_build_host`), Mac has no JRE. + +--- + +## Task 0: Contract — add `MxSparseArray` and regenerate + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (blocks all other tasks) + +**Files:** +- Modify: `src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto` (MxValue oneof ~line 1043; new messages after MxArray ~line 1063) +- Regenerate + commit: `src/ZB.MOM.WW.MxGateway.Contracts/Generated/MxaccessGateway.cs` + +**Step 1: Add the messages to the proto.** After the `MxArray` message (line 1063), add: + +```proto +// Write-only sparse array value. The gateway expands this into a full, +// default-filled MxArray before forwarding to the worker; the worker never +// receives or produces it. Unmentioned indices take the element type's +// default (reset, NOT preserved). +message MxSparseArray { + MxDataType element_data_type = 1; + uint32 total_length = 2; + repeated MxSparseElement elements = 3; +} + +message MxSparseElement { + uint32 index = 1; + MxValue value = 2; // scalar +} +``` + +**Step 2: Add the oneof arm to `MxValue`.** Inside the `oneof kind { ... }` block, after `bytes raw_value = 18;`: + +```proto + MxSparseArray sparse_array_value = 19; +``` + +**Step 3: Regenerate generated code.** + +Run (PowerShell on windev, or locally on Mac — .NET builds fine): +``` +del src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs +dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj +``` +Expected: build succeeds; `Generated/MxaccessGateway.cs` now contains `MxSparseArray`, `MxSparseElement`, and `MxValue.SparseArrayValue`. + +**Step 4: Verify net10 + net48 both compile** (the worker consumes these types via net48): +``` +dotnet build src/ZB.MOM.WW.MxGateway.slnx +``` +Expected: PASS (no CS0246 on the new types). + +**Step 5: Commit** (include regenerated `Generated/`): +```bash +git add src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto \ + src/ZB.MOM.WW.MxGateway.Contracts/Generated/MxaccessGateway.cs +git commit -m "feat(contracts): add MxSparseArray write-only value for default-fill partial writes" +``` + +--- + +## Task 1: Gateway — `SparseArrayExpander` (pure expansion + validation) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 2, Tasks 4-9 + +**Files:** +- Create: `src/ZB.MOM.WW.MxGateway.Server/Sessions/SparseArrayExpander.cs` +- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SparseArrayExpanderTests.cs` + +**Step 1: Write failing tests.** Cover: default-fill sizing + placement (one per element type is enough for two types here, rest in step 4); `total_length == 0` → `InvalidArgument`; index `>= total_length` → `InvalidArgument`; duplicate index → `InvalidArgument`; `Raw`/`Unspecified` element type → `InvalidArgument`; empty `elements` → all-defaults array of length N; timestamp default == Unix epoch. + +```csharp +using Grpc.Core; +using Mxaccess.Gateway.V1; // adjust to the generated namespace +using ZB.MOM.WW.MxGateway.Server.Sessions; +using Xunit; + +public sealed class SparseArrayExpanderTests +{ + private static MxValue Sparse(MxDataType type, uint length, params (uint Index, MxValue Value)[] els) + { + MxSparseArray sparse = new() { ElementDataType = type, TotalLength = length }; + foreach ((uint index, MxValue value) in els) + sparse.Elements.Add(new MxSparseElement { Index = index, Value = value }); + return new MxValue { SparseArrayValue = sparse }; + } + + [Fact] + public void Expand_Int32_FillsDefaultsAndPlacesValues() + { + MxValue v = Sparse(MxDataType.Integer, 4, (1, new MxValue { Int32Value = 7 })); + SparseArrayExpander.Expand(v); + Assert.Equal(MxValue.KindOneofCase.ArrayValue, v.KindCase); + Assert.Equal(new[] { 0, 7, 0, 0 }, v.ArrayValue.Int32Values.Values); + Assert.Equal((uint)4, v.ArrayValue.Dimensions[0]); + } + + [Fact] + public void Expand_EmptyElements_ProducesAllDefaults() + { + MxValue v = Sparse(MxDataType.Boolean, 3); + SparseArrayExpander.Expand(v); + Assert.Equal(new[] { false, false, false }, v.ArrayValue.BoolValues.Values); + } + + [Theory] + [InlineData(0u, 0u)] // total_length == 0 + [InlineData(2u, 5u)] // index >= total_length + public void Expand_InvalidShape_Throws(uint length, uint badIndex) + { + MxValue v = Sparse(MxDataType.Integer, length, (badIndex, new MxValue { Int32Value = 1 })); + RpcException ex = Assert.Throws(() => SparseArrayExpander.Expand(v)); + Assert.Equal(StatusCode.InvalidArgument, ex.StatusCode); + } + + [Fact] + public void Expand_DuplicateIndex_Throws() + { + MxValue v = Sparse(MxDataType.Integer, 4, + (1, new MxValue { Int32Value = 1 }), (1, new MxValue { Int32Value = 2 })); + Assert.Throws(() => SparseArrayExpander.Expand(v)); + } +} +``` + +**Step 2: Run, confirm fail.** `dotnet test src/ZB.MOM.WW.MxGateway.Tests/... --filter FullyQualifiedName~SparseArrayExpanderTests` → FAIL (type not defined). + +**Step 3: Implement `SparseArrayExpander`.** Mutates the passed `MxValue` in place, replacing `SparseArrayValue` with `ArrayValue`. Throw `RpcException(new Status(StatusCode.InvalidArgument, msg))` on any validation failure. Element-type switch must cover the supported scalar element types (`Boolean`, `Integer` → int32 or int64, `Float`, `Double`, `String`, `Time`); default/timestamp default = Unix epoch (`new Timestamp { Seconds = 0, Nanos = 0 }`); reject `Raw`/`Unknown`/`Unspecified`. Set `MxArray.Dimensions = { total_length }` and `ElementDataType`. Validate: `total_length > 0`, every index `< total_length`, no duplicate indices, each element `value` scalar kind matches `element_data_type`. + +(Mirror the typed sub-array shapes from `VariantConverter.ConvertToComArray` in the worker so the worker's existing read path is satisfied: `Int32Values`/`Int64Values`/`BoolValues`/`FloatValues`/`DoubleValues`/`StringValues`/`TimestampValues` with their `Values` repeated fields.) + +**Step 4: Add remaining element-type tests** (int64, float, double, string, time/epoch, type-mismatch element → throws). Run filter → PASS. + +**Step 5: Commit.** +```bash +git add src/ZB.MOM.WW.MxGateway.Server/Sessions/SparseArrayExpander.cs \ + src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SparseArrayExpanderTests.cs +git commit -m "feat(gateway): add SparseArrayExpander for default-fill partial array writes" +``` + +--- + +## Task 2: Gateway — `ArrayAddressNormalizer` (suffix normalization) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 1, Tasks 4-9 + +**Files:** +- Create: `src/ZB.MOM.WW.MxGateway.Server/Sessions/ArrayAddressNormalizer.cs` +- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/ArrayAddressNormalizerTests.cs` + +**Step 1: Write failing tests** using a fake/in-memory `IGalaxyHierarchyCache` whose `Current.Index.TagsByAddress` contains `"Obj.Arr[]"` (array) and `"Obj.Scalar"` (non-array). Cases: +- `"Obj.Arr"` (bare, is array) → `"Obj.Arr[]"`. +- `"Obj.Arr[]"` (already suffixed) → unchanged. +- `"Obj.Scalar"` (non-array) → unchanged. +- `"Obj.Unknown"` (not in cache / metadata cold) → unchanged (pass-through fallback). + +```csharp +[Fact] +public void Normalize_BareArrayName_AppendsSuffix() +{ + ArrayAddressNormalizer normalizer = new(FakeCacheWith("Obj.Arr[]", isArray: true)); + Assert.Equal("Obj.Arr[]", normalizer.Normalize("Obj.Arr")); +} + +[Theory] +[InlineData("Obj.Arr[]")] // already suffixed +[InlineData("Obj.Scalar")] // non-array +[InlineData("Obj.Unknown")] // not in cache → fallback pass-through +public void Normalize_LeavesUnchanged(string address) => + Assert.Equal(address, new ArrayAddressNormalizer(FakeCacheWith("Obj.Arr[]", true)).Normalize(address)); +``` + +**Step 2: Run, confirm fail.** + +**Step 3: Implement.** Constructor injects `IGalaxyHierarchyCache cache`. `Normalize(string)`: +1. If `string.IsNullOrWhiteSpace(address)` or `address.EndsWith("[]", StringComparison.Ordinal)` → return unchanged. +2. Look up `address + "[]"` in `cache.Current.Index.TagsByAddress`. If found and `lookup.Attribute?.IsArray == true` → return `address + "[]"`. +3. Otherwise return `address` unchanged. Never throw (best-effort convenience). + +**Step 4: Run filter → PASS.** + +**Step 5: Commit.** +```bash +git add src/ZB.MOM.WW.MxGateway.Server/Sessions/ArrayAddressNormalizer.cs \ + src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/ArrayAddressNormalizerTests.cs +git commit -m "feat(gateway): add ArrayAddressNormalizer for bare-name array AddItem" +``` + +--- + +## Task 3: Gateway — wire normalization + expansion into the outbound path + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (depends on Tasks 1, 2) + +**Files:** +- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/GatewaySession.cs` (constructor — inject `ArrayAddressNormalizer`; `InvokeAsync` at 947-955) +- Modify: DI registration wherever `ArrayAddressNormalizer`/`GatewaySession` deps are registered (search `Security/Authorization/ConstraintEnforcer` registration for the pattern; register `ArrayAddressNormalizer` as scoped/singleton consistent with `IGalaxyHierarchyCache`) +- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/GatewayArrayWriteWiringTests.cs` + +**Step 1: Write a failing integration test** with `FakeWorkerHarness` (pattern: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/FakeWorkerHarnessTests.cs:51-69` — `CreateConnectedPairAsync`, `ReadCommandAsync`, `ReplyToCommandAsync`). Two assertions: +1. Client sends `AddItemCommand{ item_definition = "Obj.Arr" }` (array per the test's Galaxy cache) → the `WorkerEnvelope` the fake worker reads has `item_definition == "Obj.Arr[]"`. +2. Client sends `WriteCommand{ value = MxSparseArray(Integer, 4, {1:7}) }` → the worker receives `value.ArrayValue.Int32Values.Values == [0,7,0,0]` (no `SparseArrayValue` reaches the worker). + +**Step 2: Run, confirm fail.** + +**Step 3: Implement.** At the top of `InvokeAsync(WorkerCommand command, ...)`, before forwarding, transform `command.Command` (the `MxCommand`) by `PayloadCase`: +- `AddItem` → `command.Command.AddItem.ItemDefinition = _addressNormalizer.Normalize(command.Command.AddItem.ItemDefinition);` +- `AddItem2` → same on `AddItem2.ItemDefinition`. +- `Write`/`WriteSecured` → if `cmd.Value?.KindCase == SparseArrayValue` call `SparseArrayExpander.Expand(cmd.Value)`. +- `Write2`/`WriteSecured2` → expand `Value` only (not `TimestampValue`). +- `WriteBulk`/`Write2Bulk`/`WriteSecuredBulk`/`WriteSecured2Bulk` → expand each `entry.Value`. + +Keep it a single private helper `NormalizeOutbound(MxCommand)` called once at the choke point. Because `TrackCommandReply` later reads the **same** `MxCommand` instance, the normalized `item_definition` flows into `SessionItemRegistration` with no extra change — add an assertion in the test that `TryGetItemRegistration(...).TagAddress == "Obj.Arr[]"` to lock that in. + +**Step 4: Run the wiring test + Tasks 1/2 filters → PASS.** Then run the AddItem/Write fake-worker regression group once: +``` +dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~ArrayWrite|FullyQualifiedName~SparseArray|FullyQualifiedName~ArrayAddressNormalizer" +``` + +**Step 5: Commit.** +```bash +git add src/ZB.MOM.WW.MxGateway.Server/Sessions/GatewaySession.cs \ + src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/GatewayArrayWriteWiringTests.cs +git commit -m "feat(gateway): normalize array AddItem suffix and expand sparse writes at the worker boundary" +``` + +--- + +## Tasks 4-8: Client helpers + READMEs (one task per client, parallelizable) + +Each client task does the same four things; only paths/idioms differ. **Depends on Task 0** (needs regenerated proto types). All five are parallelizable with each other and with Tasks 1-3, 9. + +Common helper contract: `WriteArrayElements(serverHandle, itemHandle, elementDataType, totalLength, elements /* index→scalar MxValue */, userId)` builds an `MxValue { SparseArrayValue = MxSparseArray{...} }` and calls the existing raw write. Add a unit test that the built command carries `sparse_array_value` with the right `total_length`/indices (serialization round-trip; no live gateway). Update the **"Array writes replace the whole array"** README section to document: default-fill semantics (unmentioned = reset to default, not preserved), the required `total_length`, and that bare-name array writes now auto-normalize. + +### Task 4: .NET client +**Classification:** standard · **~4 min** · **Parallelizable with:** Tasks 5-9, 1-3 +- Regenerate types: `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx`. +- Add helper next to `WriteAsync` in `clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs:678-688`. +- Test alongside existing client tests; README `clients/dotnet/README.md:162-170`. +- Verify: build slnx + `dotnet test` the client test project. + +### Task 5: Go client +**Classification:** standard · **~4 min** · **Parallelizable with:** Tasks 4,6-9, 1-3 +- Regenerate per `clients/go` README; helper next to `Write`/`WriteRaw` in `clients/go/mxgateway/session.go:559-581`. +- README `clients/go/README.md:139-147`. +- Verify: `gofmt`, `go build ./...`, `go test ./...` from `clients/go`. + +### Task 6: Python client +**Classification:** standard · **~4 min** · **Parallelizable with:** Tasks 4-5,7-9, 1-3 +- Regenerate per `clients/python` README; helper next to `write` in `clients/python/src/zb_mom_ww_mxgateway/session.py:469-490`. +- README `clients/python/README.md:142-150`. +- Verify: `python -m pytest` from `clients/python`. + +### Task 7: Rust client +**Classification:** standard · **~4 min** · **Parallelizable with:** Tasks 4-6,8-9, 1-3 +- Helper next to `write` in `clients/rust/src/session.rs:530-548`; conversion helpers in `clients/rust/src/value.rs`. +- README `clients/rust/README.md:162-170`. +- Verify: `cargo fmt`, `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings` from `clients/rust`. + +### Task 8: Java client +**Classification:** standard · **~5 min** · **Parallelizable with:** Tasks 4-7,9, 1-3 +- Helper next to `write`/`writeRaw` in `clients/java/.../client/MxGatewaySession.java:581-604`. +- README `clients/java/README.md:118-126`. +- **Build/test on windev (JDK 21) via an isolated `origin/` worktree — Mac has no JRE** (memory `project_java_build_host`). After gradle regen, **revert the spurious protobuf-version churn** in `clients/java/src/main/generated/.../MxaccessGateway.java` if no proto semantics beyond the new messages changed (memory `project_java_generated_churn`); keep only the real `MxSparseArray` additions. +- Verify: `gradle test` on windev. + +(Each client task ends with its own commit, e.g. `feat(client-): add WriteArrayElements default-fill helper and document semantics`.) + +--- + +## Task 9: Docs — gateway.md + value conversion + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Tasks 1-8 (depends on Task 0 only) + +**Files:** +- Modify: `gateway.md` (command/value surface — document `MxSparseArray` as a write-only value and bare-name AddItem normalization) +- Modify: the value-conversion doc under `docs/` (search for where `MxArray`/value conversion is described) — add the default-fill + epoch-default note and the parity statement (worker still does a whole-array COM write) + +**Step 1:** Add a subsection describing: `sparse_array_value` is write-only and gateway-expanded; default-fill semantics (epoch for time); `total_length` required; bare-name array writes auto-normalize to `[]` at AddItem with metadata pass-through fallback; non-goal: no preserve-unchanged merge, no element-wise COM write. + +**Step 2: Commit.** +```bash +git add gateway.md docs/ +git commit -m "docs: document MxSparseArray default-fill writes and bare-name array AddItem" +``` + +--- + +## Dependency summary + +- **Task 0** blocks everything. +- **Tasks 1, 2** depend on 0; parallel with each other. +- **Task 3** depends on 1 and 2. +- **Tasks 4-8** depend on 0; parallel with each other and with 1-3, 9. +- **Task 9** depends on 0; parallel with all. + +## Verification gates (per CLAUDE.md targeted-tests rule) + +- Run only the `--filter` for the task you touched; run the array-write fake-worker group once after Task 3. +- Java verified on windev only. .NET/Go/Rust/Python verified locally. +- Live MXAccess (opt-in, windev): after merge, one default-fill write and one bare-name array write against real COM (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). diff --git a/docs/plans/2026-06-18-array-write-ergonomics.md.tasks.json b/docs/plans/2026-06-18-array-write-ergonomics.md.tasks.json new file mode 100644 index 0000000..23f076c --- /dev/null +++ b/docs/plans/2026-06-18-array-write-ergonomics.md.tasks.json @@ -0,0 +1,16 @@ +{ + "planPath": "docs/plans/2026-06-18-array-write-ergonomics.md", + "tasks": [ + {"id": 0, "subject": "Task 0: Contract — add MxSparseArray + regenerate", "status": "pending"}, + {"id": 1, "subject": "Task 1: Gateway SparseArrayExpander", "status": "pending", "blockedBy": [0]}, + {"id": 2, "subject": "Task 2: Gateway ArrayAddressNormalizer", "status": "pending", "blockedBy": [0]}, + {"id": 3, "subject": "Task 3: Wire normalization + expansion into GatewaySession", "status": "pending", "blockedBy": [1, 2]}, + {"id": 4, "subject": "Task 4: .NET client WriteArrayElements + README", "status": "pending", "blockedBy": [0]}, + {"id": 5, "subject": "Task 5: Go client WriteArrayElements + README", "status": "pending", "blockedBy": [0]}, + {"id": 6, "subject": "Task 6: Python client write_array_elements + README", "status": "pending", "blockedBy": [0]}, + {"id": 7, "subject": "Task 7: Rust client write_array_elements + README", "status": "pending", "blockedBy": [0]}, + {"id": 8, "subject": "Task 8: Java client writeArrayElements + README (windev)", "status": "pending", "blockedBy": [0]}, + {"id": 9, "subject": "Task 9: Docs — gateway.md + value conversion", "status": "pending", "blockedBy": [0]} + ], + "lastUpdated": "2026-06-18" +}