# Native-Typed JSON for List Attributes — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.

**Goal:** Make `AttributeValueCodec` encode List values as native-typed JSON (`[10,20]`, `[true,false]`, ISO dates) while decoding both old (array-of-strings) and new forms, and normalize already-persisted data to the native form across MS SQL, site SQLite, and bundle import.

**Architecture:** Native `Encode` + read-both `Decode` in the codec is the foundation. An idempotent central MS SQL startup normalizer rewrites old-form rows; the InstanceActor re-persists native form on override-load (site SQLite); the bundle importer re-encodes on import. To give the central normalizer a reliable element type for instance-override rows, the override-write path now stamps `ElementDataType` (folds in follow-up #93/M3).

**Tech Stack:** C#/.NET 10, System.Text.Json, EF Core 10 (MS SQL + SQLite), Akka.NET, Transport bundles.

**Design doc:** `docs/plans/2026-06-16-native-typed-json-design.md` (approved).
**Branch:** `feature/native-typed-json` (off main; design committed `91b1aa1`).

**Conventions for every task:** TDD (failing test → fail → implement → pass → commit). Targeted builds/tests only — `dotnet build src/<Project>/<Project>.csproj`, `dotnet test tests/<TestProject>/<TestProject>.csproj --filter <Name>`. Full-solution build only in the final task. Do NOT create git worktrees. Commit with pathspec form (`git commit -m "…" -- <paths>`) and retry on `.git/index.lock` when implementers run concurrently.

---

### Task NJ-1: Codec — native-typed Encode + read-both Decode

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** NJ-2
**Blocked by:** none (foundation)

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs`
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs`

This changes the canonical on-disk/on-wire form for List values. It is backward compatible (Decode reads both forms) and additive on the gRPC `string value` field (List is a new type with no external consumer of the old quoted form).

**Step 1 — Update/extend tests (TDD).** Update the existing `Encode_DoubleList_IsInvariant` assertion to the native form, and add the native + backward-compat cases:

```csharp
[Fact]
public void Encode_Int32List_ProducesNativeNumbers() =>
    Assert.Equal("[10,20,30]", AttributeValueCodec.Encode(new List<int> { 10, 20, 30 }));

[Fact]
public void Encode_BoolList_ProducesNativeBooleans() =>
    Assert.Equal("[true,false]", AttributeValueCodec.Encode(new List<bool> { true, false }));

[Fact]
public void Encode_DoubleList_IsNativeAndInvariant()
{
    var original = CultureInfo.CurrentCulture;
    try
    {
        CultureInfo.CurrentCulture = new CultureInfo("de-DE");
        Assert.Equal("[1.5,2.5]", AttributeValueCodec.Encode(new List<double> { 1.5, 2.5 }));
    }
    finally { CultureInfo.CurrentCulture = original; }
}

[Fact]
public void Encode_StringList_StaysQuoted() =>
    Assert.Equal("[\"a\",\"b\"]", AttributeValueCodec.Encode(new List<string> { "a", "b" }));

[Fact]
public void Encode_DateTimeList_IsIso8601()
{
    var json = AttributeValueCodec.Encode(new List<DateTime> { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) });
    Assert.Contains("2026-06-16T00:00:00", json);   // ISO, not "06/16/2026 00:00:00"
}

[Fact]
public void Decode_NewNativeIntForm_Parses()
{
    var back = (IList<int>)AttributeValueCodec.Decode("[10,20]", DataType.List, DataType.Int32)!;
    Assert.Equal(new[] { 10, 20 }, back);
}

[Fact]
public void Decode_OldStringIntForm_BackwardCompatible()
{
    var back = (IList<int>)AttributeValueCodec.Decode("[\"10\",\"20\"]", DataType.List, DataType.Int32)!;
    Assert.Equal(new[] { 10, 20 }, back);
}

[Theory]
[InlineData("[true,false]")]   // new native bools
[InlineData("[\"True\",\"False\"]")] // old string bools
public void Decode_BoolForms_BothParse(string json)
{
    var back = (IList<bool>)AttributeValueCodec.Decode(json, DataType.List, DataType.Boolean)!;
    Assert.Equal(new[] { true, false }, back);
}
```
Keep the existing round-trip tests (Int32/Float/Double/DateTime/Bool) — they assert round-trip, not exact form, so they continue to pass. Keep `Decode_MalformedJson_Throws`.

**Step 2 — Run, expect FAIL** on the new native-form encode assertions.
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.csproj --filter AttributeValueCodecTests`

**Step 3 — Implement.** In `Encode`, replace ONLY the `IEnumerable` branch:

```csharp
case IEnumerable e:
    // Native-typed JSON: serialize the runtime collection so STJ emits numbers/bools
    // unquoted, strings quoted, DateTime as ISO-8601. Boxed as object so STJ uses the
    // runtime element type. STJ numbers/dates are culture-invariant by spec.
    return JsonSerializer.Serialize<object>(e, JsonOpts);
```
(The `null`, `string`, and `IFormattable` scalar branches are unchanged.)

In `Decode`, change the list deserialization to read `JsonElement[]` and derive a string per element:

```csharp
JsonElement[] raw;
try { raw = JsonSerializer.Deserialize<JsonElement[]>(value) ?? []; }
catch (JsonException ex) { throw new FormatException("Malformed list JSON.", ex); }

var clrType = ElementClrType(elementType.Value);
var listType = typeof(List<>).MakeGenericType(clrType);
var result = (IList)Activator.CreateInstance(listType)!;
foreach (var el in raw)
    result.Add(ParseScalar(JsonElementToString(el), elementType.Value));
return result;
```
Add the helper (handles both new native elements and old quoted-string elements):

```csharp
private static string? JsonElementToString(JsonElement el) => el.ValueKind switch
{
    JsonValueKind.String => el.GetString(),          // old form, or string-typed lists
    JsonValueKind.Null => null,                      // ParseScalar throws "may not be null"
    _ => el.GetRawText()                             // number/bool → "10" / "1.5" / "true"
};
```
`ParseScalar` is unchanged: it parses the string per element type (invariant), so `"10"` (from either form), `"true"`, and ISO/US-invariant dates all parse. Add `using System.Text.Json;` if not already present (it is).

**Step 4 — Run, expect PASS** (all AttributeValueCodec tests).

**Step 5 — Commit.**
```bash
git commit -m "feat(commons): native-typed JSON for List values; Decode reads both forms" -- \
  src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs \
  tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs
```

**Acceptance:** numbers/bools encode unquoted, strings stay quoted, DateTime is ISO; both old and new forms decode; scalars unchanged; malformed still throws.

---

### Task NJ-2: Stamp ElementDataType on instance-override writes

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** NJ-1
**Blocked by:** none

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs:146-198` (`SetAttributeOverrideAsync`)
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs` (mirror existing style)

`SetAttributeOverrideAsync` already loads the base `templateAttr` (line 159). Set `ElementDataType` from it on both the update and create branches so the persisted `InstanceAttributeOverride` row carries the element type — this is what the NJ-3 central normalizer reads for override rows (folds in follow-up #93/M3).

**Step 1 — Failing test:** setting an override on a `List`/`String` template attribute persists an `InstanceAttributeOverride` whose `ElementDataType == DataType.String`; setting an override on a scalar attribute persists `ElementDataType == null`.

**Step 2 — Run, expect FAIL.**

**Step 3 — Implement:** in the update branch, `existingOverride.ElementDataType = templateAttr.ElementDataType;` before the repo update; in the create branch, set `ElementDataType = templateAttr.ElementDataType` in the `new InstanceAttributeOverride(...)` initializer.

**Step 4 — Run, expect PASS.**

**Step 5 — Commit:** `feat(template): stamp ElementDataType on instance attribute overrides`.

---

### Task NJ-3: Central MS SQL idempotent startup normalizer

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** NJ-4, NJ-5 (after NJ-1)
**Blocked by:** NJ-1, NJ-2

**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs` (call the normalizer after the migrate/validate step)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/...` (use the existing test DbContext/SQLite-in-memory or test-double pattern the project uses — investigate the existing ConfigurationDatabase tests first)

**Behavior:** A static `ListValueNormalizer.NormalizeAsync(ScadaBridgeDbContext db, ILogger? logger, CancellationToken ct)`:
- `TemplateAttributes` where `DataType == DataType.List`: for each, compute `var native = AttributeValueCodec.Encode(AttributeValueCodec.Decode(a.Value, DataType.List, a.ElementDataType));` (parse the row's own `ElementDataType` — it is a `DataType?`). If `native != a.Value`, set `a.Value = native`.
- `InstanceAttributeOverrides` whose `ElementDataType != null` (i.e. a List override stamped by NJ-2): same re-encode against `OverrideValue`. Rows with null `ElementDataType` are skipped with a debug log (legacy/scalar rows — none exist in practice since the feature has no deployed data).
- Wrap each row in try/catch(`FormatException`): on a malformed value, log a warning and SKIP that row — continue with the rest. The normalizer MUST NOT throw out of `NormalizeAsync` (never abort startup).
- `SaveChangesAsync` once at the end. Log a summary (`Normalized {n} list attribute value(s) to native JSON`).
- Idempotent: native→native yields identical bytes, so a second run makes no changes.

**Wire-in (MigrationHelper):** after the `if (isDevelopment) { Migrate } else { validate }` block in `ApplyOrValidateMigrationsAsync`, call `await ListValueNormalizer.NormalizeAsync(dbContext, logger, cancellationToken);`. (Runs on every central startup; idempotent and safe even if both central nodes run it — concurrent UPDATEs set identical values.)

**Steps (TDD):** failing tests (a TemplateAttributes row in old form `["10","20"]` with ElementDataType Int32 → rewritten to `[10,20]`; a native row → unchanged/no-op; a String-list `["a","b"]` → unchanged; a malformed row → skipped, sibling rows still normalized; an override row with ElementDataType set → normalized, with null → skipped) → run targeted → FAIL → implement `ListValueNormalizer` + wire-in → run targeted → PASS → build ConfigurationDatabase → commit `feat(db): idempotent startup normalizer rewriting List values to native JSON`.

---

### Task NJ-4: Site SQLite active normalization on override load

**Classification:** high-risk
**Estimated implement time:** ~4 min
**Parallelizable with:** NJ-3, NJ-5 (after NJ-1)
**Blocked by:** NJ-1

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:932-960` (`HandleOverridesLoaded`)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs`

In `HandleOverridesLoaded`, the List branch already decodes the stored override via `DecodeAttributeValue(resolved, kvp.Value)`. Extend it: when the decode succeeds and the re-encoded native form differs from the stored string (i.e. the stored value is old-form), re-persist the native form:

```csharp
var decoded = DecodeAttributeValue(resolved, kvp.Value);
_attributes[kvp.Key] = decoded;
if (decoded is null && !string.IsNullOrEmpty(kvp.Value))
{
    _attributeQualities[kvp.Key] = "Bad";
}
else if (decoded is not null)
{
    var native = AttributeValueCodec.Encode(decoded);
    if (native != kvp.Value)   // stored value was old-form → normalize on disk
    {
        var key = kvp.Key;
        _storage.SetStaticOverrideAsync(_instanceUniqueName, key, native!)
            .ContinueWith(t => _logger.LogWarning(t.Exception?.GetBaseException(),
                "Failed to normalize static override {Instance}.{Attr} to native JSON", _instanceUniqueName, key),
                TaskContinuationOptions.OnlyOnFaulted);
    }
}
```
Idempotent (native→native ⇒ `native == kvp.Value` ⇒ no write). Scalars untouched. Never throws into the actor (decode already FormatException-safe; persist is fire-and-forget).

**Steps (TDD):** failing tests (an old-form List static override on load → `SetStaticOverrideAsync` called with the native form; a native-form override → NOT re-persisted; a scalar override → unaffected; malformed → Bad quality + no crash, as today) → run targeted `--filter "FullyQualifiedName~InstanceActor"` → FAIL → implement → PASS → build SiteRuntime → commit `feat(siteruntime): normalize old-form List static overrides to native JSON on load`.

---

### Task NJ-5: Normalize List values on bundle import

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** NJ-3, NJ-4 (after NJ-1)
**Blocked by:** NJ-1

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (the 3 `TemplateAttribute` construction sites from MV-12: BuildTemplate + the two in SyncTemplateAttributesAsync) — and/or `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` `FromBundleContent` if that is the single choke point for DTO→entity.
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/...`

**Behavior:** when importing a `TemplateAttributeDto` whose `DataType == DataType.List`, write the **native** Value: `var v = NormalizeListValue(dto.Value, dto.ElementDataType);` where `NormalizeListValue` does `AttributeValueCodec.Encode(AttributeValueCodec.Decode(value, DataType.List, elementType))` inside try/catch(FormatException) → on failure return the original value unchanged + log (don't fail the import; the MS SQL normalizer is a backstop). Prefer a single private helper used at all construction sites (DRY). Non-List attributes write `dto.Value` unchanged.

**Steps (TDD):** failing test (import a bundle/DTO with an old-form List attribute `["10","20"]` + ElementDataType Int32 → the persisted TemplateAttribute.Value is `[10,20]`; a String list stays quoted; a malformed value imports unchanged) → run targeted → FAIL → implement the helper + apply at construction sites → PASS → build Transport → commit `feat(transport): normalize List attribute values to native JSON on import`.

---

### Task NJ-6: Integration verification + docs

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Blocked by:** NJ-1 … NJ-5

**Files:**
- Modify: `docs/plans/2026-06-16-native-typed-json-design.md` (mark Status complete)
- Modify: `docs/requirements/Component-Commons.md` (update the `DataType`/codec note: List values are native-typed JSON; codec reads both forms)
- Modify: `docs/plans/2026-06-16-multivalue-attribute.md.tasks.json` follow-up note OR the task list — note that #93/M3 (override ElementDataType) was folded in here.

**Steps:**
1. Full-solution build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → 0/0.
2. Targeted test sweep across touched projects: Commons codec, TemplateEngine (InstanceService), ConfigurationDatabase (normalizer), SiteRuntime (InstanceActor), Transport (serializer/importer).
3. Update docs; mark design Status complete.
4. Commit: `docs: mark native-typed JSON feature complete; update Component-Commons codec note`.

**Acceptance:** full build green; all touched-project targeted tests pass; docs synced.

---

## Parallelization summary

- **Wave 1:** NJ-1 (codec) ∥ NJ-2 (override stamp) — disjoint (Commons vs TemplateEngine).
- **Wave 2 (after NJ-1, +NJ-2 for NJ-3):** NJ-3 (normalizer), NJ-4 (site), NJ-5 (import) in parallel — disjoint (ConfigurationDatabase vs SiteRuntime vs Transport).
- **Wave 3:** NJ-6 (verify + docs).

## Risk notes

- **NJ-1** changes the canonical storage/wire form — high-risk, but backward-compatible (read-both) and additive on the proto field.
- **NJ-3** writes to the config DB at startup — must be idempotent and never abort startup (per-row skip+log).
- **NJ-4** touches the actor model — the normalization write is fire-and-forget and never throws into the actor.
- There is no deployed List data yet, so the normalizers are a safety net; correctness of NJ-1 (the format) is what matters most in practice.