From 91b1aa12756ca4628988a266e4b01446950db790 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:08:38 -0400 Subject: [PATCH 01/11] docs: design for native-typed JSON List attribute values + data normalization Encode emits native-typed JSON ([10,20], [true,false], ISO dates); Decode reads both old (array-of-strings) and new forms. Existing data normalized via an idempotent central MS SQL startup normalizer, active site SQLite normalization in the InstanceActor override-load path, and normalize-on-import for bundles. Approved via brainstorming (Approach B, thorough). --- .../2026-06-16-native-typed-json-design.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/plans/2026-06-16-native-typed-json-design.md diff --git a/docs/plans/2026-06-16-native-typed-json-design.md b/docs/plans/2026-06-16-native-typed-json-design.md new file mode 100644 index 00000000..07573fc3 --- /dev/null +++ b/docs/plans/2026-06-16-native-typed-json-design.md @@ -0,0 +1,64 @@ +# Native-Typed JSON for List Attribute Values — Design + +**Date:** 2026-06-16 +**Status:** Approved (brainstorming) — ready for implementation plan +**Branch:** `feature/native-typed-json` + +## Problem + +The multi-value (List) attribute feature (shipped 2026-06-16, branch `feature/multivalue-attribute`) stores List values via `AttributeValueCodec` as a JSON **array of strings** — e.g. an `Int32` list is `["10","20","30"]` and a `Boolean` list is `["True","False"]`. This is internally consistent and round-trips, but it is not "native-typed" JSON: numbers and booleans are quoted, and `DateTime` uses a US-invariant format rather than ISO-8601. We want the canonical form to be native-typed (`[10,20,30]`, `[true,false]`, ISO dates), while existing persisted data is normalized to the new form (no dual-format data left behind). + +## Decisions + +| Decision | Choice | +|---|---| +| Encode form | Native-typed JSON: numbers/bools unquoted, strings quoted, `DateTime` as ISO-8601 string | +| Decode | **Read both** old (array-of-strings) and new (native) forms — backward compatible | +| Existing data | **Migrate** to native form across MS SQL + site SQLite + on bundle import (Approach B, thorough) | +| MS SQL mechanism | Idempotent C# **startup normalizer** (not T-SQL — type-aware JSON re-emission is fragile in SQL) | +| Site SQLite mechanism | **Active** normalization in the InstanceActor override-load path (it already has the element type) | +| Bundles | Normalize **on import** (already-exported files are external/unreachable) | + +**Reality note:** the List feature shipped this session and was not deployed to the docker cluster, so there is almost certainly **zero** old-form List data in any store yet. The migration is a safety net guaranteeing no dual-format data ever lingers, not a fix for existing broken data. + +## Architecture + +### 1. Codec (`AttributeValueCodec`, Commons) — foundation + +- **`Encode`** — only the list branch changes. Instead of mapping each element to an invariant string then serializing, serialize the typed CLR collection directly (`JsonSerializer.Serialize(enumerable, …)`) so `System.Text.Json` emits each element in its native JSON kind. STJ numbers/bools are culture-invariant by spec; `DateTime` serializes as round-trippable ISO-8601. Scalars (the `string` / `IFormattable` branches) are untouched. +- **`Decode`** — read both forms. Deserialize to `JsonElement[]` (instead of `string?[]`); for each element feed `ParseScalar` either `GetString()` (JSON string element) or `GetRawText()` (number/bool element). So `[10,20]` and `["10","20"]` both decode to `List{10,20}`; ISO and old US-invariant `DateTime` strings both parse via the existing `DateTime.Parse(…, RoundtripKind)`. A JSON `null` element still throws `FormatException` ("elements may not be null"), unchanged. +- The read-both Decode is also what makes the migration idempotent: re-encoding an already-native value yields identical bytes. + +### 2. MS SQL — idempotent central startup normalizer + +A normalization step invoked once after `dbContext.Database.MigrateAsync(...)` in `MigrationHelper.ApplyOrValidateMigrationsAsync` (active central node only). For each List row: + +- **`TemplateAttributes`** where `DataType = 'List'`: read `Value` + the row's own `ElementDataType`; compute `Encode(Decode(value, List, elementType))`; if it differs from the stored string, `UPDATE`. +- **`InstanceAttributeOverrides`** for List attributes: these rows may have a null `ElementDataType` (it is currently informational — see follow-up #93/M3), so resolve the element type via the owning instance's template attribute (instance → `TemplateId` → `TemplateAttribute` by name → `ElementDataType`). Then re-encode as above. + +Idempotent (native→native is a no-op `UPDATE`-skip), so the step is safe to leave in permanently and cheap on every subsequent startup (a scan, no writes). Per-row failures (malformed JSON, unresolved element type) are logged and skipped — normalization NEVER aborts startup (mirrors the audit/best-effort principle). The scan is bounded to List rows only. + +### 3. Site SQLite — active normalization on override load + +Site static-override values (`SiteStorageService`) are keyed by `(instance, canonicalName)` and carry no element type — the element type lives in the instance's flattened config. The natural normalization point is therefore the **InstanceActor override-load path** (`HandleOverridesLoaded`, added in MV-7), which already decodes both forms using the `ResolvedAttribute`'s `ElementDataType`. Extend it so that when a List override's stored string is in old form (i.e. `Encode(decoded)` differs from the stored string), it re-persists the native form via `SiteStorageService.SetStaticOverrideAsync`. This actively normalizes on every instance load (site startup / failover), reuses the existing decode + element type, and is idempotent. + +### 4. Bundles — normalize on import + +Already-exported `.bundle` files are external artifacts we cannot reach to rewrite; import already reads both forms via the codec. To ensure imported List values land in native form in the DB, the importer re-encodes List attribute `Value`s through the codec when writing (and the MS SQL normalizer is a backstop on next startup). No file rewriting. + +## Error handling + +- Decode of a genuinely malformed value still throws `FormatException`; the normalizers catch it per-row, log, and skip (no startup abort, no actor crash). +- The codec change is additive on the wire (`gRPC` `string value` field unchanged; `List` is a new type with no external wire consumer relying on the old quoted form). + +## Testing + +- **Codec:** native-form encode per element type (`[10,20,30]`, `[true,false]`, ISO `DateTime`, strings stay `["a","b"]`); old-form backward-compat decode (`["10","20"]` → `List`); round-trip for every element type; malformed still throws; culture-invariance preserved. +- **MS SQL normalizer:** old-form row → rewritten to native; native row → untouched (idempotency); malformed row → skipped + logged, other rows still processed; override row element-type resolved via template attribute. +- **Site SQLite / InstanceActor:** an old-form List override on load → re-persisted native (assert `SetStaticOverrideAsync` called with native form); a native override → not re-persisted (idempotent); scalar overrides unaffected. +- **Bundle import:** importing an old-form bundle lands native-form Values in the DB. + +## Out of scope / follow-ups + +- Rewriting already-exported bundle files (unreachable). +- This pairs naturally with follow-up **#93/M3** (populate `InstanceAttributeOverride.ElementDataType` on write); if done, the override normalizer could read the column directly instead of joining to the template attribute. Not required here. From 69f7c526d0b593df94dbffe4251766f46e6edd65 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:13:14 -0400 Subject: [PATCH 02/11] docs: implementation plan for native-typed JSON List values + normalization 6 tasks (NJ-1..NJ-6): native codec + read-both decode; stamp override ElementDataType (#93/M3); idempotent central startup normalizer; site override-load normalization; normalize-on-import; integration + docs. --- docs/plans/2026-06-16-native-typed-json.md | 282 ++++++++++++++++++ ...2026-06-16-native-typed-json.md.tasks.json | 14 + 2 files changed, 296 insertions(+) create mode 100644 docs/plans/2026-06-16-native-typed-json.md create mode 100644 docs/plans/2026-06-16-native-typed-json.md.tasks.json diff --git a/docs/plans/2026-06-16-native-typed-json.md b/docs/plans/2026-06-16-native-typed-json.md new file mode 100644 index 00000000..4eb36023 --- /dev/null +++ b/docs/plans/2026-06-16-native-typed-json.md @@ -0,0 +1,282 @@ +# Native-Typed JSON for List Attributes — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task. + +**Goal:** Make `AttributeValueCodec` encode List values as native-typed JSON (`[10,20]`, `[true,false]`, ISO dates) while decoding both old (array-of-strings) and new forms, and normalize already-persisted data to the native form across MS SQL, site SQLite, and bundle import. + +**Architecture:** Native `Encode` + read-both `Decode` in the codec is the foundation. An idempotent central MS SQL startup normalizer rewrites old-form rows; the InstanceActor re-persists native form on override-load (site SQLite); the bundle importer re-encodes on import. To give the central normalizer a reliable element type for instance-override rows, the override-write path now stamps `ElementDataType` (folds in follow-up #93/M3). + +**Tech Stack:** C#/.NET 10, System.Text.Json, EF Core 10 (MS SQL + SQLite), Akka.NET, Transport bundles. + +**Design doc:** `docs/plans/2026-06-16-native-typed-json-design.md` (approved). +**Branch:** `feature/native-typed-json` (off main; design committed `91b1aa1`). + +**Conventions for every task:** TDD (failing test → fail → implement → pass → commit). Targeted builds/tests only — `dotnet build src//.csproj`, `dotnet test tests//.csproj --filter `. Full-solution build only in the final task. Do NOT create git worktrees. Commit with pathspec form (`git commit -m "…" -- `) and retry on `.git/index.lock` when implementers run concurrently. + +--- + +### Task NJ-1: Codec — native-typed Encode + read-both Decode + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** NJ-2 +**Blocked by:** none (foundation) + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs` +- Modify: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs` + +This changes the canonical on-disk/on-wire form for List values. It is backward compatible (Decode reads both forms) and additive on the gRPC `string value` field (List is a new type with no external consumer of the old quoted form). + +**Step 1 — Update/extend tests (TDD).** Update the existing `Encode_DoubleList_IsInvariant` assertion to the native form, and add the native + backward-compat cases: + +```csharp +[Fact] +public void Encode_Int32List_ProducesNativeNumbers() => + Assert.Equal("[10,20,30]", AttributeValueCodec.Encode(new List { 10, 20, 30 })); + +[Fact] +public void Encode_BoolList_ProducesNativeBooleans() => + Assert.Equal("[true,false]", AttributeValueCodec.Encode(new List { true, false })); + +[Fact] +public void Encode_DoubleList_IsNativeAndInvariant() +{ + var original = CultureInfo.CurrentCulture; + try + { + CultureInfo.CurrentCulture = new CultureInfo("de-DE"); + Assert.Equal("[1.5,2.5]", AttributeValueCodec.Encode(new List { 1.5, 2.5 })); + } + finally { CultureInfo.CurrentCulture = original; } +} + +[Fact] +public void Encode_StringList_StaysQuoted() => + Assert.Equal("[\"a\",\"b\"]", AttributeValueCodec.Encode(new List { "a", "b" })); + +[Fact] +public void Encode_DateTimeList_IsIso8601() +{ + var json = AttributeValueCodec.Encode(new List { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) }); + Assert.Contains("2026-06-16T00:00:00", json); // ISO, not "06/16/2026 00:00:00" +} + +[Fact] +public void Decode_NewNativeIntForm_Parses() +{ + var back = (IList)AttributeValueCodec.Decode("[10,20]", DataType.List, DataType.Int32)!; + Assert.Equal(new[] { 10, 20 }, back); +} + +[Fact] +public void Decode_OldStringIntForm_BackwardCompatible() +{ + var back = (IList)AttributeValueCodec.Decode("[\"10\",\"20\"]", DataType.List, DataType.Int32)!; + Assert.Equal(new[] { 10, 20 }, back); +} + +[Theory] +[InlineData("[true,false]")] // new native bools +[InlineData("[\"True\",\"False\"]")] // old string bools +public void Decode_BoolForms_BothParse(string json) +{ + var back = (IList)AttributeValueCodec.Decode(json, DataType.List, DataType.Boolean)!; + Assert.Equal(new[] { true, false }, back); +} +``` +Keep the existing round-trip tests (Int32/Float/Double/DateTime/Bool) — they assert round-trip, not exact form, so they continue to pass. Keep `Decode_MalformedJson_Throws`. + +**Step 2 — Run, expect FAIL** on the new native-form encode assertions. +Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.csproj --filter AttributeValueCodecTests` + +**Step 3 — Implement.** In `Encode`, replace ONLY the `IEnumerable` branch: + +```csharp +case IEnumerable e: + // Native-typed JSON: serialize the runtime collection so STJ emits numbers/bools + // unquoted, strings quoted, DateTime as ISO-8601. Boxed as object so STJ uses the + // runtime element type. STJ numbers/dates are culture-invariant by spec. + return JsonSerializer.Serialize(e, JsonOpts); +``` +(The `null`, `string`, and `IFormattable` scalar branches are unchanged.) + +In `Decode`, change the list deserialization to read `JsonElement[]` and derive a string per element: + +```csharp +JsonElement[] raw; +try { raw = JsonSerializer.Deserialize(value) ?? []; } +catch (JsonException ex) { throw new FormatException("Malformed list JSON.", ex); } + +var clrType = ElementClrType(elementType.Value); +var listType = typeof(List<>).MakeGenericType(clrType); +var result = (IList)Activator.CreateInstance(listType)!; +foreach (var el in raw) + result.Add(ParseScalar(JsonElementToString(el), elementType.Value)); +return result; +``` +Add the helper (handles both new native elements and old quoted-string elements): + +```csharp +private static string? JsonElementToString(JsonElement el) => el.ValueKind switch +{ + JsonValueKind.String => el.GetString(), // old form, or string-typed lists + JsonValueKind.Null => null, // ParseScalar throws "may not be null" + _ => el.GetRawText() // number/bool → "10" / "1.5" / "true" +}; +``` +`ParseScalar` is unchanged: it parses the string per element type (invariant), so `"10"` (from either form), `"true"`, and ISO/US-invariant dates all parse. Add `using System.Text.Json;` if not already present (it is). + +**Step 4 — Run, expect PASS** (all AttributeValueCodec tests). + +**Step 5 — Commit.** +```bash +git commit -m "feat(commons): native-typed JSON for List values; Decode reads both forms" -- \ + src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs \ + tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs +``` + +**Acceptance:** numbers/bools encode unquoted, strings stay quoted, DateTime is ISO; both old and new forms decode; scalars unchanged; malformed still throws. + +--- + +### Task NJ-2: Stamp ElementDataType on instance-override writes + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** NJ-1 +**Blocked by:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs:146-198` (`SetAttributeOverrideAsync`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs` (mirror existing style) + +`SetAttributeOverrideAsync` already loads the base `templateAttr` (line 159). Set `ElementDataType` from it on both the update and create branches so the persisted `InstanceAttributeOverride` row carries the element type — this is what the NJ-3 central normalizer reads for override rows (folds in follow-up #93/M3). + +**Step 1 — Failing test:** setting an override on a `List`/`String` template attribute persists an `InstanceAttributeOverride` whose `ElementDataType == DataType.String`; setting an override on a scalar attribute persists `ElementDataType == null`. + +**Step 2 — Run, expect FAIL.** + +**Step 3 — Implement:** in the update branch, `existingOverride.ElementDataType = templateAttr.ElementDataType;` before the repo update; in the create branch, set `ElementDataType = templateAttr.ElementDataType` in the `new InstanceAttributeOverride(...)` initializer. + +**Step 4 — Run, expect PASS.** + +**Step 5 — Commit:** `feat(template): stamp ElementDataType on instance attribute overrides`. + +--- + +### Task NJ-3: Central MS SQL idempotent startup normalizer + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** NJ-4, NJ-5 (after NJ-1) +**Blocked by:** NJ-1, NJ-2 + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs` (call the normalizer after the migrate/validate step) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/...` (use the existing test DbContext/SQLite-in-memory or test-double pattern the project uses — investigate the existing ConfigurationDatabase tests first) + +**Behavior:** A static `ListValueNormalizer.NormalizeAsync(ScadaBridgeDbContext db, ILogger? logger, CancellationToken ct)`: +- `TemplateAttributes` where `DataType == DataType.List`: for each, compute `var native = AttributeValueCodec.Encode(AttributeValueCodec.Decode(a.Value, DataType.List, a.ElementDataType));` (parse the row's own `ElementDataType` — it is a `DataType?`). If `native != a.Value`, set `a.Value = native`. +- `InstanceAttributeOverrides` whose `ElementDataType != null` (i.e. a List override stamped by NJ-2): same re-encode against `OverrideValue`. Rows with null `ElementDataType` are skipped with a debug log (legacy/scalar rows — none exist in practice since the feature has no deployed data). +- Wrap each row in try/catch(`FormatException`): on a malformed value, log a warning and SKIP that row — continue with the rest. The normalizer MUST NOT throw out of `NormalizeAsync` (never abort startup). +- `SaveChangesAsync` once at the end. Log a summary (`Normalized {n} list attribute value(s) to native JSON`). +- Idempotent: native→native yields identical bytes, so a second run makes no changes. + +**Wire-in (MigrationHelper):** after the `if (isDevelopment) { Migrate } else { validate }` block in `ApplyOrValidateMigrationsAsync`, call `await ListValueNormalizer.NormalizeAsync(dbContext, logger, cancellationToken);`. (Runs on every central startup; idempotent and safe even if both central nodes run it — concurrent UPDATEs set identical values.) + +**Steps (TDD):** failing tests (a TemplateAttributes row in old form `["10","20"]` with ElementDataType Int32 → rewritten to `[10,20]`; a native row → unchanged/no-op; a String-list `["a","b"]` → unchanged; a malformed row → skipped, sibling rows still normalized; an override row with ElementDataType set → normalized, with null → skipped) → run targeted → FAIL → implement `ListValueNormalizer` + wire-in → run targeted → PASS → build ConfigurationDatabase → commit `feat(db): idempotent startup normalizer rewriting List values to native JSON`. + +--- + +### Task NJ-4: Site SQLite active normalization on override load + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** NJ-3, NJ-5 (after NJ-1) +**Blocked by:** NJ-1 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:932-960` (`HandleOverridesLoaded`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs` + +In `HandleOverridesLoaded`, the List branch already decodes the stored override via `DecodeAttributeValue(resolved, kvp.Value)`. Extend it: when the decode succeeds and the re-encoded native form differs from the stored string (i.e. the stored value is old-form), re-persist the native form: + +```csharp +var decoded = DecodeAttributeValue(resolved, kvp.Value); +_attributes[kvp.Key] = decoded; +if (decoded is null && !string.IsNullOrEmpty(kvp.Value)) +{ + _attributeQualities[kvp.Key] = "Bad"; +} +else if (decoded is not null) +{ + var native = AttributeValueCodec.Encode(decoded); + if (native != kvp.Value) // stored value was old-form → normalize on disk + { + var key = kvp.Key; + _storage.SetStaticOverrideAsync(_instanceUniqueName, key, native!) + .ContinueWith(t => _logger.LogWarning(t.Exception?.GetBaseException(), + "Failed to normalize static override {Instance}.{Attr} to native JSON", _instanceUniqueName, key), + TaskContinuationOptions.OnlyOnFaulted); + } +} +``` +Idempotent (native→native ⇒ `native == kvp.Value` ⇒ no write). Scalars untouched. Never throws into the actor (decode already FormatException-safe; persist is fire-and-forget). + +**Steps (TDD):** failing tests (an old-form List static override on load → `SetStaticOverrideAsync` called with the native form; a native-form override → NOT re-persisted; a scalar override → unaffected; malformed → Bad quality + no crash, as today) → run targeted `--filter "FullyQualifiedName~InstanceActor"` → FAIL → implement → PASS → build SiteRuntime → commit `feat(siteruntime): normalize old-form List static overrides to native JSON on load`. + +--- + +### Task NJ-5: Normalize List values on bundle import + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** NJ-3, NJ-4 (after NJ-1) +**Blocked by:** NJ-1 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (the 3 `TemplateAttribute` construction sites from MV-12: BuildTemplate + the two in SyncTemplateAttributesAsync) — and/or `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` `FromBundleContent` if that is the single choke point for DTO→entity. +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/...` + +**Behavior:** when importing a `TemplateAttributeDto` whose `DataType == DataType.List`, write the **native** Value: `var v = NormalizeListValue(dto.Value, dto.ElementDataType);` where `NormalizeListValue` does `AttributeValueCodec.Encode(AttributeValueCodec.Decode(value, DataType.List, elementType))` inside try/catch(FormatException) → on failure return the original value unchanged + log (don't fail the import; the MS SQL normalizer is a backstop). Prefer a single private helper used at all construction sites (DRY). Non-List attributes write `dto.Value` unchanged. + +**Steps (TDD):** failing test (import a bundle/DTO with an old-form List attribute `["10","20"]` + ElementDataType Int32 → the persisted TemplateAttribute.Value is `[10,20]`; a String list stays quoted; a malformed value imports unchanged) → run targeted → FAIL → implement the helper + apply at construction sites → PASS → build Transport → commit `feat(transport): normalize List attribute values to native JSON on import`. + +--- + +### Task NJ-6: Integration verification + docs + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none +**Blocked by:** NJ-1 … NJ-5 + +**Files:** +- Modify: `docs/plans/2026-06-16-native-typed-json-design.md` (mark Status complete) +- Modify: `docs/requirements/Component-Commons.md` (update the `DataType`/codec note: List values are native-typed JSON; codec reads both forms) +- Modify: `docs/plans/2026-06-16-multivalue-attribute.md.tasks.json` follow-up note OR the task list — note that #93/M3 (override ElementDataType) was folded in here. + +**Steps:** +1. Full-solution build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → 0/0. +2. Targeted test sweep across touched projects: Commons codec, TemplateEngine (InstanceService), ConfigurationDatabase (normalizer), SiteRuntime (InstanceActor), Transport (serializer/importer). +3. Update docs; mark design Status complete. +4. Commit: `docs: mark native-typed JSON feature complete; update Component-Commons codec note`. + +**Acceptance:** full build green; all touched-project targeted tests pass; docs synced. + +--- + +## Parallelization summary + +- **Wave 1:** NJ-1 (codec) ∥ NJ-2 (override stamp) — disjoint (Commons vs TemplateEngine). +- **Wave 2 (after NJ-1, +NJ-2 for NJ-3):** NJ-3 (normalizer), NJ-4 (site), NJ-5 (import) in parallel — disjoint (ConfigurationDatabase vs SiteRuntime vs Transport). +- **Wave 3:** NJ-6 (verify + docs). + +## Risk notes + +- **NJ-1** changes the canonical storage/wire form — high-risk, but backward-compatible (read-both) and additive on the proto field. +- **NJ-3** writes to the config DB at startup — must be idempotent and never abort startup (per-row skip+log). +- **NJ-4** touches the actor model — the normalization write is fire-and-forget and never throws into the actor. +- There is no deployed List data yet, so the normalizers are a safety net; correctness of NJ-1 (the format) is what matters most in practice. diff --git a/docs/plans/2026-06-16-native-typed-json.md.tasks.json b/docs/plans/2026-06-16-native-typed-json.md.tasks.json new file mode 100644 index 00000000..90dcdf72 --- /dev/null +++ b/docs/plans/2026-06-16-native-typed-json.md.tasks.json @@ -0,0 +1,14 @@ +{ + "planPath": "docs/plans/2026-06-16-native-typed-json.md", + "designDoc": "docs/plans/2026-06-16-native-typed-json-design.md", + "branch": "feature/native-typed-json", + "tasks": [ + {"id": 96, "ref": "NJ-1", "subject": "Codec — native-typed Encode + read-both Decode", "class": "high-risk", "status": "pending"}, + {"id": 97, "ref": "NJ-2", "subject": "Stamp ElementDataType on instance-override writes", "class": "small", "status": "pending"}, + {"id": 98, "ref": "NJ-3", "subject": "Central MS SQL idempotent startup normalizer", "class": "high-risk", "status": "pending", "blockedBy": [96, 97]}, + {"id": 99, "ref": "NJ-4", "subject": "Site SQLite active normalization on override load", "class": "high-risk", "status": "pending", "blockedBy": [96]}, + {"id": 100, "ref": "NJ-5", "subject": "Normalize List values on bundle import", "class": "standard", "status": "pending", "blockedBy": [96]}, + {"id": 101, "ref": "NJ-6", "subject": "Integration verification + docs", "class": "standard", "status": "pending", "blockedBy": [96, 97, 98, 99, 100]} + ], + "lastUpdated": "2026-06-16" +} From 180d55482b594dbf4fd45696ac200f8c6c94ae53 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:32:40 -0400 Subject: [PATCH 03/11] feat(commons): native-typed JSON for List values; Decode reads both forms --- .../Types/AttributeValueCodec.cs | 24 +++++---- .../Types/AttributeValueCodecTests.cs | 49 ++++++++++++++++++- 2 files changed, 63 insertions(+), 10 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs index ecc4cbea..b848b79e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs @@ -25,11 +25,10 @@ public static class AttributeValueCodec case string s: return s; // already canonical case IFormattable f: return f.ToString(null, CultureInfo.InvariantCulture); case IEnumerable e: - var items = e.Cast() - .Select(x => x is IFormattable xf - ? xf.ToString(null, CultureInfo.InvariantCulture) - : x?.ToString()); - return JsonSerializer.Serialize(items, JsonOpts); + // Native-typed JSON: serialize the runtime collection so System.Text.Json emits + // numbers/bools unquoted, strings quoted, and DateTime as ISO-8601. Boxed as object + // so STJ uses the runtime element type. STJ numbers/dates are culture-invariant. + return JsonSerializer.Serialize(e, JsonOpts); default: return value.ToString(); } } @@ -46,18 +45,25 @@ public static class AttributeValueCodec if (elementType is null) throw new FormatException("List attribute requires an element type."); - string?[] raw; - try { raw = JsonSerializer.Deserialize(value) ?? []; } + JsonElement[] raw; + try { raw = JsonSerializer.Deserialize(value) ?? []; } catch (JsonException ex) { throw new FormatException("Malformed list JSON.", ex); } var clrType = ElementClrType(elementType.Value); var listType = typeof(List<>).MakeGenericType(clrType); var result = (IList)Activator.CreateInstance(listType)!; - foreach (var item in raw) - result.Add(ParseScalar(item, elementType.Value)); + foreach (var el in raw) + result.Add(ParseScalar(JsonElementToString(el), elementType.Value)); return result; } + private static string? JsonElementToString(JsonElement el) => el.ValueKind switch + { + JsonValueKind.String => el.GetString(), // old form, or string-typed lists + JsonValueKind.Null => null, // ParseScalar throws "may not be null" + _ => el.GetRawText() // number/bool → "10" / "1.5" / "true" + }; + private static Type ElementClrType(DataType t) => t switch { DataType.String => typeof(string), diff --git a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs index a4aed551..15229611 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs @@ -64,7 +64,7 @@ public class AttributeValueCodecTests try { CultureInfo.CurrentCulture = new CultureInfo("de-DE"); - Assert.Equal("[\"1.5\",\"2.5\"]", + Assert.Equal("[1.5,2.5]", AttributeValueCodec.Encode(new List { 1.5, 2.5 })); } finally @@ -73,6 +73,53 @@ public class AttributeValueCodecTests } } + [Fact] + public void Encode_Int32List_ProducesNativeNumbers() => + Assert.Equal("[10,20,30]", + AttributeValueCodec.Encode(new List { 10, 20, 30 })); + + [Fact] + public void Encode_BoolList_ProducesNativeBooleans() => + Assert.Equal("[true,false]", + AttributeValueCodec.Encode(new List { true, false })); + + [Fact] + public void Encode_StringList_StaysQuoted() => + Assert.Equal("[\"a\",\"b\"]", + AttributeValueCodec.Encode(new List { "a", "b" })); + + [Fact] + public void Encode_DateTimeList_IsIso8601() + { + var json = AttributeValueCodec.Encode( + new List { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) }); + Assert.Contains("2026-06-16T00:00:00", json); + Assert.DoesNotContain("06/16/2026", json); + } + + [Fact] + public void Decode_NewNativeIntForm_Parses() + { + var back = (IList)AttributeValueCodec.Decode("[10,20]", DataType.List, DataType.Int32)!; + Assert.Equal(new[] { 10, 20 }, back); + } + + [Fact] + public void Decode_OldStringIntForm_BackwardCompatible() + { + var back = (IList)AttributeValueCodec.Decode("[\"10\",\"20\"]", DataType.List, DataType.Int32)!; + Assert.Equal(new[] { 10, 20 }, back); + } + + [Theory] + [InlineData("[true,false]")] + [InlineData("[\"True\",\"False\"]")] + public void Decode_BoolForms_BothParse(string json) + { + var back = (IList)AttributeValueCodec.Decode(json, DataType.List, DataType.Boolean)!; + Assert.Equal(new[] { true, false }, back); + } + [Fact] public void RoundTrip_Int32List() { From abe8832e9e72fe91b3c6cb43f03aeae38f99c91f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:33:15 -0400 Subject: [PATCH 04/11] feat(template): stamp ElementDataType on instance attribute overrides Set existingOverride.ElementDataType and newOverride.ElementDataType from templateAttr.ElementDataType in both the update and create branches of SetAttributeOverrideAsync, so the persisted InstanceAttributeOverride row always carries the element type for later central normalizer use (#93/M3). --- .../Services/InstanceService.cs | 4 +- .../Services/InstanceServiceTests.cs | 100 ++++++++++++++++++ 2 files changed, 103 insertions(+), 1 deletion(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs b/src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs index 1c69a847..1f859bca 100644 --- a/src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs @@ -172,6 +172,7 @@ public class InstanceService if (existingOverride != null) { existingOverride.OverrideValue = overrideValue; + existingOverride.ElementDataType = templateAttr.ElementDataType; await _repository.UpdateInstanceAttributeOverrideAsync(existingOverride, cancellationToken); await _repository.SaveChangesAsync(cancellationToken); @@ -185,7 +186,8 @@ public class InstanceService var newOverride = new InstanceAttributeOverride(attributeName) { InstanceId = instanceId, - OverrideValue = overrideValue + OverrideValue = overrideValue, + ElementDataType = templateAttr.ElementDataType }; await _repository.AddInstanceAttributeOverrideAsync(newOverride, cancellationToken); diff --git a/tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs index b1cbc66b..b7e0a7fc 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs @@ -269,6 +269,106 @@ public class InstanceServiceTests _repoMock.Verify(r => r.AddInstanceConnectionBindingAsync(It.IsAny(), It.IsAny()), Times.Exactly(2)); } + // --- NJ-2: ElementDataType stamping on SetAttributeOverrideAsync --- + + [Fact] + public async Task SetAttributeOverride_CreatePath_ListAttribute_StampsElementDataType() + { + var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 }; + _repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny())) + .ReturnsAsync(instance); + _repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny())) + .ReturnsAsync(new List + { + new("Tags") + { + IsLocked = false, + DataType = DataType.List, + ElementDataType = DataType.String + } + }); + _repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny())) + .ReturnsAsync(new List()); // no existing override → create path + _repoMock.Setup(r => r.SaveChangesAsync(It.IsAny())) + .ReturnsAsync(1); + + InstanceAttributeOverride? captured = null; + _repoMock.Setup(r => r.AddInstanceAttributeOverrideAsync(It.IsAny(), It.IsAny())) + .Callback((o, _) => captured = o); + + var result = await _sut.SetAttributeOverrideAsync(1, "Tags", "[\"a\"]", "admin"); + + Assert.True(result.IsSuccess); + Assert.NotNull(captured); + Assert.Equal(DataType.String, captured.ElementDataType); + } + + [Fact] + public async Task SetAttributeOverride_UpdatePath_ListAttribute_StampsElementDataType() + { + var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 }; + _repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny())) + .ReturnsAsync(instance); + _repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny())) + .ReturnsAsync(new List + { + new("Tags") + { + IsLocked = false, + DataType = DataType.List, + ElementDataType = DataType.String + } + }); + + var existingOverride = new InstanceAttributeOverride("Tags") { Id = 42, InstanceId = 1, OverrideValue = "[\"old\"]" }; + _repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny())) + .ReturnsAsync(new List { existingOverride }); // pre-existing → update path + _repoMock.Setup(r => r.SaveChangesAsync(It.IsAny())) + .ReturnsAsync(1); + + InstanceAttributeOverride? captured = null; + _repoMock.Setup(r => r.UpdateInstanceAttributeOverrideAsync(It.IsAny(), It.IsAny())) + .Callback((o, _) => captured = o); + + var result = await _sut.SetAttributeOverrideAsync(1, "Tags", "[\"new\"]", "admin"); + + Assert.True(result.IsSuccess); + Assert.NotNull(captured); + Assert.Equal(DataType.String, captured.ElementDataType); + } + + [Fact] + public async Task SetAttributeOverride_CreatePath_ScalarAttribute_ElementDataTypeIsNull() + { + var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 }; + _repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny())) + .ReturnsAsync(instance); + _repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny())) + .ReturnsAsync(new List + { + new("Threshold") + { + IsLocked = false, + DataType = DataType.Float, + ElementDataType = null + } + }); + _repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny())) + .ReturnsAsync(new List()); + _repoMock.Setup(r => r.SaveChangesAsync(It.IsAny())) + .ReturnsAsync(1); + + InstanceAttributeOverride? captured = null; + _repoMock.Setup(r => r.AddInstanceAttributeOverrideAsync(It.IsAny(), It.IsAny())) + .Callback((o, _) => captured = o); + + var result = await _sut.SetAttributeOverrideAsync(1, "Threshold", "3.14", "admin"); + + Assert.True(result.IsSuccess); + Assert.NotNull(captured); + Assert.Null(captured.ElementDataType); + } + [Fact] public async Task AssignToArea_AreaInDifferentSite_ReturnsFailure() { From bf80ca1388a925e9551d01696f3f0584613fffa2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:38:57 -0400 Subject: [PATCH 05/11] =?UTF-8?q?test(commons):=20NJ-1=20review=20?= =?UTF-8?q?=E2=80=94=20backward-compat=20decode=20tests=20for=20old-form?= =?UTF-8?q?=20Float/DateTime=20+=20assert=20DateTime=20list=20is=20quoted-?= =?UTF-8?q?string=20array?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../Types/AttributeValueCodecTests.cs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs index 15229611..fa717bdb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs @@ -93,10 +93,28 @@ public class AttributeValueCodecTests { var json = AttributeValueCodec.Encode( new List { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) }); + Assert.StartsWith("[\"", json); // DateTime list must be an array of quoted strings Assert.Contains("2026-06-16T00:00:00", json); Assert.DoesNotContain("06/16/2026", json); } + [Fact] + public void Decode_OldStringFloatForm_BackwardCompatible() + { + var back = (IList)AttributeValueCodec.Decode("[\"1.5\",\"2.25\"]", DataType.List, DataType.Float)!; + Assert.Equal(new[] { 1.5f, 2.25f }, back); + } + + [Fact] + public void Decode_OldStringDateTimeForm_BackwardCompatible() + { + // The pre-native codec encoded DateTime via IFormattable.ToString(null, InvariantCulture): + // "06/16/2026 12:30:45" (US-invariant, no 'T'/'Z'). New Decode must still parse it. + var back = (IList)AttributeValueCodec.Decode( + "[\"06/16/2026 12:30:45\"]", DataType.List, DataType.DateTime)!; + Assert.Equal(new DateTime(2026, 6, 16, 12, 30, 45, DateTimeKind.Unspecified), back[0]); + } + [Fact] public void Decode_NewNativeIntForm_Parses() { From 5841cec9587b7b7a7ff9cd8969e876224bf06bcc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:49:21 -0400 Subject: [PATCH 06/11] feat(siteruntime): normalize old-form List static overrides to native JSON on load --- .../Actors/InstanceActor.cs | 21 +++ .../Actors/InstanceActorTests.cs | 155 ++++++++++++++++++ 2 files changed, 176 insertions(+) diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs index c8bd3afa..64c65d3c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs @@ -948,10 +948,31 @@ public class InstanceActor : ReceiveActor if (_resolvedAttributeByName.TryGetValue(kvp.Key, out var resolved) && IsListAttribute(resolved)) { + // NJ-4: decode the stored List override (both old array-of-strings + // and native-typed forms decode) and re-persist the native form if + // the stored value is still in the OLD form. Re-encoding the decoded + // list and comparing to the stored string detects old-form values + // (native → native is byte-identical, so a native value is a no-op). + // The re-persist is fire-and-forget and never throws into the actor. var decoded = DecodeAttributeValue(resolved, kvp.Value); _attributes[kvp.Key] = decoded; if (decoded is null && !string.IsNullOrEmpty(kvp.Value)) + { _attributeQualities[kvp.Key] = "Bad"; + } + else if (decoded is not null) + { + var native = AttributeValueCodec.Encode(decoded); + if (native != kvp.Value) // stored value was old-form → normalize on disk + { + var key = kvp.Key; + _storage.SetStaticOverrideAsync(_instanceUniqueName, key, native!) + .ContinueWith(t => _logger.LogWarning(t.Exception?.GetBaseException(), + "Failed to normalize static override {Instance}.{Attr} to native JSON", + _instanceUniqueName, key), + TaskContinuationOptions.OnlyOnFaulted); + } + } } else { diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs index f93a238e..72dbb49c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs @@ -1016,4 +1016,159 @@ public class InstanceActorTests : TestKit, IDisposable Assert.Single(overrides); Assert.Equal("[]", overrides["Labels"]); } + + // ── NJ-4: old-form List static override normalization on load ──────────── + + /// + /// NJ-4: an OLD array-of-strings static override (["10","20"]) for an + /// Int32 List attribute must be re-persisted in the native form ([10,20]) + /// when the actor loads it at startup. The in-memory read still returns the + /// typed list {10,20}; the on-disk value is normalized to native JSON. + /// + [Fact] + public async Task InstanceActor_OldFormListOverride_NormalizedToNativeOnLoad() + { + await _storage.SetStaticOverrideAsync("Pump-OldForm", "Counts", "[\"10\",\"20\"]"); + + var config = new FlattenedConfiguration + { + InstanceUniqueName = "Pump-OldForm", + Attributes = + [ + new ResolvedAttribute + { + CanonicalName = "Counts", Value = "[1,2]", + DataType = "List", ElementDataType = "Int32" + } + ] + }; + + var actor = CreateInstanceActor("Pump-OldForm", config); + + // Wait for the async override load (PipeTo) + fire-and-forget normalization. + await Task.Delay(1000); + + // In-memory read returns the typed list, decoded from the old form. + actor.Tell(new GetAttributeRequest("corr-of", "Pump-OldForm", "Counts", DateTimeOffset.UtcNow)); + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.True(response.Found); + var list = Assert.IsType>(response.Value); + Assert.Equal(new[] { 10, 20 }, list); + + // The on-disk override has been normalized to the native form. + var overrides = await _storage.GetStaticOverridesAsync("Pump-OldForm"); + Assert.Single(overrides); + Assert.Equal("[10,20]", overrides["Counts"]); + } + + /// + /// NJ-4: a NATIVE-form static override ([10,20]) is already canonical, so + /// load-time normalization must be a no-op — the on-disk value is unchanged + /// (idempotent: native → native is byte-identical, so no re-persist occurs). + /// + [Fact] + public async Task InstanceActor_NativeFormListOverride_NotRePersistedOnLoad() + { + await _storage.SetStaticOverrideAsync("Pump-Native", "Counts", "[10,20]"); + + var config = new FlattenedConfiguration + { + InstanceUniqueName = "Pump-Native", + Attributes = + [ + new ResolvedAttribute + { + CanonicalName = "Counts", Value = "[1,2]", + DataType = "List", ElementDataType = "Int32" + } + ] + }; + + var actor = CreateInstanceActor("Pump-Native", config); + await Task.Delay(1000); + + actor.Tell(new GetAttributeRequest("corr-nat", "Pump-Native", "Counts", DateTimeOffset.UtcNow)); + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.True(response.Found); + var list = Assert.IsType>(response.Value); + Assert.Equal(new[] { 10, 20 }, list); + + // The native value is left untouched on disk. + var overrides = await _storage.GetStaticOverridesAsync("Pump-Native"); + Assert.Single(overrides); + Assert.Equal("[10,20]", overrides["Counts"]); + } + + /// + /// NJ-4: a scalar static override is unaffected by the List normalization path — + /// its on-disk value is left exactly as stored (no native re-encode). + /// + [Fact] + public async Task InstanceActor_ScalarOverride_NotTouchedByListNormalization() + { + await _storage.SetStaticOverrideAsync("Pump-ScalarOf", "Temperature", "200.0"); + + var config = new FlattenedConfiguration + { + InstanceUniqueName = "Pump-ScalarOf", + Attributes = + [ + new ResolvedAttribute { CanonicalName = "Temperature", Value = "100.0", DataType = "Double" } + ] + }; + + var actor = CreateInstanceActor("Pump-ScalarOf", config); + await Task.Delay(1000); + + actor.Tell(new GetAttributeRequest("corr-sof", "Pump-ScalarOf", "Temperature", DateTimeOffset.UtcNow)); + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.True(response.Found); + Assert.Equal("200.0", response.Value); + + var overrides = await _storage.GetStaticOverridesAsync("Pump-ScalarOf"); + Assert.Single(overrides); + Assert.Equal("200.0", overrides["Temperature"]); + } + + /// + /// NJ-4: a malformed stored List override (truncated JSON) must NOT crash the + /// actor and must NOT be re-persisted — it loads with Bad quality (as today), + /// the actor stays alive, and the poison on-disk value is left unchanged. + /// + [Fact] + public async Task InstanceActor_MalformedListOverride_BadQuality_NotRePersisted() + { + await _storage.SetStaticOverrideAsync("Pump-BadOf", "Counts", "[\"a\""); + + var config = new FlattenedConfiguration + { + InstanceUniqueName = "Pump-BadOf", + Attributes = + [ + new ResolvedAttribute + { + CanonicalName = "Counts", Value = "[1,2]", + DataType = "List", ElementDataType = "Int32" + } + ] + }; + + var actor = CreateInstanceActor("Pump-BadOf", config); + Watch(actor); + await Task.Delay(1000); + + actor.Tell(new GetAttributeRequest("corr-bof", "Pump-BadOf", "Counts", DateTimeOffset.UtcNow)); + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.True(response.Found); + Assert.Equal("Bad", response.Quality); + Assert.Null(response.Value); + + // The actor must still be alive — no crash from the normalization path. + ExpectNoTerminated(actor, TimeSpan.FromMilliseconds(500)); + + // The malformed value must NOT have been re-persisted (left exactly as stored). + var overrides = await _storage.GetStaticOverridesAsync("Pump-BadOf"); + Assert.Single(overrides); + Assert.Equal("[\"a\"", overrides["Counts"]); + } } From e3d804a1a6bec85a1ed1ff6e0fdb5d346badb3cd Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:50:05 -0400 Subject: [PATCH 07/11] feat(transport): normalize List attribute values to native JSON on import --- .../Import/BundleImporter.cs | 14 ++- .../Serialization/EntitySerializer.cs | 2 +- .../Serialization/ImportValueNormalizer.cs | 51 ++++++++ .../Serialization/EntitySerializerTests.cs | 116 ++++++++++++++++++ 4 files changed, 178 insertions(+), 5 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs b/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs index a93d34bc..f05852d9 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs @@ -1040,7 +1040,7 @@ public sealed class BundleImporter : IBundleImporter { t.Attributes.Add(new TemplateAttribute(a.Name) { - Value = a.Value, + Value = ImportValueNormalizer.NormalizeListValue(a.Value, a.DataType, a.ElementDataType), DataType = a.DataType, IsLocked = a.IsLocked, Description = a.Description, @@ -1115,11 +1115,17 @@ public sealed class BundleImporter : IBundleImporter // Adds + Updates. foreach (var attrDto in dto.Attributes) { + // Normalise List values to the native-typed JSON form on import so the + // comparison (and the persisted value) match what the target already + // stores natively — otherwise an idempotent re-import of an old-form + // bundle would spuriously report a Value change. + var normalizedValue = ImportValueNormalizer.NormalizeListValue( + attrDto.Value, attrDto.DataType, attrDto.ElementDataType); if (existingByName.TryGetValue(attrDto.Name, out var current)) { // Update only if any field actually changed. bool changed = - !string.Equals(current.Value, attrDto.Value, StringComparison.Ordinal) || + !string.Equals(current.Value, normalizedValue, StringComparison.Ordinal) || current.DataType != attrDto.DataType || current.IsLocked != attrDto.IsLocked || !string.Equals(current.Description, attrDto.Description, StringComparison.Ordinal) || @@ -1127,7 +1133,7 @@ public sealed class BundleImporter : IBundleImporter current.ElementDataType != attrDto.ElementDataType; if (!changed) continue; - current.Value = attrDto.Value; + current.Value = normalizedValue; current.DataType = attrDto.DataType; current.IsLocked = attrDto.IsLocked; current.Description = attrDto.Description; @@ -1157,7 +1163,7 @@ public sealed class BundleImporter : IBundleImporter { var newAttr = new TemplateAttribute(attrDto.Name) { - Value = attrDto.Value, + Value = normalizedValue, DataType = attrDto.DataType, IsLocked = attrDto.IsLocked, Description = attrDto.Description, diff --git a/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs index 22c7c31b..88b66573 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs @@ -199,7 +199,7 @@ public sealed class EntitySerializer t.Attributes.Add(new TemplateAttribute(a.Name) { TemplateId = t.Id, - Value = a.Value, + Value = ImportValueNormalizer.NormalizeListValue(a.Value, a.DataType, a.ElementDataType), DataType = a.DataType, IsLocked = a.IsLocked, Description = a.Description, diff --git a/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs new file mode 100644 index 00000000..e4211ae1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs @@ -0,0 +1,51 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Types; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.Transport.Serialization; + +/// +/// Import-time normalization of attribute values to the native-typed JSON form. +/// +/// Bundles exported before the native-typed-JSON change carry List attribute +/// values in the old quoted-element form (e.g. ["10","20"] for an +/// list). Already-exported bundle files can't be +/// rewritten, so List values are normalised on import: every DTO→entity write +/// site routes the value through so imported +/// data lands native ([10,20]). The central DB normalizer remains the +/// backstop for anything that slips through. +/// +/// +/// Non-List attributes and null/empty values pass through unchanged. A value +/// that fails to decode (malformed JSON / un-parseable element) is left exactly +/// as-is so the import still succeeds — the DB normalizer is the backstop. +/// +/// +internal static class ImportValueNormalizer +{ + /// + /// Returns the native-typed JSON form of a List attribute value, or the + /// value unchanged for non-List / null / empty / malformed inputs. + /// + /// The attribute value as carried by the bundle DTO. + /// The attribute's declared data type. + /// The List element type (null for scalars). + public static string? NormalizeListValue(string? value, DataType dataType, DataType? elementType) + { + if (dataType != DataType.List || string.IsNullOrEmpty(value)) + { + return value; + } + + try + { + return AttributeValueCodec.Encode( + AttributeValueCodec.Decode(value, DataType.List, elementType)); + } + catch (FormatException) + { + // Leave malformed values exactly as imported; the DB normalizer is + // the backstop. Never abort the import for a single bad value. + return value; + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs index b4fefbb0..51f8ab0c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs @@ -259,6 +259,122 @@ public sealed class EntitySerializerTests Assert.Equal("[\"a\",\"b\"]", rtAttr.Value); } + private static BundleContentDto MakeContentWithListAttribute( + string value, DataType elementType) + { + var template = new TemplateDto( + Name: "Pump", + FolderName: null, + BaseTemplateName: null, + Description: null, + Attributes: new[] + { + new TemplateAttributeDto( + Name: "Tags", + Value: value, + DataType: DataType.List, + IsLocked: false, + Description: null, + DataSourceReference: null, + ElementDataType: elementType), + }, + Alarms: Array.Empty(), + Scripts: Array.Empty(), + Compositions: Array.Empty()); + + return new BundleContentDto( + TemplateFolders: Array.Empty(), + Templates: new[] { template }, + SharedScripts: Array.Empty(), + ExternalSystems: Array.Empty(), + DatabaseConnections: Array.Empty(), + NotificationLists: Array.Empty(), + SmtpConfigs: Array.Empty(), + ApiMethods: Array.Empty()); + } + + [Fact] + public void Import_normalizes_old_form_Int32_list_value_to_native_json() + { + // Pre-native bundle: quoted Int32 list elements. + var dto = MakeContentWithListAttribute("[\"10\",\"20\"]", DataType.Int32); + + var aggregate = new EntitySerializer().FromBundleContent(dto); + + var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes); + Assert.Equal(DataType.List, attr.DataType); + Assert.Equal(DataType.Int32, attr.ElementDataType); + // Imported native: numbers unquoted. + Assert.Equal("[10,20]", attr.Value); + } + + [Fact] + public void Import_leaves_string_list_value_quoted() + { + var dto = MakeContentWithListAttribute("[\"a\",\"b\"]", DataType.String); + + var aggregate = new EntitySerializer().FromBundleContent(dto); + + var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes); + Assert.Equal(DataType.List, attr.DataType); + Assert.Equal(DataType.String, attr.ElementDataType); + // Strings stay quoted in native form. + Assert.Equal("[\"a\",\"b\"]", attr.Value); + } + + [Fact] + public void Import_leaves_malformed_list_value_unchanged_without_throwing() + { + // Truncated JSON array — Decode throws FormatException; the import must + // still succeed and carry the value through verbatim (DB normalizer is + // the backstop). + var dto = MakeContentWithListAttribute("[\"a\"", DataType.String); + + var aggregate = new EntitySerializer().FromBundleContent(dto); + + var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes); + Assert.Equal("[\"a\"", attr.Value); + } + + [Fact] + public void Import_leaves_scalar_attribute_value_unchanged() + { + var template = new TemplateDto( + Name: "Sensor", + FolderName: null, + BaseTemplateName: null, + Description: null, + Attributes: new[] + { + new TemplateAttributeDto( + Name: "Pressure", + Value: "42.0", + DataType: DataType.Double, + IsLocked: false, + Description: null, + DataSourceReference: null, + ElementDataType: null), + }, + Alarms: Array.Empty(), + Scripts: Array.Empty(), + Compositions: Array.Empty()); + var dto = new BundleContentDto( + TemplateFolders: Array.Empty(), + Templates: new[] { template }, + SharedScripts: Array.Empty(), + ExternalSystems: Array.Empty(), + DatabaseConnections: Array.Empty(), + NotificationLists: Array.Empty(), + SmtpConfigs: Array.Empty(), + ApiMethods: Array.Empty()); + + var aggregate = new EntitySerializer().FromBundleContent(dto); + + var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes); + Assert.Equal(DataType.Double, attr.DataType); + Assert.Equal("42.0", attr.Value); + } + [Fact] public void Roundtrip_scalar_attribute_with_null_ElementDataType_remains_null() { From f4b101b532eb7d6832498488e13a0e0d85ddfaf0 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 17:50:19 -0400 Subject: [PATCH 08/11] feat(db): idempotent startup normalizer rewriting List values to native JSON --- .../ListValueNormalizer.cs | 123 ++++++++++ .../MigrationHelper.cs | 6 + .../ListValueNormalizerTests.cs | 231 ++++++++++++++++++ 3 files changed, 360 insertions(+) create mode 100644 src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ListValueNormalizerTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs new file mode 100644 index 00000000..abac3825 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs @@ -0,0 +1,123 @@ +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.ScadaBridge.Commons.Types; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +/// +/// Idempotent central startup normalizer that rewrites already-persisted List attribute +/// values from the old array-of-strings JSON form (["10","20"]) to the new +/// native-typed form ([10,20]). +/// +/// +/// This is a safety net: at the time of writing there is no deployed List attribute data, +/// so in practice it finds nothing to rewrite. It runs on every central startup after +/// migrations apply and MUST never abort startup for data reasons — every row's work is +/// wrapped in a per-row try/catch that logs and skips on malformed data. A second run +/// finds nothing to change (native → native re-encode is byte-identical). +/// +/// +public static class ListValueNormalizer +{ + /// + /// Rewrites old-form List attribute values to the native-typed JSON form across + /// and + /// . Idempotent and + /// best-effort: malformed rows are logged and skipped, never rethrown. + /// + /// The configuration database context. + /// Optional logger for diagnostics. + /// Cancellation token. + /// A task that represents the asynchronous operation. + public static async Task NormalizeAsync( + ScadaBridgeDbContext db, + ILogger? logger = null, + CancellationToken ct = default) + { + var rewritten = 0; + + // TemplateAttributes: List rows carry the element type on the row itself. + var templateRows = await db.TemplateAttributes + .Where(a => a.DataType == DataType.List) + .ToListAsync(ct); + + foreach (var a in templateRows) + { + try + { + var native = AttributeValueCodec.Encode( + AttributeValueCodec.Decode(a.Value, DataType.List, a.ElementDataType)); + if (native != a.Value) + { + a.Value = native; + rewritten++; + } + } + catch (FormatException ex) + { + logger?.LogWarning(ex, + "List value normalizer: skipping unparseable list value for TemplateAttribute {Id}.", + a.Id); + } + } + + // InstanceAttributeOverrides: only rows that carry an element type are List rows. + // Rows with a null ElementDataType are scalar/legacy rows (no deployed List data + // exists, so none in practice) and are skipped. + var overrideRows = await db.InstanceAttributeOverrides + .Where(o => o.ElementDataType != null) + .ToListAsync(ct); + + foreach (var o in overrideRows) + { + if (o.ElementDataType is null) + { + logger?.LogDebug( + "List value normalizer: skipping InstanceAttributeOverride {Id} with no element type.", + o.Id); + continue; + } + + try + { + var native = AttributeValueCodec.Encode( + AttributeValueCodec.Decode(o.OverrideValue, DataType.List, o.ElementDataType)); + if (native != o.OverrideValue) + { + o.OverrideValue = native; + rewritten++; + } + } + catch (FormatException ex) + { + logger?.LogWarning(ex, + "List value normalizer: skipping unparseable list value for InstanceAttributeOverride {Id}.", + o.Id); + } + } + + try + { + await db.SaveChangesAsync(ct); + } + catch (Exception ex) + { + // A catastrophic DB failure on SaveChanges may propagate, but log it first so + // startup diagnostics are not silent. Per-row data problems are already handled + // above and never reach here. + logger?.LogError(ex, "List value normalizer: SaveChanges failed."); + throw; + } + + if (rewritten > 0) + { + logger?.LogInformation( + "List value normalizer: rewrote {n} attribute value(s) to native JSON.", rewritten); + } + else + { + logger?.LogDebug("List value normalizer: no attribute values required rewriting."); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs index c5b9ba73..c5b15d1a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs @@ -47,6 +47,12 @@ public static class MigrationHelper "Apply migrations using 'dotnet ef database update' or the generated SQL scripts before starting in production mode."); } } + + // Safety-net normalizer: rewrite any already-persisted List attribute values from the + // old array-of-strings JSON form to the new native-typed form. Idempotent and never + // aborts startup for data reasons (per-row skip + log). Safe even if both central + // nodes run it concurrently on startup. + await ListValueNormalizer.NormalizeAsync(dbContext, logger, cancellationToken); } private static async Task WaitForDatabaseReadyAsync( diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ListValueNormalizerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ListValueNormalizerTests.cs new file mode 100644 index 00000000..fc224c46 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ListValueNormalizerTests.cs @@ -0,0 +1,231 @@ +using Microsoft.EntityFrameworkCore; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests; + +/// +/// Tests for — the idempotent startup normalizer that +/// rewrites already-persisted List attribute values from the old array-of-strings JSON +/// form (["10","20"]) to the new native-typed form ([10,20]). +/// +public class ListValueNormalizerTests : IDisposable +{ + private readonly ScadaBridgeDbContext _context; + + public ListValueNormalizerTests() + { + _context = SqliteTestHelper.CreateInMemoryContext(); + } + + public void Dispose() + { + _context.Database.CloseConnection(); + _context.Dispose(); + } + + // Seeds a Template parent (satisfies the TemplateAttribute -> Template FK) and returns its Id. + private async Task SeedTemplateAsync() + { + var template = new Template("T1"); + _context.Templates.Add(template); + await _context.SaveChangesAsync(); + return template.Id; + } + + // Seeds Site + Template + Instance (satisfies the InstanceAttributeOverride -> Instance FK) + // and returns the Instance Id. + private async Task SeedInstanceAsync() + { + var site = new Site("Site1", "S-001"); + var template = new Template("T1"); + _context.Sites.Add(site); + _context.Templates.Add(template); + await _context.SaveChangesAsync(); + + var instance = new Instance("Inst1") { SiteId = site.Id, TemplateId = template.Id }; + _context.Instances.Add(instance); + await _context.SaveChangesAsync(); + return instance.Id; + } + + [Fact] + public async Task TemplateAttribute_OldStringForm_RewrittenToNative() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("intList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.Int32, + Value = "[\"10\",\"20\"]", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("[10,20]", reloaded.Value); + } + + [Fact] + public async Task TemplateAttribute_AlreadyNative_IsNotRewritten() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("intList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.Int32, + Value = "[10,20]", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + // No tracked entity should be marked Modified — idempotent no-op. + var tracked = _context.ChangeTracker.Entries() + .Where(e => e.State == EntityState.Modified) + .ToList(); + Assert.Empty(tracked); + + var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("[10,20]", reloaded.Value); + } + + [Fact] + public async Task TemplateAttribute_StringList_IsUnchanged() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("stringList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.String, + Value = "[\"a\",\"b\"]", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("[\"a\",\"b\"]", reloaded.Value); + } + + [Fact] + public async Task TemplateAttribute_Malformed_IsSkipped_AndSiblingStillNormalized() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("badList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.Int32, + Value = "[\"a\"", // malformed JSON + }); + _context.TemplateAttributes.Add(new TemplateAttribute("goodList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.Int32, + Value = "[\"5\"]", // old form, valid + }); + await _context.SaveChangesAsync(); + + // Must NOT throw despite the malformed row. + await ListValueNormalizer.NormalizeAsync(_context); + + var bad = await _context.TemplateAttributes.AsNoTracking() + .SingleAsync(a => a.Name == "badList"); + var good = await _context.TemplateAttributes.AsNoTracking() + .SingleAsync(a => a.Name == "goodList"); + + Assert.Equal("[\"a\"", bad.Value); // skipped, untouched + Assert.Equal("[5]", good.Value); // normalized + } + + [Fact] + public async Task TemplateAttribute_NonListRow_IsUnchanged() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("scalar") + { + TemplateId = templateId, + DataType = DataType.Int32, + ElementDataType = null, + Value = "42", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("42", reloaded.Value); + } + + [Fact] + public async Task InstanceAttributeOverride_OldStringForm_RewrittenToNative() + { + var instanceId = await SeedInstanceAsync(); + _context.InstanceAttributeOverrides.Add(new InstanceAttributeOverride("intList") + { + InstanceId = instanceId, + ElementDataType = DataType.Int32, + OverrideValue = "[\"5\"]", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + var reloaded = await _context.InstanceAttributeOverrides.AsNoTracking().SingleAsync(); + Assert.Equal("[5]", reloaded.OverrideValue); + } + + [Fact] + public async Task InstanceAttributeOverride_NullElementType_IsUntouched() + { + var instanceId = await SeedInstanceAsync(); + _context.InstanceAttributeOverrides.Add(new InstanceAttributeOverride("scalar") + { + InstanceId = instanceId, + ElementDataType = null, + OverrideValue = "42", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + + var reloaded = await _context.InstanceAttributeOverrides.AsNoTracking().SingleAsync(); + Assert.Equal("42", reloaded.OverrideValue); + } + + [Fact] + public async Task NormalizeAsync_IsIdempotent_SecondRunChangesNothing() + { + var templateId = await SeedTemplateAsync(); + _context.TemplateAttributes.Add(new TemplateAttribute("intList") + { + TemplateId = templateId, + DataType = DataType.List, + ElementDataType = DataType.Int32, + Value = "[\"10\",\"20\"]", + }); + await _context.SaveChangesAsync(); + + await ListValueNormalizer.NormalizeAsync(_context); + var afterFirst = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("[10,20]", afterFirst.Value); + + await ListValueNormalizer.NormalizeAsync(_context); + var afterSecond = await _context.TemplateAttributes.AsNoTracking().SingleAsync(); + Assert.Equal("[10,20]", afterSecond.Value); + + var tracked = _context.ChangeTracker.Entries() + .Where(e => e.State == EntityState.Modified) + .ToList(); + Assert.Empty(tracked); + } +} From feeae1371eec924c6da428e32b19de62b5ad1b37 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 18:25:42 -0400 Subject: [PATCH 09/11] fix(multivalue): NJ-3/NJ-4/NJ-5 review fixes - NJ-3: widen per-row catch to Exception (an STJ encode failure can't abort startup); drop dead null-guard already excluded by the SQL filter - NJ-4: capture logger/instanceName in locals for the fire-and-forget normalize continuation (match the sibling pattern in this actor) - NJ-5: emit a warn-log when a malformed List value is imported verbatim; thread an optional ILogger to the sync re-import site --- .../ListValueNormalizer.cs | 19 +++++++------------ .../Actors/InstanceActor.cs | 8 +++++--- .../Import/BundleImporter.cs | 8 ++++++-- .../Serialization/ImportValueNormalizer.cs | 16 ++++++++++++++-- 4 files changed, 32 insertions(+), 19 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs index abac3825..2d8659b6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs @@ -54,10 +54,13 @@ public static class ListValueNormalizer rewritten++; } } - catch (FormatException ex) + catch (Exception ex) { + // Never abort startup for a single bad row. FormatException from Decode is the + // expected case; the broad catch also covers an unexpected serialize failure + // (e.g. a JsonException on a non-finite value) so one poison row can't crash boot. logger?.LogWarning(ex, - "List value normalizer: skipping unparseable list value for TemplateAttribute {Id}.", + "List value normalizer: skipping unprocessable list value for TemplateAttribute {Id}.", a.Id); } } @@ -71,14 +74,6 @@ public static class ListValueNormalizer foreach (var o in overrideRows) { - if (o.ElementDataType is null) - { - logger?.LogDebug( - "List value normalizer: skipping InstanceAttributeOverride {Id} with no element type.", - o.Id); - continue; - } - try { var native = AttributeValueCodec.Encode( @@ -89,10 +84,10 @@ public static class ListValueNormalizer rewritten++; } } - catch (FormatException ex) + catch (Exception ex) { logger?.LogWarning(ex, - "List value normalizer: skipping unparseable list value for InstanceAttributeOverride {Id}.", + "List value normalizer: skipping unprocessable list value for InstanceAttributeOverride {Id}.", o.Id); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs index 64c65d3c..55b4e781 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs @@ -966,10 +966,12 @@ public class InstanceActor : ReceiveActor if (native != kvp.Value) // stored value was old-form → normalize on disk { var key = kvp.Key; - _storage.SetStaticOverrideAsync(_instanceUniqueName, key, native!) - .ContinueWith(t => _logger.LogWarning(t.Exception?.GetBaseException(), + var logger = _logger; + var instanceName = _instanceUniqueName; + _storage.SetStaticOverrideAsync(instanceName, key, native!) + .ContinueWith(t => logger.LogWarning(t.Exception?.GetBaseException(), "Failed to normalize static override {Instance}.{Attr} to native JSON", - _instanceUniqueName, key), + instanceName, key), TaskContinuationOptions.OnlyOnFaulted); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs b/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs index f05852d9..43750632 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs @@ -1,6 +1,7 @@ using System.IO.Compression; using System.Security.Cryptography; using System.Text.Json; +using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi; @@ -70,6 +71,7 @@ public sealed class BundleImporter : IBundleImporter private readonly IOptions _options; private readonly TimeProvider _timeProvider; private readonly SemanticValidator _semanticValidator; + private readonly ILogger? _logger; /// /// Initializes a new with all required dependencies. @@ -106,7 +108,8 @@ public sealed class BundleImporter : IBundleImporter IAuditService auditService, IAuditCorrelationContext correlationContext, ScadaBridgeDbContext dbContext, - SemanticValidator semanticValidator) + SemanticValidator semanticValidator, + ILogger? logger = null) { _bundleSerializer = bundleSerializer ?? throw new ArgumentNullException(nameof(bundleSerializer)); _manifestValidator = manifestValidator ?? throw new ArgumentNullException(nameof(manifestValidator)); @@ -124,6 +127,7 @@ public sealed class BundleImporter : IBundleImporter _correlationContext = correlationContext ?? throw new ArgumentNullException(nameof(correlationContext)); _dbContext = dbContext ?? throw new ArgumentNullException(nameof(dbContext)); _semanticValidator = semanticValidator ?? throw new ArgumentNullException(nameof(semanticValidator)); + _logger = logger; } /// @@ -1120,7 +1124,7 @@ public sealed class BundleImporter : IBundleImporter // stores natively — otherwise an idempotent re-import of an old-form // bundle would spuriously report a Value change. var normalizedValue = ImportValueNormalizer.NormalizeListValue( - attrDto.Value, attrDto.DataType, attrDto.ElementDataType); + attrDto.Value, attrDto.DataType, attrDto.ElementDataType, _logger, attrDto.Name); if (existingByName.TryGetValue(attrDto.Name, out var current)) { // Update only if any field actually changed. diff --git a/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs index e4211ae1..2fcf9772 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/ImportValueNormalizer.cs @@ -1,3 +1,4 @@ +using Microsoft.Extensions.Logging; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; @@ -29,7 +30,14 @@ internal static class ImportValueNormalizer /// The attribute value as carried by the bundle DTO. /// The attribute's declared data type. /// The List element type (null for scalars). - public static string? NormalizeListValue(string? value, DataType dataType, DataType? elementType) + /// Optional logger; a warning is emitted when a malformed value is left as-is. + /// Optional attribute name for the diagnostic message. + public static string? NormalizeListValue( + string? value, + DataType dataType, + DataType? elementType, + ILogger? logger = null, + string? attributeName = null) { if (dataType != DataType.List || string.IsNullOrEmpty(value)) { @@ -41,10 +49,14 @@ internal static class ImportValueNormalizer return AttributeValueCodec.Encode( AttributeValueCodec.Decode(value, DataType.List, elementType)); } - catch (FormatException) + catch (FormatException ex) { // Leave malformed values exactly as imported; the DB normalizer is // the backstop. Never abort the import for a single bad value. + logger?.LogWarning(ex, + "Bundle import: could not normalize List value for attribute {Attribute}; " + + "importing verbatim (the central DB normalizer is the backstop).", + attributeName ?? "(unknown)"); return value; } } From c53b621b8502e694e6d06a4939278fb6f7bff5dc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 18:27:10 -0400 Subject: [PATCH 10/11] docs: mark native-typed JSON feature complete; update Component-Commons codec note NJ-6: full solution builds 0/0; feature-targeted tests green (Commons codec 38, TemplateEngine InstanceService 17, ConfigDB normalizer 8, Transport serializer 12, SiteRuntime InstanceActor 47). Component-Commons now describes the native-typed List encoding + read-both decode + the three normalization paths. #93/M3 folded in. --- .../plans/2026-06-16-native-typed-json-design.md | 2 +- .../2026-06-16-native-typed-json.md.tasks.json | 16 +++++++++------- docs/requirements/Component-Commons.md | 2 +- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/plans/2026-06-16-native-typed-json-design.md b/docs/plans/2026-06-16-native-typed-json-design.md index 07573fc3..edc9a9e3 100644 --- a/docs/plans/2026-06-16-native-typed-json-design.md +++ b/docs/plans/2026-06-16-native-typed-json-design.md @@ -1,7 +1,7 @@ # Native-Typed JSON for List Attribute Values — Design **Date:** 2026-06-16 -**Status:** Approved (brainstorming) — ready for implementation plan +**Status:** Implemented (NJ-1 … NJ-6, branch `feature/native-typed-json`) — full solution builds 0/0; feature-targeted tests green across Commons, TemplateEngine, ConfigurationDatabase, SiteRuntime, and Transport. Follow-up **#93/M3** (populate `InstanceAttributeOverride.ElementDataType` on write) was folded into NJ-2 so the central normalizer can read the override element type directly. **Branch:** `feature/native-typed-json` ## Problem diff --git a/docs/plans/2026-06-16-native-typed-json.md.tasks.json b/docs/plans/2026-06-16-native-typed-json.md.tasks.json index 90dcdf72..ac42df4f 100644 --- a/docs/plans/2026-06-16-native-typed-json.md.tasks.json +++ b/docs/plans/2026-06-16-native-typed-json.md.tasks.json @@ -2,13 +2,15 @@ "planPath": "docs/plans/2026-06-16-native-typed-json.md", "designDoc": "docs/plans/2026-06-16-native-typed-json-design.md", "branch": "feature/native-typed-json", + "status": "complete", "tasks": [ - {"id": 96, "ref": "NJ-1", "subject": "Codec — native-typed Encode + read-both Decode", "class": "high-risk", "status": "pending"}, - {"id": 97, "ref": "NJ-2", "subject": "Stamp ElementDataType on instance-override writes", "class": "small", "status": "pending"}, - {"id": 98, "ref": "NJ-3", "subject": "Central MS SQL idempotent startup normalizer", "class": "high-risk", "status": "pending", "blockedBy": [96, 97]}, - {"id": 99, "ref": "NJ-4", "subject": "Site SQLite active normalization on override load", "class": "high-risk", "status": "pending", "blockedBy": [96]}, - {"id": 100, "ref": "NJ-5", "subject": "Normalize List values on bundle import", "class": "standard", "status": "pending", "blockedBy": [96]}, - {"id": 101, "ref": "NJ-6", "subject": "Integration verification + docs", "class": "standard", "status": "pending", "blockedBy": [96, 97, 98, 99, 100]} + {"id": 96, "ref": "NJ-1", "subject": "Codec — native-typed Encode + read-both Decode", "class": "high-risk", "status": "completed", "commits": ["180d554", "bf80ca1"]}, + {"id": 97, "ref": "NJ-2", "subject": "Stamp ElementDataType on instance-override writes (folds in #93/M3)", "class": "small", "status": "completed", "commits": ["abe8832"]}, + {"id": 98, "ref": "NJ-3", "subject": "Central MS SQL idempotent startup normalizer", "class": "high-risk", "status": "completed", "commits": ["f4b101b", "feeae13"]}, + {"id": 99, "ref": "NJ-4", "subject": "Site SQLite active normalization on override load", "class": "high-risk", "status": "completed", "commits": ["5841cec", "feeae13"]}, + {"id": 100, "ref": "NJ-5", "subject": "Normalize List values on bundle import", "class": "standard", "status": "completed", "commits": ["e3d804a", "feeae13"]}, + {"id": 101, "ref": "NJ-6", "subject": "Integration verification + docs", "class": "standard", "status": "completed"} ], - "lastUpdated": "2026-06-16" + "lastUpdated": "2026-06-16", + "note": "feature/native-typed-json was reset off these commits during concurrent other-window git work and restored to f4b101b (the NJ-1..NJ-5 tip) via reflog; NJ-3/4/5 review fixes committed as feeae13 + NJ-6 docs on top." } diff --git a/docs/requirements/Component-Commons.md b/docs/requirements/Component-Commons.md index 87853db4..929b31b7 100644 --- a/docs/requirements/Component-Commons.md +++ b/docs/requirements/Component-Commons.md @@ -26,7 +26,7 @@ Referenced by all component libraries and the Host. Commons must define shared primitive and utility types used across multiple components, including but not limited to: -- **`DataType` enum**: Enumerates the data types supported by the system: Boolean, Int32, Float, Double, String, DateTime, Binary, and **List** (a homogeneous multi-value attribute). A `List` attribute carries a companion `ElementDataType` (one of the scalar types — String, Int32, Float, Double, Boolean, DateTime; not Binary, not nested List) on `TemplateAttribute` / `InstanceAttributeOverride` and the flattened `ResolvedAttribute`. List values are stored and transmitted as a canonical JSON array via the `AttributeValueCodec` helper (scalars keep their historical invariant-culture string form); the `ElementDataType` is fixed by the defining level and cannot be changed on a derived template or instance override. +- **`DataType` enum**: Enumerates the data types supported by the system: Boolean, Int32, Float, Double, String, DateTime, Binary, and **List** (a homogeneous multi-value attribute). A `List` attribute carries a companion `ElementDataType` (one of the scalar types — String, Int32, Float, Double, Boolean, DateTime; not Binary, not nested List) on `TemplateAttribute` / `InstanceAttributeOverride` and the flattened `ResolvedAttribute`. List values are stored and transmitted as a canonical **native-typed** JSON array via the `AttributeValueCodec` helper — numbers and booleans unquoted, strings quoted, `DateTime` as ISO-8601 (e.g. `[10,20]`, `[true,false]`, `["2026-06-16T00:00:00Z"]`); scalars keep their historical invariant-culture string form. `AttributeValueCodec.Decode` is backward compatible: it reads both the native form and the earlier array-of-strings form (`["10","20"]`). Already-persisted old-form values are normalized to native on the fly — an idempotent central startup normalizer (`ListValueNormalizer`, run after migrations) rewrites MS SQL rows, the `InstanceActor` re-persists site SQLite overrides on load, and bundle import re-encodes List values. The `ElementDataType` is fixed by the defining level and cannot be changed on a derived template or instance override. - **`RetryPolicy`**: A record or immutable class describing retry behavior (max retries, fixed delay between retries). - **`Result`**: A discriminated result type that represents either a success value or an error, enabling consistent error handling across component boundaries without exceptions. - **`InstanceState` enum**: Enabled, Disabled. From dc9f31537af9244892676cbb40c22b5f216840bc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 16 Jun 2026 18:34:34 -0400 Subject: [PATCH 11/11] docs: record final-review follow-ups (deployed-snapshot normalization gap I-1; CLI native-form help example) --- docs/plans/2026-06-16-native-typed-json-design.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/plans/2026-06-16-native-typed-json-design.md b/docs/plans/2026-06-16-native-typed-json-design.md index edc9a9e3..8d39fb0e 100644 --- a/docs/plans/2026-06-16-native-typed-json-design.md +++ b/docs/plans/2026-06-16-native-typed-json-design.md @@ -60,5 +60,19 @@ Already-exported `.bundle` files are external artifacts we cannot reach to rewri ## Out of scope / follow-ups +- **Deployed-config snapshot is a fourth, un-normalized List-value store (latent gap, I-1).** + `DeployedConfigSnapshot.ConfigurationJson` + `RevisionHash` freeze the flattened config at + deploy time. The staleness/diff path (`DeploymentService.GetDeploymentComparisonAsync` → + `DiffService.AttributesEqual` ordinal compare + `RevisionHashService` SHA over the raw + `Value`) compares that frozen blob against a freshly-flattened (now native-form) config. If a + List attribute was ever *deployed* in old-form, the snapshot stays old-form → a spurious + "Changed" diff + false staleness flag until redeployed. This **cannot fire against current + data** (no List attributes were ever deployed — see the Reality Note), so it is recorded as a + known latent gap, not fixed. If hardening is wanted before List attributes are deployed at + scale: route the deserialized snapshot's List values through `Decode→Encode` in + `GetDeploymentComparisonAsync` before the diff/hash (symmetric with the other normalizers). +- CLI `template attribute` help still illustrates `--value` with a quoted string-list example; + add a native-form numeric example (e.g. `[10,20]`) so users don't hand-author quoted numbers + that get silently re-normalized. Doc-only; the quoted form still decodes. - Rewriting already-exported bundle files (unreachable). - This pairs naturally with follow-up **#93/M3** (populate `InstanceAttributeOverride.ElementDataType` on write); if done, the override normalizer could read the column directly instead of joining to the template attribute. Not required here.