Merge feature/native-typed-json: native-typed JSON for List attribute values + data normalization
List values now encode as native-typed JSON ([10,20], [true,false], ISO dates; strings stay quoted) via AttributeValueCodec; Decode reads both native and the earlier array-of-strings form for every element type. Already-persisted old-form data is normalized on the fly: idempotent central startup normalizer (ListValueNormalizer), active site-SQLite normalization on InstanceActor override-load, and normalize-on-import in the bundle importer. Instance-override writes now stamp ElementDataType (#93/M3). Full solution 0/0; feature-targeted tests green. Plan: docs/plans/2026-06-16-native-typed-json.md.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,78 @@
|
||||
# Native-Typed JSON for List Attribute Values — Design
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Status:** Implemented (NJ-1 … NJ-6, branch `feature/native-typed-json`) — full solution builds 0/0; feature-targeted tests green across Commons, TemplateEngine, ConfigurationDatabase, SiteRuntime, and Transport. Follow-up **#93/M3** (populate `InstanceAttributeOverride.ElementDataType` on write) was folded into NJ-2 so the central normalizer can read the override element type directly.
|
||||
**Branch:** `feature/native-typed-json`
|
||||
|
||||
## Problem
|
||||
|
||||
The multi-value (List) attribute feature (shipped 2026-06-16, branch `feature/multivalue-attribute`) stores List values via `AttributeValueCodec` as a JSON **array of strings** — e.g. an `Int32` list is `["10","20","30"]` and a `Boolean` list is `["True","False"]`. This is internally consistent and round-trips, but it is not "native-typed" JSON: numbers and booleans are quoted, and `DateTime` uses a US-invariant format rather than ISO-8601. We want the canonical form to be native-typed (`[10,20,30]`, `[true,false]`, ISO dates), while existing persisted data is normalized to the new form (no dual-format data left behind).
|
||||
|
||||
## Decisions
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Encode form | Native-typed JSON: numbers/bools unquoted, strings quoted, `DateTime` as ISO-8601 string |
|
||||
| Decode | **Read both** old (array-of-strings) and new (native) forms — backward compatible |
|
||||
| Existing data | **Migrate** to native form across MS SQL + site SQLite + on bundle import (Approach B, thorough) |
|
||||
| MS SQL mechanism | Idempotent C# **startup normalizer** (not T-SQL — type-aware JSON re-emission is fragile in SQL) |
|
||||
| Site SQLite mechanism | **Active** normalization in the InstanceActor override-load path (it already has the element type) |
|
||||
| Bundles | Normalize **on import** (already-exported files are external/unreachable) |
|
||||
|
||||
**Reality note:** the List feature shipped this session and was not deployed to the docker cluster, so there is almost certainly **zero** old-form List data in any store yet. The migration is a safety net guaranteeing no dual-format data ever lingers, not a fix for existing broken data.
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Codec (`AttributeValueCodec`, Commons) — foundation
|
||||
|
||||
- **`Encode`** — only the list branch changes. Instead of mapping each element to an invariant string then serializing, serialize the typed CLR collection directly (`JsonSerializer.Serialize<object>(enumerable, …)`) so `System.Text.Json` emits each element in its native JSON kind. STJ numbers/bools are culture-invariant by spec; `DateTime` serializes as round-trippable ISO-8601. Scalars (the `string` / `IFormattable` branches) are untouched.
|
||||
- **`Decode`** — read both forms. Deserialize to `JsonElement[]` (instead of `string?[]`); for each element feed `ParseScalar` either `GetString()` (JSON string element) or `GetRawText()` (number/bool element). So `[10,20]` and `["10","20"]` both decode to `List<int>{10,20}`; ISO and old US-invariant `DateTime` strings both parse via the existing `DateTime.Parse(…, RoundtripKind)`. A JSON `null` element still throws `FormatException` ("elements may not be null"), unchanged.
|
||||
- The read-both Decode is also what makes the migration idempotent: re-encoding an already-native value yields identical bytes.
|
||||
|
||||
### 2. MS SQL — idempotent central startup normalizer
|
||||
|
||||
A normalization step invoked once after `dbContext.Database.MigrateAsync(...)` in `MigrationHelper.ApplyOrValidateMigrationsAsync` (active central node only). For each List row:
|
||||
|
||||
- **`TemplateAttributes`** where `DataType = 'List'`: read `Value` + the row's own `ElementDataType`; compute `Encode(Decode(value, List, elementType))`; if it differs from the stored string, `UPDATE`.
|
||||
- **`InstanceAttributeOverrides`** for List attributes: these rows may have a null `ElementDataType` (it is currently informational — see follow-up #93/M3), so resolve the element type via the owning instance's template attribute (instance → `TemplateId` → `TemplateAttribute` by name → `ElementDataType`). Then re-encode as above.
|
||||
|
||||
Idempotent (native→native is a no-op `UPDATE`-skip), so the step is safe to leave in permanently and cheap on every subsequent startup (a scan, no writes). Per-row failures (malformed JSON, unresolved element type) are logged and skipped — normalization NEVER aborts startup (mirrors the audit/best-effort principle). The scan is bounded to List rows only.
|
||||
|
||||
### 3. Site SQLite — active normalization on override load
|
||||
|
||||
Site static-override values (`SiteStorageService`) are keyed by `(instance, canonicalName)` and carry no element type — the element type lives in the instance's flattened config. The natural normalization point is therefore the **InstanceActor override-load path** (`HandleOverridesLoaded`, added in MV-7), which already decodes both forms using the `ResolvedAttribute`'s `ElementDataType`. Extend it so that when a List override's stored string is in old form (i.e. `Encode(decoded)` differs from the stored string), it re-persists the native form via `SiteStorageService.SetStaticOverrideAsync`. This actively normalizes on every instance load (site startup / failover), reuses the existing decode + element type, and is idempotent.
|
||||
|
||||
### 4. Bundles — normalize on import
|
||||
|
||||
Already-exported `.bundle` files are external artifacts we cannot reach to rewrite; import already reads both forms via the codec. To ensure imported List values land in native form in the DB, the importer re-encodes List attribute `Value`s through the codec when writing (and the MS SQL normalizer is a backstop on next startup). No file rewriting.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Decode of a genuinely malformed value still throws `FormatException`; the normalizers catch it per-row, log, and skip (no startup abort, no actor crash).
|
||||
- The codec change is additive on the wire (`gRPC` `string value` field unchanged; `List` is a new type with no external wire consumer relying on the old quoted form).
|
||||
|
||||
## Testing
|
||||
|
||||
- **Codec:** native-form encode per element type (`[10,20,30]`, `[true,false]`, ISO `DateTime`, strings stay `["a","b"]`); old-form backward-compat decode (`["10","20"]` → `List<int>`); round-trip for every element type; malformed still throws; culture-invariance preserved.
|
||||
- **MS SQL normalizer:** old-form row → rewritten to native; native row → untouched (idempotency); malformed row → skipped + logged, other rows still processed; override row element-type resolved via template attribute.
|
||||
- **Site SQLite / InstanceActor:** an old-form List override on load → re-persisted native (assert `SetStaticOverrideAsync` called with native form); a native override → not re-persisted (idempotent); scalar overrides unaffected.
|
||||
- **Bundle import:** importing an old-form bundle lands native-form Values in the DB.
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- **Deployed-config snapshot is a fourth, un-normalized List-value store (latent gap, I-1).**
|
||||
`DeployedConfigSnapshot.ConfigurationJson` + `RevisionHash` freeze the flattened config at
|
||||
deploy time. The staleness/diff path (`DeploymentService.GetDeploymentComparisonAsync` →
|
||||
`DiffService.AttributesEqual` ordinal compare + `RevisionHashService` SHA over the raw
|
||||
`Value`) compares that frozen blob against a freshly-flattened (now native-form) config. If a
|
||||
List attribute was ever *deployed* in old-form, the snapshot stays old-form → a spurious
|
||||
"Changed" diff + false staleness flag until redeployed. This **cannot fire against current
|
||||
data** (no List attributes were ever deployed — see the Reality Note), so it is recorded as a
|
||||
known latent gap, not fixed. If hardening is wanted before List attributes are deployed at
|
||||
scale: route the deserialized snapshot's List values through `Decode→Encode` in
|
||||
`GetDeploymentComparisonAsync` before the diff/hash (symmetric with the other normalizers).
|
||||
- CLI `template attribute` help still illustrates `--value` with a quoted string-list example;
|
||||
add a native-form numeric example (e.g. `[10,20]`) so users don't hand-author quoted numbers
|
||||
that get silently re-normalized. Doc-only; the quoted form still decodes.
|
||||
- Rewriting already-exported bundle files (unreachable).
|
||||
- This pairs naturally with follow-up **#93/M3** (populate `InstanceAttributeOverride.ElementDataType` on write); if done, the override normalizer could read the column directly instead of joining to the template attribute. Not required here.
|
||||
@@ -0,0 +1,282 @@
|
||||
# Native-Typed JSON for List Attributes — Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Make `AttributeValueCodec` encode List values as native-typed JSON (`[10,20]`, `[true,false]`, ISO dates) while decoding both old (array-of-strings) and new forms, and normalize already-persisted data to the native form across MS SQL, site SQLite, and bundle import.
|
||||
|
||||
**Architecture:** Native `Encode` + read-both `Decode` in the codec is the foundation. An idempotent central MS SQL startup normalizer rewrites old-form rows; the InstanceActor re-persists native form on override-load (site SQLite); the bundle importer re-encodes on import. To give the central normalizer a reliable element type for instance-override rows, the override-write path now stamps `ElementDataType` (folds in follow-up #93/M3).
|
||||
|
||||
**Tech Stack:** C#/.NET 10, System.Text.Json, EF Core 10 (MS SQL + SQLite), Akka.NET, Transport bundles.
|
||||
|
||||
**Design doc:** `docs/plans/2026-06-16-native-typed-json-design.md` (approved).
|
||||
**Branch:** `feature/native-typed-json` (off main; design committed `91b1aa1`).
|
||||
|
||||
**Conventions for every task:** TDD (failing test → fail → implement → pass → commit). Targeted builds/tests only — `dotnet build src/<Project>/<Project>.csproj`, `dotnet test tests/<TestProject>/<TestProject>.csproj --filter <Name>`. Full-solution build only in the final task. Do NOT create git worktrees. Commit with pathspec form (`git commit -m "…" -- <paths>`) and retry on `.git/index.lock` when implementers run concurrently.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-1: Codec — native-typed Encode + read-both Decode
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** NJ-2
|
||||
**Blocked by:** none (foundation)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs`
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs`
|
||||
|
||||
This changes the canonical on-disk/on-wire form for List values. It is backward compatible (Decode reads both forms) and additive on the gRPC `string value` field (List is a new type with no external consumer of the old quoted form).
|
||||
|
||||
**Step 1 — Update/extend tests (TDD).** Update the existing `Encode_DoubleList_IsInvariant` assertion to the native form, and add the native + backward-compat cases:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Encode_Int32List_ProducesNativeNumbers() =>
|
||||
Assert.Equal("[10,20,30]", AttributeValueCodec.Encode(new List<int> { 10, 20, 30 }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_BoolList_ProducesNativeBooleans() =>
|
||||
Assert.Equal("[true,false]", AttributeValueCodec.Encode(new List<bool> { true, false }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_DoubleList_IsNativeAndInvariant()
|
||||
{
|
||||
var original = CultureInfo.CurrentCulture;
|
||||
try
|
||||
{
|
||||
CultureInfo.CurrentCulture = new CultureInfo("de-DE");
|
||||
Assert.Equal("[1.5,2.5]", AttributeValueCodec.Encode(new List<double> { 1.5, 2.5 }));
|
||||
}
|
||||
finally { CultureInfo.CurrentCulture = original; }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Encode_StringList_StaysQuoted() =>
|
||||
Assert.Equal("[\"a\",\"b\"]", AttributeValueCodec.Encode(new List<string> { "a", "b" }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_DateTimeList_IsIso8601()
|
||||
{
|
||||
var json = AttributeValueCodec.Encode(new List<DateTime> { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) });
|
||||
Assert.Contains("2026-06-16T00:00:00", json); // ISO, not "06/16/2026 00:00:00"
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_NewNativeIntForm_Parses()
|
||||
{
|
||||
var back = (IList<int>)AttributeValueCodec.Decode("[10,20]", DataType.List, DataType.Int32)!;
|
||||
Assert.Equal(new[] { 10, 20 }, back);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_OldStringIntForm_BackwardCompatible()
|
||||
{
|
||||
var back = (IList<int>)AttributeValueCodec.Decode("[\"10\",\"20\"]", DataType.List, DataType.Int32)!;
|
||||
Assert.Equal(new[] { 10, 20 }, back);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("[true,false]")] // new native bools
|
||||
[InlineData("[\"True\",\"False\"]")] // old string bools
|
||||
public void Decode_BoolForms_BothParse(string json)
|
||||
{
|
||||
var back = (IList<bool>)AttributeValueCodec.Decode(json, DataType.List, DataType.Boolean)!;
|
||||
Assert.Equal(new[] { true, false }, back);
|
||||
}
|
||||
```
|
||||
Keep the existing round-trip tests (Int32/Float/Double/DateTime/Bool) — they assert round-trip, not exact form, so they continue to pass. Keep `Decode_MalformedJson_Throws`.
|
||||
|
||||
**Step 2 — Run, expect FAIL** on the new native-form encode assertions.
|
||||
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.csproj --filter AttributeValueCodecTests`
|
||||
|
||||
**Step 3 — Implement.** In `Encode`, replace ONLY the `IEnumerable` branch:
|
||||
|
||||
```csharp
|
||||
case IEnumerable e:
|
||||
// Native-typed JSON: serialize the runtime collection so STJ emits numbers/bools
|
||||
// unquoted, strings quoted, DateTime as ISO-8601. Boxed as object so STJ uses the
|
||||
// runtime element type. STJ numbers/dates are culture-invariant by spec.
|
||||
return JsonSerializer.Serialize<object>(e, JsonOpts);
|
||||
```
|
||||
(The `null`, `string`, and `IFormattable` scalar branches are unchanged.)
|
||||
|
||||
In `Decode`, change the list deserialization to read `JsonElement[]` and derive a string per element:
|
||||
|
||||
```csharp
|
||||
JsonElement[] raw;
|
||||
try { raw = JsonSerializer.Deserialize<JsonElement[]>(value) ?? []; }
|
||||
catch (JsonException ex) { throw new FormatException("Malformed list JSON.", ex); }
|
||||
|
||||
var clrType = ElementClrType(elementType.Value);
|
||||
var listType = typeof(List<>).MakeGenericType(clrType);
|
||||
var result = (IList)Activator.CreateInstance(listType)!;
|
||||
foreach (var el in raw)
|
||||
result.Add(ParseScalar(JsonElementToString(el), elementType.Value));
|
||||
return result;
|
||||
```
|
||||
Add the helper (handles both new native elements and old quoted-string elements):
|
||||
|
||||
```csharp
|
||||
private static string? JsonElementToString(JsonElement el) => el.ValueKind switch
|
||||
{
|
||||
JsonValueKind.String => el.GetString(), // old form, or string-typed lists
|
||||
JsonValueKind.Null => null, // ParseScalar throws "may not be null"
|
||||
_ => el.GetRawText() // number/bool → "10" / "1.5" / "true"
|
||||
};
|
||||
```
|
||||
`ParseScalar` is unchanged: it parses the string per element type (invariant), so `"10"` (from either form), `"true"`, and ISO/US-invariant dates all parse. Add `using System.Text.Json;` if not already present (it is).
|
||||
|
||||
**Step 4 — Run, expect PASS** (all AttributeValueCodec tests).
|
||||
|
||||
**Step 5 — Commit.**
|
||||
```bash
|
||||
git commit -m "feat(commons): native-typed JSON for List values; Decode reads both forms" -- \
|
||||
src/ZB.MOM.WW.ScadaBridge.Commons/Types/AttributeValueCodec.cs \
|
||||
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AttributeValueCodecTests.cs
|
||||
```
|
||||
|
||||
**Acceptance:** numbers/bools encode unquoted, strings stay quoted, DateTime is ISO; both old and new forms decode; scalars unchanged; malformed still throws.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-2: Stamp ElementDataType on instance-override writes
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** NJ-1
|
||||
**Blocked by:** none
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Services/InstanceService.cs:146-198` (`SetAttributeOverrideAsync`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Services/InstanceServiceTests.cs` (mirror existing style)
|
||||
|
||||
`SetAttributeOverrideAsync` already loads the base `templateAttr` (line 159). Set `ElementDataType` from it on both the update and create branches so the persisted `InstanceAttributeOverride` row carries the element type — this is what the NJ-3 central normalizer reads for override rows (folds in follow-up #93/M3).
|
||||
|
||||
**Step 1 — Failing test:** setting an override on a `List`/`String` template attribute persists an `InstanceAttributeOverride` whose `ElementDataType == DataType.String`; setting an override on a scalar attribute persists `ElementDataType == null`.
|
||||
|
||||
**Step 2 — Run, expect FAIL.**
|
||||
|
||||
**Step 3 — Implement:** in the update branch, `existingOverride.ElementDataType = templateAttr.ElementDataType;` before the repo update; in the create branch, set `ElementDataType = templateAttr.ElementDataType` in the `new InstanceAttributeOverride(...)` initializer.
|
||||
|
||||
**Step 4 — Run, expect PASS.**
|
||||
|
||||
**Step 5 — Commit:** `feat(template): stamp ElementDataType on instance attribute overrides`.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-3: Central MS SQL idempotent startup normalizer
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** NJ-4, NJ-5 (after NJ-1)
|
||||
**Blocked by:** NJ-1, NJ-2
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ListValueNormalizer.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/MigrationHelper.cs` (call the normalizer after the migrate/validate step)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/...` (use the existing test DbContext/SQLite-in-memory or test-double pattern the project uses — investigate the existing ConfigurationDatabase tests first)
|
||||
|
||||
**Behavior:** A static `ListValueNormalizer.NormalizeAsync(ScadaBridgeDbContext db, ILogger? logger, CancellationToken ct)`:
|
||||
- `TemplateAttributes` where `DataType == DataType.List`: for each, compute `var native = AttributeValueCodec.Encode(AttributeValueCodec.Decode(a.Value, DataType.List, a.ElementDataType));` (parse the row's own `ElementDataType` — it is a `DataType?`). If `native != a.Value`, set `a.Value = native`.
|
||||
- `InstanceAttributeOverrides` whose `ElementDataType != null` (i.e. a List override stamped by NJ-2): same re-encode against `OverrideValue`. Rows with null `ElementDataType` are skipped with a debug log (legacy/scalar rows — none exist in practice since the feature has no deployed data).
|
||||
- Wrap each row in try/catch(`FormatException`): on a malformed value, log a warning and SKIP that row — continue with the rest. The normalizer MUST NOT throw out of `NormalizeAsync` (never abort startup).
|
||||
- `SaveChangesAsync` once at the end. Log a summary (`Normalized {n} list attribute value(s) to native JSON`).
|
||||
- Idempotent: native→native yields identical bytes, so a second run makes no changes.
|
||||
|
||||
**Wire-in (MigrationHelper):** after the `if (isDevelopment) { Migrate } else { validate }` block in `ApplyOrValidateMigrationsAsync`, call `await ListValueNormalizer.NormalizeAsync(dbContext, logger, cancellationToken);`. (Runs on every central startup; idempotent and safe even if both central nodes run it — concurrent UPDATEs set identical values.)
|
||||
|
||||
**Steps (TDD):** failing tests (a TemplateAttributes row in old form `["10","20"]` with ElementDataType Int32 → rewritten to `[10,20]`; a native row → unchanged/no-op; a String-list `["a","b"]` → unchanged; a malformed row → skipped, sibling rows still normalized; an override row with ElementDataType set → normalized, with null → skipped) → run targeted → FAIL → implement `ListValueNormalizer` + wire-in → run targeted → PASS → build ConfigurationDatabase → commit `feat(db): idempotent startup normalizer rewriting List values to native JSON`.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-4: Site SQLite active normalization on override load
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** NJ-3, NJ-5 (after NJ-1)
|
||||
**Blocked by:** NJ-1
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:932-960` (`HandleOverridesLoaded`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs`
|
||||
|
||||
In `HandleOverridesLoaded`, the List branch already decodes the stored override via `DecodeAttributeValue(resolved, kvp.Value)`. Extend it: when the decode succeeds and the re-encoded native form differs from the stored string (i.e. the stored value is old-form), re-persist the native form:
|
||||
|
||||
```csharp
|
||||
var decoded = DecodeAttributeValue(resolved, kvp.Value);
|
||||
_attributes[kvp.Key] = decoded;
|
||||
if (decoded is null && !string.IsNullOrEmpty(kvp.Value))
|
||||
{
|
||||
_attributeQualities[kvp.Key] = "Bad";
|
||||
}
|
||||
else if (decoded is not null)
|
||||
{
|
||||
var native = AttributeValueCodec.Encode(decoded);
|
||||
if (native != kvp.Value) // stored value was old-form → normalize on disk
|
||||
{
|
||||
var key = kvp.Key;
|
||||
_storage.SetStaticOverrideAsync(_instanceUniqueName, key, native!)
|
||||
.ContinueWith(t => _logger.LogWarning(t.Exception?.GetBaseException(),
|
||||
"Failed to normalize static override {Instance}.{Attr} to native JSON", _instanceUniqueName, key),
|
||||
TaskContinuationOptions.OnlyOnFaulted);
|
||||
}
|
||||
}
|
||||
```
|
||||
Idempotent (native→native ⇒ `native == kvp.Value` ⇒ no write). Scalars untouched. Never throws into the actor (decode already FormatException-safe; persist is fire-and-forget).
|
||||
|
||||
**Steps (TDD):** failing tests (an old-form List static override on load → `SetStaticOverrideAsync` called with the native form; a native-form override → NOT re-persisted; a scalar override → unaffected; malformed → Bad quality + no crash, as today) → run targeted `--filter "FullyQualifiedName~InstanceActor"` → FAIL → implement → PASS → build SiteRuntime → commit `feat(siteruntime): normalize old-form List static overrides to native JSON on load`.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-5: Normalize List values on bundle import
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** NJ-3, NJ-4 (after NJ-1)
|
||||
**Blocked by:** NJ-1
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (the 3 `TemplateAttribute` construction sites from MV-12: BuildTemplate + the two in SyncTemplateAttributesAsync) — and/or `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` `FromBundleContent` if that is the single choke point for DTO→entity.
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/...`
|
||||
|
||||
**Behavior:** when importing a `TemplateAttributeDto` whose `DataType == DataType.List`, write the **native** Value: `var v = NormalizeListValue(dto.Value, dto.ElementDataType);` where `NormalizeListValue` does `AttributeValueCodec.Encode(AttributeValueCodec.Decode(value, DataType.List, elementType))` inside try/catch(FormatException) → on failure return the original value unchanged + log (don't fail the import; the MS SQL normalizer is a backstop). Prefer a single private helper used at all construction sites (DRY). Non-List attributes write `dto.Value` unchanged.
|
||||
|
||||
**Steps (TDD):** failing test (import a bundle/DTO with an old-form List attribute `["10","20"]` + ElementDataType Int32 → the persisted TemplateAttribute.Value is `[10,20]`; a String list stays quoted; a malformed value imports unchanged) → run targeted → FAIL → implement the helper + apply at construction sites → PASS → build Transport → commit `feat(transport): normalize List attribute values to native JSON on import`.
|
||||
|
||||
---
|
||||
|
||||
### Task NJ-6: Integration verification + docs
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** NJ-1 … NJ-5
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/plans/2026-06-16-native-typed-json-design.md` (mark Status complete)
|
||||
- Modify: `docs/requirements/Component-Commons.md` (update the `DataType`/codec note: List values are native-typed JSON; codec reads both forms)
|
||||
- Modify: `docs/plans/2026-06-16-multivalue-attribute.md.tasks.json` follow-up note OR the task list — note that #93/M3 (override ElementDataType) was folded in here.
|
||||
|
||||
**Steps:**
|
||||
1. Full-solution build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → 0/0.
|
||||
2. Targeted test sweep across touched projects: Commons codec, TemplateEngine (InstanceService), ConfigurationDatabase (normalizer), SiteRuntime (InstanceActor), Transport (serializer/importer).
|
||||
3. Update docs; mark design Status complete.
|
||||
4. Commit: `docs: mark native-typed JSON feature complete; update Component-Commons codec note`.
|
||||
|
||||
**Acceptance:** full build green; all touched-project targeted tests pass; docs synced.
|
||||
|
||||
---
|
||||
|
||||
## Parallelization summary
|
||||
|
||||
- **Wave 1:** NJ-1 (codec) ∥ NJ-2 (override stamp) — disjoint (Commons vs TemplateEngine).
|
||||
- **Wave 2 (after NJ-1, +NJ-2 for NJ-3):** NJ-3 (normalizer), NJ-4 (site), NJ-5 (import) in parallel — disjoint (ConfigurationDatabase vs SiteRuntime vs Transport).
|
||||
- **Wave 3:** NJ-6 (verify + docs).
|
||||
|
||||
## Risk notes
|
||||
|
||||
- **NJ-1** changes the canonical storage/wire form — high-risk, but backward-compatible (read-both) and additive on the proto field.
|
||||
- **NJ-3** writes to the config DB at startup — must be idempotent and never abort startup (per-row skip+log).
|
||||
- **NJ-4** touches the actor model — the normalization write is fire-and-forget and never throws into the actor.
|
||||
- There is no deployed List data yet, so the normalizers are a safety net; correctness of NJ-1 (the format) is what matters most in practice.
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-06-16-native-typed-json.md",
|
||||
"designDoc": "docs/plans/2026-06-16-native-typed-json-design.md",
|
||||
"branch": "feature/native-typed-json",
|
||||
"status": "complete",
|
||||
"tasks": [
|
||||
{"id": 96, "ref": "NJ-1", "subject": "Codec — native-typed Encode + read-both Decode", "class": "high-risk", "status": "completed", "commits": ["180d554", "bf80ca1"]},
|
||||
{"id": 97, "ref": "NJ-2", "subject": "Stamp ElementDataType on instance-override writes (folds in #93/M3)", "class": "small", "status": "completed", "commits": ["abe8832"]},
|
||||
{"id": 98, "ref": "NJ-3", "subject": "Central MS SQL idempotent startup normalizer", "class": "high-risk", "status": "completed", "commits": ["f4b101b", "feeae13"]},
|
||||
{"id": 99, "ref": "NJ-4", "subject": "Site SQLite active normalization on override load", "class": "high-risk", "status": "completed", "commits": ["5841cec", "feeae13"]},
|
||||
{"id": 100, "ref": "NJ-5", "subject": "Normalize List values on bundle import", "class": "standard", "status": "completed", "commits": ["e3d804a", "feeae13"]},
|
||||
{"id": 101, "ref": "NJ-6", "subject": "Integration verification + docs", "class": "standard", "status": "completed"}
|
||||
],
|
||||
"lastUpdated": "2026-06-16",
|
||||
"note": "feature/native-typed-json was reset off these commits during concurrent other-window git work and restored to f4b101b (the NJ-1..NJ-5 tip) via reflog; NJ-3/4/5 review fixes committed as feeae13 + NJ-6 docs on top."
|
||||
}
|
||||
@@ -26,7 +26,7 @@ Referenced by all component libraries and the Host.
|
||||
|
||||
Commons must define shared primitive and utility types used across multiple components, including but not limited to:
|
||||
|
||||
- **`DataType` enum**: Enumerates the data types supported by the system: Boolean, Int32, Float, Double, String, DateTime, Binary, and **List** (a homogeneous multi-value attribute). A `List` attribute carries a companion `ElementDataType` (one of the scalar types — String, Int32, Float, Double, Boolean, DateTime; not Binary, not nested List) on `TemplateAttribute` / `InstanceAttributeOverride` and the flattened `ResolvedAttribute`. List values are stored and transmitted as a canonical JSON array via the `AttributeValueCodec` helper (scalars keep their historical invariant-culture string form); the `ElementDataType` is fixed by the defining level and cannot be changed on a derived template or instance override.
|
||||
- **`DataType` enum**: Enumerates the data types supported by the system: Boolean, Int32, Float, Double, String, DateTime, Binary, and **List** (a homogeneous multi-value attribute). A `List` attribute carries a companion `ElementDataType` (one of the scalar types — String, Int32, Float, Double, Boolean, DateTime; not Binary, not nested List) on `TemplateAttribute` / `InstanceAttributeOverride` and the flattened `ResolvedAttribute`. List values are stored and transmitted as a canonical **native-typed** JSON array via the `AttributeValueCodec` helper — numbers and booleans unquoted, strings quoted, `DateTime` as ISO-8601 (e.g. `[10,20]`, `[true,false]`, `["2026-06-16T00:00:00Z"]`); scalars keep their historical invariant-culture string form. `AttributeValueCodec.Decode` is backward compatible: it reads both the native form and the earlier array-of-strings form (`["10","20"]`). Already-persisted old-form values are normalized to native on the fly — an idempotent central startup normalizer (`ListValueNormalizer`, run after migrations) rewrites MS SQL rows, the `InstanceActor` re-persists site SQLite overrides on load, and bundle import re-encodes List values. The `ElementDataType` is fixed by the defining level and cannot be changed on a derived template or instance override.
|
||||
- **`RetryPolicy`**: A record or immutable class describing retry behavior (max retries, fixed delay between retries).
|
||||
- **`Result<T>`**: A discriminated result type that represents either a success value or an error, enabling consistent error handling across component boundaries without exceptions.
|
||||
- **`InstanceState` enum**: Enabled, Disabled.
|
||||
|
||||
@@ -25,11 +25,10 @@ public static class AttributeValueCodec
|
||||
case string s: return s; // already canonical
|
||||
case IFormattable f: return f.ToString(null, CultureInfo.InvariantCulture);
|
||||
case IEnumerable e:
|
||||
var items = e.Cast<object?>()
|
||||
.Select(x => x is IFormattable xf
|
||||
? xf.ToString(null, CultureInfo.InvariantCulture)
|
||||
: x?.ToString());
|
||||
return JsonSerializer.Serialize(items, JsonOpts);
|
||||
// Native-typed JSON: serialize the runtime collection so System.Text.Json emits
|
||||
// numbers/bools unquoted, strings quoted, and DateTime as ISO-8601. Boxed as object
|
||||
// so STJ uses the runtime element type. STJ numbers/dates are culture-invariant.
|
||||
return JsonSerializer.Serialize<object>(e, JsonOpts);
|
||||
default: return value.ToString();
|
||||
}
|
||||
}
|
||||
@@ -46,18 +45,25 @@ public static class AttributeValueCodec
|
||||
if (elementType is null)
|
||||
throw new FormatException("List attribute requires an element type.");
|
||||
|
||||
string?[] raw;
|
||||
try { raw = JsonSerializer.Deserialize<string?[]>(value) ?? []; }
|
||||
JsonElement[] raw;
|
||||
try { raw = JsonSerializer.Deserialize<JsonElement[]>(value) ?? []; }
|
||||
catch (JsonException ex) { throw new FormatException("Malformed list JSON.", ex); }
|
||||
|
||||
var clrType = ElementClrType(elementType.Value);
|
||||
var listType = typeof(List<>).MakeGenericType(clrType);
|
||||
var result = (IList)Activator.CreateInstance(listType)!;
|
||||
foreach (var item in raw)
|
||||
result.Add(ParseScalar(item, elementType.Value));
|
||||
foreach (var el in raw)
|
||||
result.Add(ParseScalar(JsonElementToString(el), elementType.Value));
|
||||
return result;
|
||||
}
|
||||
|
||||
private static string? JsonElementToString(JsonElement el) => el.ValueKind switch
|
||||
{
|
||||
JsonValueKind.String => el.GetString(), // old form, or string-typed lists
|
||||
JsonValueKind.Null => null, // ParseScalar throws "may not be null"
|
||||
_ => el.GetRawText() // number/bool → "10" / "1.5" / "true"
|
||||
};
|
||||
|
||||
private static Type ElementClrType(DataType t) => t switch
|
||||
{
|
||||
DataType.String => typeof(string),
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
|
||||
|
||||
/// <summary>
|
||||
/// Idempotent central startup normalizer that rewrites already-persisted List attribute
|
||||
/// values from the old array-of-strings JSON form (<c>["10","20"]</c>) to the new
|
||||
/// native-typed form (<c>[10,20]</c>).
|
||||
///
|
||||
/// <para>
|
||||
/// This is a safety net: at the time of writing there is no deployed List attribute data,
|
||||
/// so in practice it finds nothing to rewrite. It runs on every central startup after
|
||||
/// migrations apply and MUST never abort startup for data reasons — every row's work is
|
||||
/// wrapped in a per-row try/catch that logs and skips on malformed data. A second run
|
||||
/// finds nothing to change (native → native re-encode is byte-identical).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class ListValueNormalizer
|
||||
{
|
||||
/// <summary>
|
||||
/// Rewrites old-form List attribute values to the native-typed JSON form across
|
||||
/// <see cref="ScadaBridgeDbContext.TemplateAttributes"/> and
|
||||
/// <see cref="ScadaBridgeDbContext.InstanceAttributeOverrides"/>. Idempotent and
|
||||
/// best-effort: malformed rows are logged and skipped, never rethrown.
|
||||
/// </summary>
|
||||
/// <param name="db">The configuration database context.</param>
|
||||
/// <param name="logger">Optional logger for diagnostics.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public static async Task NormalizeAsync(
|
||||
ScadaBridgeDbContext db,
|
||||
ILogger? logger = null,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
var rewritten = 0;
|
||||
|
||||
// TemplateAttributes: List rows carry the element type on the row itself.
|
||||
var templateRows = await db.TemplateAttributes
|
||||
.Where(a => a.DataType == DataType.List)
|
||||
.ToListAsync(ct);
|
||||
|
||||
foreach (var a in templateRows)
|
||||
{
|
||||
try
|
||||
{
|
||||
var native = AttributeValueCodec.Encode(
|
||||
AttributeValueCodec.Decode(a.Value, DataType.List, a.ElementDataType));
|
||||
if (native != a.Value)
|
||||
{
|
||||
a.Value = native;
|
||||
rewritten++;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Never abort startup for a single bad row. FormatException from Decode is the
|
||||
// expected case; the broad catch also covers an unexpected serialize failure
|
||||
// (e.g. a JsonException on a non-finite value) so one poison row can't crash boot.
|
||||
logger?.LogWarning(ex,
|
||||
"List value normalizer: skipping unprocessable list value for TemplateAttribute {Id}.",
|
||||
a.Id);
|
||||
}
|
||||
}
|
||||
|
||||
// InstanceAttributeOverrides: only rows that carry an element type are List rows.
|
||||
// Rows with a null ElementDataType are scalar/legacy rows (no deployed List data
|
||||
// exists, so none in practice) and are skipped.
|
||||
var overrideRows = await db.InstanceAttributeOverrides
|
||||
.Where(o => o.ElementDataType != null)
|
||||
.ToListAsync(ct);
|
||||
|
||||
foreach (var o in overrideRows)
|
||||
{
|
||||
try
|
||||
{
|
||||
var native = AttributeValueCodec.Encode(
|
||||
AttributeValueCodec.Decode(o.OverrideValue, DataType.List, o.ElementDataType));
|
||||
if (native != o.OverrideValue)
|
||||
{
|
||||
o.OverrideValue = native;
|
||||
rewritten++;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger?.LogWarning(ex,
|
||||
"List value normalizer: skipping unprocessable list value for InstanceAttributeOverride {Id}.",
|
||||
o.Id);
|
||||
}
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(ct);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A catastrophic DB failure on SaveChanges may propagate, but log it first so
|
||||
// startup diagnostics are not silent. Per-row data problems are already handled
|
||||
// above and never reach here.
|
||||
logger?.LogError(ex, "List value normalizer: SaveChanges failed.");
|
||||
throw;
|
||||
}
|
||||
|
||||
if (rewritten > 0)
|
||||
{
|
||||
logger?.LogInformation(
|
||||
"List value normalizer: rewrote {n} attribute value(s) to native JSON.", rewritten);
|
||||
}
|
||||
else
|
||||
{
|
||||
logger?.LogDebug("List value normalizer: no attribute values required rewriting.");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -47,6 +47,12 @@ public static class MigrationHelper
|
||||
"Apply migrations using 'dotnet ef database update' or the generated SQL scripts before starting in production mode.");
|
||||
}
|
||||
}
|
||||
|
||||
// Safety-net normalizer: rewrite any already-persisted List attribute values from the
|
||||
// old array-of-strings JSON form to the new native-typed form. Idempotent and never
|
||||
// aborts startup for data reasons (per-row skip + log). Safe even if both central
|
||||
// nodes run it concurrently on startup.
|
||||
await ListValueNormalizer.NormalizeAsync(dbContext, logger, cancellationToken);
|
||||
}
|
||||
|
||||
private static async Task WaitForDatabaseReadyAsync(
|
||||
|
||||
@@ -948,10 +948,33 @@ public class InstanceActor : ReceiveActor
|
||||
if (_resolvedAttributeByName.TryGetValue(kvp.Key, out var resolved)
|
||||
&& IsListAttribute(resolved))
|
||||
{
|
||||
// NJ-4: decode the stored List override (both old array-of-strings
|
||||
// and native-typed forms decode) and re-persist the native form if
|
||||
// the stored value is still in the OLD form. Re-encoding the decoded
|
||||
// list and comparing to the stored string detects old-form values
|
||||
// (native → native is byte-identical, so a native value is a no-op).
|
||||
// The re-persist is fire-and-forget and never throws into the actor.
|
||||
var decoded = DecodeAttributeValue(resolved, kvp.Value);
|
||||
_attributes[kvp.Key] = decoded;
|
||||
if (decoded is null && !string.IsNullOrEmpty(kvp.Value))
|
||||
{
|
||||
_attributeQualities[kvp.Key] = "Bad";
|
||||
}
|
||||
else if (decoded is not null)
|
||||
{
|
||||
var native = AttributeValueCodec.Encode(decoded);
|
||||
if (native != kvp.Value) // stored value was old-form → normalize on disk
|
||||
{
|
||||
var key = kvp.Key;
|
||||
var logger = _logger;
|
||||
var instanceName = _instanceUniqueName;
|
||||
_storage.SetStaticOverrideAsync(instanceName, key, native!)
|
||||
.ContinueWith(t => logger.LogWarning(t.Exception?.GetBaseException(),
|
||||
"Failed to normalize static override {Instance}.{Attr} to native JSON",
|
||||
instanceName, key),
|
||||
TaskContinuationOptions.OnlyOnFaulted);
|
||||
}
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@@ -172,6 +172,7 @@ public class InstanceService
|
||||
if (existingOverride != null)
|
||||
{
|
||||
existingOverride.OverrideValue = overrideValue;
|
||||
existingOverride.ElementDataType = templateAttr.ElementDataType;
|
||||
await _repository.UpdateInstanceAttributeOverrideAsync(existingOverride, cancellationToken);
|
||||
await _repository.SaveChangesAsync(cancellationToken);
|
||||
|
||||
@@ -185,7 +186,8 @@ public class InstanceService
|
||||
var newOverride = new InstanceAttributeOverride(attributeName)
|
||||
{
|
||||
InstanceId = instanceId,
|
||||
OverrideValue = overrideValue
|
||||
OverrideValue = overrideValue,
|
||||
ElementDataType = templateAttr.ElementDataType
|
||||
};
|
||||
|
||||
await _repository.AddInstanceAttributeOverrideAsync(newOverride, cancellationToken);
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using System.IO.Compression;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
|
||||
@@ -70,6 +71,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
private readonly IOptions<TransportOptions> _options;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly SemanticValidator _semanticValidator;
|
||||
private readonly ILogger<BundleImporter>? _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleImporter"/> with all required dependencies.
|
||||
@@ -106,7 +108,8 @@ public sealed class BundleImporter : IBundleImporter
|
||||
IAuditService auditService,
|
||||
IAuditCorrelationContext correlationContext,
|
||||
ScadaBridgeDbContext dbContext,
|
||||
SemanticValidator semanticValidator)
|
||||
SemanticValidator semanticValidator,
|
||||
ILogger<BundleImporter>? logger = null)
|
||||
{
|
||||
_bundleSerializer = bundleSerializer ?? throw new ArgumentNullException(nameof(bundleSerializer));
|
||||
_manifestValidator = manifestValidator ?? throw new ArgumentNullException(nameof(manifestValidator));
|
||||
@@ -124,6 +127,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
_correlationContext = correlationContext ?? throw new ArgumentNullException(nameof(correlationContext));
|
||||
_dbContext = dbContext ?? throw new ArgumentNullException(nameof(dbContext));
|
||||
_semanticValidator = semanticValidator ?? throw new ArgumentNullException(nameof(semanticValidator));
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -1040,7 +1044,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
{
|
||||
t.Attributes.Add(new TemplateAttribute(a.Name)
|
||||
{
|
||||
Value = a.Value,
|
||||
Value = ImportValueNormalizer.NormalizeListValue(a.Value, a.DataType, a.ElementDataType),
|
||||
DataType = a.DataType,
|
||||
IsLocked = a.IsLocked,
|
||||
Description = a.Description,
|
||||
@@ -1115,11 +1119,17 @@ public sealed class BundleImporter : IBundleImporter
|
||||
// Adds + Updates.
|
||||
foreach (var attrDto in dto.Attributes)
|
||||
{
|
||||
// Normalise List values to the native-typed JSON form on import so the
|
||||
// comparison (and the persisted value) match what the target already
|
||||
// stores natively — otherwise an idempotent re-import of an old-form
|
||||
// bundle would spuriously report a Value change.
|
||||
var normalizedValue = ImportValueNormalizer.NormalizeListValue(
|
||||
attrDto.Value, attrDto.DataType, attrDto.ElementDataType, _logger, attrDto.Name);
|
||||
if (existingByName.TryGetValue(attrDto.Name, out var current))
|
||||
{
|
||||
// Update only if any field actually changed.
|
||||
bool changed =
|
||||
!string.Equals(current.Value, attrDto.Value, StringComparison.Ordinal) ||
|
||||
!string.Equals(current.Value, normalizedValue, StringComparison.Ordinal) ||
|
||||
current.DataType != attrDto.DataType ||
|
||||
current.IsLocked != attrDto.IsLocked ||
|
||||
!string.Equals(current.Description, attrDto.Description, StringComparison.Ordinal) ||
|
||||
@@ -1127,7 +1137,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
current.ElementDataType != attrDto.ElementDataType;
|
||||
if (!changed) continue;
|
||||
|
||||
current.Value = attrDto.Value;
|
||||
current.Value = normalizedValue;
|
||||
current.DataType = attrDto.DataType;
|
||||
current.IsLocked = attrDto.IsLocked;
|
||||
current.Description = attrDto.Description;
|
||||
@@ -1157,7 +1167,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
{
|
||||
var newAttr = new TemplateAttribute(attrDto.Name)
|
||||
{
|
||||
Value = attrDto.Value,
|
||||
Value = normalizedValue,
|
||||
DataType = attrDto.DataType,
|
||||
IsLocked = attrDto.IsLocked,
|
||||
Description = attrDto.Description,
|
||||
|
||||
@@ -199,7 +199,7 @@ public sealed class EntitySerializer
|
||||
t.Attributes.Add(new TemplateAttribute(a.Name)
|
||||
{
|
||||
TemplateId = t.Id,
|
||||
Value = a.Value,
|
||||
Value = ImportValueNormalizer.NormalizeListValue(a.Value, a.DataType, a.ElementDataType),
|
||||
DataType = a.DataType,
|
||||
IsLocked = a.IsLocked,
|
||||
Description = a.Description,
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Serialization;
|
||||
|
||||
/// <summary>
|
||||
/// Import-time normalization of attribute values to the native-typed JSON form.
|
||||
/// <para>
|
||||
/// Bundles exported before the native-typed-JSON change carry List attribute
|
||||
/// values in the old quoted-element form (e.g. <c>["10","20"]</c> for an
|
||||
/// <see cref="DataType.Int32"/> list). Already-exported bundle files can't be
|
||||
/// rewritten, so List values are normalised on import: every DTO→entity write
|
||||
/// site routes the value through <see cref="NormalizeListValue"/> so imported
|
||||
/// data lands native (<c>[10,20]</c>). The central DB normalizer remains the
|
||||
/// backstop for anything that slips through.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Non-List attributes and null/empty values pass through unchanged. A value
|
||||
/// that fails to decode (malformed JSON / un-parseable element) is left exactly
|
||||
/// as-is so the import still succeeds — the DB normalizer is the backstop.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class ImportValueNormalizer
|
||||
{
|
||||
/// <summary>
|
||||
/// Returns the native-typed JSON form of a List attribute value, or the
|
||||
/// value unchanged for non-List / null / empty / malformed inputs.
|
||||
/// </summary>
|
||||
/// <param name="value">The attribute value as carried by the bundle DTO.</param>
|
||||
/// <param name="dataType">The attribute's declared data type.</param>
|
||||
/// <param name="elementType">The List element type (null for scalars).</param>
|
||||
/// <param name="logger">Optional logger; a warning is emitted when a malformed value is left as-is.</param>
|
||||
/// <param name="attributeName">Optional attribute name for the diagnostic message.</param>
|
||||
public static string? NormalizeListValue(
|
||||
string? value,
|
||||
DataType dataType,
|
||||
DataType? elementType,
|
||||
ILogger? logger = null,
|
||||
string? attributeName = null)
|
||||
{
|
||||
if (dataType != DataType.List || string.IsNullOrEmpty(value))
|
||||
{
|
||||
return value;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return AttributeValueCodec.Encode(
|
||||
AttributeValueCodec.Decode(value, DataType.List, elementType));
|
||||
}
|
||||
catch (FormatException ex)
|
||||
{
|
||||
// Leave malformed values exactly as imported; the DB normalizer is
|
||||
// the backstop. Never abort the import for a single bad value.
|
||||
logger?.LogWarning(ex,
|
||||
"Bundle import: could not normalize List value for attribute {Attribute}; " +
|
||||
"importing verbatim (the central DB normalizer is the backstop).",
|
||||
attributeName ?? "(unknown)");
|
||||
return value;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -64,7 +64,7 @@ public class AttributeValueCodecTests
|
||||
try
|
||||
{
|
||||
CultureInfo.CurrentCulture = new CultureInfo("de-DE");
|
||||
Assert.Equal("[\"1.5\",\"2.5\"]",
|
||||
Assert.Equal("[1.5,2.5]",
|
||||
AttributeValueCodec.Encode(new List<double> { 1.5, 2.5 }));
|
||||
}
|
||||
finally
|
||||
@@ -73,6 +73,71 @@ public class AttributeValueCodecTests
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Encode_Int32List_ProducesNativeNumbers() =>
|
||||
Assert.Equal("[10,20,30]",
|
||||
AttributeValueCodec.Encode(new List<int> { 10, 20, 30 }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_BoolList_ProducesNativeBooleans() =>
|
||||
Assert.Equal("[true,false]",
|
||||
AttributeValueCodec.Encode(new List<bool> { true, false }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_StringList_StaysQuoted() =>
|
||||
Assert.Equal("[\"a\",\"b\"]",
|
||||
AttributeValueCodec.Encode(new List<string> { "a", "b" }));
|
||||
|
||||
[Fact]
|
||||
public void Encode_DateTimeList_IsIso8601()
|
||||
{
|
||||
var json = AttributeValueCodec.Encode(
|
||||
new List<DateTime> { new(2026, 6, 16, 0, 0, 0, DateTimeKind.Utc) });
|
||||
Assert.StartsWith("[\"", json); // DateTime list must be an array of quoted strings
|
||||
Assert.Contains("2026-06-16T00:00:00", json);
|
||||
Assert.DoesNotContain("06/16/2026", json);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_OldStringFloatForm_BackwardCompatible()
|
||||
{
|
||||
var back = (IList<float>)AttributeValueCodec.Decode("[\"1.5\",\"2.25\"]", DataType.List, DataType.Float)!;
|
||||
Assert.Equal(new[] { 1.5f, 2.25f }, back);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_OldStringDateTimeForm_BackwardCompatible()
|
||||
{
|
||||
// The pre-native codec encoded DateTime via IFormattable.ToString(null, InvariantCulture):
|
||||
// "06/16/2026 12:30:45" (US-invariant, no 'T'/'Z'). New Decode must still parse it.
|
||||
var back = (IList<DateTime>)AttributeValueCodec.Decode(
|
||||
"[\"06/16/2026 12:30:45\"]", DataType.List, DataType.DateTime)!;
|
||||
Assert.Equal(new DateTime(2026, 6, 16, 12, 30, 45, DateTimeKind.Unspecified), back[0]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_NewNativeIntForm_Parses()
|
||||
{
|
||||
var back = (IList<int>)AttributeValueCodec.Decode("[10,20]", DataType.List, DataType.Int32)!;
|
||||
Assert.Equal(new[] { 10, 20 }, back);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Decode_OldStringIntForm_BackwardCompatible()
|
||||
{
|
||||
var back = (IList<int>)AttributeValueCodec.Decode("[\"10\",\"20\"]", DataType.List, DataType.Int32)!;
|
||||
Assert.Equal(new[] { 10, 20 }, back);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("[true,false]")]
|
||||
[InlineData("[\"True\",\"False\"]")]
|
||||
public void Decode_BoolForms_BothParse(string json)
|
||||
{
|
||||
var back = (IList<bool>)AttributeValueCodec.Decode(json, DataType.List, DataType.Boolean)!;
|
||||
Assert.Equal(new[] { true, false }, back);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RoundTrip_Int32List()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for <see cref="ListValueNormalizer"/> — the idempotent startup normalizer that
|
||||
/// rewrites already-persisted List attribute values from the old array-of-strings JSON
|
||||
/// form (<c>["10","20"]</c>) to the new native-typed form (<c>[10,20]</c>).
|
||||
/// </summary>
|
||||
public class ListValueNormalizerTests : IDisposable
|
||||
{
|
||||
private readonly ScadaBridgeDbContext _context;
|
||||
|
||||
public ListValueNormalizerTests()
|
||||
{
|
||||
_context = SqliteTestHelper.CreateInMemoryContext();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_context.Database.CloseConnection();
|
||||
_context.Dispose();
|
||||
}
|
||||
|
||||
// Seeds a Template parent (satisfies the TemplateAttribute -> Template FK) and returns its Id.
|
||||
private async Task<int> SeedTemplateAsync()
|
||||
{
|
||||
var template = new Template("T1");
|
||||
_context.Templates.Add(template);
|
||||
await _context.SaveChangesAsync();
|
||||
return template.Id;
|
||||
}
|
||||
|
||||
// Seeds Site + Template + Instance (satisfies the InstanceAttributeOverride -> Instance FK)
|
||||
// and returns the Instance Id.
|
||||
private async Task<int> SeedInstanceAsync()
|
||||
{
|
||||
var site = new Site("Site1", "S-001");
|
||||
var template = new Template("T1");
|
||||
_context.Sites.Add(site);
|
||||
_context.Templates.Add(template);
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
var instance = new Instance("Inst1") { SiteId = site.Id, TemplateId = template.Id };
|
||||
_context.Instances.Add(instance);
|
||||
await _context.SaveChangesAsync();
|
||||
return instance.Id;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TemplateAttribute_OldStringForm_RewrittenToNative()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("intList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.Int32,
|
||||
Value = "[\"10\",\"20\"]",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[10,20]", reloaded.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TemplateAttribute_AlreadyNative_IsNotRewritten()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("intList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.Int32,
|
||||
Value = "[10,20]",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
// No tracked entity should be marked Modified — idempotent no-op.
|
||||
var tracked = _context.ChangeTracker.Entries<TemplateAttribute>()
|
||||
.Where(e => e.State == EntityState.Modified)
|
||||
.ToList();
|
||||
Assert.Empty(tracked);
|
||||
|
||||
var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[10,20]", reloaded.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TemplateAttribute_StringList_IsUnchanged()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("stringList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.String,
|
||||
Value = "[\"a\",\"b\"]",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[\"a\",\"b\"]", reloaded.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TemplateAttribute_Malformed_IsSkipped_AndSiblingStillNormalized()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("badList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.Int32,
|
||||
Value = "[\"a\"", // malformed JSON
|
||||
});
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("goodList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.Int32,
|
||||
Value = "[\"5\"]", // old form, valid
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
// Must NOT throw despite the malformed row.
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var bad = await _context.TemplateAttributes.AsNoTracking()
|
||||
.SingleAsync(a => a.Name == "badList");
|
||||
var good = await _context.TemplateAttributes.AsNoTracking()
|
||||
.SingleAsync(a => a.Name == "goodList");
|
||||
|
||||
Assert.Equal("[\"a\"", bad.Value); // skipped, untouched
|
||||
Assert.Equal("[5]", good.Value); // normalized
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TemplateAttribute_NonListRow_IsUnchanged()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("scalar")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.Int32,
|
||||
ElementDataType = null,
|
||||
Value = "42",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var reloaded = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("42", reloaded.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task InstanceAttributeOverride_OldStringForm_RewrittenToNative()
|
||||
{
|
||||
var instanceId = await SeedInstanceAsync();
|
||||
_context.InstanceAttributeOverrides.Add(new InstanceAttributeOverride("intList")
|
||||
{
|
||||
InstanceId = instanceId,
|
||||
ElementDataType = DataType.Int32,
|
||||
OverrideValue = "[\"5\"]",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var reloaded = await _context.InstanceAttributeOverrides.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[5]", reloaded.OverrideValue);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task InstanceAttributeOverride_NullElementType_IsUntouched()
|
||||
{
|
||||
var instanceId = await SeedInstanceAsync();
|
||||
_context.InstanceAttributeOverrides.Add(new InstanceAttributeOverride("scalar")
|
||||
{
|
||||
InstanceId = instanceId,
|
||||
ElementDataType = null,
|
||||
OverrideValue = "42",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
|
||||
var reloaded = await _context.InstanceAttributeOverrides.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("42", reloaded.OverrideValue);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NormalizeAsync_IsIdempotent_SecondRunChangesNothing()
|
||||
{
|
||||
var templateId = await SeedTemplateAsync();
|
||||
_context.TemplateAttributes.Add(new TemplateAttribute("intList")
|
||||
{
|
||||
TemplateId = templateId,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.Int32,
|
||||
Value = "[\"10\",\"20\"]",
|
||||
});
|
||||
await _context.SaveChangesAsync();
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
var afterFirst = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[10,20]", afterFirst.Value);
|
||||
|
||||
await ListValueNormalizer.NormalizeAsync(_context);
|
||||
var afterSecond = await _context.TemplateAttributes.AsNoTracking().SingleAsync();
|
||||
Assert.Equal("[10,20]", afterSecond.Value);
|
||||
|
||||
var tracked = _context.ChangeTracker.Entries<TemplateAttribute>()
|
||||
.Where(e => e.State == EntityState.Modified)
|
||||
.ToList();
|
||||
Assert.Empty(tracked);
|
||||
}
|
||||
}
|
||||
@@ -1016,4 +1016,159 @@ public class InstanceActorTests : TestKit, IDisposable
|
||||
Assert.Single(overrides);
|
||||
Assert.Equal("[]", overrides["Labels"]);
|
||||
}
|
||||
|
||||
// ── NJ-4: old-form List static override normalization on load ────────────
|
||||
|
||||
/// <summary>
|
||||
/// NJ-4: an OLD array-of-strings static override (<c>["10","20"]</c>) for an
|
||||
/// Int32 List attribute must be re-persisted in the native form (<c>[10,20]</c>)
|
||||
/// when the actor loads it at startup. The in-memory read still returns the
|
||||
/// typed list {10,20}; the on-disk value is normalized to native JSON.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task InstanceActor_OldFormListOverride_NormalizedToNativeOnLoad()
|
||||
{
|
||||
await _storage.SetStaticOverrideAsync("Pump-OldForm", "Counts", "[\"10\",\"20\"]");
|
||||
|
||||
var config = new FlattenedConfiguration
|
||||
{
|
||||
InstanceUniqueName = "Pump-OldForm",
|
||||
Attributes =
|
||||
[
|
||||
new ResolvedAttribute
|
||||
{
|
||||
CanonicalName = "Counts", Value = "[1,2]",
|
||||
DataType = "List", ElementDataType = "Int32"
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
var actor = CreateInstanceActor("Pump-OldForm", config);
|
||||
|
||||
// Wait for the async override load (PipeTo) + fire-and-forget normalization.
|
||||
await Task.Delay(1000);
|
||||
|
||||
// In-memory read returns the typed list, decoded from the old form.
|
||||
actor.Tell(new GetAttributeRequest("corr-of", "Pump-OldForm", "Counts", DateTimeOffset.UtcNow));
|
||||
var response = ExpectMsg<GetAttributeResponse>(TimeSpan.FromSeconds(5));
|
||||
Assert.True(response.Found);
|
||||
var list = Assert.IsType<List<int>>(response.Value);
|
||||
Assert.Equal(new[] { 10, 20 }, list);
|
||||
|
||||
// The on-disk override has been normalized to the native form.
|
||||
var overrides = await _storage.GetStaticOverridesAsync("Pump-OldForm");
|
||||
Assert.Single(overrides);
|
||||
Assert.Equal("[10,20]", overrides["Counts"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// NJ-4: a NATIVE-form static override (<c>[10,20]</c>) is already canonical, so
|
||||
/// load-time normalization must be a no-op — the on-disk value is unchanged
|
||||
/// (idempotent: native → native is byte-identical, so no re-persist occurs).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task InstanceActor_NativeFormListOverride_NotRePersistedOnLoad()
|
||||
{
|
||||
await _storage.SetStaticOverrideAsync("Pump-Native", "Counts", "[10,20]");
|
||||
|
||||
var config = new FlattenedConfiguration
|
||||
{
|
||||
InstanceUniqueName = "Pump-Native",
|
||||
Attributes =
|
||||
[
|
||||
new ResolvedAttribute
|
||||
{
|
||||
CanonicalName = "Counts", Value = "[1,2]",
|
||||
DataType = "List", ElementDataType = "Int32"
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
var actor = CreateInstanceActor("Pump-Native", config);
|
||||
await Task.Delay(1000);
|
||||
|
||||
actor.Tell(new GetAttributeRequest("corr-nat", "Pump-Native", "Counts", DateTimeOffset.UtcNow));
|
||||
var response = ExpectMsg<GetAttributeResponse>(TimeSpan.FromSeconds(5));
|
||||
Assert.True(response.Found);
|
||||
var list = Assert.IsType<List<int>>(response.Value);
|
||||
Assert.Equal(new[] { 10, 20 }, list);
|
||||
|
||||
// The native value is left untouched on disk.
|
||||
var overrides = await _storage.GetStaticOverridesAsync("Pump-Native");
|
||||
Assert.Single(overrides);
|
||||
Assert.Equal("[10,20]", overrides["Counts"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// NJ-4: a scalar static override is unaffected by the List normalization path —
|
||||
/// its on-disk value is left exactly as stored (no native re-encode).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task InstanceActor_ScalarOverride_NotTouchedByListNormalization()
|
||||
{
|
||||
await _storage.SetStaticOverrideAsync("Pump-ScalarOf", "Temperature", "200.0");
|
||||
|
||||
var config = new FlattenedConfiguration
|
||||
{
|
||||
InstanceUniqueName = "Pump-ScalarOf",
|
||||
Attributes =
|
||||
[
|
||||
new ResolvedAttribute { CanonicalName = "Temperature", Value = "100.0", DataType = "Double" }
|
||||
]
|
||||
};
|
||||
|
||||
var actor = CreateInstanceActor("Pump-ScalarOf", config);
|
||||
await Task.Delay(1000);
|
||||
|
||||
actor.Tell(new GetAttributeRequest("corr-sof", "Pump-ScalarOf", "Temperature", DateTimeOffset.UtcNow));
|
||||
var response = ExpectMsg<GetAttributeResponse>(TimeSpan.FromSeconds(5));
|
||||
Assert.True(response.Found);
|
||||
Assert.Equal("200.0", response.Value);
|
||||
|
||||
var overrides = await _storage.GetStaticOverridesAsync("Pump-ScalarOf");
|
||||
Assert.Single(overrides);
|
||||
Assert.Equal("200.0", overrides["Temperature"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// NJ-4: a malformed stored List override (truncated JSON) must NOT crash the
|
||||
/// actor and must NOT be re-persisted — it loads with Bad quality (as today),
|
||||
/// the actor stays alive, and the poison on-disk value is left unchanged.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task InstanceActor_MalformedListOverride_BadQuality_NotRePersisted()
|
||||
{
|
||||
await _storage.SetStaticOverrideAsync("Pump-BadOf", "Counts", "[\"a\"");
|
||||
|
||||
var config = new FlattenedConfiguration
|
||||
{
|
||||
InstanceUniqueName = "Pump-BadOf",
|
||||
Attributes =
|
||||
[
|
||||
new ResolvedAttribute
|
||||
{
|
||||
CanonicalName = "Counts", Value = "[1,2]",
|
||||
DataType = "List", ElementDataType = "Int32"
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
var actor = CreateInstanceActor("Pump-BadOf", config);
|
||||
Watch(actor);
|
||||
await Task.Delay(1000);
|
||||
|
||||
actor.Tell(new GetAttributeRequest("corr-bof", "Pump-BadOf", "Counts", DateTimeOffset.UtcNow));
|
||||
var response = ExpectMsg<GetAttributeResponse>(TimeSpan.FromSeconds(5));
|
||||
Assert.True(response.Found);
|
||||
Assert.Equal("Bad", response.Quality);
|
||||
Assert.Null(response.Value);
|
||||
|
||||
// The actor must still be alive — no crash from the normalization path.
|
||||
ExpectNoTerminated(actor, TimeSpan.FromMilliseconds(500));
|
||||
|
||||
// The malformed value must NOT have been re-persisted (left exactly as stored).
|
||||
var overrides = await _storage.GetStaticOverridesAsync("Pump-BadOf");
|
||||
Assert.Single(overrides);
|
||||
Assert.Equal("[\"a\"", overrides["Counts"]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -269,6 +269,106 @@ public class InstanceServiceTests
|
||||
_repoMock.Verify(r => r.AddInstanceConnectionBindingAsync(It.IsAny<InstanceConnectionBinding>(), It.IsAny<CancellationToken>()), Times.Exactly(2));
|
||||
}
|
||||
|
||||
// --- NJ-2: ElementDataType stamping on SetAttributeOverrideAsync ---
|
||||
|
||||
[Fact]
|
||||
public async Task SetAttributeOverride_CreatePath_ListAttribute_StampsElementDataType()
|
||||
{
|
||||
var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 };
|
||||
_repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(instance);
|
||||
_repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<TemplateAttribute>
|
||||
{
|
||||
new("Tags")
|
||||
{
|
||||
IsLocked = false,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.String
|
||||
}
|
||||
});
|
||||
_repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<InstanceAttributeOverride>()); // no existing override → create path
|
||||
_repoMock.Setup(r => r.SaveChangesAsync(It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(1);
|
||||
|
||||
InstanceAttributeOverride? captured = null;
|
||||
_repoMock.Setup(r => r.AddInstanceAttributeOverrideAsync(It.IsAny<InstanceAttributeOverride>(), It.IsAny<CancellationToken>()))
|
||||
.Callback<InstanceAttributeOverride, CancellationToken>((o, _) => captured = o);
|
||||
|
||||
var result = await _sut.SetAttributeOverrideAsync(1, "Tags", "[\"a\"]", "admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotNull(captured);
|
||||
Assert.Equal(DataType.String, captured.ElementDataType);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SetAttributeOverride_UpdatePath_ListAttribute_StampsElementDataType()
|
||||
{
|
||||
var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 };
|
||||
_repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(instance);
|
||||
_repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<TemplateAttribute>
|
||||
{
|
||||
new("Tags")
|
||||
{
|
||||
IsLocked = false,
|
||||
DataType = DataType.List,
|
||||
ElementDataType = DataType.String
|
||||
}
|
||||
});
|
||||
|
||||
var existingOverride = new InstanceAttributeOverride("Tags") { Id = 42, InstanceId = 1, OverrideValue = "[\"old\"]" };
|
||||
_repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<InstanceAttributeOverride> { existingOverride }); // pre-existing → update path
|
||||
_repoMock.Setup(r => r.SaveChangesAsync(It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(1);
|
||||
|
||||
InstanceAttributeOverride? captured = null;
|
||||
_repoMock.Setup(r => r.UpdateInstanceAttributeOverrideAsync(It.IsAny<InstanceAttributeOverride>(), It.IsAny<CancellationToken>()))
|
||||
.Callback<InstanceAttributeOverride, CancellationToken>((o, _) => captured = o);
|
||||
|
||||
var result = await _sut.SetAttributeOverrideAsync(1, "Tags", "[\"new\"]", "admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotNull(captured);
|
||||
Assert.Equal(DataType.String, captured.ElementDataType);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SetAttributeOverride_CreatePath_ScalarAttribute_ElementDataTypeIsNull()
|
||||
{
|
||||
var instance = new Instance("Inst1") { Id = 1, TemplateId = 1 };
|
||||
_repoMock.Setup(r => r.GetInstanceByIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(instance);
|
||||
_repoMock.Setup(r => r.GetAttributesByTemplateIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<TemplateAttribute>
|
||||
{
|
||||
new("Threshold")
|
||||
{
|
||||
IsLocked = false,
|
||||
DataType = DataType.Float,
|
||||
ElementDataType = null
|
||||
}
|
||||
});
|
||||
_repoMock.Setup(r => r.GetOverridesByInstanceIdAsync(1, It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new List<InstanceAttributeOverride>());
|
||||
_repoMock.Setup(r => r.SaveChangesAsync(It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(1);
|
||||
|
||||
InstanceAttributeOverride? captured = null;
|
||||
_repoMock.Setup(r => r.AddInstanceAttributeOverrideAsync(It.IsAny<InstanceAttributeOverride>(), It.IsAny<CancellationToken>()))
|
||||
.Callback<InstanceAttributeOverride, CancellationToken>((o, _) => captured = o);
|
||||
|
||||
var result = await _sut.SetAttributeOverrideAsync(1, "Threshold", "3.14", "admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotNull(captured);
|
||||
Assert.Null(captured.ElementDataType);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AssignToArea_AreaInDifferentSite_ReturnsFailure()
|
||||
{
|
||||
|
||||
@@ -259,6 +259,122 @@ public sealed class EntitySerializerTests
|
||||
Assert.Equal("[\"a\",\"b\"]", rtAttr.Value);
|
||||
}
|
||||
|
||||
private static BundleContentDto MakeContentWithListAttribute(
|
||||
string value, DataType elementType)
|
||||
{
|
||||
var template = new TemplateDto(
|
||||
Name: "Pump",
|
||||
FolderName: null,
|
||||
BaseTemplateName: null,
|
||||
Description: null,
|
||||
Attributes: new[]
|
||||
{
|
||||
new TemplateAttributeDto(
|
||||
Name: "Tags",
|
||||
Value: value,
|
||||
DataType: DataType.List,
|
||||
IsLocked: false,
|
||||
Description: null,
|
||||
DataSourceReference: null,
|
||||
ElementDataType: elementType),
|
||||
},
|
||||
Alarms: Array.Empty<TemplateAlarmDto>(),
|
||||
Scripts: Array.Empty<TemplateScriptDto>(),
|
||||
Compositions: Array.Empty<TemplateCompositionDto>());
|
||||
|
||||
return new BundleContentDto(
|
||||
TemplateFolders: Array.Empty<TemplateFolderDto>(),
|
||||
Templates: new[] { template },
|
||||
SharedScripts: Array.Empty<SharedScriptDto>(),
|
||||
ExternalSystems: Array.Empty<ExternalSystemDto>(),
|
||||
DatabaseConnections: Array.Empty<DatabaseConnectionDto>(),
|
||||
NotificationLists: Array.Empty<NotificationListDto>(),
|
||||
SmtpConfigs: Array.Empty<SmtpConfigDto>(),
|
||||
ApiMethods: Array.Empty<ApiMethodDto>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Import_normalizes_old_form_Int32_list_value_to_native_json()
|
||||
{
|
||||
// Pre-native bundle: quoted Int32 list elements.
|
||||
var dto = MakeContentWithListAttribute("[\"10\",\"20\"]", DataType.Int32);
|
||||
|
||||
var aggregate = new EntitySerializer().FromBundleContent(dto);
|
||||
|
||||
var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes);
|
||||
Assert.Equal(DataType.List, attr.DataType);
|
||||
Assert.Equal(DataType.Int32, attr.ElementDataType);
|
||||
// Imported native: numbers unquoted.
|
||||
Assert.Equal("[10,20]", attr.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Import_leaves_string_list_value_quoted()
|
||||
{
|
||||
var dto = MakeContentWithListAttribute("[\"a\",\"b\"]", DataType.String);
|
||||
|
||||
var aggregate = new EntitySerializer().FromBundleContent(dto);
|
||||
|
||||
var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes);
|
||||
Assert.Equal(DataType.List, attr.DataType);
|
||||
Assert.Equal(DataType.String, attr.ElementDataType);
|
||||
// Strings stay quoted in native form.
|
||||
Assert.Equal("[\"a\",\"b\"]", attr.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Import_leaves_malformed_list_value_unchanged_without_throwing()
|
||||
{
|
||||
// Truncated JSON array — Decode throws FormatException; the import must
|
||||
// still succeed and carry the value through verbatim (DB normalizer is
|
||||
// the backstop).
|
||||
var dto = MakeContentWithListAttribute("[\"a\"", DataType.String);
|
||||
|
||||
var aggregate = new EntitySerializer().FromBundleContent(dto);
|
||||
|
||||
var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes);
|
||||
Assert.Equal("[\"a\"", attr.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Import_leaves_scalar_attribute_value_unchanged()
|
||||
{
|
||||
var template = new TemplateDto(
|
||||
Name: "Sensor",
|
||||
FolderName: null,
|
||||
BaseTemplateName: null,
|
||||
Description: null,
|
||||
Attributes: new[]
|
||||
{
|
||||
new TemplateAttributeDto(
|
||||
Name: "Pressure",
|
||||
Value: "42.0",
|
||||
DataType: DataType.Double,
|
||||
IsLocked: false,
|
||||
Description: null,
|
||||
DataSourceReference: null,
|
||||
ElementDataType: null),
|
||||
},
|
||||
Alarms: Array.Empty<TemplateAlarmDto>(),
|
||||
Scripts: Array.Empty<TemplateScriptDto>(),
|
||||
Compositions: Array.Empty<TemplateCompositionDto>());
|
||||
var dto = new BundleContentDto(
|
||||
TemplateFolders: Array.Empty<TemplateFolderDto>(),
|
||||
Templates: new[] { template },
|
||||
SharedScripts: Array.Empty<SharedScriptDto>(),
|
||||
ExternalSystems: Array.Empty<ExternalSystemDto>(),
|
||||
DatabaseConnections: Array.Empty<DatabaseConnectionDto>(),
|
||||
NotificationLists: Array.Empty<NotificationListDto>(),
|
||||
SmtpConfigs: Array.Empty<SmtpConfigDto>(),
|
||||
ApiMethods: Array.Empty<ApiMethodDto>());
|
||||
|
||||
var aggregate = new EntitySerializer().FromBundleContent(dto);
|
||||
|
||||
var attr = Assert.Single(Assert.Single(aggregate.Templates).Attributes);
|
||||
Assert.Equal(DataType.Double, attr.DataType);
|
||||
Assert.Equal("42.0", attr.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Roundtrip_scalar_attribute_with_null_ElementDataType_remains_null()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user