dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)

Browse Source 
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.

Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.

Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.

gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.

Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
This commit is contained in:
Joseph Doherty
2026-08-15 17:00:19 -04:00
parent 2fd60ecce1
commit 9dbd5e96f7
5 changed files with 298 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigComposerTests.cs
+40
View File
@@ -1,3 +1,6 @@
using System.IO.Compression;
using System.Security.Cryptography;
using System.Text;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -86,6 +89,43 @@ public sealed class ConfigComposerTests : ControlPlaneActorTestBase
artifact.RevisionHash.ShouldMatch("^[0-9a-f]{64}$");
}
/// <summary>
/// Task 8: the emitted blob is gzip AND the revision hash is the SHA-256 of THOSE bytes. The second
/// half is the load-bearing one — a <c>FetchAndCache</c> node verifies the bytes it fetched against
/// the <c>RevisionHash</c> its dispatch carried, so hashing the uncompressed JSON instead would fail
/// every fetch in the fleet while every hash-shape test above stayed green.
/// </summary>
[Fact]
public async Task Blob_is_gzip_and_the_revision_hash_is_over_the_compressed_bytes()
{
var f = NewInMemoryDbFactory();
await using (var db = f.CreateDbContext())
{
db.ServerClusters.Add(NewCluster("c1"));
await db.SaveChangesAsync();
}
var artifact = await ConfigComposer.SnapshotAndFlattenAsync(f.CreateDbContext());
artifact.Blob[0].ShouldBe((byte)0x1F);
artifact.Blob[1].ShouldBe((byte)0x8B);
artifact.RevisionHash.ShouldBe(Convert.ToHexStringLower(SHA256.HashData(artifact.Blob)));
// …which is exactly what ConfigComposer.HashOf (the fetcher's derivation) computes over the same bytes.
artifact.RevisionHash.ShouldBe(ConfigComposer.HashOf(artifact.Blob));
// Falsifiability control: the hash is NOT over the JSON. Decompress and hash that — it must differ,
// or "hash of the compressed bytes" would be an untested claim on a blob that happened to round-trip.
using var inflated = new MemoryStream();
await using (var gz = new GZipStream(new MemoryStream(artifact.Blob), CompressionMode.Decompress))
{
await gz.CopyToAsync(inflated);
}
var json = inflated.ToArray();
Encoding.UTF8.GetString(json).ShouldStartWith("{");
Convert.ToHexStringLower(SHA256.HashData(json)).ShouldNotBe(artifact.RevisionHash);
artifact.Blob.Length.ShouldBeLessThan(json.Length);
}
/// <summary>
/// v3: verifies that <see cref="ConfigComposer.SnapshotAndFlattenAsync"/> serialises a
/// <see cref="Device"/>'s endpoint into the artifact blob and that
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/ParsedDeploymentArtifactTests.cs
+109
View File
@@ -1,4 +1,6 @@
using System.IO.Compression;
using System.Text;
using System.Text.Json;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
@@ -340,6 +342,113 @@ public sealed class ParsedDeploymentArtifactTests
DeploymentArtifact.ParseComposition(empty, NodeC1).EquipmentNodes.ShouldBeEmpty();
}
[Fact]
public void Open_reads_gzip_compressed_artifacts()
{
// Task 8: ConfigComposer now gzips the artifact BEFORE hashing it, so every blob that reaches a node
// (SQL row, LocalDb cache chunk, gRPC fetch) is gzip. Decompression lives in exactly one place —
// Open() — so this parity assertion covers every static wrapper and every consumer at once.
var plain = Blob;
var gzipped = Gzip(plain);
gzipped[0].ShouldBe((byte)0x1F);
gzipped[1].ShouldBe((byte)0x8B);
gzipped.Length.ShouldBeLessThan(plain.Length); // the whole point
using var parsedGzip = DeploymentArtifact.Open(gzipped);
using var parsedPlain = DeploymentArtifact.Open(plain);
parsedGzip.ResolveClusterScope(NodeC1).ShouldBe(parsedPlain.ResolveClusterScope(NodeC1));
parsedGzip.ParseDriverInstances().ShouldBe(parsedPlain.ParseDriverInstances());
parsedGzip.ParseDriverInstances(NodeC1).ShouldBe(parsedPlain.ParseDriverInstances(NodeC1));
AssertSameAuthority(parsedGzip.ParseDataPlaneAuthority(NodeC1), parsedPlain.ParseDataPlaneAuthority(NodeC1));
AssertSameComposition(parsedGzip.ParseComposition(), parsedPlain.ParseComposition());
AssertSameComposition(parsedGzip.ParseComposition(NodeC1), parsedPlain.ParseComposition(NodeC1));
// …and through the statics, which is what most call sites still use.
DeploymentArtifact.ParseDriverInstances(gzipped, NodeC1)
.ShouldBe(DeploymentArtifact.ParseDriverInstances(plain, NodeC1));
AssertSameComposition(
DeploymentArtifact.ParseComposition(gzipped, NodeC1),
DeploymentArtifact.ParseComposition(plain, NodeC1));
}
[Fact]
public void Open_still_reads_uncompressed_artifacts()
{
// Backward compatibility: every pre-upgrade Deployment row (and every cached artifact chunk written
// before the upgrade) holds raw JSON. A JSON artifact can never start with 0x1F, so the magic sniff
// is unambiguous and these bytes take the identical path they always did.
using var parsed = DeploymentArtifact.Open(Blob);
parsed.ParseDriverInstances().Select(s => s.DriverInstanceId).ShouldBe(new[] { "DI-modbus", "DI-s7" });
parsed.ParseComposition(NodeC1).RawTags.ShouldNotBeEmpty();
}
[Fact]
public void Static_wrappers_stay_lenient_on_corrupt_gzip()
{
// Corrupt/truncated gzip must degrade exactly like malformed JSON does — the decompressor's
// InvalidDataException is translated to JsonException so the wrappers' existing lenient catch
// (and every apply path's failure handling) covers it unchanged.
var truncated = Gzip(Blob)[..24];
DeploymentArtifact.ParseDriverInstances(truncated).ShouldBeEmpty();
DeploymentArtifact.ParseDriverInstances(truncated, NodeC1).ShouldBeEmpty();
DeploymentArtifact.ResolveClusterScope(truncated, NodeC1).Mode.ShouldBe(ClusterFilterMode.None);
DeploymentArtifact.ParseDataPlaneAuthority(truncated, NodeC1).Keys.ShouldBeEmpty();
DeploymentArtifact.ParseComposition(truncated).EquipmentNodes.ShouldBeEmpty();
DeploymentArtifact.ParseComposition(truncated, NodeC1).EquipmentNodes.ShouldBeEmpty();
// Open() itself is explicit about unusable bytes, same as it is for malformed JSON.
Should.Throw<JsonException>(() => DeploymentArtifact.Open(truncated));
}
[Fact]
public void Decompression_size_guard_refuses_bombs()
{
// A hostile/corrupt blob must not be able to OOM the node: gzip of a highly repetitive payload
// reaches ~1000:1, so a few-hundred-KB artifact row could inflate to hundreds of MB. The production
// limit is 512 MB, which no honest artifact approaches — crafting a real 512 MB expansion in a unit
// test would burn seconds and gigabytes, so the limit is an internal seam this test lowers instead.
var original = DeploymentArtifact.MaxDecompressedBytes;
try
{
var payload = new byte[64 * 1024]; // all zeroes: compresses to a few dozen bytes
var bomb = Gzip(payload);
bomb.Length.ShouldBeLessThan(1024); // the small-in / huge-out shape the guard exists for
DeploymentArtifact.MaxDecompressedBytes = 4096;
var ex = Should.Throw<JsonException>(() => DeploymentArtifact.Open(bomb));
ex.Message.ShouldContain("4096");
// And the lenient statics swallow it exactly like malformed JSON.
DeploymentArtifact.ParseComposition(bomb).EquipmentNodes.ShouldBeEmpty();
DeploymentArtifact.ParseDriverInstances(bomb).ShouldBeEmpty();
// Falsifiability control: under the production limit these same bytes decompress fine (and then
// fail as JSON, not as a bomb) — so the assertion above is the guard firing, not gzip failing.
DeploymentArtifact.MaxDecompressedBytes = original;
Should.Throw<JsonException>(() => DeploymentArtifact.Open(bomb))
.Message.ShouldNotContain("refusing to inflate");
}
finally
{
DeploymentArtifact.MaxDecompressedBytes = original;
}
}
private static byte[] Gzip(byte[] plain)
{
using var output = new MemoryStream();
using (var gz = new GZipStream(output, CompressionLevel.Fastest, leaveOpen: true))
{
gz.Write(plain, 0, plain.Length);
}
return output.ToArray();
}
private static void AssertSameComposition(AddressSpaceComposition actual, AddressSpaceComposition expected)
{
actual.UnsAreas.ShouldBe(expected.UnsAreas);