perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)

ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.

Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.

Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.

gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.

Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
This commit is contained in:
Joseph Doherty
2026-08-15 17:00:19 -04:00
parent 2fd60ecce1
commit 9dbd5e96f7
5 changed files with 298 additions and 12 deletions
@@ -1,3 +1,6 @@
using System.IO.Compression;
using System.Security.Cryptography;
using System.Text;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -86,6 +89,43 @@ public sealed class ConfigComposerTests : ControlPlaneActorTestBase
artifact.RevisionHash.ShouldMatch("^[0-9a-f]{64}$");
}
/// <summary>
/// Task 8: the emitted blob is gzip AND the revision hash is the SHA-256 of THOSE bytes. The second
/// half is the load-bearing one — a <c>FetchAndCache</c> node verifies the bytes it fetched against
/// the <c>RevisionHash</c> its dispatch carried, so hashing the uncompressed JSON instead would fail
/// every fetch in the fleet while every hash-shape test above stayed green.
/// </summary>
[Fact]
public async Task Blob_is_gzip_and_the_revision_hash_is_over_the_compressed_bytes()
{
var f = NewInMemoryDbFactory();
await using (var db = f.CreateDbContext())
{
db.ServerClusters.Add(NewCluster("c1"));
await db.SaveChangesAsync();
}
var artifact = await ConfigComposer.SnapshotAndFlattenAsync(f.CreateDbContext());
artifact.Blob[0].ShouldBe((byte)0x1F);
artifact.Blob[1].ShouldBe((byte)0x8B);
artifact.RevisionHash.ShouldBe(Convert.ToHexStringLower(SHA256.HashData(artifact.Blob)));
// …which is exactly what ConfigComposer.HashOf (the fetcher's derivation) computes over the same bytes.
artifact.RevisionHash.ShouldBe(ConfigComposer.HashOf(artifact.Blob));
// Falsifiability control: the hash is NOT over the JSON. Decompress and hash that — it must differ,
// or "hash of the compressed bytes" would be an untested claim on a blob that happened to round-trip.
using var inflated = new MemoryStream();
await using (var gz = new GZipStream(new MemoryStream(artifact.Blob), CompressionMode.Decompress))
{
await gz.CopyToAsync(inflated);
}
var json = inflated.ToArray();
Encoding.UTF8.GetString(json).ShouldStartWith("{");
Convert.ToHexStringLower(SHA256.HashData(json)).ShouldNotBe(artifact.RevisionHash);
artifact.Blob.Length.ShouldBeLessThan(json.Length);
}
/// <summary>
/// v3: verifies that <see cref="ConfigComposer.SnapshotAndFlattenAsync"/> serialises a
/// <see cref="Device"/>'s endpoint into the artifact blob and that