perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)

ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.

Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.

Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.

gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.

Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
This commit is contained in:
Joseph Doherty
2026-08-15 17:00:19 -04:00
parent 2fd60ecce1
commit 9dbd5e96f7
5 changed files with 298 additions and 12 deletions
@@ -1,3 +1,4 @@
using System.IO.Compression;
using System.Security.Cryptography;
using System.Text.Json;
using Microsoft.EntityFrameworkCore;
@@ -8,9 +9,10 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
/// <summary>
/// Pure snapshot composer: reads the current live-edit state from <see cref="OtOpcUaConfigDbContext"/>
/// and serialises it into a deterministic byte[] artifact + SHA-256 hex revision hash. Determinism
/// comes from sorting every collection by its natural key before serialising, so two snapshots over
/// the same DB state always produce the same hash regardless of EF row ordering.
/// and serialises it into a deterministic, gzip-compressed byte[] artifact + SHA-256 hex revision hash.
/// Determinism comes from sorting every collection by its natural key before serialising, so two snapshots
/// over the same DB state always produce the same hash regardless of EF row ordering; compression wraps the
/// final bytes and reorders nothing. The hash is taken over the COMPRESSED bytes — see <see cref="Pack"/>.
/// </summary>
public static class ConfigComposer
{
@@ -61,9 +63,49 @@ public static class ConfigComposer
ScriptedAlarms = await db.ScriptedAlarms.AsNoTracking().OrderBy(x => x.ScriptedAlarmId).ToListAsync(ct),
};
var blob = JsonSerializer.SerializeToUtf8Bytes(snapshot, JsonOptions);
var hash = Convert.ToHexStringLower(SHA256.HashData(blob));
return new ConfigArtifact(blob, hash);
return Pack(JsonSerializer.SerializeToUtf8Bytes(snapshot, JsonOptions));
}
/// <summary>
/// Compress the serialised snapshot and hash the COMPRESSED bytes — the emitted artifact and its
/// revision hash, in one step, so the two can never be derived from different byte sequences.
/// </summary>
/// <remarks>
/// <para>
/// <b>Compression happens before hashing on purpose.</b> The invariant the whole deploy path
/// rests on is <c>RevisionHash == SHA-256(stored/wire bytes)</c>: a <c>FetchAndCache</c> node
/// verifies the bytes it fetched against the hash its dispatch already carried, and the LocalDb
/// cache stores those same bytes. Hashing the uncompressed JSON instead would break every one of
/// those checks. Nothing between here and <c>DeploymentArtifact.Open</c> looks inside the blob,
/// which is what makes the change invisible to the cache and the gRPC serve path.
/// </para>
/// <para>
/// gzip with the standard magic <c>1F 8B</c> and no framing of our own: valid JSON can never
/// start with <c>0x1F</c>, so the read side sniffs the first two bytes and pre-upgrade
/// uncompressed rows keep parsing unchanged. <see cref="CompressionLevel.Fastest"/> because the
/// artifact is highly repetitive JSON — the marginal ratio from a slower level is small next to
/// the ~10–20x the first level already delivers.
/// </para>
/// <para>
/// gzip output is not guaranteed byte-stable across .NET releases. Harmless here: the hash is
/// computed from the bytes emitted at compose time and never re-derived from the JSON, so an
/// implementation change costs at most one spurious "changed" revision after a runtime upgrade.
/// Determinism of the snapshot itself is unaffected — compression wraps the final bytes and
/// reorders nothing.
/// </para>
/// </remarks>
/// <param name="json">The deterministic UTF-8 JSON snapshot.</param>
/// <returns>The compressed artifact blob and the SHA-256 hex digest OF THAT BLOB.</returns>
internal static ConfigArtifact Pack(byte[] json)
{
using var output = new MemoryStream(json.Length / 8);
using (var gzip = new GZipStream(output, CompressionLevel.Fastest, leaveOpen: true))
{
gzip.Write(json, 0, json.Length);
}
var blob = output.ToArray();
return new ConfigArtifact(blob, Convert.ToHexStringLower(SHA256.HashData(blob)));
}
/// <summary>Returns the SHA-256 hex digest of the supplied artifact bytes (lowercase, no prefix).</summary>