fix(site): DeployedSecretsMirror — ledger-owned rows recover past clock skew; real-cipher recovery test

Fix round 2 (review task-9-rereview-1.md), both Minors:

1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
   OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
   the local row happens to be a tombstone. A live local row this node already owns but whose
   ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
   re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
   is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
   someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
   message is now only ever produced for a name that is NOT in the ledger. New test:
   Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
   re-stamp followed by a genuinely new central revision converges and re-invalidates the
   cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
   against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
   DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
   IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
   encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
   recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
   AEAD-bound" property against the real cipher rather than by source inspection alone.
   Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
   ZB.MOM.WW.Secrets.Abstractions).

Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
This commit is contained in:
Joseph Doherty
2026-09-28 19:19:22 -04:00
parent 4d7d6456e3
commit 838fb4177b
4 changed files with 280 additions and 37 deletions
@@ -141,6 +141,51 @@ public class DeployedSecretsMirrorTests : IDisposable
Assert.Equal(1, mirrored.Single(m => m.Name == "a").Revision);
}
[Fact]
public async Task Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision()
{
// Fix round 2 (review "clock skew" finding): a tombstone-recovery bump can leave a local
// row stamped AHEAD of a central UpdatedUtc that arrives afterward (site clock skew, or
// simply a central edit made before the recovery but shipped after it). The row is still
// OWNED by this mirror's ledger, so a later genuinely-new central revision must not be
// misreported as an out-of-band "newer local value" — it must converge.
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
var (mirror, storage, store, _, invalidator) = await CreateAsync(time: time);
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
await mirror.ReconcileAsync(CancellationToken.None);
// Remove then re-add — but the SITE CLOCK IS SKEWED FAR INTO THE FUTURE while the
// recovery bump happens (a bad RTC/NTP fault), so the recovered row's UpdatedUtc lands
// far ahead of anything central will ship for a long while.
store.Now = new DateTimeOffset(2026, 9, 28, 12, 0, 0, TimeSpan.Zero);
await storage.UpsertDeployedSecretsAsync([], "d2");
await mirror.ReconcileAsync(CancellationToken.None); // tombstones "a"
time.Now = new DateTimeOffset(2099, 1, 1, 0, 0, 0, TimeSpan.Zero); // skewed site clock
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d3");
var recoveredPass = await mirror.ReconcileAsync(CancellationToken.None);
Assert.Equal(["a"], recoveredPass.Applied);
Assert.True(store.Rows["a"].UpdatedUtc.Year >= 2099); // confirms the skewed stamp landed
// Site clock recovers to normal. Central ships a genuinely NEW revision (different
// ciphertext, i.e. an actual rotation) with an ordinary timestamp — OLDER than the
// skewed local stamp, so a naive ApplyReplicatedAsync would no-op against it.
time.Now = new DateTimeOffset(2026, 9, 29, 0, 0, 0, TimeSpan.Zero);
var rotatedCiphertext = new byte[] { 7, 7, 7 };
await storage.UpsertDeployedSecretsAsync([Art("a", 2, ciphertext: rotatedCiphertext)], "d4");
var convergePass = await mirror.ReconcileAsync(CancellationToken.None);
Assert.Equal(["a"], convergePass.Applied);
Assert.Empty(convergePass.Failed); // NOT "newer local value" — this mirror owns the name
Assert.Equal(rotatedCiphertext, store.Rows["a"].Ciphertext);
Assert.Contains("a", invalidator.Invalidated);
var mirrored = await storage.GetMirroredSecretsAsync();
Assert.Equal(2, mirrored.Single(m => m.Name == "a").Revision);
}
[Fact]
public async Task Reconcile_ReAppliesOnlyWhenRevisionChanges()
{
@@ -523,10 +568,10 @@ public class DeployedSecretsMirrorTests : IDisposable
}
private static SecretArtifact Art(
string name, long revision, string kek = "kek-1", string contentType = "Text") => new(
string name, long revision, string kek = "kek-1", string contentType = "Text", byte[]? ciphertext = null) => new(
Name: name,
ContentType: contentType,
Ciphertext: [1, 2, 3, 4, 5],
Ciphertext: ciphertext ?? [1, 2, 3, 4, 5],
Nonce: [10, 11, 12],
Tag: [20, 21, 22, 23],
WrappedDek: [30, 31, 32, 33, 34, 35],