fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)

Success now requires BOTH persistence commit AND an InstanceActorInitialized
readiness signal from the actor's PreStart — persistence can commit before the
actor's async init has run or failed, so persistence alone must not report
Success. An actor that dies during init never signals readiness; the Terminated
fallback fails that deployment and rolls back the optimistic state. The
persisted-row rollback is deferred until the store commits so it cannot race the
optimistic write.

Deviation from plan Task 15: the plan's swallow-only guard did not handle the
persistence-first ordering (empirically the store commits before the Terminated
signal, so Success was reported for a dead actor). Added the readiness handshake
to make the join deterministic.
This commit is contained in:
Joseph Doherty
2026-07-09 00:33:40 -04:00
parent db5ad02cc9
commit 957db62df1
4 changed files with 237 additions and 18 deletions
@@ -360,4 +360,38 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
r.Message.Contains("terminated unexpectedly", StringComparison.OrdinalIgnoreCase));
}, TimeSpan.FromSeconds(10));
}
[Fact]
public async Task Deploy_WhoseActorDiesDuringInit_ReportsFailed_NotSuccess()
{
// A deploy whose config compiles (passing the compile gate) but whose Instance
// Actor dies during init must report Failed — NOT the Success it used to report
// once SQLite persisted, leaving the deployer believing a dead instance is live
// (S6). The reply must be exactly-once.
var actor = CreateDeploymentManager();
await Task.Delay(500); // empty startup
actor.Tell(new DeployInstanceCommand(
"dep-dies", "dies-on-init", "r1",
CtorThrowingConfig("dies-on-init"), "admin", DateTimeOffset.UtcNow));
var resp = ExpectMsg<DeploymentStatusResponse>(TimeSpan.FromSeconds(10));
Assert.Equal(DeploymentStatus.Failed, resp.Status);
// Exactly one reply — no late Success from the persistence path.
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
// The optimistically-persisted config row is rolled back so a restart does not
// re-create the failing actor. The rollback is fire-and-forget once the store
// commits, so poll for it rather than reading once.
List<DeployedInstance> configs = [];
for (var i = 0; i < 50; i++)
{
configs = await _storage.GetAllDeployedConfigsAsync();
if (configs.All(c => c.InstanceUniqueName != "dies-on-init"))
break;
await Task.Delay(100);
}
Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "dies-on-init");
}
}