Commit Graph

5 Commits

Author SHA1 Message Date
Joseph Doherty 02503a4a9f fix(controlplane): ConfigPublishCoordinator survives an unreachable ConfigDb at singleton start (#868)
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.

- PreStart now only subscribes and kicks off an async recovery load piped back
  to the actor. A failure retries with exponential backoff (1 s doubling to
  30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
  stashes every message (DispatchDeployment logged as queued, ApplyAck too);
  they are replayed in order after the recovered state is applied. Recovery
  decisions (re-dispatch / time out) now run inline before the replay instead
  of via self-tells, and a duplicate dispatch of the in-flight deployment with
  a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
  "config-publish" ready-tier health check: Degraded while loading (with the
  attempt count and last error), Healthy when ready or not hosted here.

Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:28:39 -05:00
Joseph Doherty b211911f98 test: migrate AdminUI/ControlPlane/Runtime/Host tests to xunit.v3 (#756)
v2-ci / unit-tests (push) Failing after 13m22s
Akka.TestKit.Xunit2 -> Akka.TestKit.Xunit 1.5.62 (xunit.v3 TestKit; needs xunit.v3 3.2.2, set via
VersionOverride on the four projects). Removes the xunit 2.9.3 pin. IAsyncLifetime returns ValueTask,
SqlServerFactAttribute gains the source-info ctor, and xUnit1051 call sites pass
TestContext.Current.CancellationToken. Test counts unchanged.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 08:57:33 -05:00
Joseph Doherty 5097444d56 fix(controlplane): condition the dispatch flip on the guard's RowVersion; one deadline source; align IdentityModel pins
Re-review 1: HandleDispatch's terminal guard and its AwaitingApplyAcks flip read the row separately, so a
lost-dispatch closure committed between them was silently reopened. The flip now carries the RowVersion the
guard read; a change in between is a concurrency conflict (nothing seeded, nothing published, coordinator
reset). Pinned on real SQL Server with a closure injected between the two reads (fails without the fix).

AdminOps' apply deadline and the coordinator's now both read DeployTimings.ApplyDeadline, passed explicitly
to both singletons. Microsoft.IdentityModel.Protocols(.OpenIdConnect) pinned to 8.11.0 and referenced
directly by the Security project, so the whole IdentityModel family resolves 8.11.0 (fresh restore clean
of NU190x/NU1605).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:08:42 -04:00
Joseph Doherty 9466b8b242 fix(controlplane): close a lost dispatch from the in-flight check; guard terminal rows; blob-free status writes
I7: a Dispatching row committed after the coordinator's PreStart whose Tell never arrived had no owner and
refused every later deploy. The in-flight check (now fully off the admin singleton's thread, serialised by the
actor-local compose flag alone) closes a Dispatching row with zero NodeDeploymentState rows that is past the
apply deadline as TimedOut (optimistic on RowVersion) and proceeds.

HandleDispatch ignores a dispatch for a Sealed/PartiallyFailed/TimedOut row, so a late or duplicated dispatch
cannot flip it back. Coordinator status/reason writes attach a key+RowVersion stub instead of loading the
multi-MB ArtifactBlob; proven on the real SQL Server (blob untouched, seal lands).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:46:54 -04:00
Joseph Doherty b437c34f92 fix(controlplane): recover a lost dispatch instead of wedging the deploy pipeline
A Deployment row left in Dispatching with no NodeDeploymentState rows (HandleDispatch threw
before its SaveChanges, or the process died between the AdminOps row write and the
coordinator's Tell) was recovered with an empty expected set; the deadline then hit the
0 == 0 "race" branch and returned without touching the row, so every later StartDeployment was
refused AnotherDeploymentInFlight until someone hand-edited SQL.

- PreStart: a recovered row with no expected-ack rows is a lost dispatch. Inside the apply
  deadline it is re-driven through HandleDispatch (bounded: CreatedAtUtc never moves); past it
  the row is closed TimedOut with a reason naming the condition.
- HandleDispatch is idempotent on a re-dispatch (rows already seeded ARE the expected set).
- HandleDeadline takes the race branch only when _expectedAcks.Count > 0.
- The PreStart recovery read projects ArtifactBlob away (R03:3.5).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:22:20 -04:00