fix(controlplane): ConfigPublishCoordinator survives an unreachable ConfigDb at singleton start (#868)

PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.

- PreStart now only subscribes and kicks off an async recovery load piped back
  to the actor. A failure retries with exponential backoff (1 s doubling to
  30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
  stashes every message (DispatchDeployment logged as queued, ApplyAck too);
  they are replayed in order after the recovered state is applied. Recovery
  decisions (re-dispatch / time out) now run inline before the replay instead
  of via self-tells, and a duplicate dispatch of the in-flight deployment with
  a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
  "config-publish" ready-tier health check: Degraded while loading (with the
  attempt count and last error), Healthy when ready or not hosted here.

Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 12:28:39 -05:00
parent 8e97fc94fd
commit 02503a4a9f
9 changed files with 569 additions and 27 deletions
@@ -112,7 +112,11 @@ public static class ServiceCollectionExtensions
// coordinator. Deliberately NOT an ActorSelection.ResolveOne here: that blocks
// inside a props factory and races the spawn it is waiting for.
var meshRouter = registry.Get<CentralCommunicationActorKey>();
return ConfigPublishCoordinator.Props(dbFactory, applyDeadline: DeployTimings.ApplyDeadline, meshRouter: meshRouter);
// lmxopcua#868: optional — through IServiceProvider for GetService's null-for-absent contract
// (the Akka resolver makes no such promise), so harnesses that never register it still boot.
var readiness = resolver.GetService<IServiceProvider>()?.GetService<ConfigPublishCoordinatorReadiness>();
return ConfigPublishCoordinator.Props(
dbFactory, applyDeadline: DeployTimings.ApplyDeadline, meshRouter: meshRouter, readiness: readiness);
},
singletonOptions);