ConfigPublishCoordinator dies permanently if ConfigDb is unreachable when its singleton starts, and every deploy then hangs #868
Assignees
dohertj2 (dohertj2)
Clear assignees
dohertj2 (dohertj2)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: dohertj2/lmxopcua#868
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Observed on docker-dev on 2026-11-11, after an OrbStack restart in which central-1 started before SQL was reachable.
ConfigPublishCoordinator.PreStartqueries the database synchronously. If that throws, the cluster singleton dies, is never restarted, and every deployment waits forever. Restarting central-1 recovered it.Fix: move the DB load out of PreStart and retry with backoff (or use a supervisor with restart and backoff), and report Degraded on health until it loads. Add a test with an unreachable DB at start that then recovers.
Done after four review rounds. The coordinator loads asynchronously with backoff and stashes messages meanwhile. Other changes: recovered fully-acked deployments seal; acks are persisted before they count, with bounded retry; permanent ack-write failures are recognised only from a SQL error-number allowlist; node rows stay Applying until the outcome is known; terminal states are written once; failure reasons are truncated to the column length; cache-booted and stale-recovered revisions are treated as unconfirmed; a config-publish health check is added. Host.IntegrationTests 299/299. Pushed to master in
e857135e..7a27ef1f; full suite green (20 projects, 0 failures, 0 warnings).