docs(deploy): degraded-apply ack amendment (#601)

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 17:35:55 -04:00
parent f066fb9c80
commit 82ef89416f
3 changed files with 43 additions and 0 deletions
+36
View File
@@ -585,6 +585,42 @@ Two properties are load-bearing and easy to break:
2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.
### What an `Applied` ack means: the degraded-apply amendment (Gitea #601)
Until #601 the node sent `ApplyAck(Applied)` the moment its revision advanced — *before*
`OpcUaPublishActor` had even dequeued the address-space rebuild — so an apply that swallowed sink faults
(the degraded non-rebuild apply that deliberately does not advance `_lastApplied`) still sealed the
deployment green fleet-wide. Now:
- **The Applied ack waits for the address-space outcome.** `DriverHostActor` sends
`RebuildAddressSpace(ReportOutcome: true)`; `OpcUaPublishActor` replies `RebuildCompleted(Degraded, Reason)`
on every path (clean / sink faults / failed rebuild / pipeline throw / no usable artifact — an empty plan
is clean), and only then is the ack sent. It is a plain `Tell` + reply message — no `Ask`, no `await`, so
nothing resumes off the actor thread.
- **A missing outcome is degraded, never clean.** If no reply arrives within
`DriverHostActor.DefaultApplyOutcomeTimeout` (60 s — inside the coordinator's 2-minute apply deadline,
which also has to cover the artifact read/fetch), the ack goes out `Applied` + degraded
(*"did not report an outcome within 60s"*). A node with no publish actor wired acks clean at once, as before.
- **The ack carries an ADDITIVE degraded flag on both transports.** `ApplyAck` gained trailing
`Degraded = false, DegradedReason = null` (an old node's DPS JSON decodes clean); `ApplyAckMessage` gained
`bool degraded = 6; optional string degraded_reason = 7;` (an old node's gRPC ack decodes clean). The
outcome stays `APPLIED` — a degraded node *did* apply the revision.
- **Central records it without a status-model change.** `ConfigPublishCoordinator.PersistNodeAck` keeps the
node's `NodeDeploymentState.Status = Applied` and writes `FailureReason = "Applied degraded: <reason>"`
(`ConfigPublishCoordinator.IsDegraded` = Applied **and** a reason). The deployment still **seals** —
every node runs the revision, and "latest sealed" drives drift detection and the publish actor's fallback,
so leaving it unsealed would show the fleet a revision behind what it runs — but `Deployment.FailureReason`
names the degraded nodes and the seal logs at Warning. Recovered from the rows on singleton failover. A
degraded ack alongside a `Failed` one is still `PartiallyFailed`. `/fleet` renders the node as
**Applied (degraded)** (caution chip); `/deployments` renders **Sealed (degraded)** with the reason as a tooltip.
- **Re-deploying a degraded revision repairs it.** The node's idempotent "already at this revision" short-
circuit is skipped while the current revision's apply is degraded or its outcome is still pending — the
dispatch re-applies instead, the publish actor re-diffs the un-advanced composition and re-ensures the
missing nodes, and the ack reports the new outcome.
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
always did — no worse than today.
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the