docs(deploy): degraded-apply ack amendment (#601)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -585,6 +585,42 @@ Two properties are load-bearing and easy to break:
|
||||
2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
|
||||
orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.
|
||||
|
||||
### What an `Applied` ack means: the degraded-apply amendment (Gitea #601)
|
||||
|
||||
Until #601 the node sent `ApplyAck(Applied)` the moment its revision advanced — *before*
|
||||
`OpcUaPublishActor` had even dequeued the address-space rebuild — so an apply that swallowed sink faults
|
||||
(the degraded non-rebuild apply that deliberately does not advance `_lastApplied`) still sealed the
|
||||
deployment green fleet-wide. Now:
|
||||
|
||||
- **The Applied ack waits for the address-space outcome.** `DriverHostActor` sends
|
||||
`RebuildAddressSpace(ReportOutcome: true)`; `OpcUaPublishActor` replies `RebuildCompleted(Degraded, Reason)`
|
||||
on every path (clean / sink faults / failed rebuild / pipeline throw / no usable artifact — an empty plan
|
||||
is clean), and only then is the ack sent. It is a plain `Tell` + reply message — no `Ask`, no `await`, so
|
||||
nothing resumes off the actor thread.
|
||||
- **A missing outcome is degraded, never clean.** If no reply arrives within
|
||||
`DriverHostActor.DefaultApplyOutcomeTimeout` (60 s — inside the coordinator's 2-minute apply deadline,
|
||||
which also has to cover the artifact read/fetch), the ack goes out `Applied` + degraded
|
||||
(*"did not report an outcome within 60s"*). A node with no publish actor wired acks clean at once, as before.
|
||||
- **The ack carries an ADDITIVE degraded flag on both transports.** `ApplyAck` gained trailing
|
||||
`Degraded = false, DegradedReason = null` (an old node's DPS JSON decodes clean); `ApplyAckMessage` gained
|
||||
`bool degraded = 6; optional string degraded_reason = 7;` (an old node's gRPC ack decodes clean). The
|
||||
outcome stays `APPLIED` — a degraded node *did* apply the revision.
|
||||
- **Central records it without a status-model change.** `ConfigPublishCoordinator.PersistNodeAck` keeps the
|
||||
node's `NodeDeploymentState.Status = Applied` and writes `FailureReason = "Applied degraded: <reason>"`
|
||||
(`ConfigPublishCoordinator.IsDegraded` = Applied **and** a reason). The deployment still **seals** —
|
||||
every node runs the revision, and "latest sealed" drives drift detection and the publish actor's fallback,
|
||||
so leaving it unsealed would show the fleet a revision behind what it runs — but `Deployment.FailureReason`
|
||||
names the degraded nodes and the seal logs at Warning. Recovered from the rows on singleton failover. A
|
||||
degraded ack alongside a `Failed` one is still `PartiallyFailed`. `/fleet` renders the node as
|
||||
**Applied (degraded)** (caution chip); `/deployments` renders **Sealed (degraded)** with the reason as a tooltip.
|
||||
- **Re-deploying a degraded revision repairs it.** The node's idempotent "already at this revision" short-
|
||||
circuit is skipped while the current revision's apply is degraded or its outcome is still pending — the
|
||||
dispatch re-applies instead, the publish actor re-diffs the un-advanced composition and re-ensures the
|
||||
missing nodes, and the ack reports the new outcome.
|
||||
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
|
||||
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
|
||||
always did — no worse than today.
|
||||
|
||||
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
|
||||
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
|
||||
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
|
||||
|
||||
Reference in New Issue
Block a user