From 82ef89416f9396f714f141417b872895cbcfe95c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 8 Oct 2026 17:35:55 -0400 Subject: [PATCH] docs(deploy): degraded-apply ack amendment (#601) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- CLAUDE.md | 5 +++++ docs/Configuration.md | 2 ++ docs/Redundancy.md | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 42f5921c..d1a2535d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -184,6 +184,11 @@ is still default-closed; what changed is which deltas it whitelists. re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct, bounded resync). A **rebuild** still advances even with failures. +- **…and the deploy ack now SAYS so (#601).** The Applied ack waits for `OpcUaPublishActor`'s + `RebuildCompleted` (60 s timeout ⇒ degraded, never clean) and carries an additive `Degraded`/`DegradedReason` + on both ack transports; central seals but records `NodeDeploymentState.FailureReason = "Applied degraded: …"` + (`/fleet` "Applied (degraded)"), and a re-dispatch of a degraded revision re-applies instead of short-circuiting. + See `docs/Redundancy.md` §"What an `Applied` ack means". - **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is what makes every whitelist above safe. - **Announcements:** `AnnounceAddedNodes` announces surgically-added **Raw** containers/tags diff --git a/docs/Configuration.md b/docs/Configuration.md index 58cf17c3..5e254b77 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -175,6 +175,8 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only **`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`. +The ack itself is sent only once the node's address-space apply outcome is known, and carries an additive `degraded` flag (`ApplyAckMessage` fields 6/7; trailing `ApplyAck.Degraded`/`DegradedReason` on DPS) — an old node's ack decodes clean, exactly as before. See [Redundancy.md § "What an `Applied` ack means"](Redundancy.md#what-an-applied-ack-means-the-degraded-apply-amendment-gitea-601) (Gitea #601). + | Key | Type | Default | Meaning | |---|---|---|---| | `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. | diff --git a/docs/Redundancy.md b/docs/Redundancy.md index d031b520..5373b455 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -585,6 +585,42 @@ Two properties are load-bearing and easy to break: 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era. +### What an `Applied` ack means: the degraded-apply amendment (Gitea #601) + +Until #601 the node sent `ApplyAck(Applied)` the moment its revision advanced — *before* +`OpcUaPublishActor` had even dequeued the address-space rebuild — so an apply that swallowed sink faults +(the degraded non-rebuild apply that deliberately does not advance `_lastApplied`) still sealed the +deployment green fleet-wide. Now: + +- **The Applied ack waits for the address-space outcome.** `DriverHostActor` sends + `RebuildAddressSpace(ReportOutcome: true)`; `OpcUaPublishActor` replies `RebuildCompleted(Degraded, Reason)` + on every path (clean / sink faults / failed rebuild / pipeline throw / no usable artifact — an empty plan + is clean), and only then is the ack sent. It is a plain `Tell` + reply message — no `Ask`, no `await`, so + nothing resumes off the actor thread. +- **A missing outcome is degraded, never clean.** If no reply arrives within + `DriverHostActor.DefaultApplyOutcomeTimeout` (60 s — inside the coordinator's 2-minute apply deadline, + which also has to cover the artifact read/fetch), the ack goes out `Applied` + degraded + (*"did not report an outcome within 60s"*). A node with no publish actor wired acks clean at once, as before. +- **The ack carries an ADDITIVE degraded flag on both transports.** `ApplyAck` gained trailing + `Degraded = false, DegradedReason = null` (an old node's DPS JSON decodes clean); `ApplyAckMessage` gained + `bool degraded = 6; optional string degraded_reason = 7;` (an old node's gRPC ack decodes clean). The + outcome stays `APPLIED` — a degraded node *did* apply the revision. +- **Central records it without a status-model change.** `ConfigPublishCoordinator.PersistNodeAck` keeps the + node's `NodeDeploymentState.Status = Applied` and writes `FailureReason = "Applied degraded: "` + (`ConfigPublishCoordinator.IsDegraded` = Applied **and** a reason). The deployment still **seals** — + every node runs the revision, and "latest sealed" drives drift detection and the publish actor's fallback, + so leaving it unsealed would show the fleet a revision behind what it runs — but `Deployment.FailureReason` + names the degraded nodes and the seal logs at Warning. Recovered from the rows on singleton failover. A + degraded ack alongside a `Failed` one is still `PartiallyFailed`. `/fleet` renders the node as + **Applied (degraded)** (caution chip); `/deployments` renders **Sealed (degraded)** with the reason as a tooltip. +- **Re-deploying a degraded revision repairs it.** The node's idempotent "already at this revision" short- + circuit is skipped while the current revision's apply is degraded or its outcome is still pending — the + dispatch re-applies instead, the publish actor re-diffs the un-advanced composition and re-ensures the + missing nodes, and the ack reports the new outcome. +- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before + (clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it + always did — no worse than today. + Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from `Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the