Merge b640-601: honest deploy acks — Applied only after the address-space outcome, degraded flag + reason (#601)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe # Conflicts: # src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
This commit is contained in:
@@ -175,6 +175,8 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
|
||||
**`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`.
|
||||
|
||||
The ack itself is sent only once the node's address-space apply outcome is known, and carries an additive `degraded` flag (`ApplyAckMessage` fields 6/7; trailing `ApplyAck.Degraded`/`DegradedReason` on DPS) — an old node's ack decodes clean, exactly as before. See [Redundancy.md § "What an `Applied` ack means"](Redundancy.md#what-an-applied-ack-means-the-degraded-apply-amendment-gitea-601) (Gitea #601).
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. |
|
||||
|
||||
@@ -585,6 +585,54 @@ Two properties are load-bearing and easy to break:
|
||||
2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
|
||||
orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.
|
||||
|
||||
### What an `Applied` ack means: the degraded-apply amendment (Gitea #601)
|
||||
|
||||
Until #601 the node sent `ApplyAck(Applied)` the moment its revision advanced — *before*
|
||||
`OpcUaPublishActor` had even dequeued the address-space rebuild — so an apply that swallowed sink faults
|
||||
(the degraded non-rebuild apply that deliberately does not advance `_lastApplied`) still sealed the
|
||||
deployment green fleet-wide. Now:
|
||||
|
||||
- **The Applied ack waits for the address-space outcome.** `DriverHostActor` sends
|
||||
`RebuildAddressSpace(ReportOutcome: true)`; `OpcUaPublishActor` replies `RebuildCompleted(Degraded, Reason)`
|
||||
on every path (clean / sink faults / failed rebuild / pipeline throw / no usable artifact — an empty plan
|
||||
is clean), and only then is the ack sent. It is a plain `Tell` + reply message — no `Ask`, no `await`, so
|
||||
nothing resumes off the actor thread.
|
||||
- **A missing outcome is degraded, never clean.** If no reply arrives within
|
||||
`DriverHostActor.DefaultApplyOutcomeTimeout` the ack goes out `Applied` + degraded, with a reason starting
|
||||
*"address-space apply outcome not reported in time"* — distinct from a real sink failure (which carries
|
||||
`failedNodes=` / "rebuild failed" / "threw"), so an operator can tell a slow or lost apply from a broken one.
|
||||
The timeout is **derived**, not a free constant: `DeployTimings.ApplyDeadline` (Commons, 2 min — the same
|
||||
value `ConfigPublishCoordinator.DefaultApplyDeadline` reads) minus `DeployTimings.ApplyOutcomeSafetyMargin`
|
||||
(15 s) = **105 s**, so a slow-but-clean rebuild of a large address space is not labelled degraded and the
|
||||
ack still reaches central before its deadline. Change the deadline there and both sides move together.
|
||||
Accepted edge: the timer starts after the artifact read + driver reconcile, so if those alone exceed the
|
||||
15 s margin a never-answered apply can still time the deployment out. A node with no publish actor wired
|
||||
acks clean at once, as before.
|
||||
- **The ack carries an ADDITIVE degraded flag on both transports.** `ApplyAck` gained trailing
|
||||
`Degraded = false, DegradedReason = null` (an old node's DPS JSON decodes clean); `ApplyAckMessage` gained
|
||||
`bool degraded = 6; optional string degraded_reason = 7;` (an old node's gRPC ack decodes clean). The
|
||||
outcome stays `APPLIED` — a degraded node *did* apply the revision.
|
||||
- **Central records it without a status-model change.** `ConfigPublishCoordinator.PersistNodeAck` keeps the
|
||||
node's `NodeDeploymentState.Status = Applied` and writes `FailureReason = "Applied degraded: <reason>"`
|
||||
(`ConfigPublishCoordinator.IsDegraded` = Applied **and** a reason). The deployment still **seals** —
|
||||
every node runs the revision, and "latest sealed" drives drift detection and the publish actor's fallback,
|
||||
so leaving it unsealed would show the fleet a revision behind what it runs — but `Deployment.FailureReason`
|
||||
names the degraded nodes and the seal logs at Warning. Recovered from the rows on singleton failover. A
|
||||
degraded ack alongside a `Failed` one is still `PartiallyFailed`. `/fleet` renders the node as
|
||||
**Applied (degraded)** (caution chip); `/deployments` renders **Sealed (degraded)** with the reason as a tooltip.
|
||||
- **Re-deploying a degraded revision repairs it.** The node's idempotent "already at this revision" short-
|
||||
circuit is skipped while the current revision's apply is degraded or its outcome is still pending — the
|
||||
dispatch re-applies instead, the publish actor re-diffs the un-advanced composition and re-ensures the
|
||||
missing nodes, and the ack reports the new outcome. A degraded **rebuild** is the other shape: it DOES
|
||||
advance `_lastApplied`, so the identical re-deploy diffs to an empty plan — the publish actor remembers
|
||||
that the last advance was degraded and, on that empty plan, re-ensures the whole composition (idempotent,
|
||||
no second rebuild), reporting clean only if that pass is clean.
|
||||
- **A node with a ConfigDb also writes the reason on its own row** (`ApplyAck.DegradedReasonPrefix`), because
|
||||
central only records it when it *accepts* the ack — a timed-out deployment's late ack is discarded.
|
||||
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
|
||||
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
|
||||
always did — no worse than today.
|
||||
|
||||
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
|
||||
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
|
||||
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
|
||||
|
||||
Reference in New Issue
Block a user