Joseph Doherty
7b5a5a6f34
Normative first: Component-StoreAndForward.md:83 specified the whole chunked, ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It is rewritten rather than removed — the failure modes it reasoned about still exist, they are just bounded differently — and it now states the duplicate-delivery bound explicitly: a message can be delivered twice only when the OLD primary delivered it and the status change had not yet replicated when the gate flipped. One flush interval plus the in-flight ack, and unlike the old model it does NOT grow with backlog depth or with how long a node was absent. The N1 directional-authority and N5 orphan-row hazards are recorded as structurally gone, not merely unguarded. Frame-size known-issue amended: its 2026-06-26 resolution replaced the intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch itself, so the 128 KB Akka frame constraint no longer applies to that hop in any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which batches by ROW COUNT), including that the failure mode differs — an oversized gRPC message is rejected, not silently dropped. Deployment docs gain the two operational constraints that have no home in code: a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat handler that made a mixed-version pair converge went with the replicator, and a mixed pair now diverges silently), and a node offline beyond TombstoneRetention can resurrect deleted rows on rejoin. Both CLAUDE.md files corrected — each still said Phase 2 was NOT started. Definition of done closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures, 0 skips). Two DoD items needed amending rather than ticking: the stale-symbol grep still matches 4 lines, all deliberate comment prose recording what was deleted (a literal zero would delete the explanations that stop the old design coming back), and the live gate has 10 evidence items, not the 9 the checklist claimed. Deletes the phase2 resume-state scratch doc, which said to delete it on landing. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
192 lines
12 KiB
Markdown
192 lines
12 KiB
Markdown
# Deploy hangs (120 s) when the flattened instance config exceeds Akka's default frame size
|
||
|
||
**Date:** 2026-06-26 · **Status:** RESOLVED (2026-06-26) · **Severity:** High (silently un-deployable instances) · **Area:** Deployment Manager / Cluster Communication
|
||
|
||
## Resolution (2026-06-26)
|
||
|
||
Fixed via the **notify-and-fetch** rework (the primary recommendation below), not the frame-size stopgap. The full flattened config no longer travels inside any Akka message: central stages it in a `PendingDeployment` row and sends a small `RefreshDeploymentCommand`; the site fetches the config over HTTP using a per-deployment token. Both frame-trapped Akka hops are fixed — the central→site deploy hop **and** the site active→standby `SiteReplicationActor` hop (which carried the same config across an intra-site Akka hop with the same silent-drop trap). The secondary `AskTimeoutException` mis-classification (below) is fixed, and a startup `SiteReconciliationActor` self-heal plus a topology-page performance fix shipped alongside.
|
||
|
||
- **Design:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch-design.md`](../plans/2026-06-26-deploy-config-notify-and-fetch-design.md)
|
||
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
|
||
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
|
||
|
||
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
|
||
|
||
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
|
||
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
|
||
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
|
||
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
|
||
more, so the 128 000-byte frame constraint does not apply to it in any form.
|
||
|
||
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
|
||
still fetches over HTTP, so that half of the original fix stands.
|
||
|
||
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
|
||
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
|
||
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
|
||
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
|
||
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
|
||
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
|
||
|
||
Note the failure mode differs from the one documented below: an oversized gRPC message is
|
||
**rejected**, not silently dropped.
|
||
|
||
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
|
||
|
||
## Summary
|
||
|
||
Deploying an instance whose **flattened configuration JSON is large enough** that the
|
||
`DeployInstanceCommand` Akka message exceeds Akka.Remote's default `maximum-frame-size`
|
||
(**128 KB / `128000b`**) causes the message to be **silently dropped in transit** between the
|
||
central and site nodes. The central's deploy `Ask` then times out after exactly **120 s**
|
||
(`Communication:DeploymentTimeout`), the deployment is recorded as failed, and **the instance
|
||
cannot be (re)deployed at all** until the config shrinks back under the limit.
|
||
|
||
It was discovered while adding a **third composition of the same base template** (a generic
|
||
`DelmiaReceiver` added to `CvdReactor`, which already composes `LeftDelmiaReceiver` +
|
||
`RightDelmiaReceiver`). Each composition fully expands the composed template's members into the
|
||
flattened config, so the 3rd copy tipped the message over 128 KB.
|
||
|
||
## How it manifests
|
||
|
||
- CLI/UI deploy returns `{"error":"Deployment error: Timeout after 120.00 seconds","code":"COMMAND_FAILED"}` (or a 504 via the management API) after ~120 s.
|
||
- `deploy status` shows the record going `InProgress (1) → Failed (3)` with `errorMessage = "Deployment error: Timeout after 120.00 seconds"`.
|
||
- The **running instance is unaffected** — deploys are atomic, so it keeps serving its last successful deployment.
|
||
- Smaller configs deploy normally and fast (the working baseline here, the 2-receiver side-based config, completed in **0.04 s**).
|
||
|
||
## Root cause
|
||
|
||
The deploy ships the **entire flattened config inline** as a JSON string inside the Akka message,
|
||
and Akka.Remote's frame size is left at the un-configured default:
|
||
|
||
1. **Full config carried in the message** — `DeploymentService.cs` serializes the flattened config
|
||
(`JsonSerializer.Serialize(flattenedConfig)`) and puts it in
|
||
`DeployInstanceCommand.FlattenedConfigurationJson`
|
||
(`src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Deployment/DeployInstanceCommand.cs`).
|
||
2. **Sent central → site over Akka** — via ClusterClient
|
||
(`CentralCommunicationActor` → `ClusterClient.Send("/user/site-communication", …)`).
|
||
3. **No frame-size override** — `AkkaHostedService.BuildHocon`
|
||
(`src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs`, ~lines 240-244) sets only
|
||
`dot-netty.tcp { hostname; port }`. There is **no `maximum-frame-size`** and **no appsettings
|
||
knob** for it, so the Akka.Remote 1.5.62 default of `128000b` applies (and
|
||
`log-frame-size-exceeding` defaults to `off`, so there isn't even a warning).
|
||
4. **Amplifier** — no custom Akka serializer / `serialization-bindings` is configured, so the
|
||
already-serialized `configJson` is JSON-escaped **again** inside the envelope by the default
|
||
serializer (every `"` → `\"`), roughly doubling that portion of the on-wire payload. So the raw
|
||
flattened JSON only needs to be ~60-70 KB to blow the 128 KB frame.
|
||
|
||
When the encoded frame exceeds the limit, the transport drops **that one message** without tearing
|
||
down the association (heartbeats/telemetry keep flowing, so the site still reports healthy), the
|
||
site never receives the deploy, and the central's `Ask` (timeout = `DeploymentTimeout` = 2 min)
|
||
expires.
|
||
|
||
### Why the 3rd same-base composition is the trigger
|
||
|
||
The flattener (`FlatteningService`) fully expands **each** composition's attributes/alarms/scripts,
|
||
path-qualified — so N compositions of the same base ≈ N× those members in the config. The config
|
||
grows roughly linearly with composition count; here **2 `DelmiaReceiver` copies sat under the limit,
|
||
the 3rd pushed it over**. Removing the composed *scripts* did not help, because the *attribute* set
|
||
alone (×3) already exceeded the frame.
|
||
|
||
## Evidence (live logs, wonder-app-vd03, 2026-06-26)
|
||
|
||
Central log:
|
||
```
|
||
07:44:28.033 [INF] Sending deployment 82a39ef5… for instance Z28061Sim to site site-a
|
||
… (no further deploy activity; only DB/health/telemetry) …
|
||
07:46:28.038 [ERR] Deployment 82a39ef5… for instance Z28061Sim failed
|
||
07:46:28.039 [ERR] Management command MgmtDeployInstanceCommand failed
|
||
07:46:28.039 [WRN] Dead letter: ManagementError from akka://…/temp/3d to deadLetters
|
||
```
|
||
Site log (same window): **nothing** — no deploy receipt, no error. The last deploy the site ever
|
||
received was the smaller side-based config:
|
||
```
|
||
05:51:46.429 [INF] Deploying instance Z28061Sim, deploymentId=e531d216… ← worked
|
||
```
|
||
No `OversizedPayloadException`, serialization error, or deserialization error appears on **either**
|
||
node — consistent with a silent transport-level frame drop. The differentiator is purely config size
|
||
(2-receiver config delivered & applied; 3-receiver config never arrived).
|
||
|
||
## Secondary bug (found during the same investigation)
|
||
|
||
`AskTimeoutException` does **not** derive from `System.TimeoutException`, so the catch in
|
||
`DeploymentService.cs` (~line 312) mis-classifies a genuine site Ask-timeout as a generic
|
||
`"Deployment error: …"` instead of the `"Communication failure: …"` / timeout branch. Consequence:
|
||
the DeploymentManager-006 "query-site-before-redeploy" reconciliation (keyed off the
|
||
`Communication failure:` prefix) does **not** trigger after these failures. Fix the classification so
|
||
`AskTimeoutException`/`OperationCanceledException` are treated as timeouts.
|
||
|
||
## Recommended fix
|
||
|
||
> **Shipped:** the **Primary** option below (notify-and-fetch) was implemented and merged on 2026-06-26 — see the Resolution section at the top. The stopgap frame-size bump and the config-only workaround were NOT taken.
|
||
|
||
### Primary (recommended): notify-and-fetch instead of push
|
||
|
||
Stop shipping the full flattened config inside the Akka message. Instead:
|
||
|
||
1. Central persists the pending flattened config (it already persists `DeployedConfigSnapshots` on
|
||
success — extend to a pending/by-`deploymentId` store).
|
||
2. Central sends the site a **small** "apply deployment `<deploymentId>` for instance `<id>`
|
||
(revisionHash `<hash>`)" notification.
|
||
3. The **site pulls** the config by `deploymentId` and applies it, then sends a completion/status
|
||
response.
|
||
|
||
This decouples the Akka message size from the config size entirely (no instance is ever
|
||
un-deployable due to size), and it fits the **existing pull-based gRPC telemetry pattern** the site
|
||
already uses (`PullSiteCalls`, `PullAuditEvents` over `SiteStreamService`). A new
|
||
`PullDeploymentConfig`/`SiteStreamService` RPC (or reuse of the management API) is the natural home.
|
||
Keep the completion ack so the central still gets authoritative success/failure.
|
||
|
||
### Stopgap (until the above ships)
|
||
|
||
Add an explicit frame size + matching buffers to the `dot-netty.tcp` block in
|
||
`AkkaHostedService.BuildHocon`, e.g.:
|
||
```hocon
|
||
remote {
|
||
dot-netty.tcp {
|
||
hostname = …
|
||
port = …
|
||
maximum-frame-size = 4000000b # 4 MB
|
||
send-buffer-size = 8000000b
|
||
receive-buffer-size = 8000000b
|
||
log-frame-size-exceeding = 1000000b # warn before silently dropping
|
||
}
|
||
}
|
||
```
|
||
This is a **code** change (no appsettings override exists) and must be deployed to **both** central
|
||
and site (they share the Host); rebuild + redeploy per `deploy/wonder-app-vd03/RUNBOOK.md`
|
||
(Upgrading). Consider exposing it via `appsettings`/`CommunicationOptions` so it's tunable without a
|
||
rebuild. Also fix the `AskTimeoutException` classification.
|
||
|
||
### Config-only workaround (specific to this case)
|
||
|
||
For the generic-`DelmiaReceiver` rollout specifically: removing the `Left`/`Right` `DelmiaReceiver`
|
||
compositions and keeping **only** the single generic `DelmiaReceiver` makes the config *smaller* than
|
||
the working 2-receiver baseline, so it deploys under the current limit — and it's the intended
|
||
"no sides" end state anyway (each machine bound to its own generic Galaxy receiver,
|
||
`DelmiaReceiver_037/038/039`). This unblocks without an engine change, but does not fix the
|
||
underlying fragility for any other instance that legitimately needs a large config.
|
||
|
||
## Current operational state (to clean up)
|
||
|
||
> **Engine fix resolves the blocker.** With notify-and-fetch merged, a large flattened config is no longer size-capped, so the generic-`DelmiaReceiver` rollout (and the `Download` handshake re-add) can proceed on `Z28061Sim` without hitting this bug. The items below are the operational re-rollout steps, now unblocked — they are NOT engine work.
|
||
|
||
- `Z28061Sim` is running on deployment **id 55** (side-based, with the earlier `side`-crash fix) and
|
||
is healthy, but is **not currently redeployable**: `CvdReactor` has the extra generic
|
||
`DelmiaRecv`→`DelmiaReceiver_037` composition + binding, so every redeploy hits this bug.
|
||
- To restore redeployability, either apply the config-only workaround above (consolidate to one
|
||
generic receiver) or roll back the generic composition + restore the side-based
|
||
`ProcessRecipeDownload`.
|
||
- The `Download` handshake method (mirroring `MESReceiver.MoveIn`) is currently removed from the
|
||
`DelmiaReceiver` base template (template 12); re-add it once a deployable path is chosen.
|
||
|
||
## Files
|
||
|
||
- `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon — missing `maximum-frame-size`)
|
||
- `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (inline config serialize; AskTimeout mis-classification)
|
||
- `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Deployment/DeployInstanceCommand.cs` (`FlattenedConfigurationJson`)
|
||
- `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (ClusterClient send)
|
||
- `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs` (`DeploymentTimeout = 2 min`)
|
||
- `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/FlatteningService.cs` (per-composition member expansion)
|
||
</content>
|