fix(secrets-akka): announce from the mailbox with a sender; chunk anti-entropy payloads (scadaproj#45)
Secrets-011: HandleAnnounceTick pipes the manifest read back to the actor and publishes the announce from the mailbox with Self as sender. Under a genuinely async store it used to publish from a pool continuation with NoSender, so every peer replied to deadLetters; a faulted manifest read was dropped silently and is now logged. Secrets-012: the exchange's push rows, the pull reply and the pull request are split into messages of at most 64 KB (measured with the wire encoding), half Akka.Remote's default 128000-byte frame, so a cold node converges against a large store. A single over-budget row is still sent alone; the manifest announce is not split (documented in clustered-secrets.md). Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -171,6 +171,20 @@ One manifest per node per interval: names, revisions, timestamps, tombstone flag
|
||||
Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen
|
||||
`AnnounceInterval` rather than shorten it.
|
||||
|
||||
### Message size and Akka.Remote's frame limit
|
||||
|
||||
Akka.Remote drops any message larger than `akka.remote.dot-netty.tcp.maximum-frame-size` (default
|
||||
128000 bytes). Since `0.6.6` the rows an exchange pushes, the rows a pull is answered with, and the
|
||||
pull request itself are split into messages of at most ~64 KB each, so a cold node catching up on a
|
||||
peer's whole inventory converges however many secrets there are. Two bounds remain:
|
||||
|
||||
- **One very large secret** (a single row whose ciphertext alone exceeds the budget, e.g. a large
|
||||
certificate bundle) is still sent as one message. If it is over the frame size it never replicates;
|
||||
raise `maximum-frame-size` on every node.
|
||||
- **The manifest announce is not split** — a peer computes what to push against the *whole* manifest.
|
||||
An entry is ~100 bytes, so the default frame holds roughly a thousand secrets. Beyond that, raise
|
||||
`maximum-frame-size`.
|
||||
|
||||
---
|
||||
|
||||
## Choosing between them
|
||||
|
||||
Reference in New Issue
Block a user