fix(secrets-akka): announce from the mailbox with a sender; chunk anti-entropy payloads (scadaproj#45)

Secrets-011: HandleAnnounceTick pipes the manifest read back to the actor and publishes the
announce from the mailbox with Self as sender. Under a genuinely async store it used to publish
from a pool continuation with NoSender, so every peer replied to deadLetters; a faulted manifest
read was dropped silently and is now logged.

Secrets-012: the exchange's push rows, the pull reply and the pull request are split into
messages of at most 64 KB (measured with the wire encoding), half Akka.Remote's default 128000-byte
frame, so a cold node converges against a large store. A single over-budget row is still sent
alone; the manifest announce is not split (documented in clustered-secrets.md).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:16:57 -05:00
parent 5e2d3dd1bc
commit 849640f467
4 changed files with 327 additions and 22 deletions
@@ -171,6 +171,20 @@ One manifest per node per interval: names, revisions, timestamps, tombstone flag
Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen
`AnnounceInterval` rather than shorten it.
### Message size and Akka.Remote's frame limit
Akka.Remote drops any message larger than `akka.remote.dot-netty.tcp.maximum-frame-size` (default
128000 bytes). Since `0.6.6` the rows an exchange pushes, the rows a pull is answered with, and the
pull request itself are split into messages of at most ~64 KB each, so a cold node catching up on a
peer's whole inventory converges however many secrets there are. Two bounds remain:
- **One very large secret** (a single row whose ciphertext alone exceeds the budget, e.g. a large
certificate bundle) is still sent as one message. If it is over the frame size it never replicates;
raise `maximum-frame-size` on every node.
- **The manifest announce is not split** — a peer computes what to push against the *whole* manifest.
An entry is ~100 bytes, so the default frame holds roughly a thousand secrets. Beyond that, raise
`maximum-frame-size`.
---
## Choosing between them