Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.
- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
fails silently when broken, so it is enforced loudly. NOTE: the gitignored
deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
failure-detection timings, incl. a falsifiability control proving the OLD
peer-first ordering never forms.
Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.
Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.
Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.
Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.
Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.
Deletes the phase2 resume-state scratch doc, which said to delete it on landing.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Gitea renders mermaid inline, so the flow/state/hierarchy/DAG diagrams
move to text-in-markdown: auto-layout (removes the manual overlap-prone
draw.io step), diffable source, no committed binaries, and a dark-text
theme so labels stay legible. Keep draw.io PNGs only for the two complex
bespoke diagrams (logical architecture, env2 topology) where pixel
control still wins. All 24 mermaid blocks validated by rendering.
Replace ASCII-art diagrams across the README and docs/ with editable
.drawio sources plus exported PNGs, so the diagrams render clearly in
rendered markdown and can be maintained/regenerated instead of being
hand-edited as fragile text art. Non-diagram blocks (code, folder
trees, UI wireframes) were left as text.
Joseph Doherty