# Architecture Review — Central–Site Communication & Store-and-Forward

Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.Communication`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward`, the telemetry transport paths that ride them, and the corresponding test projects.
Date: 2026-07-08.

## Scope

Read in full: `Component-Communication.md`, `Component-StoreAndForward.md`; all 9 S&F source files (`StoreAndForwardService.cs`, `StoreAndForwardStorage.cs`, `ReplicationService.cs`, `NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, options/messages/DI); all Communication source (`CentralCommunicationActor.cs`, `SiteCommunicationActor.cs`, `CommunicationService.cs`, `DebugStreamService.cs`, `DebugStreamBridgeActor.cs`, `StreamRelayActor.cs`, `SiteStreamGrpcServer.cs`, `SiteStreamGrpcClient.cs`, `SiteStreamGrpcClientFactory.cs`, `ReconcileService.cs`, `PendingDeploymentPurgeActor.cs`, `CommunicationOptions.cs`); the transport ends of the telemetry pipelines (`AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs`, `ClusterClientSiteAuditClient.cs`); and the Host wiring that composes them (`Host/Actors/AkkaHostedService.cs` — `BuildHocon`, `RegisterCentralActors`, `RegisterSiteActorsAsync`), plus `SiteRuntime/Actors/SiteReplicationActor.cs` and the `Notify.Send` enqueue path in `SiteRuntime/Scripts/ScriptRuntimeContext.cs`. Central-side ingest internals (NotificationOutboxActor, AuditLogIngestActor, SiteCallAuditActor) are another reviewer's domain; only their transport contracts are assessed here.

## Maturity Verdict

This is a carefully engineered, unusually well-documented subsystem that is strong *in the small* — idempotency keys threaded end-to-end (NotificationId, EventId, TrackedOperationId), Sender captured before every PipeTo, compare-and-swap storage updates guarding the sweep against operator races, additive-only proto discipline with contract tests, and deliberate, written-down failure semantics on nearly every path. However, it has a small number of serious *in-the-large* lifecycle/topology defects that the (otherwise broad) test suite is structurally blind to, because the risky collaborators are mocked away: the ClusterClient recreate-on-address-change path will almost certainly crash-loop the `CentralCommunicationActor` in production, the standby site node runs the full S&F delivery pipeline with no active-node gate (systematic duplicate-delivery exposure the spec explicitly says should only occur on failover), and one debug session's gRPC failover disposes the shared channel out from under every other session to the same site. The retry sweep's serial, unbounded design also collapses under a realistic central outage backlog. Verdict: **high code quality, mid maturity as a distributed system — 3–4 targeted fixes away from solid.**

---

## Findings — Stability

### [Critical] ClusterClient recreate-on-address-change causes actor-name collision and a permanent restart loop
`CentralCommunicationActor.HandleSiteAddressCacheLoaded` stops the old client and immediately creates a new one under the same name:

- `CentralCommunicationActor.cs:514-521` — `Context.Stop(_siteClients[siteId].Client)` followed synchronously by `_siteClientFactory.Create(...)`.
- `CentralCommunicationActor.cs:36-40` (`DefaultSiteClientFactory`) — `system.ActorOf(ClusterClient.Props(settings), $"site-client-{siteId}")`: a **named top-level actor**.

`Context.Stop` is asynchronous; the name `site-client-{siteId}` stays reserved until the old actor fully terminates. Creating the replacement in the same message handling will throw `InvalidActorNameException` essentially every time. The `Create` call is **outside** the surrounding try/catch (which only guards `ActorPath.Parse`, `CentralCommunicationActor.cs:492-505`), so the exception escapes the handler → the user-guardian default directive **restarts** the actor → the new incarnation's `_siteClients` dictionary is empty, but all old `site-client-*` actors are still alive at the system level. Every subsequent 60-second refresh (`PreStart`, `CentralCommunicationActor.cs:564-569`) then tries to create clients for **all** sites, collides on the reserved names, and crashes again. Failure scenario: an operator edits one site's NodeA address in the Central UI → central loses ClusterClient routing to *every* site (all `SiteEnvelope`s dropped at `HandleSiteEnvelope`, `CentralCommunicationActor.cs:398-413`) until the central process is restarted. This is the exact path the design doc promises works: "ClusterClient instances are recreated when contact points change" (`Component-Communication.md:238`).

Secondary issue on the same lines: `siteId` is interpolated into an actor name unvalidated. A `SiteIdentifier` containing `/`, whitespace, or other non-path characters throws `InvalidActorNameException` on first creation with the same restart-loop consequence — contrast with `SiteStreamGrpcServer.SubscribeInstance`, which validates `correlation_id` with `ActorPath.IsValidPathElement` for precisely this reason (`SiteStreamGrpcServer.cs:229-234`).

Why tests miss it: `CentralCommunicationActorTests` substitutes `ISiteClientFactory` with an NSubstitute mock returning TestProbes (`tests/.../CentralCommunicationActorTests.cs:42-51`) — the real naming/lifecycle machinery is never exercised anywhere.

Fix shape: unique names (`site-client-{siteId}-{seq}` via a counter, like `SiteStreamGrpcServer._actorCounter`), or watch `Terminated` and defer recreation, plus sanitize/validate the site id, plus one test using `DefaultSiteClientFactory`.

### [High] Standby site node runs the full S&F delivery pipeline — duplicate delivery is systematic, not a failover residue
The design says the standby only *applies* replicated operations and "on failover, the new active node resumes delivery" (`Component-StoreAndForward.md:77-80`). The code starts delivery on **both** nodes:

- `AkkaHostedService.cs:981` — `await storeAndForwardService.StartAsync()` runs in `RegisterSiteActorsAsync` on **every** site node, starting the retry timer (`StoreAndForwardService.cs:337-341`).
- `AkkaHostedService.cs:989, 998, 1033` — all three delivery handlers (ExternalSystem, CachedDbWrite, Notification) are registered unconditionally on every node.
- `SiteReplicationActor.cs:280-291` — replicated `Add` operations are applied to the standby's SQLite, so the standby's sweep has real Pending rows to deliver.

There is no active-node/leader gate anywhere in `StoreAndForwardService.RetryPendingMessagesAsync` (`StoreAndForwardService.cs:563-589`) or in the delivery handlers (`ExternalSystemClient.DeliverBufferedAsync`, `ExternalSystemGateway/ExternalSystemClient.cs:173-224` — checked, no gate). Consequences:

1. **Duplicate deliveries in steady operation.** Both nodes' sweeps become due for the same message at the same wall-clock moment (the replicated row carries the same `last_attempt_at`). Whichever delivers first replicates a `Remove`, but if both attempts are in flight concurrently — a window the size of the delivery duration (an HTTP call, often ≥ hundreds of ms) plus replication latency — the target receives the call **twice**. Over a long outage with many buffered messages, duplicates recur every recovery. Notifications survive this (central dedups on `NotificationId` insert-if-not-exists), but `ExternalSystem` calls and `CachedDbWrite`s do not — "idempotency is the caller's responsibility" was scoped to *rare failover* duplicates, not steady-state ones.
2. **Doubled retry traffic** against an already-struggling target during every outage.
3. Both nodes emit `CachedCallTelemetry` attempts for the same operation with independent retry counts, muddying the central audit picture.

Fix shape: gate the sweep (and ideally the queue-depth gauge registration) behind the same leader-check used by `SiteCommunicationActor.DefaultIsActiveCheck` (`SiteCommunicationActor.cs:498-507`), re-evaluated per sweep tick so failover resumes delivery automatically.

### [High] One debug session's gRPC failover disposes the shared channel under every other session to the same site
`SiteStreamGrpcClientFactory` caches **one** `SiteStreamGrpcClient` per site, keyed by site id, replacing (and disposing) it whenever a caller asks for a different endpoint (`SiteStreamGrpcClientFactory.cs:52-78`). `DebugStreamBridgeActor` holds **per-session** node preference and flips endpoint on every stream error (`DebugStreamBridgeActor.cs:416, 468-483`). With two concurrent debug sessions to the same site:

- Session A (on NodeA) hits an error and flips to NodeB → `GetOrCreate(site, NodeB)` **disposes** the NodeA-bound client. `SiteStreamGrpcClient.ReleaseResources` cancels **every** registered subscription CTS and disposes the channel (`SiteStreamGrpcClient.cs:314-324`) — killing session B's healthy stream.
- Session B's `SubscribeAsync` observes the cancellation as an error → B flips to NodeA → disposes the NodeB client → kills A's brand-new stream. The two sessions ping-pong, burning both sessions' 3-retry budgets (`DebugStreamBridgeActor.cs:45`) and terminating both.

Also, `CleanupGrpc` (`DebugStreamBridgeActor.cs:486-495`) calls `GetOrCreate` during teardown — if the cached client is bound to the other endpoint, cleanup *creates a fresh channel* (and disposes the cached one) merely to call `Unsubscribe`, with the same collateral effect.

Fix shape: key the cache by `(siteId, endpoint)` (both node channels can coexist), or reference-count clients, or give each bridge actor its own client instance. The single-flip failover for a *site-wide* address change can stay in the factory; per-session failover must not mutate shared state.

### [Medium] Replication operations can be applied out of order
`ReplicationService.FireAndForget` wraps each operation in its own `Task.Run` (`ReplicationService.cs:134-149`). Two operations for the same message issued in quick succession (Add then Remove; Park then operator Requeue) are handed to the thread pool as independent tasks — the pool does not preserve ordering, so the `replicationActor.Tell(...)` calls (handler wired at `AkkaHostedService.cs:897-901`) can happen inverted. Akka's per-sender/receiver ordering guarantee only applies *after* the Tells, so the standby can apply `Remove` (no-op, row absent) then `Add` → an orphan Pending row that the ungated standby sweep (finding above) or a failover will re-deliver. The `Task.Run` buys nothing — the handler is a non-blocking Tell returning `Task.CompletedTask`; invoking it inline (with a try/catch) preserves ordering by construction.

### [Medium] Replication failures are logged at Debug — a dead replication channel is invisible
`ReplicationService.cs:144-147` logs a failed replication at `LogDebug`. If the handler starts failing systematically (peer unreachable in a way `SendToPeer` doesn't detect, serialization fault of `StoreAndForwardMessage`), the standby's buffer silently diverges and the next failover loses the entire un-replicated delta — with zero operator-visible signal at default log levels and no counter metric. `SiteReplicationActor.SendToPeer`'s no-peer drop is also Debug (`SiteReplicationActor.cs:139-149`). At minimum Warning + a telemetry counter; the repo already has `ScadaBridgeTelemetry` plumbing for exactly this kind of gauge.

### [Medium] Park/Requeue replication is a blind UPDATE — lost on a standby missing the row
`ApplyReplicatedOperationAsync` applies `Park` and `Requeue` via `UpdateMessageAsync` (`ReplicationService.cs:121-130`), which is `UPDATE ... WHERE id = @id` (`StoreAndForwardStorage.cs:210-232`) — zero rows affected is silent. If the original `Add` was lost (fire-and-forget), the park state never materializes on the standby: after failover the message is simply *gone* from the parked view (operator can't retry/discard it) even though the full message payload was in hand in the replication operation. An upsert (INSERT OR REPLACE) for Park/Requeue would self-heal the lost-Add case for free.

### [Medium] Notifications park after ~25 minutes of central unreachability, and the "no limit" escape hatch is unused
`Notify.Send` enqueues with no `maxRetries` (`ScriptRuntimeContext.cs:2197-2203`) → `DefaultMaxRetries = 50` (`StoreAndForwardOptions.cs:21`) at the 30 s default interval = ~25 minutes of central outage before a notification parks and requires per-message operator action. `Component-StoreAndForward.md:52` explicitly documents `maxRetries: 0` as the escape hatch for callers that "genuinely require unbounded retry" — but the *only* notification producer doesn't use it, and no host config raises the cap for the Notification category specifically. A central maintenance window longer than 25 minutes strands every notification raised during it. Compounding: the serial sweep (Performance #1) makes effective intervals much longer than 30 s under backlog, but retries are counted per sweep-attempt, so the wall-clock park horizon is at least honest — the operational cliff is the manual un-parking.

### [Medium] Corrupt buffered notification is discarded and reported as delivered
`NotificationForwarder.DeliverAsync` returns `true` (→ engine removes the row) when the payload fails to deserialize (`NotificationForwarder.cs:79-87`). The message is permanently lost with only a site-log Warning: no parked row, no central `Notifications` row, no audit row. Returning `false` (permanent failure → **park**) would preserve the payload for forensics at zero protocol cost — parking is exactly the designed home for undeliverable-but-preserved messages.

### [Low] Central heartbeat state is not replicated to the peer central node
`HandleSiteHealthReport` fans reports to the peer via DistributedPubSub (`CentralCommunicationActor.cs:360-373`), but `HandleHeartbeat` marks only the local aggregator (`CentralCommunicationActor.cs:349-353`). ClusterClient delivers each site's heartbeat to one central node; the other's aggregator sees none. After a central failover the new active node briefly shows all sites offline until fresh heartbeats arrive (≤ heartbeat interval + offline threshold). Minor, but inconsistent with the care taken for health reports.

### [Low] `SubscribeInstance` concurrency-limit check is check-then-act
`SiteStreamGrpcServer.cs:243-254`: `_activeStreams.Count >= _maxConcurrentStreams` then insert — two concurrent subscribes at the boundary both pass. Bounded overshoot only; not worth locking, worth a comment.

---

## Findings — Performance

### [High] The retry sweep is serial, unbounded, and single-laned across categories — it collapses under backlog
Three compounding design choices:

- `GetMessagesForRetryAsync` has **no LIMIT** — it loads every due row into memory each sweep (`StoreAndForwardStorage.cs:183-203`).
- `RetryPendingMessagesAsync` delivers strictly one-at-a-time (`StoreAndForwardService.cs:571-579`) while `_retryInProgress` blocks the next sweep (`StoreAndForwardService.cs:566`).
- Every notification forward attempt against a down central costs a full 30 s Ask timeout (`NotificationForwardTimeout`, `CommunicationOptions.cs:36`; `NotificationForwarder.cs:93-95`).

Failure scenario: central is down for an hour and scripts have buffered 200 notifications. One sweep = 200 × 30 s ≈ 100 minutes of serial timeouts — during which no `ExternalSystem` or `CachedDbWrite` retries happen either (single lane, head-of-line blocking across categories and targets: one slow external system delays notification forwarding to a *healthy* central and vice versa). There is no per-target short-circuit — N messages to the same dead target incur N consecutive timeouts per sweep. The queue drains far slower than it fills; retry counts tick toward parking on a wildly distorted cadence.

Fix shape: `LIMIT` on the due query; group by (category, target) and short-circuit remaining messages for a target after its first transient failure in a sweep; optionally parallelize across targets with a small cap. Any one of the three helps materially.

### [Medium] No explicit Akka serializer configuration — every cross-cluster contract rides default reflective JSON
`BuildHocon` (`AkkaHostedService.cs:214-271`) configures remoting, SBR, and failure detectors, but **no** `akka.actor.serializers` / `serialization-bindings`. All command/control and telemetry traffic — deployment configs, `IngestAuditEventsCommand` batches (entities converted *from* compact proto DTOs back to POCOs before the Akka hop, `ClusterClientSiteAuditClient.cs:72-85`), `ReplicationOperation` with full message payloads — is serialized by Akka.NET's default Newtonsoft JSON serializer with embedded CLR type manifests. Costs: (a) hot-path CPU/allocation and payload bloat on the highest-volume channel (audit telemetry); (b) the documented "additive-only message contract evolution" rule (`CLAUDE.md`, Commons conventions) is only half the story — wire compatibility is also coupled to *type and namespace identity*, so a rename/move of a Commons message type breaks rolling cross-version central↔site communication with no test to catch it. `MessageContractTests` asserts correlation-id presence only (`tests/.../MessageContractTests.cs`), not serialized round-trip stability. Ironic detail: the telemetry path builds efficient proto DTOs, then converts them back to POCOs specifically to cross the ClusterClient hop.

### [Low] S&F SQLite: no WAL, no explicit busy_timeout, connection-per-operation
Every `StoreAndForwardStorage` method opens a fresh `SqliteConnection` (e.g. `StoreAndForwardStorage.cs:137-139, 185-186`) with a bare `Data Source=` connection string (`StoreAndForward/ServiceCollectionExtensions.cs:22-24`) — default rollback-journal mode. Concurrent writers exist by design: script-thread enqueues, the sweep, replication applies (standby), and gRPC `PullSiteCalls`/parked-query reads. WAL + `busy_timeout` pragma would remove writer/reader blocking; Microsoft.Data.Sqlite's pooling makes connection-per-op tolerable but the pragmas are one line each in `InitializeAsync`.

### [Low] Due-check does per-row `julianday()` string parsing on ISO-8601 timestamps
`StoreAndForwardStorage.cs:195-197` computes due-ness with `(julianday('now') - julianday(last_attempt_at)) * 86400000` against `"O"`-format strings with 7-digit fractional seconds and `+00:00` offsets — relying on SQLite's lenient datetime parser, non-indexable, and re-parsed for every Pending row on every 10 s sweep. Storing epoch milliseconds would be cheaper, indexable, and immune to format drift.

### [Low] Audit-observer notification is awaited inline in the sweep
`NotifyCachedCallObserverAsync` is awaited per attempt inside `RetryMessageAsync` (`StoreAndForwardService.cs:620-626, 688-694, 711-717`). It's exception-isolated (good), but not latency-isolated: a slow observer (SQLite audit write) stretches the already-serial sweep. Fire-and-forget with isolation, or post to a channel, would decouple it.

---

## Findings — Conventions

### [Medium] Spec-vs-code drift in the S&F component doc
Two material divergences from `Component-StoreAndForward.md`:
1. **Standby-passive model** (doc lines 77-80) vs. the ungated standby delivery pipeline (Stability #2). The doc's "a few duplicate deliveries [on failover]... acceptable trade-off" framing materially understates actual behavior.
2. **`Notify.Send` latency**: the doc/CLAUDE.md say Send "returns a NotificationId status handle immediately", but `EnqueueAsync`'s immediate-delivery attempt (`StoreAndForwardService.cs:500-529`) runs the `NotificationForwarder` Ask inline on the script's await — when central is down, every `Notify.Send` blocks its script execution for up to `NotificationForwardTimeout` (30 s) before buffering. Either document the worst case or make the Notification category enqueue with `attemptImmediateDelivery: false` + an immediate sweep kick.

### [Low] `StreamRelayActor.WriteToChannel` has a dead branch and silent loss
`TryWrite` on a bounded channel in `DropOldest` mode never returns `false` — it evicts the oldest item and succeeds. The "Channel full, dropping event" warning (`StreamRelayActor.cs:105-111`) can never fire, while the *actual* loss (the evicted oldest event) is completely unlogged. Lossy-under-backpressure is fine for debug view per spec (`Component-Communication.md:59`), but the observability is inverted: dead warning, silent real drop. Count evictions (channel `ItemDropped` callback via `CreateBounded(options, itemDropped)`) or log periodically.

### [Low] Duplicated Ask-timeout constants and divergent fault semantics between the two ingest transports
`CentralCommunicationActor.DefaultAuditIngestAskTimeout` copies `SiteStreamGrpcServer.AuditIngestAskTimeout` by hand (`CentralCommunicationActor.cs:114-124` vs `SiteStreamGrpcServer.cs:45`), and the two handlers deliberately differ on fault behavior (propagate `Status.Failure` vs swallow-to-empty-ack — documented at `CentralCommunicationActor.cs:104-111`). Both are acknowledged in comments, but two transports for the same operation with different failure semantics and copy-pasted tuning is drift waiting to happen (see Underdeveloped, dual transports).

### [Low] Heartbeat `IsActive` uses leader-check while singletons follow oldest-node
`SiteCommunicationActor.DefaultIsActiveCheck` reports active = cluster leader (`SiteCommunicationActor.cs:498-507`). Akka cluster singletons run on the *oldest* qualifying node; leader and oldest usually coincide in a 2-node stable cluster but can diverge transiently. The comment says it deliberately mirrors `ActiveNodeGate`, so it's a consistent repo convention — but the convention itself can misreport which node hosts the DeploymentManager singleton. Worth one shared helper with an oldest-member check.

### [Low] `TransportHeartbeatInterval` does double duty
The Akka.Remote transport failure-detector interval (`BuildHocon`, `AkkaHostedService.cs:246-249`) is also the *application-level* site→central heartbeat cadence (`SiteCommunicationActor.cs:421-425`). These are unrelated concerns (spec treats them separately: `Component-Communication.md:268-274` vs health reporting); tuning one silently retunes the other.

### Positives worth recording
- Tell/Ask discipline matches the repo rule everywhere: Ask only at system boundaries (`CommunicationService`, forwarder, telemetry clients), Tell/Forward on hot paths with Sender preservation for reply routing (`SiteCommunicationActor.cs:270-388`, `CentralCommunicationActor.cs:398-413`).
- Correlation IDs on every request/response contract, verified by tests.
- The at-least-once notification handoff is textbook: ack-after-persist at central, script-generated `NotificationId` as buffered-row id *and* idempotency key, re-stamped provenance at the forwarder (`NotificationForwarder.cs:123-160`).
- The sweep's CAS updates (`UpdateMessageIfStatusAsync`, `StoreAndForwardStorage.cs:243-269`) correctly close the operator-retry/discard vs sweep race, replicated with correct captured state (`StoreAndForwardService.cs:861-892`).
- gRPC server stream lifecycle is thorough: readiness + shutdown gating, correlation-id validation, duplicate-stream replacement, session-lifetime cap, leak-proofed Subscribe failure cleanup, own-entry-only removal (`SiteStreamGrpcServer.cs:211-327`).
- Stream-first debug subscription with buffered gap-window replay and per-entity timestamp dedup (`DebugStreamBridgeActor.cs:134-176, 286-329`) is a correct solution to the snapshot/stream ordering problem, including the flapping-stream retry-budget stability window.
- Additive proto evolution is real and tested (`ProtoContractTests`, `ProtoRoundtripTests`), with the manual regen process documented.

---

## Underdeveloped Areas

1. **Dual transports for the same ingest operations.** `IngestAuditEvents`/`IngestCachedTelemetry` exist both as gRPC unary RPCs (`SiteStreamGrpcServer.cs:330-433`) and as ClusterClient commands (`CentralCommunicationActor.cs:262-305`); production push uses ClusterClient (`ClusterClientSiteAuditClient`), leaving the gRPC ingest surface production-idle (doc admits: "the gRPC-receiving counterpart", `Component-Communication.md:87`). Two code paths, two fault semantics, double the test/maintenance surface — consolidation is pending and should be scheduled.
2. **No real-collaborator test for the ClusterClient cache lifecycle.** `DefaultSiteClientFactory` is untested; address-change recreation is only tested through mocks — precisely where the Critical finding lives.
3. **No test or design treatment of standby-node sweep behavior.** Neither the S&F tests nor an integration test asserts that only the active node delivers; the docker-cluster smoke topology would expose it with a two-node site and a delayed target.
4. **Replication has no anti-entropy/resync.** A standby down for an hour rejoins and applies only *new* operations; the missed delta diverges forever (until rows drain naturally). The doc covers only the "last few operations" loss case. A periodic checksum or full-row reconciliation (the pattern already exists for audit rows via `PullAuditEvents`) is absent for the S&F buffer itself.
5. **Parked rows have unbounded retention.** By design parked messages persist until operator action (`Component-StoreAndForward.md:110-116`), but there is no aging alert, cap, or archival — a forgotten site accumulates parked rows indefinitely with only a health-metric count.
6. **Reconciliation cursor edge.** `PullAuditEvents`/`PullSiteCalls` batch continuation relies on central advancing a `since_utc` cursor from the last returned timestamp (`SiteStreamGrpcServer.cs:478-484, 563-572`); rows sharing a boundary timestamp are protected only by EventId dedup / upsert-on-newer at central — fine, but the site-side read contract (`>` vs `>=`) isn't pinned by a test in this project.
7. **`DebugStreamService._sessions` on central failover** — sessions are process-local with no persistence (documented as acceptable: engineer re-establishes), but the SignalR consumer contract (`OnStreamTerminated`) is the only notification; a central restart drops sessions with no signal at all. Known/accepted, listed for completeness.

---

## Prioritized Recommendations

1. **[Critical] Fix ClusterClient recreation** (`CentralCommunicationActor.cs:514-521`, `DefaultSiteClientFactory`): unique per-incarnation actor names (suffix a counter) or Terminated-watch before re-create; validate/sanitize `siteId` for actor-name safety; wrap `Create` in the per-site try/catch; add a test using the real factory that edits a site's address twice.
2. **[High] Gate the S&F retry sweep to the active node** — check leader/oldest status at the top of `RetryPendingMessagesAsync` (or gate `StartAsync`'s timer), re-evaluated per tick so failover resumes delivery within one sweep interval. Add a two-node test.
3. **[High] Stop disposing shared gRPC clients on per-session failover** — key `SiteStreamGrpcClientFactory` by `(site, endpoint)` (allow both node channels concurrently), reserve dispose-and-replace for site removal/address edits driven by the address cache, not by individual bridge actors.
4. **[High] Bound and de-serialize the retry sweep** — `LIMIT` the due query, short-circuit per target after the first transient failure in a sweep, and consider a small parallelism cap across distinct targets; separately consider a much shorter forward timeout for the notification category or `attemptImmediateDelivery: false` on `Notify.Send` to unblock script threads.
5. **[Medium] Make replication trustworthy**: replace `Task.Run` fire-and-forget with an inline Tell (ordering), log failures at Warning with a counter metric, and make Park/Requeue applies upserts so a lost Add self-heals.
6. **[Medium] Pin the wire format**: configure an explicit serializer (or at least add serializer round-trip compatibility tests for the cross-cluster Commons contracts) so the additive-only rule is actually enforced end-to-end; long-term, stop converting proto DTOs back to POCOs for the ClusterClient hop.
7. **[Medium] Unstrand notifications**: pass `maxRetries: 0` (documented no-limit) or a category-specific high cap on the `Notify.Send` enqueue path, or auto-requeue parked Notification-category rows when central connectivity recovers.
8. **[Medium] Park (don't silently drop) corrupt notification payloads** in `NotificationForwarder.DeliverAsync` — return `false` instead of `true`.
9. **[Low] SQLite hygiene**: enable WAL + busy_timeout in `InitializeAsync`; migrate timestamps to epoch ms for the due-check.
10. **[Low] Observability polish**: count DropOldest evictions in `StreamRelayActor`; replicate heartbeat marks to the peer central node (or document the failover blind window); split the app heartbeat interval from the transport failure-detector setting.
