ScadaBridge/archreview/02-communication-store-and-forward.md

Architecture Review — Central–Site Communication & Store-and-Forward

Reviewer domain: src/ZB.MOM.WW.ScadaBridge.Communication, src/ZB.MOM.WW.ScadaBridge.StoreAndForward, the telemetry transport paths that ride them, and the corresponding test projects. Date: 2026-07-08.

Scope

Read in full: Component-Communication.md, Component-StoreAndForward.md; all 9 S&F source files (StoreAndForwardService.cs, StoreAndForwardStorage.cs, ReplicationService.cs, NotificationForwarder.cs, ParkedMessageHandlerActor.cs, options/messages/DI); all Communication source (CentralCommunicationActor.cs, SiteCommunicationActor.cs, CommunicationService.cs, DebugStreamService.cs, DebugStreamBridgeActor.cs, StreamRelayActor.cs, SiteStreamGrpcServer.cs, SiteStreamGrpcClient.cs, SiteStreamGrpcClientFactory.cs, ReconcileService.cs, PendingDeploymentPurgeActor.cs, CommunicationOptions.cs); the transport ends of the telemetry pipelines (AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs, ClusterClientSiteAuditClient.cs); and the Host wiring that composes them (Host/Actors/AkkaHostedService.cs — BuildHocon, RegisterCentralActors, RegisterSiteActorsAsync), plus SiteRuntime/Actors/SiteReplicationActor.cs and the Notify.Send enqueue path in SiteRuntime/Scripts/ScriptRuntimeContext.cs. Central-side ingest internals (NotificationOutboxActor, AuditLogIngestActor, SiteCallAuditActor) are another reviewer's domain; only their transport contracts are assessed here.

Maturity Verdict

This is a carefully engineered, unusually well-documented subsystem that is strong in the small — idempotency keys threaded end-to-end (NotificationId, EventId, TrackedOperationId), Sender captured before every PipeTo, compare-and-swap storage updates guarding the sweep against operator races, additive-only proto discipline with contract tests, and deliberate, written-down failure semantics on nearly every path. However, it has a small number of serious in-the-large lifecycle/topology defects that the (otherwise broad) test suite is structurally blind to, because the risky collaborators are mocked away: the ClusterClient recreate-on-address-change path will almost certainly crash-loop the CentralCommunicationActor in production, the standby site node runs the full S&F delivery pipeline with no active-node gate (systematic duplicate-delivery exposure the spec explicitly says should only occur on failover), and one debug session's gRPC failover disposes the shared channel out from under every other session to the same site. The retry sweep's serial, unbounded design also collapses under a realistic central outage backlog. Verdict: high code quality, mid maturity as a distributed system — 3–4 targeted fixes away from solid.

---

Findings — Stability

[Critical] ClusterClient recreate-on-address-change causes actor-name collision and a permanent restart loop

CentralCommunicationActor.HandleSiteAddressCacheLoaded stops the old client and immediately creates a new one under the same name:

• CentralCommunicationActor.cs:514-521 — Context.Stop(_siteClients[siteId].Client) followed synchronously by _siteClientFactory.Create(...).
• CentralCommunicationActor.cs:36-40 (DefaultSiteClientFactory) — system.ActorOf(ClusterClient.Props(settings), $"site-client-{siteId}"): a named top-level actor.

Context.Stop is asynchronous; the name site-client-{siteId} stays reserved until the old actor fully terminates. Creating the replacement in the same message handling will throw InvalidActorNameException essentially every time. The Create call is outside the surrounding try/catch (which only guards ActorPath.Parse, CentralCommunicationActor.cs:492-505), so the exception escapes the handler → the user-guardian default directive restarts the actor → the new incarnation's _siteClients dictionary is empty, but all old site-client-* actors are still alive at the system level. Every subsequent 60-second refresh (PreStart, CentralCommunicationActor.cs:564-569) then tries to create clients for all sites, collides on the reserved names, and crashes again. Failure scenario: an operator edits one site's NodeA address in the Central UI → central loses ClusterClient routing to every site (all SiteEnvelopes dropped at HandleSiteEnvelope, CentralCommunicationActor.cs:398-413) until the central process is restarted. This is the exact path the design doc promises works: "ClusterClient instances are recreated when contact points change" (Component-Communication.md:238).

Secondary issue on the same lines: siteId is interpolated into an actor name unvalidated. A SiteIdentifier containing /, whitespace, or other non-path characters throws InvalidActorNameException on first creation with the same restart-loop consequence — contrast with SiteStreamGrpcServer.SubscribeInstance, which validates correlation_id with ActorPath.IsValidPathElement for precisely this reason (SiteStreamGrpcServer.cs:229-234).

Why tests miss it: CentralCommunicationActorTests substitutes ISiteClientFactory with an NSubstitute mock returning TestProbes (tests/.../CentralCommunicationActorTests.cs:42-51) — the real naming/lifecycle machinery is never exercised anywhere.

Fix shape: unique names (site-client-{siteId}-{seq} via a counter, like SiteStreamGrpcServer._actorCounter), or watch Terminated and defer recreation, plus sanitize/validate the site id, plus one test using DefaultSiteClientFactory.

[High] Standby site node runs the full S&F delivery pipeline — duplicate delivery is systematic, not a failover residue

The design says the standby only applies replicated operations and "on failover, the new active node resumes delivery" (Component-StoreAndForward.md:77-80). The code starts delivery on both nodes:

• AkkaHostedService.cs:981 — await storeAndForwardService.StartAsync() runs in RegisterSiteActorsAsync on every site node, starting the retry timer (StoreAndForwardService.cs:337-341).
• AkkaHostedService.cs:989, 998, 1033 — all three delivery handlers (ExternalSystem, CachedDbWrite, Notification) are registered unconditionally on every node.
• SiteReplicationActor.cs:280-291 — replicated Add operations are applied to the standby's SQLite, so the standby's sweep has real Pending rows to deliver.

There is no active-node/leader gate anywhere in StoreAndForwardService.RetryPendingMessagesAsync (StoreAndForwardService.cs:563-589) or in the delivery handlers (ExternalSystemClient.DeliverBufferedAsync, ExternalSystemGateway/ExternalSystemClient.cs:173-224 — checked, no gate). Consequences:

1. Duplicate deliveries in steady operation. Both nodes' sweeps become due for the same message at the same wall-clock moment (the replicated row carries the same last_attempt_at). Whichever delivers first replicates a Remove, but if both attempts are in flight concurrently — a window the size of the delivery duration (an HTTP call, often ≥ hundreds of ms) plus replication latency — the target receives the call twice. Over a long outage with many buffered messages, duplicates recur every recovery. Notifications survive this (central dedups on NotificationId insert-if-not-exists), but ExternalSystem calls and CachedDbWrites do not — "idempotency is the caller's responsibility" was scoped to rare failover duplicates, not steady-state ones.
2. Doubled retry traffic against an already-struggling target during every outage.
3. Both nodes emit CachedCallTelemetry attempts for the same operation with independent retry counts, muddying the central audit picture.

Fix shape: gate the sweep (and ideally the queue-depth gauge registration) behind the same leader-check used by SiteCommunicationActor.DefaultIsActiveCheck (SiteCommunicationActor.cs:498-507), re-evaluated per sweep tick so failover resumes delivery automatically.

[High] One debug session's gRPC failover disposes the shared channel under every other session to the same site

SiteStreamGrpcClientFactory caches one SiteStreamGrpcClient per site, keyed by site id, replacing (and disposing) it whenever a caller asks for a different endpoint (SiteStreamGrpcClientFactory.cs:52-78). DebugStreamBridgeActor holds per-session node preference and flips endpoint on every stream error (DebugStreamBridgeActor.cs:416, 468-483). With two concurrent debug sessions to the same site:

• Session A (on NodeA) hits an error and flips to NodeB → GetOrCreate(site, NodeB) disposes the NodeA-bound client. SiteStreamGrpcClient.ReleaseResources cancels every registered subscription CTS and disposes the channel (SiteStreamGrpcClient.cs:314-324) — killing session B's healthy stream.
• Session B's SubscribeAsync observes the cancellation as an error → B flips to NodeA → disposes the NodeB client → kills A's brand-new stream. The two sessions ping-pong, burning both sessions' 3-retry budgets (DebugStreamBridgeActor.cs:45) and terminating both.

Also, CleanupGrpc (DebugStreamBridgeActor.cs:486-495) calls GetOrCreate during teardown — if the cached client is bound to the other endpoint, cleanup creates a fresh channel (and disposes the cached one) merely to call Unsubscribe, with the same collateral effect.

Fix shape: key the cache by (siteId, endpoint) (both node channels can coexist), or reference-count clients, or give each bridge actor its own client instance. The single-flip failover for a site-wide address change can stay in the factory; per-session failover must not mutate shared state.

[Medium] Replication operations can be applied out of order

ReplicationService.FireAndForget wraps each operation in its own Task.Run (ReplicationService.cs:134-149). Two operations for the same message issued in quick succession (Add then Remove; Park then operator Requeue) are handed to the thread pool as independent tasks — the pool does not preserve ordering, so the replicationActor.Tell(...) calls (handler wired at AkkaHostedService.cs:897-901) can happen inverted. Akka's per-sender/receiver ordering guarantee only applies after the Tells, so the standby can apply Remove (no-op, row absent) then Add → an orphan Pending row that the ungated standby sweep (finding above) or a failover will re-deliver. The Task.Run buys nothing — the handler is a non-blocking Tell returning Task.CompletedTask; invoking it inline (with a try/catch) preserves ordering by construction.

[Medium] Replication failures are logged at Debug — a dead replication channel is invisible

ReplicationService.cs:144-147 logs a failed replication at LogDebug. If the handler starts failing systematically (peer unreachable in a way SendToPeer doesn't detect, serialization fault of StoreAndForwardMessage), the standby's buffer silently diverges and the next failover loses the entire un-replicated delta — with zero operator-visible signal at default log levels and no counter metric. SiteReplicationActor.SendToPeer's no-peer drop is also Debug (SiteReplicationActor.cs:139-149). At minimum Warning + a telemetry counter; the repo already has ScadaBridgeTelemetry plumbing for exactly this kind of gauge.

[Medium] Park/Requeue replication is a blind UPDATE — lost on a standby missing the row

ApplyReplicatedOperationAsync applies Park and Requeue via UpdateMessageAsync (ReplicationService.cs:121-130), which is UPDATE ... WHERE id = @id (StoreAndForwardStorage.cs:210-232) — zero rows affected is silent. If the original Add was lost (fire-and-forget), the park state never materializes on the standby: after failover the message is simply gone from the parked view (operator can't retry/discard it) even though the full message payload was in hand in the replication operation. An upsert (INSERT OR REPLACE) for Park/Requeue would self-heal the lost-Add case for free.

[Medium] Notifications park after ~25 minutes of central unreachability, and the "no limit" escape hatch is unused

Notify.Send enqueues with no maxRetries (ScriptRuntimeContext.cs:2197-2203) → DefaultMaxRetries = 50 (StoreAndForwardOptions.cs:21) at the 30 s default interval = ~25 minutes of central outage before a notification parks and requires per-message operator action. Component-StoreAndForward.md:52 explicitly documents maxRetries: 0 as the escape hatch for callers that "genuinely require unbounded retry" — but the only notification producer doesn't use it, and no host config raises the cap for the Notification category specifically. A central maintenance window longer than 25 minutes strands every notification raised during it. Compounding: the serial sweep (Performance #1) makes effective intervals much longer than 30 s under backlog, but retries are counted per sweep-attempt, so the wall-clock park horizon is at least honest — the operational cliff is the manual un-parking.

[Medium] Corrupt buffered notification is discarded and reported as delivered

NotificationForwarder.DeliverAsync returns true (→ engine removes the row) when the payload fails to deserialize (NotificationForwarder.cs:79-87). The message is permanently lost with only a site-log Warning: no parked row, no central Notifications row, no audit row. Returning false (permanent failure → park) would preserve the payload for forensics at zero protocol cost — parking is exactly the designed home for undeliverable-but-preserved messages.

[Low] Central heartbeat state is not replicated to the peer central node

HandleSiteHealthReport fans reports to the peer via DistributedPubSub (CentralCommunicationActor.cs:360-373), but HandleHeartbeat marks only the local aggregator (CentralCommunicationActor.cs:349-353). ClusterClient delivers each site's heartbeat to one central node; the other's aggregator sees none. After a central failover the new active node briefly shows all sites offline until fresh heartbeats arrive (≤ heartbeat interval + offline threshold). Minor, but inconsistent with the care taken for health reports.

[Low] SubscribeInstance concurrency-limit check is check-then-act

SiteStreamGrpcServer.cs:243-254: _activeStreams.Count >= _maxConcurrentStreams then insert — two concurrent subscribes at the boundary both pass. Bounded overshoot only; not worth locking, worth a comment.

---

Findings — Performance

[High] The retry sweep is serial, unbounded, and single-laned across categories — it collapses under backlog

Three compounding design choices:

• GetMessagesForRetryAsync has no LIMIT — it loads every due row into memory each sweep (StoreAndForwardStorage.cs:183-203).
• RetryPendingMessagesAsync delivers strictly one-at-a-time (StoreAndForwardService.cs:571-579) while _retryInProgress blocks the next sweep (StoreAndForwardService.cs:566).
• Every notification forward attempt against a down central costs a full 30 s Ask timeout (NotificationForwardTimeout, CommunicationOptions.cs:36; NotificationForwarder.cs:93-95).

Failure scenario: central is down for an hour and scripts have buffered 200 notifications. One sweep = 200 × 30 s ≈ 100 minutes of serial timeouts — during which no ExternalSystem or CachedDbWrite retries happen either (single lane, head-of-line blocking across categories and targets: one slow external system delays notification forwarding to a healthy central and vice versa). There is no per-target short-circuit — N messages to the same dead target incur N consecutive timeouts per sweep. The queue drains far slower than it fills; retry counts tick toward parking on a wildly distorted cadence.

Fix shape: LIMIT on the due query; group by (category, target) and short-circuit remaining messages for a target after its first transient failure in a sweep; optionally parallelize across targets with a small cap. Any one of the three helps materially.

[Medium] No explicit Akka serializer configuration — every cross-cluster contract rides default reflective JSON

BuildHocon (AkkaHostedService.cs:214-271) configures remoting, SBR, and failure detectors, but no akka.actor.serializers / serialization-bindings. All command/control and telemetry traffic — deployment configs, IngestAuditEventsCommand batches (entities converted from compact proto DTOs back to POCOs before the Akka hop, ClusterClientSiteAuditClient.cs:72-85), ReplicationOperation with full message payloads — is serialized by Akka.NET's default Newtonsoft JSON serializer with embedded CLR type manifests. Costs: (a) hot-path CPU/allocation and payload bloat on the highest-volume channel (audit telemetry); (b) the documented "additive-only message contract evolution" rule (CLAUDE.md, Commons conventions) is only half the story — wire compatibility is also coupled to type and namespace identity, so a rename/move of a Commons message type breaks rolling cross-version central↔site communication with no test to catch it. MessageContractTests asserts correlation-id presence only (tests/.../MessageContractTests.cs), not serialized round-trip stability. Ironic detail: the telemetry path builds efficient proto DTOs, then converts them back to POCOs specifically to cross the ClusterClient hop.

[Low] S&F SQLite: no WAL, no explicit busy_timeout, connection-per-operation

Every StoreAndForwardStorage method opens a fresh SqliteConnection (e.g. StoreAndForwardStorage.cs:137-139, 185-186) with a bare Data Source= connection string (StoreAndForward/ServiceCollectionExtensions.cs:22-24) — default rollback-journal mode. Concurrent writers exist by design: script-thread enqueues, the sweep, replication applies (standby), and gRPC PullSiteCalls/parked-query reads. WAL + busy_timeout pragma would remove writer/reader blocking; Microsoft.Data.Sqlite's pooling makes connection-per-op tolerable but the pragmas are one line each in InitializeAsync.

[Low] Due-check does per-row julianday() string parsing on ISO-8601 timestamps

StoreAndForwardStorage.cs:195-197 computes due-ness with (julianday('now') - julianday(last_attempt_at)) * 86400000 against "O"-format strings with 7-digit fractional seconds and +00:00 offsets — relying on SQLite's lenient datetime parser, non-indexable, and re-parsed for every Pending row on every 10 s sweep. Storing epoch milliseconds would be cheaper, indexable, and immune to format drift.

[Low] Audit-observer notification is awaited inline in the sweep

NotifyCachedCallObserverAsync is awaited per attempt inside RetryMessageAsync (StoreAndForwardService.cs:620-626, 688-694, 711-717). It's exception-isolated (good), but not latency-isolated: a slow observer (SQLite audit write) stretches the already-serial sweep. Fire-and-forget with isolation, or post to a channel, would decouple it.

---

Findings — Conventions

[Medium] Spec-vs-code drift in the S&F component doc

Two material divergences from Component-StoreAndForward.md:

1. Standby-passive model (doc lines 77-80) vs. the ungated standby delivery pipeline (Stability #2). The doc's "a few duplicate deliveries [on failover]... acceptable trade-off" framing materially understates actual behavior.
2. Notify.Send latency: the doc/CLAUDE.md say Send "returns a NotificationId status handle immediately", but EnqueueAsync's immediate-delivery attempt (StoreAndForwardService.cs:500-529) runs the NotificationForwarder Ask inline on the script's await — when central is down, every Notify.Send blocks its script execution for up to NotificationForwardTimeout (30 s) before buffering. Either document the worst case or make the Notification category enqueue with attemptImmediateDelivery: false + an immediate sweep kick.

[Low] StreamRelayActor.WriteToChannel has a dead branch and silent loss

TryWrite on a bounded channel in DropOldest mode never returns false — it evicts the oldest item and succeeds. The "Channel full, dropping event" warning (StreamRelayActor.cs:105-111) can never fire, while the actual loss (the evicted oldest event) is completely unlogged. Lossy-under-backpressure is fine for debug view per spec (Component-Communication.md:59), but the observability is inverted: dead warning, silent real drop. Count evictions (channel ItemDropped callback via CreateBounded(options, itemDropped)) or log periodically.

[Low] Duplicated Ask-timeout constants and divergent fault semantics between the two ingest transports

CentralCommunicationActor.DefaultAuditIngestAskTimeout copies SiteStreamGrpcServer.AuditIngestAskTimeout by hand (CentralCommunicationActor.cs:114-124 vs SiteStreamGrpcServer.cs:45), and the two handlers deliberately differ on fault behavior (propagate Status.Failure vs swallow-to-empty-ack — documented at CentralCommunicationActor.cs:104-111). Both are acknowledged in comments, but two transports for the same operation with different failure semantics and copy-pasted tuning is drift waiting to happen (see Underdeveloped, dual transports).

[Low] Heartbeat IsActive uses leader-check while singletons follow oldest-node

SiteCommunicationActor.DefaultIsActiveCheck reports active = cluster leader (SiteCommunicationActor.cs:498-507). Akka cluster singletons run on the oldest qualifying node; leader and oldest usually coincide in a 2-node stable cluster but can diverge transiently. The comment says it deliberately mirrors ActiveNodeGate, so it's a consistent repo convention — but the convention itself can misreport which node hosts the DeploymentManager singleton. Worth one shared helper with an oldest-member check.

[Low] TransportHeartbeatInterval does double duty

The Akka.Remote transport failure-detector interval (BuildHocon, AkkaHostedService.cs:246-249) is also the application-level site→central heartbeat cadence (SiteCommunicationActor.cs:421-425). These are unrelated concerns (spec treats them separately: Component-Communication.md:268-274 vs health reporting); tuning one silently retunes the other.

Positives worth recording

• Tell/Ask discipline matches the repo rule everywhere: Ask only at system boundaries (CommunicationService, forwarder, telemetry clients), Tell/Forward on hot paths with Sender preservation for reply routing (SiteCommunicationActor.cs:270-388, CentralCommunicationActor.cs:398-413).
• Correlation IDs on every request/response contract, verified by tests.
• The at-least-once notification handoff is textbook: ack-after-persist at central, script-generated NotificationId as buffered-row id and idempotency key, re-stamped provenance at the forwarder (NotificationForwarder.cs:123-160).
• The sweep's CAS updates (UpdateMessageIfStatusAsync, StoreAndForwardStorage.cs:243-269) correctly close the operator-retry/discard vs sweep race, replicated with correct captured state (StoreAndForwardService.cs:861-892).
• gRPC server stream lifecycle is thorough: readiness + shutdown gating, correlation-id validation, duplicate-stream replacement, session-lifetime cap, leak-proofed Subscribe failure cleanup, own-entry-only removal (SiteStreamGrpcServer.cs:211-327).
• Stream-first debug subscription with buffered gap-window replay and per-entity timestamp dedup (DebugStreamBridgeActor.cs:134-176, 286-329) is a correct solution to the snapshot/stream ordering problem, including the flapping-stream retry-budget stability window.
• Additive proto evolution is real and tested (ProtoContractTests, ProtoRoundtripTests), with the manual regen process documented.

---

Underdeveloped Areas

1. Dual transports for the same ingest operations. IngestAuditEvents/IngestCachedTelemetry exist both as gRPC unary RPCs (SiteStreamGrpcServer.cs:330-433) and as ClusterClient commands (CentralCommunicationActor.cs:262-305); production push uses ClusterClient (ClusterClientSiteAuditClient), leaving the gRPC ingest surface production-idle (doc admits: "the gRPC-receiving counterpart", Component-Communication.md:87). Two code paths, two fault semantics, double the test/maintenance surface — consolidation is pending and should be scheduled.
2. No real-collaborator test for the ClusterClient cache lifecycle. DefaultSiteClientFactory is untested; address-change recreation is only tested through mocks — precisely where the Critical finding lives.
3. No test or design treatment of standby-node sweep behavior. Neither the S&F tests nor an integration test asserts that only the active node delivers; the docker-cluster smoke topology would expose it with a two-node site and a delayed target.
4. Replication has no anti-entropy/resync. A standby down for an hour rejoins and applies only new operations; the missed delta diverges forever (until rows drain naturally). The doc covers only the "last few operations" loss case. A periodic checksum or full-row reconciliation (the pattern already exists for audit rows via PullAuditEvents) is absent for the S&F buffer itself.
5. Parked rows have unbounded retention. By design parked messages persist until operator action (Component-StoreAndForward.md:110-116), but there is no aging alert, cap, or archival — a forgotten site accumulates parked rows indefinitely with only a health-metric count.
6. Reconciliation cursor edge. PullAuditEvents/PullSiteCalls batch continuation relies on central advancing a since_utc cursor from the last returned timestamp (SiteStreamGrpcServer.cs:478-484, 563-572); rows sharing a boundary timestamp are protected only by EventId dedup / upsert-on-newer at central — fine, but the site-side read contract (> vs >=) isn't pinned by a test in this project.
7. DebugStreamService._sessions on central failover — sessions are process-local with no persistence (documented as acceptable: engineer re-establishes), but the SignalR consumer contract (OnStreamTerminated) is the only notification; a central restart drops sessions with no signal at all. Known/accepted, listed for completeness.

---

Prioritized Recommendations

1. [Critical] Fix ClusterClient recreation (CentralCommunicationActor.cs:514-521, DefaultSiteClientFactory): unique per-incarnation actor names (suffix a counter) or Terminated-watch before re-create; validate/sanitize siteId for actor-name safety; wrap Create in the per-site try/catch; add a test using the real factory that edits a site's address twice.
2. [High] Gate the S&F retry sweep to the active node — check leader/oldest status at the top of RetryPendingMessagesAsync (or gate StartAsync's timer), re-evaluated per tick so failover resumes delivery within one sweep interval. Add a two-node test.
3. [High] Stop disposing shared gRPC clients on per-session failover — key SiteStreamGrpcClientFactory by (site, endpoint) (allow both node channels concurrently), reserve dispose-and-replace for site removal/address edits driven by the address cache, not by individual bridge actors.
4. [High] Bound and de-serialize the retry sweep — LIMIT the due query, short-circuit per target after the first transient failure in a sweep, and consider a small parallelism cap across distinct targets; separately consider a much shorter forward timeout for the notification category or attemptImmediateDelivery: false on Notify.Send to unblock script threads.
5. [Medium] Make replication trustworthy: replace Task.Run fire-and-forget with an inline Tell (ordering), log failures at Warning with a counter metric, and make Park/Requeue applies upserts so a lost Add self-heals.
6. [Medium] Pin the wire format: configure an explicit serializer (or at least add serializer round-trip compatibility tests for the cross-cluster Commons contracts) so the additive-only rule is actually enforced end-to-end; long-term, stop converting proto DTOs back to POCOs for the ClusterClient hop.
7. [Medium] Unstrand notifications: pass maxRetries: 0 (documented no-limit) or a category-specific high cap on the Notify.Send enqueue path, or auto-requeue parked Notification-category rows when central connectivity recovers.
8. [Medium] Park (don't silently drop) corrupt notification payloads in NotificationForwarder.DeliverAsync — return false instead of true.
9. [Low] SQLite hygiene: enable WAL + busy_timeout in InitializeAsync; migrate timestamps to epoch ms for the due-check.
10. [Low] Observability polish: count DropOldest evictions in StreamRelayActor; replicate heartbeat marks to the peer central node (or document the failover blind window); split the app heartbeat interval from the transport failure-detector setting.