docs(store-and-forward): replication text is LocalDb CDC; document lane paging + gauge reseed (review 2026-10-21 02/U4a)

docs/requirements/Component-StoreAndForward.md

@@ -19,8 +19,8 @@ Site clusters only. The central cluster does not buffer messages.
 - Park messages that exhaust their retry limit (dead-letter).
 - Persist buffered messages to local SQLite for durability.
 - Emit cached-call lifecycle telemetry to the central Site Call Audit component via the `ICachedCallLifecycleObserver` hook (one notification per attempt outcome) so the audit pipeline can record each status transition.
-- Replicate buffered messages to the standby node via application-level replication over Akka.NET remoting.
-- On failover, the standby node takes over delivery from its replicated copy.
+- Buffered messages are replicated to the standby node as rows of the LocalDb-replicated `sf_messages` table (CDC over the LocalDb gRPC sync stream).
+- On failover, the standby node takes over delivery from its replicated copy of `sf_messages`.
 - Respond to remote queries from central for parked message management (list, retry, discard), including central-driven Retry/Discard of parked cached calls.
 
 ## Message Lifecycle
@@ -34,7 +34,7 @@ flowchart TD
     C -->|Failure| E[Buffer message]
     E --> F[Retry loop<br/>per retry policy]
     F --> G{Retry outcome}
-    G -->|Success| H([Remove from buffer<br/>+ notify standby])
+    G -->|Success| H([Remove from buffer<br/>+ CDC to standby])
     G -->|Max retries exhausted| I([Park message<br/>dead-letter])
 
     classDef ok fill:#d5e8d4,stroke:#82b366,color:#111111;
@@ -89,9 +89,9 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
 ## Persistence
 
 - Buffered messages are persisted to a **local SQLite database** on each site node.
-- The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
-- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
-- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
+- The active node persists locally; every buffer mutation (add, remove, park) is a row change on `sf_messages`, which LocalDb CDC captures and ships to the standby **asynchronously**. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
+- The standby node receives those row changes into its own copy of the table but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby's copy is kept warm purely for a future failover.
+- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent changes may not have replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its status change not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
 - On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
 - **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
 
@@ -123,9 +123,13 @@ On every tracking-table status transition, the site emits a `CachedCallTelemetry
 
 The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel<T>` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry.
 
+### Queue-depth gauge
+
+`scadabridge.store_and_forward.queue.depth` reads a cached counter adjusted by this node's own enqueue/deliver/park paths. Rows that arrive via LocalDb CDC (standby) or change on the peer never touch it, so every 6th retry sweep (including the first) re-seeds it from the database's Pending count; the reported value is clamped at 0 (review 2026-10-21 02/C3).
+
 ### Retry Sweep Indexing
 
-`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
+`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. The sweep page is ranked **per `(category, target)` lane** (`ROW_NUMBER() OVER (PARTITION BY category, target ORDER BY created_at)`, ordered by lane rank then `created_at`, limited to `SweepBatchLimit`), so a dead target with hundreds of due rows contributes one row per round like every other lane instead of owning the whole page and starving healthy targets (review 2026-10-21 02/C2); per-target FIFO is preserved. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
 
 ## Parked Message Management
 
@@ -153,7 +157,7 @@ Each buffered message stores:
 ## Dependencies
 
 - **SQLite**: Local persistence on each node.
-- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks.
+- **Communication Layer**: remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks. (Standby replication is not carried by Communication — it is LocalDb CDC.)
 - **External System Gateway**: Delivers external system API calls.
 - **Central–Site Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over Central–Site Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
 - **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.