diff --git a/docs/structuregaps.md b/docs/structuregaps.md index baf8d30..a23dcc5 100644 --- a/docs/structuregaps.md +++ b/docs/structuregaps.md @@ -7,7 +7,7 @@ | Metric | Go Server | .NET Port | Ratio | |--------|-----------|-----------|-------| | Source lines | ~130K (109 files) | ~39.1K (215 files) | 3.3x | -| Test methods | 2,937 `func Test*` | 5,914 `[Fact]`/`[Theory]` (~6,532 with parameterized) | 2.0x | +| Test methods | 2,937 `func Test*` | 6,920 `[Fact]`/`[Theory]` (~7,500+ with parameterized) | 2.4x | | Go tests mapped | 2,646 / 2,937 (90.1%) | — | — | | .NET tests linked to Go | — | 4,250 / 5,808 (73.2%) | — | | Largest source file | filestore.go (12,593 lines) | NatsServer.cs (1,883 lines) | 6.7x | @@ -56,48 +56,29 @@ S2 compression integrated into block write/read path using `S2Codec`. Message-le Full crash recovery with block scanning, index rebuilding, TTL wheel recovery, lost-data tracking, and MaxAge enforcement on recovery. Tests: `FileStoreCrashRecoveryTests.cs`. -### 1.5 Checksum Validation (HIGH) +### 1.5 Checksum Validation — IMPLEMENTED -Go uses highway hash for per-message checksums and validates integrity on read. +**Implemented in:** `MsgBlock.cs`, `FileStore.cs` (Task 1, Phase 1) -**Missing:** -- `msgBlock.lchk[8]byte` — last checksum buffer -- `msgBlock.hh *highwayhash.Digest64` — highway hasher -- `msgBlock.lastChecksum()` (line 2204) — returns last computed checksum -- Message checksum validation during `msgFromBufEx()` (line ~8180) +Per-block last-checksum tracking and read-path validation using XxHash64. Validates message integrity on read with configurable `validateOnRead` flag. Tests: `FileStoreChecksumTests.cs`. -**.NET status**: `MessageRecord` has no checksum field. No integrity checks on read. +### 1.6 Atomic File Overwrites — IMPLEMENTED -### 1.6 Atomic File Overwrites (HIGH) +**Implemented in:** `AtomicFileWriter.cs`, `FileStore.cs` (Task 2, Phase 1) -Go uses temp file + rename for crash-safe state updates. +`AtomicFileWriter` with temp-file-plus-rename pattern and `SemaphoreSlim` write lock for crash-safe state persistence. Tests: `AtomicFileWriterTests.cs`. -**Missing:** -- `_writeFullState()` (line 10599) — atomic state serialization with temp file + rename -- State file write protection with mutual exclusion (`wfsmu`/`wfsrun`) +### 1.7 Tombstone & Deletion Tracking — IMPLEMENTED -**.NET status**: Writes directly without atomic guarantees. +**Implemented in:** `SequenceSet.cs`, `MsgBlock.cs`, `FileStore.cs` (Task 3, Phase 1) -### 1.7 Tombstone & Deletion Tracking (HIGH) +Sparse `SequenceSet` with range compression replacing `HashSet`, secure erase via `RandomNumberGenerator.Fill`, and tombstone record persistence with recovery during `RebuildIndex()`. Tests: `SequenceSetTests.cs`, `FileStoreTombstoneTrackingTests.cs`. -**Missing:** -- `removeMsg()` (line 5267) — removes message with optional secure erase -- `removeMsgFromBlock()` (line 5307) — removes from specific block -- `eraseMsg()` (line 5890) — overwrites deleted message with random data (secure erase) -- Sparse `avl.SequenceSet` for deletion tracking (Go uses AVL; .NET uses `HashSet`) -- Tombstone record persistence (deleted messages lost on crash in .NET) +### 1.8 Multi-Block Write Cache — IMPLEMENTED -### 1.8 Multi-Block Write Cache (MEDIUM) +**Implemented in:** `FileStore.cs` (Task 4, Phase 1) -**Missing Go functions:** -- `setupWriteCache()` (line 4443) — sets up write cache -- `finishedWithCache()` (line 4477) — marks cache complete -- `expireCache()` / `expireCacheLocked()` (line 6148+) — expires idle cache -- `resetCacheExpireTimer()` / `startCacheExpireTimer()` — manages cache timeout -- Elastic reference machinery (line 6704, `ecache.Strengthen()` / `WeakenAfter()`) -- `flushPendingMsgs()` / `flushPendingMsgsLocked()` (line 7560+) — background flush - -**.NET status**: Synchronous flush on rotation; no background flusher, no cache expiration. +`WriteCacheManager` inner class with `PeriodicTimer` background flusher, bounded cache by size (64MB default), TTL eviction, and integration with `StoreMsg` and `RotateBlock`. Tests: `WriteCacheTests.cs`. ### 1.9 Missing IStreamStore Interface Methods — IMPLEMENTED @@ -105,15 +86,11 @@ Go uses temp file + rename for crash-safe state updates. All missing interface methods added to `IStreamStore` and implemented in both `FileStore` and `MemStore`: `StoreRawMsg`, `LoadNextMsgMulti`, `LoadPrevMsg`, `LoadPrevMsgMulti`, `NumPendingMulti`, `SyncDeleted`, `EncodedStreamState`, `Utilization`, `FlushAllPending`, `RegisterStorageUpdates`, `RegisterStorageRemoveMsg`, `RegisterProcessJetStreamMsg`, `ResetState`, `UpdateConfig`, `Delete(bool)`, `Stop()`. Tests: `StreamStoreInterfaceTests.cs`. -### 1.10 Query/Filter Operations (MEDIUM) +### 1.10 Query/Filter Operations — IMPLEMENTED -Go uses trie-based subject indexing (`SubjectTree[T]`) for O(log n) lookups. .NET uses linear LINQ scans. +**Implemented in:** `FileStore.cs` (Task 5, Phase 1) -**Missing:** -- `FilteredState()` (line 3191) — optimized filtered state with caching -- `checkSkipFirstBlock()` family (lines 3241–3287) -- `numFiltered*()` family (lines 3308–3413) -- `LoadMsg()` (line 8308) — block-aware message loading +Optimized `FilteredState` with wildcard subjects, block-aware binary search for `LoadMsg`, `CheckSkipFirstBlock` optimization for range queries, and `NumFiltered` with caching using `SubjectMatch.IsMatch()`. Tests: `FileStoreFilterQueryTests.cs`. --- @@ -145,14 +122,11 @@ Stream and consumer assignment processing including create, update, delete, and Inflight stream and consumer proposal tracking with ops count, deleted flag, and assignment capture. Tests: `InflightDedupTests.cs`. -### 2.4 Peer Management & Stream Moves (HIGH) +### 2.4 Peer Management & Stream Moves — IMPLEMENTED -**Missing:** -- `processAddPeer()` (lines 2290–2340) -- `processRemovePeer()` (lines 2342–2393) -- `removePeerFromStream()` / `removePeerFromStreamLocked()` (lines 2396–2439) -- `remapStreamAssignment()` (lines 7077–7111) -- Stream move operations in `jsClusteredStreamUpdateRequest()` (lines 7757+) +**Implemented in:** `RaftMembership.cs`, `RaftPeerState.cs`, `JetStreamMetaGroup.cs` (Task 12, Phase 2) + +Peer add/remove processing, stream peer removal with remap, and stream move operations. Tests: `PeerManagementTests.cs`. ### 2.5 Leadership Transition — IMPLEMENTED @@ -166,55 +140,41 @@ Meta-level, stream-level, and consumer-level leader change processing with step- Meta snapshot encode/decode with S2 compression, apply snapshot with delta computation, and stream/consumer change collection. Tests: `SnapshotRecoveryTests.cs`. -### 2.7 Entry Application Pipeline (HIGH) +### 2.7 Entry Application Pipeline — IMPLEMENTED -**Missing:** -- `applyMetaEntries()` (lines 2474–2609, 136 lines) — handles all entry types -- `applyStreamEntries()` (lines 3645–4067, 423 lines) — per-stream entries -- `applyStreamMsgOp()` (lines 4068–4261, 194 lines) — per-message ops -- `applyConsumerEntries()` (lines 6351–6621, 271 lines) — per-consumer entries +**Implemented in:** `CommitQueue.cs`, `RaftReplicator.cs`, `JetStreamMetaGroup.cs` (Task 13, Phase 2) -### 2.8 Topology-Aware Placement (MEDIUM) +Meta, stream, and consumer entry application with per-message ops. Tests: `EntryApplicationTests.cs`. -**Missing from PlacementEngine (currently 80 lines vs Go's 312 lines for `selectPeerGroup()`):** -- Unique tag enforcement (`JetStreamUniqueTag`) -- HA asset limits per peer -- Dynamic available storage calculation -- Tag inclusion/exclusion with prefix handling -- Weighted node selection based on availability -- Mixed-mode detection -- `tieredStreamAndReservationCount()` (lines 7524–7546) -- `createGroupForConsumer()` (lines 8783–8923, 141 lines) -- `jsClusteredStreamLimitsCheck()` (lines 7599–7618) +### 2.8 Topology-Aware Placement — IMPLEMENTED -### 2.9 RAFT Group Creation & Lifecycle (MEDIUM) +**Implemented in:** `PlacementEngine.cs`, `AssetPlacementPlanner.cs` (Task 14, Phase 2) -**Missing:** -- `createRaftGroup()` (lines 2659–2789, 131 lines) -- `raftGroup.isMember()` (lines 2611–2621) -- `raftGroup.setPreferred()` (lines 2623–2656) +Unique tag enforcement, HA asset limits, dynamic storage calculation, tag inclusion/exclusion, weighted selection, mixed-mode detection, and consumer group creation. Tests: `TopologyPlacementTests.cs`. -### 2.10 Assignment Encoding/Decoding (MEDIUM) +### 2.9 RAFT Group Creation & Lifecycle — IMPLEMENTED -**Missing (no serialization for RAFT persistence):** -- `encodeAddStreamAssignment()`, `encodeUpdateStreamAssignment()`, `encodeDeleteStreamAssignment()` -- `decodeStreamAssignment()`, `decodeStreamAssignmentConfig()` -- `encodeAddConsumerAssignment()`, `encodeDeleteConsumerAssignment()` -- `encodeAddConsumerAssignmentCompressed()`, `decodeConsumerAssignment()` -- `decodeConsumerAssignmentCompressed()`, `decodeConsumerAssignmentConfig()` +**Implemented in:** `StreamReplicaGroup.cs`, `RaftGroup` (Task 15, Phase 2) -### 2.11 Unsupported Asset Handling (LOW) +RAFT group creation, `isMember`, `setPreferred`, and lifecycle management. Tests: `RaftGroupLifecycleTests.cs`. -**Missing:** -- `unsupportedStreamAssignment` type (lines 186–247) — graceful handling of version-incompatible streams -- `unsupportedConsumerAssignment` type (lines 268–330) +### 2.10 Assignment Encoding/Decoding — IMPLEMENTED -### 2.12 Clustered API Handlers (MEDIUM) +**Implemented in:** `AssignmentCodec.cs` (Task 16, Phase 2) -**Missing:** -- `jsClusteredStreamRequest()` (lines 7620–7701, 82 lines) -- `jsClusteredStreamUpdateRequest()` (lines 7757–8265, 509 lines) -- System-level subscriptions for result processing, peer removal, leadership step-down +Stream and consumer assignment encode/decode with S2 compression support for RAFT persistence. Tests: `AssignmentSerializationTests.cs`. + +### 2.11 Unsupported Asset Handling — IMPLEMENTED + +**Implemented in:** `JetStreamMetaGroup.cs` (Task 17, Phase 2) + +Graceful handling of version-incompatible stream and consumer assignments with version checking. Tests: `UnsupportedAssetTests.cs`. + +### 2.12 Clustered API Handlers — IMPLEMENTED + +**Implemented in:** `ClusteredRequestProcessor.cs` (Task 18, Phase 2) + +Clustered stream create/update/delete request handlers with result processing subscriptions. Tests: `ClusteredApiTests.cs`. --- @@ -225,14 +185,11 @@ Meta snapshot encode/decode with S2 compression, apply snapshot with delta compu **Current .NET total**: ~1,292 lines **Gap factor**: 5.2x -### 3.1 Core Message Delivery Loop (CRITICAL) +### 3.1 Core Message Delivery Loop — IMPLEMENTED -**Missing**: `loopAndGatherMsgs()` (Go lines ~1400–1700) — the main consumer dispatch loop that: -- Polls stream store for new messages matching filter -- Applies redelivery logic -- Handles num_pending calculations -- Manages delivery interest tracking -- Responds to stream updates +**Implemented in:** `PushConsumerEngine.cs` (Task 25, Phase 3) + +Main consumer dispatch loop with stream store polling, filter matching, redelivery logic, num_pending calculations, delivery interest tracking, and stream update response. Tests: `DeliveryLoopTests.cs`. ### 3.2 Pull Request Pipeline — IMPLEMENTED @@ -252,12 +209,11 @@ Background ack processing with `+ACK`, `-NAK`, `+TERM`, `+WPI` frame parsing, ha Min-heap/priority queue redelivery with deadline ordering, batch dispatch, per-sequence redelivery state, and rate limiting. Tests: `RedeliverySchedulerTests.cs`. -### 3.5 Idle Heartbeat & Flow Control (HIGH) +### 3.5 Idle Heartbeat & Flow Control — IMPLEMENTED -**Missing**: -- `sendIdleHeartbeat()` (Go line ~5222) — heartbeat with pending counts and stall headers -- `sendFlowControl()` (Go line ~5495) — dedicated flow control handler -- Flow control reply generation with pending counts +**Implemented in:** `PushConsumerEngine.cs` (Task 26, Phase 3) + +Idle heartbeat with pending counts and stall headers, dedicated flow control handler, and flow control reply generation. Tests: `IdleHeartbeatTests.cs`. ### 3.6 Priority Group Pinning — IMPLEMENTED @@ -271,36 +227,41 @@ Pin ID generation and assignment, pin timeout timers, `Nats-Pin-Id` header respo PauseUntil deadline tracking, pause expiry timer, pause/resume advisory generation, and pause state in consumer info response. Tests: `PauseResumeTests.cs`. -### 3.8 Delivery Interest Tracking (HIGH) +### 3.8 Delivery Interest Tracking — IMPLEMENTED -**Missing**: Dynamic delivery interest monitoring: -- Subject interest change tracking -- Gateway interest checks -- Subscribe/unsubscribe tracking for delivery subject -- Interest-driven consumer cleanup (`deleteNotActive`) +**Implemented in:** `DeliveryInterestTracker.cs` (Task 27, Phase 3) -### 3.9 Max Deliveries Enforcement (MEDIUM) +Dynamic delivery interest monitoring with subject interest change tracking, gateway interest checks, subscribe/unsubscribe tracking, and interest-driven consumer cleanup. Tests: `DeliveryInterestTests.cs`. -`AckProcessor` checks basic threshold but missing: -- Advisory event generation on exceeding max delivers -- Per-delivery-count policy selection -- `NotifyDeliveryExceeded` advisory +### 3.9 Max Deliveries Enforcement — IMPLEMENTED -### 3.10 Filter Subject Skip Tracking (MEDIUM) +**Implemented in:** `RedeliveryTracker.cs`, `AckProcessor.cs` (Task 28, Phase 3) -**Missing**: Efficient filter matching with compiled regex, skip list tracking, `UpdateSkipped` state updates. +Advisory event generation on exceeding max delivers, per-delivery-count policy selection, and `NotifyDeliveryExceeded` advisory. Tests: `MaxDeliveriesTests.cs`. -### 3.11 Sample/Observe Mode (MEDIUM) +### 3.10 Filter Subject Skip Tracking — IMPLEMENTED -**Missing**: Sample frequency parsing (`"1%"`), stochastic sampling, latency measurement, latency advisory generation. +**Implemented in:** `FilterSkipTracker.cs` (Task 29, Phase 3) -### 3.12 Reset to Sequence (MEDIUM) +Efficient filter matching with compiled regex, skip list tracking, and `UpdateSkipped` state updates. Tests: `FilterSkipTests.cs`. -**Missing**: `processResetReq()` (Go line ~4241) — consumer state reset to specific sequence. +### 3.11 Sample/Observe Mode — IMPLEMENTED -### 3.13 Rate Limiting (MEDIUM) +**Implemented in:** `SampleTracker.cs` (Task 30, Phase 3) -`PushConsumerEngine` has basic rate limiting but missing accurate token bucket, dynamic updates, per-message delay calculation. +Sample frequency parsing, stochastic sampling, latency measurement, and latency advisory generation. Tests: `SampleModeTests.cs`. + +### 3.12 Reset to Sequence — IMPLEMENTED + +**Implemented in:** `PushConsumerEngine.cs`, `PullConsumerEngine.cs` (Task 31, Phase 3) + +Consumer state reset to specific sequence via `processResetReq()`. Tests: `ConsumerResetTests.cs`. + +### 3.13 Rate Limiting — IMPLEMENTED + +**Implemented in:** `TokenBucketRateLimiter.cs` (Task 32, Phase 3) + +Accurate token bucket with dynamic updates and per-message delay calculation. Tests: `TokenBucketTests.cs`. ### 3.14 Cluster-Aware Pending Requests — IMPLEMENTED @@ -497,32 +458,41 @@ SemaphoreSlim-based per-subscription flow control with `TryAcquireAsync`/`Acquir **NET implementation**: ~1,374 lines across 11 handler files **Gap factor**: 5.8x -### 7.1 Leader Forwarding (HIGH) +### 7.1 Leader Forwarding — IMPLEMENTED -API requests arriving at non-leader nodes must be forwarded to the current leader. Not implemented. +**Implemented in:** `ClusteredRequestProcessor.cs` (Task 19, Phase 2) -### 7.2 Clustered API Handlers (HIGH) +API request forwarding from non-leader nodes to current leader. Tests: `LeaderForwardingTests.cs`. -**Missing**: -- `jsClusteredStreamRequest()` — stream create in clustered mode -- `jsClusteredStreamUpdateRequest()` — update/delete/move/scale (509 lines in Go) -- Cluster-aware consumer create/delete handlers +### 7.2 Clustered API Handlers — IMPLEMENTED -### 7.3 API Rate Limiting & Deduplication (MEDIUM) +**Implemented in:** `ClusteredRequestProcessor.cs` (Task 20, Phase 2) -No request deduplication or rate limiting. +Clustered stream create/update/delete, consumer create/delete handlers. Tests: `ClusteredApiTests.cs`. -### 7.4 Snapshot & Restore API (MEDIUM) +### 7.3 API Rate Limiting & Deduplication — IMPLEMENTED -No snapshot/restore API endpoints. +**Implemented in:** `ApiRateLimiter.cs` (Task 21, Phase 2) -### 7.5 Consumer Pause/Resume API (MEDIUM) +Request deduplication and rate limiting for JetStream API. Tests: `ApiRateLimiterTests.cs`. -No consumer pause/resume API endpoint. +### 7.4 Snapshot & Restore API — IMPLEMENTED -### 7.6 Advisory Event Publication (LOW) +**Implemented in:** `StreamSnapshotService.cs` (Task 22, Phase 2) -No advisory events for API operations. +Snapshot and restore API endpoints. Tests: `SnapshotApiTests.cs`. + +### 7.5 Consumer Pause/Resume API — IMPLEMENTED + +**Implemented in:** `ConsumerApiHandlers.cs` (Task 23, Phase 2) + +Consumer pause/resume API endpoint. Tests: `ConsumerPauseApiTests.cs`. + +### 7.6 Advisory Event Publication — IMPLEMENTED + +**Implemented in:** `AdvisoryPublisher.cs` (Task 24, Phase 2) + +Advisory events for API operations (stream create/delete/update, consumer create/delete). Tests: `AdvisoryEventTests.cs`. --- @@ -544,29 +514,41 @@ Binary WAL with header, CRC32 integrity, append/sync/compact/load operations. Cr Two-phase joint consensus per RAFT paper Section 4: `BeginJointConsensus`, `CommitJointConsensus`, `CalculateJointQuorum` requiring majority from both Cold and Cnew. Tests: `RaftJointConsensusTests.cs`. -### 8.3 InstallSnapshot Streaming (HIGH) +### 8.3 InstallSnapshot Streaming — IMPLEMENTED -`RaftSnapshot` is fully loaded into memory. Go streams snapshots in chunks with progress tracking. +**Implemented in:** `RaftSnapshot.cs`, `RaftSnapshotStore.cs`, `SnapshotChunkEnumerator.cs` (Task 6, Phase 1) -### 8.4 Leadership Transfer (MEDIUM) +Chunk-based snapshot streaming with CRC32 validation and progress tracking. Tests: `RaftSnapshotStreamingTests.cs`. -No graceful leader step-down mechanism for maintenance operations. +### 8.4 Leadership Transfer — IMPLEMENTED -### 8.5 Log Compaction (MEDIUM) +**Implemented in:** `RaftNode.cs` (Task 7, Phase 1) -Basic compaction exists but no configurable retention policies. +Graceful leader step-down mechanism for maintenance operations. Tests: `RaftLeadershipTransferTests.cs`. -### 8.6 Quorum Check Before Proposing (MEDIUM) +### 8.5 Log Compaction — IMPLEMENTED -May propose entries when quorum is unreachable. +**Implemented in:** `CompactionPolicy.cs`, `RaftNode.cs` (Task 8, Phase 1) -### 8.7 Read-Only Query Optimization (LOW) +Configurable retention policies for log compaction. Tests: `RaftCompactionPolicyTests.cs`. -All queries append log entries; no ReadIndex optimization for read-only commands. +### 8.6 Quorum Check Before Proposing — IMPLEMENTED -### 8.8 Election Timeout Jitter (LOW) +**Implemented in:** `RaftNode.cs` (Task 9, Phase 1) -May not have sufficient randomized jitter to prevent split votes. +Pre-proposal quorum reachability check to avoid proposing when quorum is unreachable. Tests: `RaftQuorumCheckTests.cs`. + +### 8.7 Read-Only Query Optimization — IMPLEMENTED + +**Implemented in:** `RaftNode.cs` (Task 10, Phase 1) + +ReadIndex optimization for read-only commands without appending log entries. Tests: `RaftReadIndexTests.cs`. + +### 8.8 Election Timeout Jitter — IMPLEMENTED + +**Implemented in:** `RaftNode.cs`, `RaftTermState.cs` (Task 11, Phase 1) + +Randomized election timeout jitter to prevent split votes. Tests: `RaftElectionJitterTests.cs`. --- @@ -839,7 +821,7 @@ Implemented in: `WebSocketTlsConfig.cs` (Task 93). Tests: `WebSocketTlsTests.cs` ### Tier 2: Capability Limiters (HIGH) — ALL IMPLEMENTED -6. **Consumer delivery loop** (Gap 3.1) — remaining (core message dispatch loop) +6. ~~**Consumer delivery loop** (Gap 3.1)~~ — IMPLEMENTED (Task 25) 7. ~~**Consumer pull request pipeline** (Gap 3.2)~~ — IMPLEMENTED (Task 16) 8. ~~**Consumer ack/redelivery** (Gaps 3.3–3.4)~~ — IMPLEMENTED (Tasks 14–15) 9. ~~**Stream mirror/source retry** (Gaps 4.1–4.2, 4.9)~~ — IMPLEMENTED (Task 21) @@ -879,8 +861,8 @@ All 2,937 Go tests have been categorized: - **19 skipped** (0.6%) — intentionally deferred - **0 unmapped** — none remaining -The .NET test suite has **5,914 test methods** (expanding to ~6,532 with parameterized `[Theory]` test cases). Of these, **4,250** (73.2%) are linked back to Go test counterparts via the `test_mappings` table. The remaining tests are .NET-specific — covering implementations (SubjectTree, ConfigProcessor, NatsConfLexer, RAFT WAL, joint consensus, etc.) that don't have 1:1 Go counterparts. +The .NET test suite has **6,920 test methods** (expanding to ~7,500+ with parameterized `[Theory]` test cases). Of these, **4,250** (61.4%) are linked back to Go test counterparts via the `test_mappings` table. The remaining tests are .NET-specific — covering implementations (SubjectTree, ConfigProcessor, NatsConfLexer, RAFT WAL, joint consensus, etc.) that don't have 1:1 Go counterparts. -The production gaps implementation (2026-02-25/26) added **106 new test methods** across 10 new test files covering FileStore block management, RAFT persistence, cluster coordination, consumer delivery, stream lifecycle, client protocol, MQTT persistence, config reload, and route/gateway discovery. +The production and remaining gaps implementations (2026-02-25/26) added **~1,006 new test methods** across 93 new test files covering all 15 gap subsystems: FileStore, RAFT, JetStream cluster, API, consumer engines, stream lifecycle, client protocol, MQTT, account management, monitoring, events, gateway, route, leaf node, and WebSocket. Many tests validate API contracts against stubs/mocks rather than testing actual distributed behavior. The gap is in test *depth*, not test *count*.