Compare commits

...

8 Commits

Author SHA1 Message Date
Joseph Doherty a1fc600d84 docs: add optimization planning documents 2026-03-13 10:19:56 -04:00
Joseph Doherty fb0d31c615 docs: refresh benchmark comparison after SubList optimization 2026-03-13 10:18:52 -04:00
Joseph Doherty 900a4b0923 Merge branch 'codex/sublist-allocation-reduction' 2026-03-13 10:15:46 -04:00
Joseph Doherty d1f22255d7 docs: record SubList allocation strategy 2026-03-13 10:08:50 -04:00
Joseph Doherty 0126234fa6 perf: pool SubList match builders and cleanup scans 2026-03-13 10:06:24 -04:00
Joseph Doherty 5876ad7dfa perf: reduce SubList token string churn 2026-03-13 09:53:37 -04:00
Joseph Doherty 348bec36b2 perf: replace SubList routed-sub string keys 2026-03-13 09:51:11 -04:00
Joseph Doherty 08bd34c529 test: lock SubList remote-key and match behavior 2026-03-13 09:49:54 -04:00
14 changed files with 1794 additions and 317 deletions
+115 -170
View File
@@ -1,6 +1,6 @@
# SubList # SubList
`SubList` is the subscription routing trie. Every published message triggers a `Match()` call to find all interested subscribers. `SubList` stores subscriptions indexed by their subject tokens and returns a `SubListResult` containing both plain subscribers and queue groups. `SubList` is the subscription routing trie for the core server. Every publish path calls `Match()` to find the local plain subscribers and queue groups interested in a subject. The type also tracks remote route and gateway interest so clustering code can answer `HasRemoteInterest(...)` and `MatchRemote(...)` queries without a second routing structure.
Go reference: `golang/nats-server/server/sublist.go` Go reference: `golang/nats-server/server/sublist.go`
@@ -8,32 +8,30 @@ Go reference: `golang/nats-server/server/sublist.go`
## Thread Safety ## Thread Safety
`SubList` uses a `ReaderWriterLockSlim` (`_lock`) with the following locking discipline: `SubList` uses a single `ReaderWriterLockSlim` (`_lock`) to protect trie mutation, remote-interest bookkeeping, and cache state.
| Operation | Lock | | Operation | Lock |
|-----------|------| |-----------|------|
| `Count` read | Read lock | | Cache hit in `Match()` | Read lock |
| `Match()` — cache hit | Read lock only | | Cache miss in `Match()` | Write lock |
| `Match()` — cache miss | Write lock (to update cache) | | `Insert()` / `Remove()` / `RemoveBatch()` | Write lock |
| `Insert()` | Write lock | | Remote-interest mutation (`ApplyRemoteSub`, `UpdateRemoteQSub`, cleanup) | Write lock |
| `Remove()` | Write lock | | Read-only queries (`Count`, `HasRemoteInterest`, `MatchRemote`, `Stats`) | Read lock |
Cache misses in `Match()` require a write lock because the cache must be updated after the trie traversal. To avoid a race between the read-lock check and the write-lock update, `Match()` uses double-checked locking: after acquiring the write lock, it checks the cache again before doing trie work. `Match()` uses generation-based double-checked locking. It first checks the cache under a read lock, then retries under the write lock before traversing the trie and updating the cache.
--- ---
## Trie Structure ## Trie Structure
The trie is built from two private classes, `TrieLevel` and `TrieNode`, nested inside `SubList`. The trie is built from `TrieLevel` and `TrieNode`:
### `TrieLevel` and `TrieNode`
```csharp ```csharp
private sealed class TrieLevel private sealed class TrieLevel
{ {
public readonly Dictionary<string, TrieNode> Nodes = new(StringComparer.Ordinal); public readonly Dictionary<string, TrieNode> Nodes = new(StringComparer.Ordinal);
public TrieNode? Pwc; // partial wildcard (*) public TrieNode? Pwc;
public TrieNode? Fwc; // full wildcard (>) public TrieNode? Fwc;
} }
private sealed class TrieNode private sealed class TrieNode
@@ -41,202 +39,149 @@ private sealed class TrieNode
public TrieLevel? Next; public TrieLevel? Next;
public readonly HashSet<Subscription> PlainSubs = []; public readonly HashSet<Subscription> PlainSubs = [];
public readonly Dictionary<string, HashSet<Subscription>> QueueSubs = new(StringComparer.Ordinal); public readonly Dictionary<string, HashSet<Subscription>> QueueSubs = new(StringComparer.Ordinal);
public bool PackedListEnabled;
public bool IsEmpty => PlainSubs.Count == 0 && QueueSubs.Count == 0 &&
(Next == null || (Next.Nodes.Count == 0 && Next.Pwc == null && Next.Fwc == null));
} }
``` ```
Each level in the trie represents one token position in a subject. A `TrieLevel` holds: - `Nodes` stores literal-token edges by exact token string.
- `Pwc` stores the `*` edge for the current token position.
- `Fwc` stores the `>` edge for the current token position.
- `PlainSubs` stores non-queue subscriptions attached to the terminal node.
- `QueueSubs` groups queue subscriptions by queue name at the terminal node.
- `Nodes` — a dictionary keyed by literal token string, mapping to the `TrieNode` for that token. Uses `StringComparer.Ordinal` for performance. The root of the trie is `_root`, a `TrieLevel` with no parent node.
- `Pwc` — the node for the `*` wildcard at this level, or `null` if no `*` subscriptions exist at this depth.
- `Fwc` — the node for the `>` wildcard at this level, or `null` if no `>` subscriptions exist at this depth.
A `TrieNode` sits at the boundary between two levels. It holds the subscriptions registered for subjects whose last token leads to this node: ---
- `PlainSubs` — a `HashSet<Subscription>` of plain (non-queue) subscribers. ## Token Traversal
- `QueueSubs` — a dictionary from queue name to the set of members in that queue group. Uses `StringComparer.Ordinal`.
- `Next` — the next `TrieLevel` for deeper token positions. `null` for leaf nodes.
- `IsEmpty``true` when the node and all its descendants have no subscriptions. Used during `Remove()` to prune dead branches.
The trie root is a `TrieLevel` (`_root`) with no parent node. `TokenEnumerator` walks a subject string token-by-token using `ReadOnlySpan<char>` slices, so traversal itself does not allocate. Literal-token insert and remove paths use `TryGetLiteralNode(...)` plus `SubjectMatch.TokenEquals(...)` to reuse the existing trie key string when the token is already present, instead of calling `token.ToString()` on every hop.
### `TokenEnumerator` That keeps literal-subject maintenance allocation-lean while preserving the current `Dictionary<string, TrieNode>` storage model.
`TokenEnumerator` is a `ref struct` that splits a subject string by `.` without allocating. It operates on a `ReadOnlySpan<char>` derived from the original string. ---
## Local Subscription Operations
### Insert
`Insert(Subscription sub)` walks the trie one token at a time:
- `*` follows or creates `Pwc`
- `>` follows or creates `Fwc` and terminates further token traversal
- literal tokens follow or create `Nodes[token]`
The terminal node stores the subscription in either `PlainSubs` or the appropriate queue-group bucket in `QueueSubs`. Every successful insert increments `_generation`, which invalidates cached match results.
### Remove
`Remove(Subscription sub)` and `RemoveBatch(IEnumerable<Subscription>)` walk the same subject path, remove the subscription from the terminal node, and then prune empty trie nodes on the way back out. Removing a subscription also bumps `_generation`, so any stale cached result is ignored on the next lookup.
---
## Remote Interest Bookkeeping
Remote route and gateway subscriptions are stored separately from the local trie in `_remoteSubs`:
```csharp ```csharp
private ref struct TokenEnumerator private readonly Dictionary<RoutedSubKey, RemoteSubscription> _remoteSubs = [];
{
private ReadOnlySpan<char> _remaining;
public TokenEnumerator(string subject)
{
_remaining = subject.AsSpan();
Current = default;
}
public ReadOnlySpan<char> Current { get; private set; }
public TokenEnumerator GetEnumerator() => this;
public bool MoveNext()
{
if (_remaining.IsEmpty)
return false;
int sep = _remaining.IndexOf(SubjectMatch.Sep);
if (sep < 0)
{
Current = _remaining;
_remaining = default;
}
else
{
Current = _remaining[..sep];
_remaining = _remaining[(sep + 1)..];
}
return true;
}
}
``` ```
`TokenEnumerator` implements the `foreach` pattern directly (via `GetEnumerator()` returning `this`), so it can be used in `foreach` loops without boxing. `Insert()` uses it during trie traversal to avoid string allocations per token. `RoutedSubKey` is a compact value key:
---
## Insert
`Insert(Subscription sub)` adds a subscription to the trie under a write lock.
The method walks the trie one token at a time using `TokenEnumerator`. For each token:
- If the token is `*`, it creates or follows `level.Pwc`.
- If the token is `>`, it creates or follows `level.Fwc` and sets `sawFwc = true` to reject further tokens.
- Otherwise it creates or follows `level.Nodes[token]`.
At each step, `node.Next` is created if absent, and `level` advances to `node.Next`.
After all tokens are consumed, the subscription is added to the terminal node:
- Plain subscription: `node.PlainSubs.Add(sub)`.
- Queue subscription: `node.QueueSubs[sub.Queue].Add(sub)`, creating the inner `HashSet<Subscription>` if this is the first member of that group.
`_count` is incremented and `AddToCache` is called to update any cached results that would now include this subscription.
---
## Remove
`Remove(Subscription sub)` removes a subscription from the trie under a write lock.
The method walks the trie along the subscription's subject, recording the path as a `List<(TrieLevel, TrieNode, string token, bool isPwc, bool isFwc)>`. If any node along the path is missing, the method returns without error (the subscription was never inserted).
After locating the terminal node, the subscription is removed from `PlainSubs` or from the appropriate `QueueSubs` group. If the queue group becomes empty, its entry is removed from the dictionary.
If removal succeeds:
- `_count` is decremented.
- `RemoveFromCache` is called to invalidate affected cache entries.
- The path list is walked backwards. At each step, if `node.IsEmpty` is `true`, the node is removed from its parent level (`Pwc = null`, `Fwc = null`, or `Nodes.Remove(token)`). This prunes dead branches so the trie does not accumulate empty nodes over time.
---
## Match
`Match(string subject)` is called for every published message. It returns a `SubListResult` containing all matching plain and queue subscriptions.
### Cache check and fallback
```csharp ```csharp
public SubListResult Match(string subject) internal readonly record struct RoutedSubKey(
{ string RouteId,
// Check cache under read lock first. string Account,
_lock.EnterReadLock(); string Subject,
try string? Queue);
{
if (_cache != null && _cache.TryGetValue(subject, out var cached))
return cached;
}
finally
{
_lock.ExitReadLock();
}
// Cache miss -- tokenize and match under write lock (needed for cache update).
var tokens = Tokenize(subject);
if (tokens == null)
return SubListResult.Empty;
_lock.EnterWriteLock();
try
{
// Re-check cache after acquiring write lock.
if (_cache != null && _cache.TryGetValue(subject, out var cached))
return cached;
var plainSubs = new List<Subscription>();
var queueSubs = new List<List<Subscription>>();
MatchLevel(_root, tokens, 0, plainSubs, queueSubs);
...
if (_cache != null)
{
_cache[subject] = result;
if (_cache.Count > CacheMax) { /* sweep */ }
}
return result;
}
finally { _lock.ExitWriteLock(); }
}
``` ```
On a read-lock cache hit, `Match()` returns immediately with no trie traversal. On a miss, `Tokenize()` splits the subject before acquiring the write lock (subjects with empty tokens return `SubListResult.Empty` immediately). The write lock is then taken and the cache is checked again before invoking `MatchLevel`. This replaces the earlier `"route|account|subject|queue"` composite string model. The change removes repeated string concatenation, `Split('|')`, and runtime reparsing in remote cleanup paths.
### `MatchLevel` traversal Remote-interest APIs:
`MatchLevel` is a recursive method that descends the trie matching tokens against the subject array. At each level, for each remaining token position: - `ApplyRemoteSub(...)` inserts or removes a `RemoteSubscription`
- `UpdateRemoteQSub(...)` updates queue weight for an existing remote queue subscription
- `RemoveRemoteSubs(routeId)` removes all remote interest for a disconnected route
- `RemoveRemoteSubsForAccount(routeId, account)` removes only one route/account slice
- `HasRemoteInterest(account, subject)` answers whether any remote subscription matches
- `MatchRemote(account, subject)` returns the expanded weighted remote matches
1. If `level.Fwc` is set, all subscriptions from that node are added to the result. The `>` wildcard matches all remaining tokens, so no further recursion is needed for this branch. Cleanup paths collect matching `RoutedSubKey` values into a reusable per-thread list and then remove them, avoiding `_remoteSubs.ToArray()` snapshots on every sweep.
2. If `level.Pwc` is set, `MatchLevel` recurses with the next token index and `pwc.Next` as the new level. This handles `*` matching the current token.
3. A literal dictionary lookup on `level.Nodes[tokens[i]]` advances the level pointer for the next iteration.
After all tokens are consumed, subscriptions from the final literal node and the final `*` position (if present at the last level) are added to the result. The `*` case at the last token requires explicit handling because the loop exits before the recursive call for `*` can execute. ---
`AddNodeToResults` flattens a node's `PlainSubs` into the accumulator list and merges its `QueueSubs` groups into the queue accumulator, combining groups by name across multiple matching nodes. ## Match Pipeline
`Match(string subject)` is the hot path.
1. Increment `_matches`
2. Read the current `_generation`
3. Try the cache under a read lock
4. On cache miss, tokenize the subject and retry under the write lock
5. Traverse the trie and build a `SubListResult`
6. Cache the result with the generation that produced it
Cached entries are stored as:
```csharp
private readonly record struct CachedResult(SubListResult Result, long Generation);
```
A cache entry is valid only if its stored generation matches the current `_generation`. Any local or remote-interest mutation increments `_generation`, so stale entries are ignored automatically.
### Match Builder
Cache misses use a reusable per-thread `MatchBuilder` instead of allocating fresh nested `List<List<Subscription>>` structures on every traversal. The builder:
- reuses a `List<Subscription>` for plain subscribers
- reuses queue-group lists across matches
- merges queue matches by queue name during traversal
- materializes the public `SubListResult` arrays only once at the end
This keeps the public contract unchanged while removing temporary match-building churn from the publish path.
### Intentional Remaining Allocations
The current implementation still allocates in two places by design:
- the tokenized `string[]` produced by `Tokenize(subject)` on cache misses
- the final `Subscription[]` and `Subscription[][]` arrays stored in `SubListResult`
Those allocations are part of the current public result shape and cache model.
--- ---
## Cache Strategy ## Cache Strategy
The cache is a `Dictionary<string, SubListResult>` keyed by the literal published subject. All operations use `StringComparer.Ordinal`. The cache is a `Dictionary<string, CachedResult>` keyed by literal publish subject with `StringComparer.Ordinal`.
**Size limits:** The cache holds at most `CacheMax` (1024) entries. When `_cache.Count` exceeds this, a sweep removes entries until the count reaches `CacheSweep` (256). The sweep takes the first `count - 256` keys from the dictionary — no LRU ordering is maintained. - `CacheMax = 1024`
- `CacheSweep = 256`
**`AddToCache`** is called from `Insert()` to keep cached results consistent after adding a subscription: When the cache grows past `CacheMax`, `SubListCacheSweeper` schedules a sweep that removes enough keys to return to the target size. The sweep is intentionally simple; it is not LRU.
- For a literal subscription subject, `AddToCache` does a direct lookup. If the exact key is in the cache, it creates a new `SubListResult` with the subscription appended and replaces the cached entry.
- For a wildcard subscription subject, `AddToCache` scans all cached keys and updates any entry whose key is matched by `SubjectMatch.MatchLiteral(key, subject)`.
**`RemoveFromCache`** is called from `Remove()` to invalidate cached results after removing a subscription: The cache stores fully materialized `SubListResult` instances because publish callers need stable array-based results immediately after lookup.
- For a literal subscription subject, `RemoveFromCache` removes the exact cache key.
- For a wildcard subscription subject, `RemoveFromCache` removes all cached keys matched by the pattern. Because it is difficult to reconstruct the correct result without a full trie traversal, invalidation is preferred over update.
The asymmetry between `AddToCache` (updates in place) and `RemoveFromCache` (invalidates) avoids a second trie traversal on removal at the cost of a cache miss on the next `Match()` for those keys.
--- ---
## Disposal ## Statistics and Monitoring
`SubList` implements `IDisposable`. `Dispose()` releases the `ReaderWriterLockSlim`: `Stats()` exposes:
```csharp - subscription count
public void Dispose() => _lock.Dispose(); - cache entry count
``` - insert/remove/match counts
- cache hit rate
- fanout statistics derived from cached results
`SubList` instances are owned by `NatsServer` and disposed during server shutdown. These counters are used by tests and monitoring code to validate routing behavior and cache effectiveness.
--- ---
## Related Documentation ## Related Documentation
- [Subscriptions Overview](../Subscriptions/Overview.md) - [Overview](Overview.md)
<!-- Last verified against codebase: 2026-02-22 --> <!-- Last verified against codebase: 2026-03-13 -->
+54 -69
View File
@@ -1,47 +1,10 @@
# Go vs .NET NATS Server — Benchmark Comparison # Go vs .NET NATS Server — Benchmark Comparison
Benchmark run: 2026-03-13 10:06 AM America/Indiana/Indianapolis. The latest refresh used the benchmark project README command (`dotnet test tests/NATS.Server.Benchmark.Tests --filter "Category=Benchmark" -v normal --logger "console;verbosity=detailed"`) and completed successfully as a `.NET`-only run. The Go/.NET comparison tables below remain the last Go-capable comparison baseline. Benchmark run: 2026-03-13 10:16 AM America/Indiana/Indianapolis. Both servers ran on the same machine using the benchmark project README command (`dotnet test tests/NATS.Server.Benchmark.Tests --filter "Category=Benchmark" -v normal --logger "console;verbosity=detailed"`). Test parallelization remained disabled inside the benchmark assembly.
**Environment:** Apple M4, .NET SDK 10.0.101, README benchmark command run in the benchmark project's default `Debug` configuration, Go toolchain installed but the current full-suite run emitted only `.NET` result blocks. **Environment:** Apple M4, .NET SDK 10.0.101, benchmark README command run in the benchmark project's default `Debug` configuration, Go toolchain installed, Go reference server built from `golang/nats-server/`.
--- ---
## Latest README Run (.NET only)
The current refresh came from `/tmp/bench-output.txt` using the benchmark project README workflow. Because the run did not emit any Go comparison blocks, the values below are the latest `.NET`-only numbers from that run, and the historical Go/.NET comparison tables are preserved below instead of being overwritten with mixed-source ratios.
### Core and JetStream
| Benchmark | .NET msg/s | .NET MB/s | Notes |
|-----------|------------|-----------|-------|
| Single Publisher (16B) | 1,392,442 | 21.2 | README full-suite run |
| Single Publisher (128B) | 1,491,226 | 182.0 | README full-suite run |
| PubSub 1:1 (16B) | 717,731 | 11.0 | README full-suite run |
| PubSub 1:1 (16KB) | 28,450 | 444.5 | README full-suite run |
| Fan-Out 1:4 (128B) | 1,451,748 | 177.2 | README full-suite run |
| Multi 4Px4S (128B) | 244,878 | 29.9 | README full-suite run |
| Request-Reply Single (128B) | 6,840 | 0.8 | P50 142.5 us, P99 203.9 us |
| Request-Reply 10Cx2S (16B) | 22,844 | 0.3 | P50 421.1 us, P99 602.1 us |
| JS Sync Publish (16B Memory) | 12,619 | 0.2 | README full-suite run |
| JS Async Publish (128B File) | 46,631 | 5.7 | README full-suite run |
| JS Ordered Consumer (128B) | 108,057 | 13.2 | README full-suite run |
| JS Durable Fetch (128B) | 490,090 | 59.8 | README full-suite run |
### Parser Microbenchmarks
| Benchmark | Ops/s | MB/s | Alloc |
|-----------|-------|------|-------|
| Parser PING | 5,756,370 | 32.9 | 0.0 B/op |
| Parser PUB | 2,537,973 | 96.8 | 40.0 B/op |
| Parser HPUB | 2,298,811 | 122.8 | 40.0 B/op |
| Parser PUB split payload | 2,049,535 | 78.2 | 176.0 B/op |
### Current Run Highlights
1. The parser microbenchmarks show the hot path is already at zero allocation for `PING`, with contiguous `PUB` and `HPUB` still paying a small fixed cost for retained field copies.
2. Split-payload `PUB` remains meaningfully more allocation-heavy than contiguous `PUB` because the parser must preserve unread payload state across reads and then materialize contiguous memory at the current client boundary.
3. The README-driven suite was a `.NET`-only refresh, so the comparative Go/.NET ratios below should still be treated as the last Go-capable baseline rather than current same-run ratios.
--- ---
## Core NATS — Pub/Sub Throughput ## Core NATS — Pub/Sub Throughput
@@ -50,27 +13,27 @@ The current refresh came from `/tmp/bench-output.txt` using the benchmark projec
| Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) | | Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) |
|---------|----------|---------|------------|-----------|-----------------| |---------|----------|---------|------------|-----------|-----------------|
| 16 B | 2,252,242 | 34.4 | 1,610,807 | 24.6 | 0.72x | | 16 B | 2,258,647 | 34.5 | 1,275,230 | 19.5 | 0.56x |
| 128 B | 2,199,267 | 268.5 | 1,661,014 | 202.8 | 0.76x | | 128 B | 2,251,274 | 274.8 | 1,661,668 | 202.8 | 0.74x |
### Publisher + Subscriber (1:1) ### Publisher + Subscriber (1:1)
| Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) | | Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) |
|---------|----------|---------|------------|-----------|-----------------| |---------|----------|---------|------------|-----------|-----------------|
| 16 B | 313,790 | 4.8 | 909,298 | 13.9 | **2.90x** | | 16 B | 296,374 | 4.5 | 875,105 | 13.4 | **2.95x** |
| 16 KB | 41,153 | 643.0 | 38,287 | 598.2 | 0.93x | | 16 KB | 32,111 | 501.7 | 30,030 | 469.2 | 0.94x |
### Fan-Out (1 Publisher : 4 Subscribers) ### Fan-Out (1 Publisher : 4 Subscribers)
| Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) | | Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) |
|---------|----------|---------|------------|-----------|-----------------| |---------|----------|---------|------------|-----------|-----------------|
| 128 B | 3,217,684 | 392.8 | 1,817,860 | 221.9 | 0.57x | | 128 B | 2,387,889 | 291.5 | 1,780,888 | 217.4 | 0.75x |
### Multi-Publisher / Multi-Subscriber (4P x 4S) ### Multi-Publisher / Multi-Subscriber (4P x 4S)
| Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) | | Payload | Go msg/s | Go MB/s | .NET msg/s | .NET MB/s | Ratio (.NET/Go) |
|---------|----------|---------|------------|-----------|-----------------| |---------|----------|---------|------------|-----------|-----------------|
| 128 B | 2,101,337 | 256.5 | 1,527,330 | 186.4 | 0.73x | | 128 B | 1,079,112 | 131.7 | 953,596 | 116.4 | 0.88x |
--- ---
@@ -80,13 +43,13 @@ The current refresh came from `/tmp/bench-output.txt` using the benchmark projec
| Payload | Go msg/s | .NET msg/s | Ratio | Go P50 (us) | .NET P50 (us) | Go P99 (us) | .NET P99 (us) | | Payload | Go msg/s | .NET msg/s | Ratio | Go P50 (us) | .NET P50 (us) | Go P99 (us) | .NET P99 (us) |
|---------|----------|------------|-------|-------------|---------------|-------------|---------------| |---------|----------|------------|-------|-------------|---------------|-------------|---------------|
| 128 B | 9,450 | 7,662 | 0.81x | 103.2 | 128.9 | 145.6 | 170.8 | | 128 B | 8,506 | 7,182 | 0.84x | 114.9 | 135.2 | 161.2 | 189.8 |
### 10 Clients, 2 Services (Queue Group) ### 10 Clients, 2 Services (Queue Group)
| Payload | Go msg/s | .NET msg/s | Ratio | Go P50 (us) | .NET P50 (us) | Go P99 (us) | .NET P99 (us) | | Payload | Go msg/s | .NET msg/s | Ratio | Go P50 (us) | .NET P50 (us) | Go P99 (us) | .NET P99 (us) |
|---------|----------|------------|-------|-------------|---------------|-------------|---------------| |---------|----------|------------|-------|-------------|---------------|-------------|---------------|
| 16 B | 31,094 | 26,144 | 0.84x | 316.9 | 368.7 | 439.2 | 559.7 | | 16 B | 26,610 | 22,533 | 0.85x | 367.7 | 425.3 | 487.4 | 622.5 |
--- ---
@@ -94,10 +57,10 @@ The current refresh came from `/tmp/bench-output.txt` using the benchmark projec
| Mode | Payload | Storage | Go msg/s | .NET msg/s | Ratio (.NET/Go) | | Mode | Payload | Storage | Go msg/s | .NET msg/s | Ratio (.NET/Go) |
|------|---------|---------|----------|------------|-----------------| |------|---------|---------|----------|------------|-----------------|
| Synchronous | 16 B | Memory | 17,533 | 14,373 | 0.82x | | Synchronous | 16 B | Memory | 13,756 | 9,954 | 0.72x |
| Async (batch) | 128 B | File | 198,237 | 60,416 | 0.30x | | Async (batch) | 128 B | File | 171,761 | 50,711 | 0.30x |
> **Note:** Async file store publish improved from 174 msg/s to 60K msg/s (347x improvement) after two rounds of FileStore-level optimizations plus profiling overhead removal. Remaining 3.3x gap is GC pressure from per-message allocations. > **Note:** Async file-store publish remains the largest JetStream gap at 0.30x. The bottleneck is still the storage write path and the remaining managed allocation pressure around persisted message state.
--- ---
@@ -105,10 +68,32 @@ The current refresh came from `/tmp/bench-output.txt` using the benchmark projec
| Mode | Go msg/s | .NET msg/s | Ratio (.NET/Go) | | Mode | Go msg/s | .NET msg/s | Ratio (.NET/Go) |
|------|----------|------------|-----------------| |------|----------|------------|-----------------|
| Ordered ephemeral consumer | 748,671 | 114,021 | 0.15x | | Ordered ephemeral consumer | 135,704 | 107,168 | 0.79x |
| Durable consumer fetch | 662,471 | 488,520 | 0.74x | | Durable consumer fetch | 533,441 | 375,652 | 0.70x |
> **Note:** Durable fetch improved from 0.13x → 0.60x → **0.74x** after Round 6 optimizations (batch flush, ackReply stack formatting, cached CompiledFilter, pooled fetch list). Ordered consumer ratio dropped due to Go benchmark improvement (748K vs 156K in earlier runs); .NET throughput is stable at ~110K msg/s. > **Note:** Ordered-consumer results in this run are much closer to parity than earlier snapshots. That suggests prior Go-side variance was material; `.NET` throughput is still clustered around ~107K msg/s.
---
## Hot Path Microbenchmarks (.NET only)
### SubList
| Benchmark | .NET msg/s | .NET MB/s | Alloc |
|-----------|------------|-----------|-------|
| SubList Exact Match (128 subjects) | 17,746,607 | 236.9 | 0.00 B/op |
| SubList Wildcard Match | 18,811,278 | 251.2 | 0.00 B/op |
| SubList Queue Match | 20,624,510 | 157.4 | 0.00 B/op |
| SubList Remote Interest | 264,725 | 4.3 | 0.00 B/op |
### Parser
| Benchmark | Ops/s | MB/s | Alloc |
|-----------|-------|------|-------|
| Parser PING | 5,598,176 | 32.0 | 0.0 B/op |
| Parser PUB | 2,701,645 | 103.1 | 40.0 B/op |
| Parser HPUB | 2,177,745 | 116.3 | 40.0 B/op |
| Parser PUB split payload | 1,702,439 | 64.9 | 176.0 B/op |
--- ---
@@ -116,25 +101,25 @@ The current refresh came from `/tmp/bench-output.txt` using the benchmark projec
| Category | Ratio Range | Assessment | | Category | Ratio Range | Assessment |
|----------|-------------|------------| |----------|-------------|------------|
| Pub-only throughput | 0.72x0.76x | Good — within 2x | | Pub-only throughput | 0.56x0.74x | Mixed — 128 B is solid, 16 B still trails materially |
| Pub/sub (small payload) | **2.90x** | .NET outperforms Go — direct buffer path eliminates all per-message overhead | | Pub/sub (small payload) | **2.95x** | .NET outperforms Go decisively |
| Pub/sub (large payload) | 0.93x | Near parity | | Pub/sub (large payload) | 0.94x | Near parity |
| Fan-out | 0.57x | Improved from 0.18x → 0.44x → 0.66x; batch flush applied but serial delivery remains | | Fan-out | 0.75x | Good improvement; still limited by serial delivery |
| Multi pub/sub | 0.73x | Improved from 0.49x → 0.84x; variance from system load | | Multi pub/sub | 0.88x | Close to parity in this run |
| Request/reply latency | 0.81x0.84x | Good — improved from 0.77x | | Request/reply latency | 0.84x0.85x | Good |
| JetStream sync publish | 0.82x | Good | | JetStream sync publish | 0.72x | Good |
| JetStream async file publish | 0.30x | Improved from 0.00x — storage write path dominates | | JetStream async file publish | 0.30x | Storage write path still dominates |
| JetStream ordered consume | 0.15x | .NET stable ~110K; Go variance high (156K749K) | | JetStream ordered consume | 0.79x | Much closer to parity in this run |
| JetStream durable fetch | **0.74x** | **Improved from 0.60x** — batch flush + ackReply optimization | | JetStream durable fetch | 0.70x | Good |
### Key Observations ### Key Observations
1. **Small-payload 1:1 pub/sub outperforms Go by ~3x** (909K vs 314K msg/s). The per-client direct write buffer with `stackalloc` header formatting eliminates all per-message heap allocations and channel overhead. 1. **Small-payload 1:1 pub/sub still beats Go by ~3x** (875K vs 296K msg/s). The direct write path continues to pay off when message fanout is simple and payloads are tiny.
2. **Durable consumer fetch improved to 0.74x** (489K vs 662K msg/s) — Round 6 batch flush signaling and `string.Create`-based ack reply formatting reduced per-message overhead significantly. 2. **Fan-out and multi pub/sub both improved in this run** to 0.75x and 0.88x respectively. The remaining gap is still consistent with Go's more naturally parallel fanout model.
3. **Fan-out holds at ~0.57x** despite batch flush optimization. The remaining gap is goroutine-level parallelism (Go fans out per-client via goroutines; .NET delivers serially). The batch flush reduces wakeup overhead but doesn't add concurrency. 3. **Ordered consumer moved up to 0.79x** (107K vs 136K msg/s). That is materially stronger than earlier runs and suggests previous Go-side variance was distorting the comparison more than the `.NET` consumer path itself.
4. **Request/reply improved to 0.81x0.84x** — deferred flush benefits single-message delivery paths too. 4. **Durable fetch remains solid at 0.70x**. The Round 6 fetch-path work is still holding, but there is room left in consumer dispatch and storage reads.
5. **JetStream file store async publish: 0.30x** — remaining gap is GC pressure from per-message `StoredMessage` objects and `byte[]` copies (Change 2 deferred due to scope: 80+ sites in FileStore.cs need migration). 5. **Async file-store publish is still the largest server-level gap at 0.30x**. The storage layer remains the highest-value runtime target after parser and SubList hot-path cleanup.
6. **JetStream ordered consumer: 0.15x** — ratio drop is due to Go benchmark variance (749K in this run vs 156K previously); .NET throughput stable at ~110K msg/s. Further investigation needed for the Go variability. 6. **The new SubList microbenchmarks show effectively zero temporary allocation per operation** for exact, wildcard, queue, and remote-interest lookups in the current implementation. Parser contiguous hot paths also remain small and stable, while split-payload `PUB` still pays a higher copy cost.
--- ---
@@ -0,0 +1,331 @@
# FileStore Payload And Index Optimization Implementation Plan
> **For Codex:** REQUIRED SUB-SKILLS: Use `using-git-worktrees` to create an isolated workspace before Task 1, then use `executeplan` to implement this plan task-by-task. After verification is complete, merge the finished branch back into `main`.
**Goal:** Reduce JetStream FileStore memory churn and repeated full scans by tightening payload ownership, splitting compact metadata from large payload buffers, and replacing LINQ-based maintenance work with explicit indexes and loops.
**Architecture:** Start by freezing current behavior across `AppendAsync`, `StoreMsg`, retention, snapshots, and recovery. Then introduce compact metadata/index structures, remove avoidable duplicate payload buffers, replace repeated `_messages` scans with maintained indexes, and finish by updating recovery/snapshot paths plus benchmark coverage.
**Tech Stack:** .NET 10, C#, JetStream storage stack, `ReadOnlyMemory<byte>`, pooled buffers where safe, xUnit, existing JetStream benchmark harness.
---
## Scope Anchors
- Primary source: `src/NATS.Server/JetStream/Storage/FileStore.cs`
- Supporting sources:
- `src/NATS.Server/JetStream/Storage/MsgBlock.cs`
- `src/NATS.Server/JetStream/Storage/StoredMessage.cs`
- `src/NATS.Server/JetStream/Storage/MessageRecord.cs`
- Existing contract tests:
- `tests/NATS.Server.JetStream.Tests/StreamStoreContractTests.cs`
- `tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs`
- Existing FileStore coverage:
- `tests/NATS.Server.JetStream.Tests/FileStoreTests.cs`
- `tests/NATS.Server.JetStream.Tests/JetStreamStoreIndexTests.cs`
- `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreCompressionTests.cs`
- `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreCrashRecoveryTests.cs`
- `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreTombstoneTests.cs`
- Documentation to update: `Documentation/JetStream/Overview.md`
- Benchmark project: `tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj`
- Benchmark comparison doc: `benchmarks_comparison.md`
## Task 0: Create an isolated git worktree and verify the baseline
**Files:**
- Modify: `.gitignore` only if the chosen local worktree directory is not already ignored
**Step 1: Choose the worktree location using the repo convention**
- Check for an existing `.worktrees/` directory first, then `worktrees/`.
- If neither exists, check repo guidance before creating one.
- Prefer a project-local `.worktrees/` directory when available.
**Step 2: Verify the worktree directory is ignored before creating anything**
- Run:
```bash
git check-ignore -q .worktrees || git check-ignore -q worktrees
```
- Expected: one configured worktree directory is ignored.
- If neither directory is ignored, add the chosen directory to `.gitignore`, commit that change on `main`, and then continue.
**Step 3: Create a dedicated branch and worktree for this plan**
- Run:
```bash
git worktree add .worktrees/filestore-payload-index-optimization -b codex/filestore-payload-index-optimization
```
- Expected: a new isolated checkout exists at `.worktrees/filestore-payload-index-optimization`.
**Step 4: Move into the worktree and verify the starting baseline**
- Run:
```bash
cd .worktrees/filestore-payload-index-optimization
dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj -c Release
```
- Expected: PASS before implementation starts.
- If the baseline fails, stop and resolve whether to proceed before changing FileStore code.
**Step 5: Commit only the worktree bootstrap change if one was required**
- Run only if `.gitignore` had to change:
```bash
git add .gitignore
git commit -m "chore: ignore local worktree directory"
```
## Task 1: Freeze store behavior and add scan/ownership regression tests
**Files:**
- Modify: `tests/NATS.Server.JetStream.Tests/JetStreamStoreIndexTests.cs`
- Modify: `tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs`
- Create: `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreOptimizationGuardTests.cs`
**Step 1: Add failing tests for the targeted optimization boundaries**
- Cover:
- `AppendAsync` retaining logical payload behavior
- `StoreMsg` with headers + payload
- `LoadLastBySubjectAsync`
- `TrimToMaxMessages`
- `PurgeEx`
- snapshot/recovery round-trips
**Step 2: Add tests that lock first/last sequence bookkeeping**
- Ensure `_firstSeq`, `_last`, and subject-last lookup behavior remain correct after removes, purges, compaction, and recovery.
**Step 3: Run focused JetStream tests to prove the new tests fail first**
- Run: `dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj --filter "FullyQualifiedName~FileStoreOptimizationGuardTests|FullyQualifiedName~JetStreamStoreIndexTests|FullyQualifiedName~StoreInterfaceTests" -c Release`
- Expected: FAIL only in the newly added optimization-guard tests.
**Step 4: Commit the failing-test baseline**
- Run:
```bash
git add tests/NATS.Server.JetStream.Tests/JetStreamStoreIndexTests.cs tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreOptimizationGuardTests.cs
git commit -m "test: lock FileStore optimization boundaries"
```
## Task 2: Introduce compact metadata/index types and remove full-scan bookkeeping
**Files:**
- Create: `src/NATS.Server/JetStream/Storage/StoredMessageIndex.cs`
- Modify: `src/NATS.Server/JetStream/Storage/FileStore.cs`
- Modify: `src/NATS.Server/JetStream/Storage/StoredMessage.cs`
**Step 1: Split compact indexing metadata from payload-bearing message objects**
- Add a small immutable metadata/index type that tracks at least:
- sequence
- subject
- logical payload length
- timestamp
- subject-local links or last-seen markers if needed
**Step 2: Replace repeated `Min()` / `Max()` / full-value scans with maintained state**
- Maintain first live sequence, last live sequence, and last-by-subject values incrementally rather than recomputing them with LINQ.
**Step 3: Run targeted index tests**
- Run: `dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj --filter "FullyQualifiedName~JetStreamStoreIndexTests|FullyQualifiedName~FileStoreOptimizationGuardTests" -c Release`
- Expected: PASS.
**Step 4: Commit the metadata/index layer**
- Run:
```bash
git add src/NATS.Server/JetStream/Storage/StoredMessageIndex.cs src/NATS.Server/JetStream/Storage/FileStore.cs src/NATS.Server/JetStream/Storage/StoredMessage.cs
git commit -m "perf: add compact FileStore index metadata"
```
## Task 3: Remove duplicate payload ownership in append and store paths
**Files:**
- Modify: `src/NATS.Server/JetStream/Storage/FileStore.cs`
- Modify: `src/NATS.Server/JetStream/Storage/MsgBlock.cs`
- Modify: `src/NATS.Server/JetStream/Storage/MessageRecord.cs`
- Modify: `tests/NATS.Server.JetStream.Tests/FileStoreTests.cs`
**Step 1: Rework `AppendAsync` and `StoreMsg` payload flow**
- Stop eagerly keeping both a transformed persisted payload and a second fully duplicated managed payload when the same buffer/view can safely back both responsibilities.
- Keep correctness for compression, encryption, and header-bearing records explicit.
**Step 2: Remove concatenated header+payload arrays where possible**
- Let record encoding paths consume header and payload spans directly instead of always building `combined = new byte[...]`.
- Leave a copy in place only where the persistence or recovery contract actually requires one.
**Step 3: Run targeted persistence tests**
- Run: `dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj --filter "FullyQualifiedName~FileStoreTests|FullyQualifiedName~FileStoreCompressionTests|FullyQualifiedName~FileStoreEncryptionTests" -c Release`
- Expected: PASS.
**Step 4: Commit the payload-ownership refactor**
- Run:
```bash
git add src/NATS.Server/JetStream/Storage/FileStore.cs src/NATS.Server/JetStream/Storage/MsgBlock.cs src/NATS.Server/JetStream/Storage/MessageRecord.cs tests/NATS.Server.JetStream.Tests/FileStoreTests.cs
git commit -m "perf: reduce FileStore duplicate payload buffers"
```
## Task 4: Replace LINQ-heavy maintenance operations with explicit indexed paths
**Files:**
- Modify: `src/NATS.Server/JetStream/Storage/FileStore.cs`
- Modify: `tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs`
- Modify: `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreCrashRecoveryTests.cs`
- Modify: `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreTombstoneTests.cs`
**Step 1: Rewrite hot maintenance methods**
- Replace LINQ-based implementations in:
- `LoadLastBySubjectAsync`
- `TrimToMaxMessages`
- `PurgeEx`
- snapshot/recovery recomputation paths
- Use explicit loops and maintained indexes first; only add more elaborate per-subject structures if profiling still demands them.
**Step 2: Preserve recovery and tombstone correctness**
- Verify delete markers, TTL rebuilds, compaction, and sequence-gap handling still match the current parity tests.
**Step 3: Run targeted JetStream storage suites**
- Run: `dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj --filter "FullyQualifiedName~StoreInterfaceTests|FullyQualifiedName~FileStoreCrashRecoveryTests|FullyQualifiedName~FileStoreTombstoneTests" -c Release`
- Expected: PASS.
**Step 4: Commit the maintenance-path rewrite**
- Run:
```bash
git add src/NATS.Server/JetStream/Storage/FileStore.cs tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreCrashRecoveryTests.cs tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreTombstoneTests.cs
git commit -m "perf: replace FileStore full scans with indexed loops"
```
## Task 5: Add benchmark coverage, update docs, and run full verification
**Files:**
- Create: `tests/NATS.Server.Benchmark.Tests/JetStream/FileStoreAppendBenchmarks.cs`
- Modify: `Documentation/JetStream/Overview.md`
**Step 1: Add FileStore-focused benchmarks**
- Cover:
- append throughput
- sync publish
- load-last-by-subject
- purge/trim maintenance overhead
- Record allocation deltas before/after.
**Step 2: Update JetStream documentation**
- Document how FileStore now separates metadata/index concerns from payload storage and where copies still remain by design.
**Step 3: Run full verification**
- Run: `dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj --filter "FullyQualifiedName~FileStore|FullyQualifiedName~SyncPublish|FullyQualifiedName~AsyncPublish" -c Release`
- Expected: PASS; benchmark output shows fewer allocations in append-heavy scenarios.
**Step 4: Commit docs and benchmarks**
- Run:
```bash
git add tests/NATS.Server.Benchmark.Tests/JetStream/FileStoreAppendBenchmarks.cs Documentation/JetStream/Overview.md
git commit -m "docs: record FileStore payload and index strategy"
```
## Task 6: Merge the verified worktree branch back into `main`
**Files:**
- No source-file changes expected unless the merge surfaces conflicts that require a follow-up fix
**Step 1: Confirm the worktree branch is clean and fully verified**
- Re-run the Task 5 verification commands in the worktree if anything changed after the final commit.
- Run:
```bash
git status --short
```
- Expected: no uncommitted changes.
**Step 2: Update `main` before merging**
- From the primary checkout, run:
```bash
git switch main
git pull --ff-only
```
- Expected: local `main` matches the latest remote state.
**Step 3: Merge the finished branch back to `main`**
- Run:
```bash
git merge --ff-only codex/filestore-payload-index-optimization
```
- Expected: `main` fast-forwards to include the completed FileStore optimization commits.
- If fast-forward is not possible, rebase `codex/filestore-payload-index-optimization` onto `main`, re-run verification, and then repeat this step.
**Step 4: Confirm `main` still passes after the merge**
- Run:
```bash
dotnet test tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj -c Release
```
- Expected: PASS on `main`.
**Step 5: Remove the temporary worktree after merge confirmation**
- Run:
```bash
git worktree remove .worktrees/filestore-payload-index-optimization
git branch -d codex/filestore-payload-index-optimization
```
- Expected: the temporary checkout is removed and the topic branch is no longer needed locally.
## Task 7: Run the benchmark suite per the benchmark README and update the comparison document
**Files:**
- Modify: `benchmarks_comparison.md`
- Reference: `tests/NATS.Server.Benchmark.Tests/README.md`
**Step 1: Run the full benchmark suite with the README-prescribed command**
- From `main` after Task 6 succeeds, run:
```bash
dotnet test tests/NATS.Server.Benchmark.Tests \
--filter "Category=Benchmark" \
-v normal \
--logger "console;verbosity=detailed" 2>&1 | tee /tmp/bench-output.txt
```
- Expected: the benchmark suite completes and writes comparison blocks to `/tmp/bench-output.txt`.
**Step 2: Extract the benchmark results from the captured output**
- Review the `Standard Output Messages` sections in `/tmp/bench-output.txt`.
- Capture the updated values for:
- core pub/sub throughput
- request/reply latency
- JetStream sync publish
- JetStream async file publish
- ordered consumer throughput
- durable consumer fetch throughput
**Step 3: Update `benchmarks_comparison.md`**
- Update:
- the benchmark run date on the first line
- environment details if they changed
- all affected tables with the new msg/s, MB/s, ratio, and latency values
- the Summary and Key Observations text if the new ratios materially change the assessment
**Step 4: Verify the comparison document changes are the only remaining edits**
- Run:
```bash
git status --short
```
- Expected: only `benchmarks_comparison.md` is modified at this point unless the benchmark run surfaced a legitimate follow-up issue to capture separately.
**Step 5: Commit the benchmark comparison refresh**
- Run:
```bash
git add benchmarks_comparison.md
git commit -m "docs: update benchmark comparison after FileStore optimization"
```
## Completion Checklist
- [ ] Implementation started from an isolated git worktree on `codex/filestore-payload-index-optimization`.
- [ ] `AppendAsync` and `StoreMsg` avoid unnecessary duplicate payload ownership.
- [ ] `LoadLastBySubjectAsync`, `TrimToMaxMessages`, and `PurgeEx` no longer rely on repeated LINQ full scans.
- [ ] First/last/live-sequence bookkeeping is maintained incrementally.
- [ ] JetStream storage, recovery, compression, encryption, and tombstone tests remain green.
- [ ] FileStore-focused benchmark coverage exists in `tests/NATS.Server.Benchmark.Tests/JetStream/`.
- [ ] `Documentation/JetStream/Overview.md` explains the updated storage/index model.
- [ ] Verified work has been merged back into `main` and the temporary worktree has been removed.
- [ ] Full benchmark suite has been run from `main` using the command in `tests/NATS.Server.Benchmark.Tests/README.md`.
- [ ] `benchmarks_comparison.md` has been updated to reflect the new benchmark results.
## Concise Execution Checklist For The Current Codebase
- [ ] Create `codex/filestore-payload-index-optimization` in `.worktrees/filestore-payload-index-optimization` and verify `tests/NATS.Server.JetStream.Tests/NATS.Server.JetStream.Tests.csproj` passes before changes.
- [ ] Add optimization-guard coverage in `tests/NATS.Server.JetStream.Tests/JetStreamStoreIndexTests.cs`, `tests/NATS.Server.JetStream.Tests/JetStream/Storage/StoreInterfaceTests.cs`, and new `tests/NATS.Server.JetStream.Tests/JetStream/Storage/FileStoreOptimizationGuardTests.cs`.
- [ ] Rework the current FileStore hot paths in `src/NATS.Server/JetStream/Storage/FileStore.cs`: `AppendAsync`, `LoadLastBySubjectAsync`, `TrimToMaxMessages`, `StoreMsg`, and `PurgeEx`.
- [ ] Introduce compact FileStore indexing metadata in new `src/NATS.Server/JetStream/Storage/StoredMessageIndex.cs` and adjust `src/NATS.Server/JetStream/Storage/StoredMessage.cs` accordingly.
- [ ] Remove avoidable payload duplication in `src/NATS.Server/JetStream/Storage/FileStore.cs`, `src/NATS.Server/JetStream/Storage/MsgBlock.cs`, and `src/NATS.Server/JetStream/Storage/MessageRecord.cs`.
- [ ] Keep JetStream storage parity green by re-running the existing storage-focused suites under `tests/NATS.Server.JetStream.Tests/JetStream/Storage/`, especially compression, crash recovery, tombstones, and store interface coverage.
- [ ] Add FileStore benchmark coverage alongside the existing JetStream benchmark classes in `tests/NATS.Server.Benchmark.Tests/JetStream/`.
- [ ] Update `Documentation/JetStream/Overview.md` to describe the new payload/index split and the remaining intentional copy boundaries.
- [ ] Merge the verified topic branch back into `main`, re-run JetStream tests on `main`, then remove the temporary worktree.
- [ ] Run the full benchmark suite exactly as documented in `tests/NATS.Server.Benchmark.Tests/README.md` and update `benchmarks_comparison.md` with the new measurements.
@@ -0,0 +1,244 @@
# Parser Span Retention Implementation Plan
> **For Codex:** REQUIRED SUB-SKILLS: Use `using-git-worktrees` to create an isolated worktree before making changes, `executeplan` to implement this plan task-by-task, and `finishing-a-development-branch` to merge the verified work back to `main` when implementation is complete.
**Goal:** Reduce parser hot-path allocations by keeping protocol fields and payloads in byte-oriented views until a caller explicitly needs materialized `string` or copied `byte[]` values.
**Architecture:** Introduce a byte-first parser representation alongside the current `ParsedCommand` contract, then migrate `NatsClient` and adjacent hot paths to consume the new representation without changing wire behavior. Preserve compatibility through an adapter layer so functional parity stays stable while allocation-heavy paths move to spans, pooled buffers, and sequence slices.
**Tech Stack:** .NET 10, C#, `System.Buffers`, `System.IO.Pipelines`, `ReadOnlySequence<byte>`, `SequenceReader<byte>`, xUnit, existing benchmark test harness.
---
## Scope Anchors
- Primary source: `src/NATS.Server/Protocol/NatsParser.cs`
- Primary consumer: `src/NATS.Server/NatsClient.cs`
- Existing parser tests: `tests/NATS.Server.Core.Tests/ParserTests.cs`
- Existing snippet/parity tests: `tests/NATS.Server.Core.Tests/Protocol/ProtocolParserSnippetGapParityTests.cs`
- Documentation to update: `Documentation/Protocol/Parser.md`
- Benchmark project: `tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj`
- Benchmark run instructions: `tests/NATS.Server.Benchmark.Tests/README.md`
- Benchmark comparison report: `benchmarks_comparison.md`
## Task 0: Create an isolated git worktree for the parser optimization work
**Files:**
- Verify: `.worktrees/`
- Modify if needed: `.gitignore`
**Step 1: Verify the preferred worktree directory is available and ignored**
- Check that `.worktrees/` exists and is ignored by git before creating a project-local worktree.
- If `.worktrees/` is not ignored, add it to `.gitignore`, then commit that repository hygiene fix before continuing.
**Step 2: Create the feature worktree on a `codex/` branch**
- Run:
```bash
git worktree add .worktrees/codex-parser-span-retention -b codex/parser-span-retention
cd .worktrees/codex-parser-span-retention
```
- Expected: a new isolated worktree exists at `.worktrees/codex-parser-span-retention` on branch `codex/parser-span-retention`.
**Step 3: Verify the worktree starts from a clean, passing baseline**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj -c Release`
- Expected: PASS. If this fails, stop and resolve or explicitly confirm whether to proceed from a failing baseline.
**Step 4: Commit any required worktree setup fix**
- Only if `.gitignore` changed, run:
```bash
git add .gitignore
git commit -m "chore: ignore local worktree directories"
```
## Task 1: Freeze parser behavior and add allocation-focused tests
**Files:**
- Modify: `tests/NATS.Server.Core.Tests/ParserTests.cs`
- Modify: `tests/NATS.Server.Core.Tests/Protocol/ProtocolParserSnippetGapParityTests.cs`
- Create: `tests/NATS.Server.Core.Tests/Protocol/ParserSpanRetentionTests.cs`
**Step 1: Add failing tests for byte-first parser behavior**
- Cover `PUB`, `HPUB`, `CONNECT`, and `INFO` with assertions that the new parser path can expose field data without forcing immediate `string` materialization.
- Add split-payload cases to prove the parser preserves pending payload state across reads.
**Step 2: Add compatibility tests for existing `ParsedCommand` behavior**
- Keep current semantics for `Type`, `Subject`, `ReplyTo`, `Queue`, `Sid`, `HeaderSize`, and `Payload`.
- Ensure malformed protocol inputs still throw `ProtocolViolationException` with existing snippets/messages.
**Step 3: Run targeted tests to verify the new tests fail first**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~ParserTests|FullyQualifiedName~ParserSpanRetentionTests|FullyQualifiedName~ProtocolParserSnippetGapParityTests" -c Release`
- Expected: FAIL in the newly added parser span-retention tests only.
**Step 4: Commit the failing-test baseline**
- Run:
```bash
git add tests/NATS.Server.Core.Tests/ParserTests.cs tests/NATS.Server.Core.Tests/Protocol/ProtocolParserSnippetGapParityTests.cs tests/NATS.Server.Core.Tests/Protocol/ParserSpanRetentionTests.cs
git commit -m "test: lock parser span-retention behavior"
```
## Task 2: Introduce byte-oriented parser view types
**Files:**
- Create: `src/NATS.Server/Protocol/ParsedCommandView.cs`
- Modify: `src/NATS.Server/Protocol/NatsParser.cs`
**Step 1: Add a hot-path parser view contract**
- Create a `ref struct` or small `readonly struct` representation for command views that can carry:
- operation kind
- subject/reply/queue/SID as spans or sequence-backed views
- payload as `ReadOnlySequence<byte>` or `ReadOnlyMemory<byte>` when contiguous
- header size and max-messages metadata
**Step 2: Add an adapter to the current `ParsedCommand` shape**
- Keep the public/internal `ParsedCommand` entry point usable for existing consumers and tests.
- Centralize materialization so `Encoding.ASCII.GetString(...)` and `ToArray()` happen in one adapter layer instead of inside every parse branch.
**Step 3: Re-run parser tests**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~Parser" -c Release`
- Expected: FAIL only in branches not yet migrated to the new view path.
**Step 4: Commit the parser-view scaffolding**
- Run:
```bash
git add src/NATS.Server/Protocol/ParsedCommandView.cs src/NATS.Server/Protocol/NatsParser.cs
git commit -m "feat: add byte-oriented parser view contract"
```
## Task 3: Rework control-line parsing and pending payload state
**Files:**
- Modify: `src/NATS.Server/Protocol/NatsParser.cs`
**Step 1: Remove early string materialization from control-line parsing**
- Change `ParsePub`, `ParseHPub`, `ParseSub`, `ParseUnsub`, `ParseConnect`, and `ParseInfo` to keep raw byte slices in the hot parser path.
- Replace `_pendingSubject` and `_pendingReplyTo` string fields with byte-oriented pending state.
**Step 2: Avoid unconditional payload copies**
- Update `TryReadPayload()` so single-segment payloads can flow through as borrowed memory/slices.
- Copy only when the payload is multi-segment or when the compatibility adapter explicitly requires a standalone buffer.
**Step 3: Replace repeated tiny literal allocations**
- Stop using per-call `u8.ToArray()`-style buffers for CRLF and other fixed protocol tokens inside this parser path.
- Add shared static buffers where appropriate.
**Step 4: Run targeted regression tests**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~ParserTests|FullyQualifiedName~ProtocolParserSnippetGapParityTests" -c Release`
- Expected: PASS.
**Step 5: Commit the parser hot-path rewrite**
- Run:
```bash
git add src/NATS.Server/Protocol/NatsParser.cs
git commit -m "perf: keep parser state in bytes until materialization"
```
## Task 4: Migrate `NatsClient` to the new parser path without changing behavior
**Files:**
- Modify: `src/NATS.Server/NatsClient.cs`
- Modify: `tests/NATS.Server.Core.Tests/ParserTests.cs`
- Modify: `tests/NATS.Server.Core.Tests/Protocol/ClientProtocolGoParityTests.cs`
**Step 1: Consume parser views first, materialize only at command handling boundaries**
- Update `ProcessCommandsAsync` and any parser call sites so hot `PUB`/`HPUB` handling can read subject, reply, and payload from the byte-oriented representation.
- Keep logging/tracing behavior intact, but ensure tracing is the only reason strings are created on trace-enabled paths.
**Step 2: Preserve feature parity**
- Verify header parsing, payload size checks, connect/info handling, and slow-consumer/error behavior still match current tests.
**Step 3: Run consumer-facing protocol tests**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~ClientProtocolGoParityTests|FullyQualifiedName~ParserTests" -c Release`
- Expected: PASS.
**Step 4: Commit the consumer migration**
- Run:
```bash
git add src/NATS.Server/NatsClient.cs tests/NATS.Server.Core.Tests/ParserTests.cs tests/NATS.Server.Core.Tests/Protocol/ClientProtocolGoParityTests.cs
git commit -m "perf: consume parser command views in client hot path"
```
## Task 5: Add benchmarks, document the change, run full verification, and refresh the benchmark comparison report
**Files:**
- Create: `tests/NATS.Server.Benchmark.Tests/Protocol/ParserHotPathBenchmarks.cs`
- Modify: `Documentation/Protocol/Parser.md`
- Modify: `benchmarks_comparison.md`
**Step 1: Add parser-focused benchmark coverage**
- Add microbenchmarks for:
- `PING` / `PONG`
- `PUB`
- `HPUB`
- split payload reads
- Capture throughput and allocation deltas before/after.
**Step 2: Update protocol documentation**
- Document the new parser view + adapter split, why strings are deferred, and where payload copying is still intentionally required.
**Step 3: Run full verification**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj --filter "FullyQualifiedName~Parser" -c Release`
- Expected: PASS; benchmark output shows reduced allocations relative to the baseline run.
**Step 4: Run the full benchmark suite per the benchmark project README**
- Run:
```bash
dotnet test tests/NATS.Server.Benchmark.Tests \
--filter "Category=Benchmark" \
-v normal \
--logger "console;verbosity=detailed" 2>&1 | tee /tmp/bench-output.txt
```
- Expected: benchmark comparison output is captured in `/tmp/bench-output.txt`, including the "Standard Output Messages" blocks described in `tests/NATS.Server.Benchmark.Tests/README.md`.
**Step 5: Update `benchmarks_comparison.md` with the new benchmark results**
- Extract the comparison blocks from `/tmp/bench-output.txt`.
- Update `benchmarks_comparison.md` with the latest msg/s, MB/s, ratio, and latency values.
- Update the benchmark date, environment description, Summary table, and Key Observations so they match the new run.
**Step 6: Commit the verification, docs, and benchmark report refresh**
- Run:
```bash
git add tests/NATS.Server.Benchmark.Tests/Protocol/ParserHotPathBenchmarks.cs Documentation/Protocol/Parser.md benchmarks_comparison.md
git commit -m "docs: record parser hot-path allocation strategy"
```
## Task 6: Merge the verified parser work back to `main` and clean up the worktree
**Files:**
- No source changes expected
**Step 1: Confirm the feature branch is fully verified before merge**
- Reuse the verification from Task 5. Do not merge if the core tests, parser benchmark tests, or full benchmark suite run did not complete successfully.
**Step 2: Merge `codex/parser-span-retention` back into `main`**
- Return to the primary repository worktree and run:
```bash
git checkout main
git pull
git merge codex/parser-span-retention
```
- Expected: the parser optimization commits merge cleanly into `main`.
**Step 3: Re-run verification on the merged `main` branch**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj --filter "FullyQualifiedName~Parser" -c Release`
- Confirm `benchmarks_comparison.md` reflects the results captured from the Task 5 full benchmark suite.
- Expected: PASS on merged `main`.
**Step 4: Delete the feature branch and remove the worktree**
- Run:
```bash
git branch -d codex/parser-span-retention
git worktree remove .worktrees/codex-parser-span-retention
```
- Expected: only `main` remains checked out in the primary workspace, and the temporary parser worktree is removed.
## Completion Checklist
- [ ] Parser optimization work was implemented in an isolated `.worktrees/codex-parser-span-retention` worktree on branch `codex/parser-span-retention`.
- [ ] `NatsParser` no longer materializes hot-path `string` values during parse unless the compatibility adapter requests them.
- [ ] Single-segment payloads can pass through without an unconditional `byte[]` copy.
- [ ] Existing parser and protocol behavior remains green in core tests.
- [ ] Parser-focused benchmark coverage exists in `tests/NATS.Server.Benchmark.Tests/Protocol/`.
- [ ] The full benchmark suite was run using the workflow from `tests/NATS.Server.Benchmark.Tests/README.md`.
- [ ] `benchmarks_comparison.md` was updated to reflect the latest benchmark run.
- [ ] `Documentation/Protocol/Parser.md` explains the byte-first parser architecture.
- [ ] Verified parser optimization commits were merged back into `main`.
@@ -0,0 +1,300 @@
# SubList Allocation Reduction Implementation Plan
> **For Codex:** REQUIRED SUB-SKILLS: Use `using-git-worktrees` to create an isolated worktree before making changes, `executeplan` to implement this plan task-by-task, and `finishing-a-development-branch` to merge the verified work back to `main` when implementation is complete.
**Goal:** Reduce publish-path allocation and lookup overhead in `SubList` by removing composite string keys, minimizing `token.ToString()` churn, and tightening `Match()` result building without changing subscription semantics.
**Architecture:** First lock the current trie, cache, and remote-interest behavior with targeted tests. Then replace routed-sub bookkeeping with a dedicated value key, remove string split/rebuild work from remote cleanup, and finally optimize trie traversal and match result construction with span-friendly helpers and pooled builders.
**Tech Stack:** .NET 10, C#, `ReaderWriterLockSlim`, span-based token parsing, xUnit, existing clustering/gateway parity suites, benchmark test harness.
---
## Scope Anchors
- Primary source: `src/NATS.Server/Subscriptions/SubList.cs`
- Supporting source: `src/NATS.Server/Subscriptions/SubjectMatch.cs`
- Existing core tests: `tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs`
- Existing ctor/notification tests: `tests/NATS.Server.Core.Tests/Subscriptions/SubListCtorAndNotificationParityTests.cs`
- Route cleanup tests: `tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs`
- Gateway/route interest tests:
- `tests/NATS.Server.Gateways.Tests/Gateways/GatewayInterestModeTests.cs`
- `tests/NATS.Server.Clustering.Tests/Routes/RouteInterestIdempotencyTests.cs`
- `tests/NATS.Server.Clustering.Tests/Routes/RouteSubscriptionTests.cs`
- Documentation to update: `Documentation/Subscriptions/SubList.md`
- Benchmark workflow reference: `tests/NATS.Server.Benchmark.Tests/README.md`
- Benchmark comparison document: `benchmarks_comparison.md`
## Task 0: Create an isolated git worktree for the SubList optimization work
**Files:**
- Verify: `.worktrees/`
- Modify if needed: `.gitignore`
**Step 1: Verify the preferred worktree directory is available and ignored**
- Check that `.worktrees/` exists and is ignored by git before creating a project-local worktree.
- If `.worktrees/` is not ignored, add it to `.gitignore`, then commit that repository hygiene fix before continuing.
**Step 2: Create the feature worktree on a `codex/` branch**
- Run:
```bash
git worktree add .worktrees/codex-sublist-allocation-reduction -b codex/sublist-allocation-reduction
cd .worktrees/codex-sublist-allocation-reduction
```
- Expected: a new isolated worktree exists at `.worktrees/codex-sublist-allocation-reduction` on branch `codex/sublist-allocation-reduction`.
**Step 3: Verify the worktree starts from a clean, passing baseline**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~SubList" -c Release`
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj --filter "FullyQualifiedName~RouteRemoteSubCleanupParityBatch2Tests|FullyQualifiedName~RouteInterestIdempotencyTests|FullyQualifiedName~RouteSubscriptionTests" -c Release`
- Run: `dotnet test tests/NATS.Server.Gateways.Tests/NATS.Server.Gateways.Tests.csproj --filter "FullyQualifiedName~GatewayInterestModeTests|FullyQualifiedName~GatewayInterestIdempotencyTests|FullyQualifiedName~GatewayForwardingTests" -c Release`
- Expected: PASS. If this fails, stop and resolve or explicitly confirm whether to proceed from a failing baseline.
**Step 4: Commit any required worktree setup fix**
- Only if `.gitignore` changed, run:
```bash
git add .gitignore
git commit -m "chore: ignore local worktree directories"
```
## Task 1: Lock behavior around remote interest, cleanup, and matching
**Files:**
- Modify: `tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs`
- Modify: `tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs`
- Create: `tests/NATS.Server.Core.Tests/Subscriptions/SubListAllocationGuardTests.cs`
**Step 1: Add failing tests for routed-sub bookkeeping changes**
- Cover:
- applying the same remote subscription twice
- removing remote subscriptions by route and by route/account
- queue-weight updates
- exact and wildcard remote-interest queries
**Step 2: Add tests for match result stability**
- Ensure `Match()` still returns correct plain and queue subscription sets for exact, `*`, and `>` subjects.
- Add a test that specifically locks cache behavior across generation bumps.
**Step 3: Run the focused tests to prove the new coverage fails first**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~SubList" -c Release`
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj --filter "FullyQualifiedName~RouteRemoteSubCleanupParityBatch2Tests|FullyQualifiedName~RouteInterestIdempotencyTests" -c Release`
- Expected: FAIL only in the newly added allocation-guard or key-behavior tests.
**Step 4: Commit the failing-test baseline**
- Run:
```bash
git add tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs tests/NATS.Server.Core.Tests/Subscriptions/SubListAllocationGuardTests.cs tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs
git commit -m "test: lock SubList remote-key and match behavior"
```
## Task 2: Replace composite routed-sub strings with a dedicated value key
**Files:**
- Create: `src/NATS.Server/Subscriptions/RoutedSubKey.cs`
- Modify: `src/NATS.Server/Subscriptions/SubList.cs`
- Modify: `tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs`
**Step 1: Introduce a strongly typed routed-sub key**
- Add a small immutable value type for `(RouteId, Account, Subject, Queue)`.
- Use it as the dictionary key for `_remoteSubs` instead of the `"route|account|subject|queue"` composite string.
**Step 2: Remove string split/rebuild helpers from hot paths**
- Replace `BuildRoutedSubKey(...)`, `GetAccNameFromRoutedSubKey(...)`, and `GetRoutedSubKeyInfo(...)` usage in runtime paths with the new value key.
- Keep compatibility helper coverage only if other call sites still require string-facing helpers temporarily.
**Step 3: Run remote-interest tests**
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj --filter "FullyQualifiedName~RouteRemoteSubCleanupParityBatch2Tests|FullyQualifiedName~RouteSubscriptionTests" -c Release`
- Run: `dotnet test tests/NATS.Server.Gateways.Tests/NATS.Server.Gateways.Tests.csproj --filter "FullyQualifiedName~GatewayInterestModeTests|FullyQualifiedName~GatewayInterestIdempotencyTests" -c Release`
- Expected: PASS.
**Step 4: Commit the key-model refactor**
- Run:
```bash
git add src/NATS.Server/Subscriptions/RoutedSubKey.cs src/NATS.Server/Subscriptions/SubList.cs tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs
git commit -m "perf: replace SubList routed-sub string keys"
```
## Task 3: Remove avoidable string churn from trie traversal
**Files:**
- Modify: `src/NATS.Server/Subscriptions/SubList.cs`
- Modify: `src/NATS.Server/Subscriptions/SubjectMatch.cs`
- Modify: `tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs`
**Step 1: Rework token traversal helpers**
- Add a shared subject token walker that can expose tokens as spans and only allocate when a trie node insertion truly needs a durable string key.
- Remove repeated `token.ToString()` in traversal paths where lookups can operate on a transient token view first.
**Step 2: Keep exact-subject match paths allocation-lean**
- Prefer span/token comparison helpers for exact-match and wildcard traversal logic.
- Leave wildcard semantics unchanged.
**Step 3: Run core subscription tests**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~SubListGoParityTests|FullyQualifiedName~SubListCtorAndNotificationParityTests|FullyQualifiedName~SubListParityBatch2Tests" -c Release`
- Expected: PASS.
**Step 4: Commit trie traversal cleanup**
- Run:
```bash
git add src/NATS.Server/Subscriptions/SubList.cs src/NATS.Server/Subscriptions/SubjectMatch.cs tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs
git commit -m "perf: reduce SubList token string churn"
```
## Task 4: Pool `Match()` result building and remove cleanup copies
**Files:**
- Modify: `src/NATS.Server/Subscriptions/SubList.cs`
- Modify: `tests/NATS.Server.Core.Tests/Subscriptions/SubListAllocationGuardTests.cs`
- Modify: `tests/NATS.Server.Clustering.Tests/Routes/RouteSubscriptionTests.cs`
**Step 1: Replace temporary per-match collections**
- Rework `Match()` so temporary result building uses pooled builders or `ArrayBufferWriter<T>` instead of fresh nested `List<T>` allocations on every call.
- Preserve the current public `SubListResult` shape unless profiling proves a larger contract change is justified.
**Step 2: Remove `ToArray()` cleanup passes over `_remoteSubs`**
- Update `RemoveRemoteSubs(...)` and `RemoveRemoteSubsForAccount(...)` to avoid eager dictionary array copies.
- Ensure removal remains correct under the existing lock discipline.
**Step 3: Run cross-module regression**
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj --filter "FullyQualifiedName~RouteSubscriptionTests|FullyQualifiedName~RouteInterestIdempotencyTests" -c Release`
- Run: `dotnet test tests/NATS.Server.Gateways.Tests/NATS.Server.Gateways.Tests.csproj --filter "FullyQualifiedName~GatewayInterestModeTests|FullyQualifiedName~GatewayForwardingTests" -c Release`
- Expected: PASS.
**Step 4: Commit match-builder changes**
- Run:
```bash
git add src/NATS.Server/Subscriptions/SubList.cs tests/NATS.Server.Core.Tests/Subscriptions/SubListAllocationGuardTests.cs tests/NATS.Server.Clustering.Tests/Routes/RouteSubscriptionTests.cs
git commit -m "perf: pool SubList match builders and cleanup scans"
```
## Task 5: Add benchmark coverage, update docs, and run full verification
**Files:**
- Create: `tests/NATS.Server.Benchmark.Tests/CorePubSub/SubListMatchBenchmarks.cs`
- Modify: `Documentation/Subscriptions/SubList.md`
**Step 1: Add focused `SubList` benchmarks**
- Measure exact-match, wildcard-match, queue-sub, and remote-interest scenarios.
- Capture throughput and allocations before/after the refactor.
**Step 2: Update subscription documentation**
- Document the new routed-sub key model, the allocation strategy for trie matching, and any remaining intentional copies.
**Step 3: Run full verification**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Gateways.Tests/NATS.Server.Gateways.Tests.csproj -c Release`
- Run: `dotnet test tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj --filter "FullyQualifiedName~SubList" -c Release`
- Expected: PASS; benchmark output shows reduced per-match allocations.
**Step 4: Commit the documentation and benchmark work**
- Run:
```bash
git add tests/NATS.Server.Benchmark.Tests/CorePubSub/SubListMatchBenchmarks.cs Documentation/Subscriptions/SubList.md
git commit -m "docs: record SubList allocation strategy"
```
## Task 6: Merge the verified SubList work back to `main` and clean up the worktree
**Files:**
- No source changes expected
**Step 1: Confirm the feature branch is fully verified before merge**
- Reuse the verification from Task 5. Do not merge if the core, clustering, gateway, or benchmark test commands are failing.
**Step 2: Merge `codex/sublist-allocation-reduction` back into `main`**
- Return to the primary repository worktree and run:
```bash
git checkout main
git pull
git merge codex/sublist-allocation-reduction
```
- Expected: the SubList optimization commits merge cleanly into `main`.
**Step 3: Re-run verification on the merged `main` branch**
- Run: `dotnet test tests/NATS.Server.Core.Tests/NATS.Server.Core.Tests.csproj --filter "FullyQualifiedName~SubList" -c Release`
- Run: `dotnet test tests/NATS.Server.Clustering.Tests/NATS.Server.Clustering.Tests.csproj --filter "FullyQualifiedName~RouteRemoteSubCleanupParityBatch2Tests|FullyQualifiedName~RouteInterestIdempotencyTests|FullyQualifiedName~RouteSubscriptionTests" -c Release`
- Run: `dotnet test tests/NATS.Server.Gateways.Tests/NATS.Server.Gateways.Tests.csproj --filter "FullyQualifiedName~GatewayInterestModeTests|FullyQualifiedName~GatewayInterestIdempotencyTests|FullyQualifiedName~GatewayForwardingTests" -c Release`
- Run: `dotnet test tests/NATS.Server.Benchmark.Tests/NATS.Server.Benchmark.Tests.csproj --filter "FullyQualifiedName~SubList" -c Release`
- Expected: PASS on merged `main`.
**Step 4: Delete the feature branch and remove the worktree**
- Run:
```bash
git branch -d codex/sublist-allocation-reduction
git worktree remove .worktrees/codex-sublist-allocation-reduction
```
- Expected: only `main` remains checked out in the primary workspace, and the temporary SubList worktree is removed.
## Task 7: Run the full benchmark suite per the benchmark README and update `benchmarks_comparison.md`
**Files:**
- Verify workflow against: `tests/NATS.Server.Benchmark.Tests/README.md`
- Modify: `benchmarks_comparison.md`
**Step 1: Run the benchmark test project using the repository-documented command**
- From the primary repository worktree on verified `main`, run:
```bash
dotnet test tests/NATS.Server.Benchmark.Tests \
--filter "Category=Benchmark" \
-v normal \
--logger "console;verbosity=detailed" 2>&1 | tee /tmp/bench-output.txt
```
- Expected: the full benchmark suite completes and writes detailed comparison output to `/tmp/bench-output.txt`.
**Step 2: Extract the benchmark comparison blocks from the captured output**
- Open `/tmp/bench-output.txt` and pull the side-by-side comparison blocks from the `Standard Output Messages` sections for:
- core pub-only
- core 1:1 pub/sub
- core fan-out
- core multi pub/sub
- request/reply
- JetStream publish
- JetStream consumption
**Step 3: Update `benchmarks_comparison.md`**
- Refresh:
- the benchmark run date on the first line
- the environment description if toolchain or machine details changed
- throughput, MB/s, ratio, and latency values in the benchmark tables
- summary assessments and key observations if the ratios materially changed
- Keep the narrative tied to measured results from `/tmp/bench-output.txt`; do not preserve stale claims that no longer match the numbers.
**Step 4: Commit the benchmark comparison refresh**
- Run:
```bash
git add benchmarks_comparison.md
git commit -m "docs: refresh benchmark comparison after SubList optimization"
```
- Expected: `main` contains the merged SubList optimization work plus the refreshed benchmark comparison document.
## Completion Checklist
- [ ] SubList optimization work was implemented in an isolated `.worktrees/codex-sublist-allocation-reduction` worktree on branch `codex/sublist-allocation-reduction`.
- [ ] `_remoteSubs` no longer uses composite string keys in runtime paths.
- [ ] Remote cleanup paths no longer depend on `Split('|')` and `_remoteSubs.ToArray()`.
- [ ] Trie traversal materially reduces `token.ToString()` churn.
- [ ] `Match()` uses pooled or allocation-lean temporary builders.
- [ ] Core, clustering, and gateway parity tests remain green.
- [ ] `Documentation/Subscriptions/SubList.md` explains the new key and match strategy.
- [ ] Verified SubList optimization commits were merged back into `main`.
- [ ] The full benchmark test project was run on merged `main` per `tests/NATS.Server.Benchmark.Tests/README.md`.
- [ ] `benchmarks_comparison.md` was updated to match the latest benchmark output.
## Concise Execution Checklist (Current Codebase)
- [ ] Start from the current repo root and leave unrelated MQTT worktree changes untouched.
- [ ] Create `.worktrees/codex-sublist-allocation-reduction` on `codex/sublist-allocation-reduction`.
- [ ] Verify the current SubList baseline with:
- `tests/NATS.Server.Core.Tests/Subscriptions/SubListGoParityTests.cs`
- `tests/NATS.Server.Core.Tests/Subscriptions/SubListCtorAndNotificationParityTests.cs`
- `tests/NATS.Server.Core.Tests/Subscriptions/SubListParityBatch2Tests.cs`
- `tests/NATS.Server.Clustering.Tests/Routes/RouteRemoteSubCleanupParityBatch2Tests.cs`
- `tests/NATS.Server.Clustering.Tests/Routes/RouteInterestIdempotencyTests.cs`
- `tests/NATS.Server.Clustering.Tests/Routes/RouteSubscriptionTests.cs`
- `tests/NATS.Server.Gateways.Tests/Gateways/GatewayInterestModeTests.cs`
- `tests/NATS.Server.Gateways.Tests/Gateways/GatewayForwardingTests.cs`
- [ ] Add new guard coverage in `tests/NATS.Server.Core.Tests/Subscriptions/SubListAllocationGuardTests.cs`.
- [ ] Refactor `src/NATS.Server/Subscriptions/SubList.cs` to use a new `src/NATS.Server/Subscriptions/RoutedSubKey.cs`.
- [ ] Reduce token string churn in `src/NATS.Server/Subscriptions/SubList.cs` and `src/NATS.Server/Subscriptions/SubjectMatch.cs`.
- [ ] Add a new benchmark file at `tests/NATS.Server.Benchmark.Tests/CorePubSub/SubListMatchBenchmarks.cs` beside the existing CorePubSub benchmarks.
- [ ] Update `Documentation/Subscriptions/SubList.md`.
- [ ] Run full verification for core, clustering, gateway, and focused SubList benchmark tests.
- [ ] Merge `codex/sublist-allocation-reduction` into `main` and re-run the merged verification.
- [ ] Run the full benchmark suite using `tests/NATS.Server.Benchmark.Tests/README.md` guidance and refresh `benchmarks_comparison.md`.
+297
View File
@@ -0,0 +1,297 @@
# .NET 10 Optimization Opportunities for `NATS.Server`
This document identifies the highest-value places in the current .NET port that are still leaving performance on the table relative to what modern .NET 10 can do well. The focus is runtime behavior in the current codebase, not generic style guidance.
The ranking is based on likely payoff in NATS workloads:
1. Protocol parsing and per-message delivery paths
2. Subscription matching and routing fanout
3. JetStream storage hot paths
4. Route, leaf, MQTT, and monitoring paths with avoidable allocation churn
Several areas already use `Span<T>`, `ReadOnlyMemory<byte>`, `SequenceReader<byte>`, and stack allocation correctly. The remaining gaps are mostly where the code falls back to `string`, `byte[]`, `List<T>`, `ToArray()`, LINQ, or repeated serialization work on hot paths.
## Detailed Implementation Plans
- [Parser span-retention plan](docs/plans/2026-03-13-optimizations_parser-plan.md)
- [SubList allocation-reduction plan](docs/plans/2026-03-13-optimizations_sublist-plan.md)
- [FileStore payload-and-index plan](docs/plans/2026-03-13-optimizations_filestore-plan.md)
## Highest ROI
### 1. Keep parser state in bytes/spans longer
- Files:
- `src/NATS.Server/Protocol/NatsParser.cs`
- `src/NATS.Server/NatsClient.cs`
- Current issue:
- `NatsParser` tokenizes control lines with spans, but then converts subjects, reply subjects, queue names, SIDs, and JSON payloads into `string` and `byte[]` immediately.
- `TryReadPayload()` always allocates a new `byte[]` and copies the payload, even when the underlying `ReadOnlySequence<byte>` is already usable.
- `ParseConnect()` and `ParseInfo()` call `ToArray()` on the JSON portion.
- Why it matters:
- This runs for every client protocol command.
- The parser sits directly on the publish/subscribe hot path, so small per-command allocations scale badly under fan-in.
- Recommended optimization:
- Introduce a split parsed representation:
- a hot-path `ref struct` or `readonly struct` view carrying `ReadOnlySpan<byte>` / `ReadOnlySequence<byte>` slices for subject, reply, SID, queue, and payload
- a slower materialization path only when code actually needs `string`
- Store pending parser state as byte slices or pooled byte segments instead of `_pendingSubject` / `_pendingReplyTo` strings.
- For single-segment payloads, hand through a `ReadOnlyMemory<byte>` slice rather than copying to a new array.
- Only copy multi-segment payloads when required.
- Use `SearchValues<byte>` for whitespace scanning and command detection instead of manual per-byte branching where it simplifies repeated searches.
- .NET 10 techniques:
- `ref struct`
- `ReadOnlySpan<byte>`
- `ReadOnlySequence<byte>`
- `SearchValues<byte>`
- `Encoding.ASCII.GetString(ReadOnlySpan<byte>)` only at materialization boundaries
- Risk / complexity:
- Medium to high. This touches command parsing contracts and downstream consumers.
- Worth doing first because it reduces allocations before messages enter the rest of the server.
### 2. Remove string-heavy trie traversal in `SubList`
- Files:
- `src/NATS.Server/Subscriptions/SubList.cs`
- `src/NATS.Server/Subscriptions/SubjectMatch.cs`
- Current issue:
- Insert/remove paths repeatedly call `token.ToString()`.
- Routed subscription keys are synthesized as `"route|account|subject|queue"` strings and later split back with `Split('|')`.
- Match path tokenization and cache population allocate arrays/lists and depend on string tokens.
- `RemoveRemoteSubs()` and `RemoveRemoteSubsForAccount()` call `_remoteSubs.ToArray()` and re-parse keys on every sweep.
- Why it matters:
- `SubList.Match()` is one of the most performance-sensitive operations in the server.
- Remote interest tracking becomes more expensive as the route/leaf topology grows.
- Recommended optimization:
- Replace composite routed-sub string keys with a dedicated value key:
- `readonly record struct RoutedSubKey(string RouteId, string Account, string Subject, string? Queue)`
- or a plain `readonly struct` with a custom comparer if profiling shows hash/comparison cost matters
- Keep tokenized subjects in a pooled or cached token form for exact subjects.
- Investigate a span-based token walker for matching so exact-subject lookups avoid `string[]` creation entirely.
- Replace temporary `List<Subscription>` / `List<List<Subscription>>` creation in `Match()` with pooled builders or `ArrayBufferWriter<T>`.
- For remote-sub cleanup, iterate dictionary entries without `ToArray()` and avoid `Split`.
- .NET 10 techniques:
- `readonly struct` / `readonly record struct` for composite keys
- `ReadOnlySpan<char>` token parsing
- pooled builders via `ArrayPool<T>` or `ArrayBufferWriter<T>`
- Risk / complexity:
- Medium. The data model change is straightforward; changing trie matching internals requires careful parity testing.
### 3. Eliminate avoidable message duplication in `FileStore`
- Files:
- `src/NATS.Server/JetStream/Storage/FileStore.cs`
- `src/NATS.Server/JetStream/Storage/MsgBlock.cs`
- `src/NATS.Server/JetStream/Storage/StoredMessage.cs`
- Current issue:
- `AppendAsync()` transforms payload for persistence and often also keeps another managed copy in `_messages`.
- `StoreMsg()` creates a combined `byte[]` for headers + payload.
- Many maintenance operations (`TrimToMaxMessages`, `PurgeEx`, `LoadLastBySubjectAsync`, `ListAsync`) use LINQ over `_messages.Values`, causing iterator allocations and repeated scans.
- Snapshot creation base64-encodes transformed payloads, forcing extra copies.
- Why it matters:
- JetStream storage code runs continuously under persistence-heavy workloads.
- It is both allocation-sensitive and memory-residency-sensitive.
- Recommended optimization:
- Split stored payload representation into:
- persisted payload bytes
- logical payload view
- optional headers view
- Avoid constructing concatenated header+payload arrays when the record format can encode both spans directly.
- Rework `StoredMessage` so hot metadata stays compact; consider a smaller `readonly struct` for indexes/metadata while payload storage remains reference-based.
- Replace LINQ scans in hot maintenance paths with explicit loops.
- Add per-subject indexes or rolling pointers for operations currently implemented as full scans when those operations are expected to be common.
- .NET 10 techniques:
- `ReadOnlyMemory<byte>` slices over shared buffers
- `readonly struct` for compact metadata/index entries
- explicit loops over LINQ in storage hot paths
- `CollectionsMarshal` where safe for dictionary/list access in tight loops
- Risk / complexity:
- High. This area needs careful correctness validation for retention, snapshots, and recovery.
- High payoff for persistent streams.
### 4. Reduce formatting and copy overhead in route and leaf message sends
- Files:
- `src/NATS.Server/Routes/RouteConnection.cs`
- `src/NATS.Server/LeafNodes/LeafConnection.cs`
- Current issue:
- Control lines are built with string interpolation, converted with `Encoding.ASCII.GetBytes`, then written separately from payload and trailer.
- `"\r\n"u8.ToArray()` allocates for every send.
- Batch protocol send methods build a `StringBuilder`, then allocate one big ASCII byte array.
- Why it matters:
- Cluster routes and leaf nodes are high-throughput transport paths in real deployments.
- This code is not as hot as client publish fanout, but it is hot enough to matter under clustered load.
- Recommended optimization:
- Mirror the client path:
- encode control lines into stackalloc or pooled byte buffers with span formatting
- write control + payload + CRLF via scatter-gather (`ReadOnlyMemory<byte>[]`) or a reusable outbound buffer
- Replace repeated CRLF arrays with a static `ReadOnlyMemory<byte>` / `ReadOnlySpan<byte>`.
- For route sub protocol batches, encode directly into an `ArrayBufferWriter<byte>` instead of `StringBuilder` -> string -> bytes.
- .NET 10 techniques:
- `string.Create` or direct span formatting into pooled buffers
- `ArrayBufferWriter<byte>`
- scatter-gather writes where transport permits
- Risk / complexity:
- Medium. Mostly localized refactoring with low semantic risk.
## Medium ROI
### 5. Stop using LINQ-heavy materialization in monitoring endpoints
- Files:
- `src/NATS.Server/Monitoring/ConnzHandler.cs`
- `src/NATS.Server/Monitoring/SubszHandler.cs`
- Current issue:
- Monitoring paths repeatedly call `ToArray()`, `Select()`, `Where()`, `OrderBy()`, `Skip()`, and `Take()`.
- `SubszHandler` builds full subscription lists even when the request only needs counts.
- `ConnzHandler` repeatedly rematerializes arrays while filtering and sorting.
- Why it matters:
- Monitoring endpoints are not the publish hot path, but they can become disruptive on busy servers with many clients/subscriptions.
- These allocations are easy to avoid.
- Recommended optimization:
- Separate count-only and detail-request paths.
- Use single-pass loops and pooled temporary lists.
- Delay expensive subscription detail expansion until after paging when possible.
- Consider returning immutable snapshots generated incrementally by the server for common monitor queries.
- .NET 10 techniques:
- explicit loops
- pooled arrays/lists
- `CollectionsMarshal.AsSpan()` for internal list traversal where safe
- Risk / complexity:
- Low to medium.
### 6. Modernize MQTT packet writing and text parsing
- Files:
- `src/NATS.Server/Mqtt/MqttPacketWriter.cs`
- `src/NATS.Server/Mqtt/MqttProtocolParser.cs`
- Current issue:
- `MqttPacketWriter` returns fresh `byte[]` instances for every string/packet write.
- Remaining-length encoding returns `scratch[..index].ToArray()`.
- `ParseLine()` uses `Trim()`, `StartsWith()`, `Split()`, slicing, and string-based parsing throughout.
- Why it matters:
- MQTT is a side protocol, so this is not the top optimization target.
- Still worth fixing because the code is currently allocation-heavy and straightforward to improve.
- Recommended optimization:
- Add `TryWrite...` APIs that write into caller-provided `Span<byte>` / `IBufferWriter<byte>`.
- Keep remaining-length bytes on the stack and copy directly into the final destination buffer.
- Rework `ParseLine()` to operate on `ReadOnlySpan<char>` and avoid `Split`.
- .NET 10 techniques:
- `Span<byte>`
- `IBufferWriter<byte>`
- `ReadOnlySpan<char>`
- `SearchValues<char>` for token separators if useful
- Risk / complexity:
- Low.
### 7. Replace full scans over `_messages` with maintained indexes where operations are common
- Files:
- `src/NATS.Server/JetStream/Storage/FileStore.cs`
- Current issue:
- `LoadLastBySubjectAsync()` scans all messages, filters, sorts descending, and picks the first result.
- `TrimToMaxMessages()` repeatedly calls `_messages.Keys.Min()`.
- `PurgeEx()` materializes candidate lists before deletion.
- Why it matters:
- These are algorithmic inefficiencies, not just allocation issues.
- They become more visible as streams grow.
- Recommended optimization:
- Maintain lightweight indexes:
- last sequence by subject
- first/last live sequence tracking without `Min()` / `Max()` scans
- optionally per-subject linked or sorted sequence sets for purge/retention operations
- If full indexing is too large a change, replace repeated LINQ scans with single-pass loops immediately.
- .NET 10 techniques:
- compact metadata structs
- tighter dictionary usage
- fewer transient enumerators
- Risk / complexity:
- Medium.
### 8. Reduce repeated small allocations in protocol constants and control frames
- Files:
- `src/NATS.Server/Protocol/NatsParser.cs`
- `src/NATS.Server/Routes/RouteConnection.cs`
- `src/NATS.Server/LeafNodes/LeafConnection.cs`
- other transport helpers
- Current issue:
- Some constants are still materialized via `ToArray()` rather than held as static `byte[]` or `ReadOnlyMemory<byte>`.
- Control frames repeatedly build temporary arrays for tiny literals.
- Why it matters:
- These are cheap wins and remove noisy allocation churn.
- Recommended optimization:
- Standardize on shared static byte literals for CRLF and fixed protocol tokens.
- Audit for repeated `u8.ToArray()` or `Encoding.ASCII.GetBytes` on invariant text.
- .NET 10 techniques:
- static cached buffers
- span-based concatenation into reusable destinations
- Risk / complexity:
- Low.
## Lower ROI Or Caution Areas
### 9. Be selective about introducing more structs
- Files:
- cross-cutting
- Current issue:
- Some parts of the code would benefit from value types, but others already contain references (`string`, `byte[]`, `ReadOnlyMemory<byte>`, dictionaries) where converting whole models to structs would increase copying and call-site complexity.
- Recommendation:
- Good struct candidates:
- composite dictionary keys
- compact metadata/index entries
- parser token views
- queue or routing bookkeeping records
- Poor struct candidates:
- large mutable models
- objects with many reference-type fields
- stateful connection objects
- Why it matters:
- “Use more structs” is only a win when the values are small, immutable, and heavily allocated.
### 10. Avoid premature replacement of already-good memory APIs
- Files:
- `src/NATS.Server/NatsClient.cs`
- `src/NATS.Server/IO/OutboundBufferPool.cs`
- several JetStream codecs
- Current issue:
- There has already been meaningful optimization work in direct write buffering and pooled outbound paths.
- Replacing these with more exotic abstractions without profiling could regress behavior.
- Recommendation:
- Prefer extending the current buffer-pool and direct-write patterns into routes, leaves, and parser payload handling before redesigning the client write path again.
## Suggested Implementation Order
1. `NatsParser` hot-path byte retention and reduced payload copying
2. `SubList` key/token allocation cleanup and remote-sub key redesign
3. Route/leaf outbound buffer encoding cleanup
4. `FileStore` hot-path de-LINQ and payload/index refactoring
5. Monitoring endpoint de-materialization
6. MQTT writer/parser span-based cleanup
## What To Measure Before And After
Use the benchmark project and targeted microbenchmarks to measure:
- allocations per `PUB`, `SUB`, `UNSUB`, `CONNECT`
- allocations per delivered message under fanout
- `SubList.Match()` throughput and allocations for:
- exact subjects
- wildcard subjects
- queue subscriptions
- remote interest present
- JetStream append throughput and bytes allocated per append
- route/leaf forwarded-message allocations
- monitoring endpoint allocations for large client/subscription sets
## Summary
The best remaining gains are not from sprinkling `Span<T>` everywhere. They come from carrying byte-oriented data further through the hot paths, removing composite-string bookkeeping, reducing duplicate payload ownership in JetStream storage, and eliminating materialization-heavy helper code around routing and monitoring.
If you only do three things, do these first:
1. Rework `NatsParser` to avoid early `string` / `byte[]` creation.
2. Replace `SubList` composite string keys and string-heavy token handling.
3. Refactor `FileStore` and route/leaf send paths to reduce duplicate buffers and transient formatting allocations.
@@ -0,0 +1,7 @@
namespace NATS.Server.Subscriptions;
internal readonly record struct RoutedSubKey(string RouteId, string Account, string Subject, string? Queue)
{
public static RoutedSubKey FromRemoteSubscription(RemoteSubscription sub)
=> new(sub.RouteId, sub.Account, sub.Subject, sub.Queue);
}
+173 -65
View File
@@ -12,11 +12,15 @@ public sealed class SubList : IDisposable
{ {
private const int CacheMax = 1024; private const int CacheMax = 1024;
private const int CacheSweep = 256; private const int CacheSweep = 256;
[ThreadStatic]
private static MatchBuilder? s_matchBuilder;
[ThreadStatic]
private static List<RoutedSubKey>? s_remoteSubRemovalKeys;
private readonly ReaderWriterLockSlim _lock = new(); private readonly ReaderWriterLockSlim _lock = new();
private readonly TrieLevel _root = new(); private readonly TrieLevel _root = new();
private readonly SubListCacheSweeper _sweeper = new(); private readonly SubListCacheSweeper _sweeper = new();
private readonly Dictionary<string, RemoteSubscription> _remoteSubs = new(StringComparer.Ordinal); private readonly Dictionary<RoutedSubKey, RemoteSubscription> _remoteSubs = [];
private Dictionary<string, CachedResult>? _cache = new(StringComparer.Ordinal); private Dictionary<string, CachedResult>? _cache = new(StringComparer.Ordinal);
private uint _count; private uint _count;
private volatile bool _disposed; private volatile bool _disposed;
@@ -31,8 +35,6 @@ public sealed class SubList : IDisposable
private readonly Dictionary<string, List<Action<bool>>> _queueRemoveNotifications = new(StringComparer.Ordinal); private readonly Dictionary<string, List<Action<bool>>> _queueRemoveNotifications = new(StringComparer.Ordinal);
private readonly record struct CachedResult(SubListResult Result, long Generation); private readonly record struct CachedResult(SubListResult Result, long Generation);
internal readonly record struct RoutedSubKeyInfo(string RouteId, string Account, string Subject, string? Queue);
public event Action<InterestChange>? InterestChanged; public event Action<InterestChange>? InterestChanged;
public SubList() public SubList()
@@ -178,7 +180,7 @@ public sealed class SubList : IDisposable
_lock.EnterWriteLock(); _lock.EnterWriteLock();
try try
{ {
var key = BuildRoutedSubKey(sub.RouteId, sub.Account, sub.Subject, sub.Queue); var key = RoutedSubKey.FromRemoteSubscription(sub);
var changed = false; var changed = false;
if (sub.IsRemoval) if (sub.IsRemoval)
{ {
@@ -223,7 +225,7 @@ public sealed class SubList : IDisposable
_lock.EnterWriteLock(); _lock.EnterWriteLock();
try try
{ {
var key = BuildRoutedSubKey(sub.RouteId, sub.Account, sub.Subject, sub.Queue); var key = RoutedSubKey.FromRemoteSubscription(sub);
if (!_remoteSubs.TryGetValue(key, out var existing)) if (!_remoteSubs.TryGetValue(key, out var existing))
return; return;
@@ -240,51 +242,36 @@ public sealed class SubList : IDisposable
} }
} }
internal static string BuildRoutedSubKey(string routeId, string account, string subject, string? queue)
=> $"{routeId}|{account}|{subject}|{queue}";
internal static string? GetAccNameFromRoutedSubKey(string routedSubKey)
=> GetRoutedSubKeyInfo(routedSubKey)?.Account;
internal static RoutedSubKeyInfo? GetRoutedSubKeyInfo(string routedSubKey)
{
if (string.IsNullOrWhiteSpace(routedSubKey))
return null;
var parts = routedSubKey.Split('|');
if (parts.Length != 4)
return null;
if (parts[0].Length == 0 || parts[1].Length == 0 || parts[2].Length == 0)
return null;
var queue = parts[3].Length == 0 ? null : parts[3];
return new RoutedSubKeyInfo(parts[0], parts[1], parts[2], queue);
}
public int RemoveRemoteSubs(string routeId) public int RemoveRemoteSubs(string routeId)
{ {
_lock.EnterWriteLock(); _lock.EnterWriteLock();
try try
{ {
var removalKeys = RentRemoteSubRemovalKeys();
var removed = 0; var removed = 0;
foreach (var kvp in _remoteSubs.ToArray()) foreach (var (key, _) in _remoteSubs)
{ {
var info = GetRoutedSubKeyInfo(kvp.Key); if (!string.Equals(key.RouteId, routeId, StringComparison.Ordinal))
if (info == null || !string.Equals(info.Value.RouteId, routeId, StringComparison.Ordinal))
continue; continue;
if (_remoteSubs.Remove(kvp.Key)) removalKeys.Add(key);
}
foreach (var key in removalKeys)
{
if (_remoteSubs.Remove(key, out var removedSub))
{ {
removed++; removed++;
InterestChanged?.Invoke(new InterestChange( InterestChanged?.Invoke(new InterestChange(
InterestChangeKind.RemoteRemoved, InterestChangeKind.RemoteRemoved,
kvp.Value.Subject, removedSub.Subject,
kvp.Value.Queue, removedSub.Queue,
kvp.Value.Account)); removedSub.Account));
} }
} }
removalKeys.Clear();
if (removed > 0) if (removed > 0)
Interlocked.Increment(ref _generation); Interlocked.Increment(ref _generation);
@@ -301,30 +288,34 @@ public sealed class SubList : IDisposable
_lock.EnterWriteLock(); _lock.EnterWriteLock();
try try
{ {
var removalKeys = RentRemoteSubRemovalKeys();
var removed = 0; var removed = 0;
foreach (var kvp in _remoteSubs.ToArray()) foreach (var (key, _) in _remoteSubs)
{ {
var info = GetRoutedSubKeyInfo(kvp.Key); if (!string.Equals(key.RouteId, routeId, StringComparison.Ordinal)
if (info == null) || !string.Equals(key.Account, account, StringComparison.Ordinal))
continue;
if (!string.Equals(info.Value.RouteId, routeId, StringComparison.Ordinal)
|| !string.Equals(info.Value.Account, account, StringComparison.Ordinal))
{ {
continue; continue;
} }
if (_remoteSubs.Remove(kvp.Key)) removalKeys.Add(key);
}
foreach (var key in removalKeys)
{
if (_remoteSubs.Remove(key, out var removedSub))
{ {
removed++; removed++;
InterestChanged?.Invoke(new InterestChange( InterestChanged?.Invoke(new InterestChange(
InterestChangeKind.RemoteRemoved, InterestChangeKind.RemoteRemoved,
kvp.Value.Subject, removedSub.Subject,
kvp.Value.Queue, removedSub.Queue,
kvp.Value.Account)); removedSub.Account));
} }
} }
removalKeys.Clear();
if (removed > 0) if (removed > 0)
Interlocked.Increment(ref _generation); Interlocked.Increment(ref _generation);
@@ -391,9 +382,9 @@ public sealed class SubList : IDisposable
} }
else else
{ {
var key = token.ToString(); if (!TryGetLiteralNode(level, token, out _, out node))
if (!level.Nodes.TryGetValue(key, out node))
{ {
var key = token.ToString();
node = new TrieNode(); node = new TrieNode();
level.Nodes[key] = node; level.Nodes[key] = node;
} }
@@ -503,13 +494,20 @@ public sealed class SubList : IDisposable
} }
else else
{ {
level.Nodes.TryGetValue(token.ToString(), out node); if (!TryGetLiteralNode(level, token, out var existingToken, out node))
return false;
pathList.Add((level, node, existingToken, isPwc: false, isFwc: false));
if (node.Next == null)
return false; // corrupted trie state
level = node.Next;
continue;
} }
if (node == null) if (node == null)
return false; // not found return false; // not found
var tokenStr = token.ToString(); var tokenStr = isPwc ? "*" : ">";
pathList.Add((level, node, tokenStr, isPwc, isFwc)); pathList.Add((level, node, tokenStr, isPwc, isFwc));
if (node.Next == null) if (node.Next == null)
return false; // corrupted trie state return false; // corrupted trie state
@@ -587,22 +585,9 @@ public sealed class SubList : IDisposable
return cached.Result; return cached.Result;
} }
var plainSubs = new List<Subscription>(); var builder = RentMatchBuilder();
var queueSubs = new List<List<Subscription>>(); MatchLevel(_root, tokens, 0, builder);
MatchLevel(_root, tokens, 0, plainSubs, queueSubs); var result = builder.ToResult();
SubListResult result;
if (plainSubs.Count == 0 && queueSubs.Count == 0)
{
result = SubListResult.Empty;
}
else
{
var queueSubsArr = new Subscription[queueSubs.Count][];
for (int i = 0; i < queueSubs.Count; i++)
queueSubsArr[i] = queueSubs[i].ToArray();
result = new SubListResult(plainSubs.ToArray(), queueSubsArr);
}
if (_cache != null) if (_cache != null)
{ {
@@ -681,6 +666,37 @@ public sealed class SubList : IDisposable
return removed; return removed;
} }
private static bool TryGetLiteralNode(TrieLevel level, ReadOnlySpan<char> token, out string existingToken, out TrieNode node)
{
foreach (var (candidate, existingNode) in level.Nodes)
{
if (!SubjectMatch.TokenEquals(token, candidate))
continue;
existingToken = candidate;
node = existingNode;
return true;
}
existingToken = string.Empty;
node = null!;
return false;
}
private static MatchBuilder RentMatchBuilder()
{
var builder = s_matchBuilder ??= new MatchBuilder();
builder.Reset();
return builder;
}
private static List<RoutedSubKey> RentRemoteSubRemovalKeys()
{
var keys = s_remoteSubRemovalKeys ??= [];
keys.Clear();
return keys;
}
private bool HasExactQueueInterestNoLock(string subject, string queue) private bool HasExactQueueInterestNoLock(string subject, string queue)
{ {
var subs = new List<Subscription>(); var subs = new List<Subscription>();
@@ -827,6 +843,42 @@ public sealed class SubList : IDisposable
AddNodeToResults(pwc, plainSubs, queueSubs); AddNodeToResults(pwc, plainSubs, queueSubs);
} }
private static void MatchLevel(TrieLevel? level, string[] tokens, int tokenIndex, MatchBuilder builder)
{
TrieNode? pwc = null;
TrieNode? node = null;
for (int i = tokenIndex; i < tokens.Length; i++)
{
if (level == null)
return;
if (level.Fwc != null)
AddNodeToResults(level.Fwc, builder);
pwc = level.Pwc;
if (pwc != null)
MatchLevel(pwc.Next, tokens, i + 1, builder);
node = null;
if (level.Nodes.TryGetValue(tokens[i], out var found))
{
node = found;
level = node.Next;
}
else
{
level = null;
}
}
if (node != null)
AddNodeToResults(node, builder);
if (pwc != null)
AddNodeToResults(pwc, builder);
}
private static void AddNodeToResults(TrieNode node, private static void AddNodeToResults(TrieNode node,
List<Subscription> plainSubs, List<List<Subscription>> queueSubs) List<Subscription> plainSubs, List<List<Subscription>> queueSubs)
{ {
@@ -858,6 +910,19 @@ public sealed class SubList : IDisposable
} }
} }
private static void AddNodeToResults(TrieNode node, MatchBuilder builder)
{
builder.PlainSubs.AddRange(node.PlainSubs);
foreach (var (queueName, subs) in node.QueueSubs)
{
if (subs.Count == 0)
continue;
builder.AddQueueGroup(queueName, subs);
}
}
public SubListStats Stats() public SubListStats Stats()
{ {
_lock.EnterReadLock(); _lock.EnterReadLock();
@@ -1373,4 +1438,47 @@ public sealed class SubList : IDisposable
public bool IsEmpty => PlainSubs.Count == 0 && QueueSubs.Count == 0 && public bool IsEmpty => PlainSubs.Count == 0 && QueueSubs.Count == 0 &&
(Next == null || (Next.Nodes.Count == 0 && Next.Pwc == null && Next.Fwc == null)); (Next == null || (Next.Nodes.Count == 0 && Next.Pwc == null && Next.Fwc == null));
} }
private sealed class MatchBuilder
{
private readonly Dictionary<string, int> _queueIndexes = new(StringComparer.Ordinal);
private readonly List<List<Subscription>> _queueGroups = [];
private int _queueGroupCount;
public List<Subscription> PlainSubs { get; } = [];
public void Reset()
{
PlainSubs.Clear();
_queueIndexes.Clear();
for (var i = 0; i < _queueGroupCount; i++)
_queueGroups[i].Clear();
_queueGroupCount = 0;
}
public void AddQueueGroup(string queueName, HashSet<Subscription> subs)
{
if (!_queueIndexes.TryGetValue(queueName, out var index))
{
index = _queueGroupCount++;
_queueIndexes[queueName] = index;
if (index == _queueGroups.Count)
_queueGroups.Add([]);
}
_queueGroups[index].AddRange(subs);
}
public SubListResult ToResult()
{
if (PlainSubs.Count == 0 && _queueGroupCount == 0)
return SubListResult.Empty;
var queueSubsArr = new Subscription[_queueGroupCount][];
for (var i = 0; i < _queueGroupCount; i++)
queueSubsArr[i] = _queueGroups[i].ToArray();
return new SubListResult(PlainSubs.ToArray(), queueSubsArr);
}
}
} }
@@ -249,6 +249,9 @@ public static class SubjectMatch
return tokens.Count == test.Count; return tokens.Count == test.Count;
} }
internal static bool TokenEquals(ReadOnlySpan<char> token, string candidate)
=> token.SequenceEqual(candidate);
private static bool TokensCanMatch(ReadOnlySpan<char> t1, ReadOnlySpan<char> t2) private static bool TokensCanMatch(ReadOnlySpan<char> t1, ReadOnlySpan<char> t2)
{ {
if (t1.Length == 1 && (t1[0] == Pwc || t1[0] == Fwc)) if (t1.Length == 1 && (t1[0] == Pwc || t1[0] == Fwc))
@@ -0,0 +1,112 @@
using System.Diagnostics;
using NATS.Server.Benchmark.Tests.Harness;
using NATS.Server.Subscriptions;
using Xunit.Abstractions;
namespace NATS.Server.Benchmark.Tests.CorePubSub;
public class SubListMatchBenchmarks(ITestOutputHelper output)
{
[Fact]
[Trait("Category", "Benchmark")]
public void SubListExactMatch_128Subjects()
{
using var subList = new SubList();
for (var i = 0; i < 128; i++)
subList.Insert(new Subscription { Subject = $"bench.exact.{i}", Sid = i.ToString() });
var (result, allocatedBytes) = Measure("SubList Exact Match (128 subjects)", "DotNet", "bench.exact.64".Length, 250_000, () =>
{
_ = subList.Match("bench.exact.64");
});
BenchmarkResultWriter.WriteSingle(output, result);
WriteAllocationSummary(allocatedBytes, result.TotalMessages);
}
[Fact]
[Trait("Category", "Benchmark")]
public void SubListWildcardMatch_FanIn()
{
using var subList = new SubList();
subList.Insert(new Subscription { Subject = "orders.created", Sid = "1" });
subList.Insert(new Subscription { Subject = "orders.*", Sid = "2" });
subList.Insert(new Subscription { Subject = "orders.>", Sid = "3" });
subList.Insert(new Subscription { Subject = "orders.created.us", Sid = "4" });
var (result, allocatedBytes) = Measure("SubList Wildcard Match", "DotNet", "orders.created".Length, 250_000, () =>
{
_ = subList.Match("orders.created");
});
BenchmarkResultWriter.WriteSingle(output, result);
WriteAllocationSummary(allocatedBytes, result.TotalMessages);
}
[Fact]
[Trait("Category", "Benchmark")]
public void SubListQueueMatch_MergedGroups()
{
using var subList = new SubList();
subList.Insert(new Subscription { Subject = "jobs.run", Queue = "workers", Sid = "1" });
subList.Insert(new Subscription { Subject = "jobs.*", Queue = "workers", Sid = "2" });
subList.Insert(new Subscription { Subject = "jobs.>", Queue = "audit", Sid = "3" });
var (result, allocatedBytes) = Measure("SubList Queue Match", "DotNet", "jobs.run".Length, 250_000, () =>
{
_ = subList.Match("jobs.run");
});
BenchmarkResultWriter.WriteSingle(output, result);
WriteAllocationSummary(allocatedBytes, result.TotalMessages);
}
[Fact]
[Trait("Category", "Benchmark")]
public void SubListRemoteInterest_WildcardLookup()
{
using var subList = new SubList();
for (var i = 0; i < 64; i++)
subList.ApplyRemoteSub(new RemoteSubscription($"remote.{i}.*", null, $"r{i}", "A"));
var (result, allocatedBytes) = Measure("SubList Remote Interest", "DotNet", "remote.42.created".Length, 250_000, () =>
{
_ = subList.HasRemoteInterest("A", "remote.42.created");
});
BenchmarkResultWriter.WriteSingle(output, result);
WriteAllocationSummary(allocatedBytes, result.TotalMessages);
}
private static (BenchmarkResult Result, long AllocatedBytes) Measure(string name, string serverType, int bytesPerOperation, int iterations, Action operation)
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
for (var i = 0; i < 1_000; i++)
operation();
var before = GC.GetAllocatedBytesForCurrentThread();
var sw = Stopwatch.StartNew();
for (var i = 0; i < iterations; i++)
operation();
sw.Stop();
var allocatedBytes = GC.GetAllocatedBytesForCurrentThread() - before;
return (new BenchmarkResult
{
Name = name,
ServerType = serverType,
TotalMessages = iterations,
TotalBytes = (long)iterations * bytesPerOperation,
Duration = sw.Elapsed,
}, allocatedBytes);
}
private void WriteAllocationSummary(long allocatedBytes, long iterations)
{
output.WriteLine($"Allocated: {allocatedBytes:N0} B total | {allocatedBytes / (double)iterations:F2} B/op");
output.WriteLine("");
}
}
@@ -10,21 +10,14 @@ namespace NATS.Server.Clustering.Tests.Routes;
public class RouteRemoteSubCleanupParityBatch2Tests public class RouteRemoteSubCleanupParityBatch2Tests
{ {
[Fact] [Fact]
public void Routed_sub_key_helpers_parse_account_and_queue_fields() public void Routed_sub_key_exposes_route_account_subject_and_queue_fields()
{ {
var key = SubList.BuildRoutedSubKey("R1", "A", "orders.*", "q1"); var key = RoutedSubKey.FromRemoteSubscription(new RemoteSubscription("orders.*", "q1", "R1", "A"));
SubList.GetAccNameFromRoutedSubKey(key).ShouldBe("A"); key.RouteId.ShouldBe("R1");
key.Account.ShouldBe("A");
var info = SubList.GetRoutedSubKeyInfo(key); key.Subject.ShouldBe("orders.*");
info.ShouldNotBeNull(); key.Queue.ShouldBe("q1");
info.Value.RouteId.ShouldBe("R1");
info.Value.Account.ShouldBe("A");
info.Value.Subject.ShouldBe("orders.*");
info.Value.Queue.ShouldBe("q1");
SubList.GetRoutedSubKeyInfo("invalid").ShouldBeNull();
SubList.GetAccNameFromRoutedSubKey("invalid").ShouldBeNull();
} }
[Fact] [Fact]
@@ -47,6 +40,23 @@ public class RouteRemoteSubCleanupParityBatch2Tests
sl.HasRemoteInterest("B", "orders.created").ShouldBeTrue(); sl.HasRemoteInterest("B", "orders.created").ShouldBeTrue();
} }
[Fact]
public void Applying_same_remote_subscription_twice_is_idempotent_for_interest_tracking()
{
using var sl = new SubList();
var changes = new List<InterestChange>();
sl.InterestChanged += changes.Add;
var sub = new RemoteSubscription("orders.*", "workers", "r1", "A");
sl.ApplyRemoteSub(sub);
sl.ApplyRemoteSub(sub);
sl.HasRemoteInterest("A", "orders.created").ShouldBeTrue();
sl.MatchRemote("A", "orders.created").Count.ShouldBe(1);
changes.Count(change => change.Kind == InterestChangeKind.RemoteAdded).ShouldBe(1);
}
[Fact] [Fact]
public async Task Route_disconnect_cleans_remote_interest_without_explicit_rs_minus() public async Task Route_disconnect_cleans_remote_interest_without_explicit_rs_minus()
{ {
@@ -208,6 +208,50 @@ public class RouteSubscriptionTests
} }
} }
[Fact]
public async Task Removing_one_subject_keeps_other_remote_interest_intact()
{
var cluster = Guid.NewGuid().ToString("N");
var a = await StartServerAsync(MakeClusterOpts(cluster));
var b = await StartServerAsync(MakeClusterOpts(cluster, a.Server.ClusterListen!));
try
{
await WaitForRouteFormation(a.Server, b.Server);
await using var nc = new NatsConnection(new NatsOpts
{
Url = $"nats://127.0.0.1:{a.Server.Port}",
});
await nc.ConnectAsync();
await using var sub1 = await nc.SubscribeCoreAsync<string>("multi.one");
await using var sub2 = await nc.SubscribeCoreAsync<string>("multi.two");
await nc.PingAsync();
await WaitForCondition(() => b.Server.HasRemoteInterest("multi.one") && b.Server.HasRemoteInterest("multi.two"));
b.Server.HasRemoteInterest("multi.one").ShouldBeTrue();
b.Server.HasRemoteInterest("multi.two").ShouldBeTrue();
await sub1.DisposeAsync();
await nc.PingAsync();
await WaitForCondition(() => !b.Server.HasRemoteInterest("multi.one"));
b.Server.HasRemoteInterest("multi.one").ShouldBeFalse();
b.Server.HasRemoteInterest("multi.two").ShouldBeTrue();
await sub2.DisposeAsync();
await nc.PingAsync();
await WaitForCondition(() => !b.Server.HasRemoteInterest("multi.two"));
b.Server.HasRemoteInterest("multi.two").ShouldBeFalse();
}
finally
{
await DisposeServers(a, b);
}
}
// Go: RS+ wire protocol parsing (low-level) // Go: RS+ wire protocol parsing (low-level)
[Fact] [Fact]
public async Task RSplus_frame_registers_remote_interest_via_wire() public async Task RSplus_frame_registers_remote_interest_via_wire()
@@ -0,0 +1,64 @@
using System.Reflection;
using NATS.Server.Subscriptions;
namespace NATS.Server.Core.Tests;
public class SubListAllocationGuardTests
{
[Fact]
public void Remote_subscription_dictionary_uses_dedicated_routed_sub_key_type()
{
var field = typeof(SubList).GetField("_remoteSubs", BindingFlags.Instance | BindingFlags.NonPublic);
field.ShouldNotBeNull();
field.FieldType.IsGenericType.ShouldBeTrue();
field.FieldType.GetGenericArguments()[0].Name.ShouldBe("RoutedSubKey");
}
[Fact]
public void Has_remote_interest_supports_exact_and_wildcard_subjects_per_account()
{
using var sl = new SubList();
sl.ApplyRemoteSub(new RemoteSubscription("orders.created", null, "r1", "A"));
sl.ApplyRemoteSub(new RemoteSubscription("orders.*", null, "r2", "A"));
sl.ApplyRemoteSub(new RemoteSubscription("payments.*", null, "r3", "B"));
sl.HasRemoteInterest("A", "orders.created").ShouldBeTrue();
sl.HasRemoteInterest("A", "orders.updated").ShouldBeTrue();
sl.HasRemoteInterest("A", "payments.created").ShouldBeFalse();
sl.HasRemoteInterest("B", "payments.posted").ShouldBeTrue();
sl.HasRemoteInterest("B", "orders.created").ShouldBeFalse();
}
[Fact]
public void Match_remote_reflects_queue_weight_updates_for_existing_remote_queue_sub()
{
using var sl = new SubList();
var sub = new RemoteSubscription("orders.*", "workers", "r1", "A", QueueWeight: 1);
sl.ApplyRemoteSub(sub);
sl.MatchRemote("A", "orders.created").Count.ShouldBe(1);
sl.UpdateRemoteQSub(sub with { QueueWeight = 4 });
var matches = sl.MatchRemote("A", "orders.created");
matches.Count.ShouldBe(4);
matches.ShouldAllBe(match => match.Queue == "workers");
}
[Fact]
public void Match_merges_queue_groups_from_multiple_matching_nodes_by_queue_name()
{
using var sl = new SubList();
sl.Insert(new Subscription { Subject = "orders.created", Queue = "workers", Sid = "1" });
sl.Insert(new Subscription { Subject = "orders.*", Queue = "workers", Sid = "2" });
sl.Insert(new Subscription { Subject = "orders.>", Queue = "audit", Sid = "3" });
var result = sl.Match("orders.created");
result.PlainSubs.ShouldBeEmpty();
result.QueueSubs.Length.ShouldBe(2);
result.QueueSubs.Single(group => group[0].Queue == "workers").Length.ShouldBe(2);
result.QueueSubs.Single(group => group[0].Queue == "audit").Length.ShouldBe(1);
}
}
@@ -199,6 +199,33 @@ public class SubListGoParityTests
sl.Match("foo.bar").PlainSubs.Length.ShouldBe(3); sl.Match("foo.bar").PlainSubs.Length.ShouldBe(3);
} }
[Fact]
public void Cache_generation_bump_rebuilds_match_result_after_insert_and_remove()
{
var sl = new SubList();
var exact = MakeSub("foo.bar", sid: "1");
var wildcard = MakeSub("foo.*", sid: "2");
sl.Insert(exact);
var first = sl.Match("foo.bar");
var second = sl.Match("foo.bar");
ReferenceEquals(first, second).ShouldBeTrue();
first.PlainSubs.Select(sub => sub.Sid).ShouldBe(["1"]);
sl.Insert(wildcard);
var afterInsert = sl.Match("foo.bar");
ReferenceEquals(afterInsert, first).ShouldBeFalse();
afterInsert.PlainSubs.Select(sub => sub.Sid).OrderBy(x => x).ToArray().ShouldBe(["1", "2"]);
sl.Remove(wildcard);
var afterRemove = sl.Match("foo.bar");
ReferenceEquals(afterRemove, afterInsert).ShouldBeFalse();
afterRemove.PlainSubs.Select(sub => sub.Sid).ShouldBe(["1"]);
}
/// <summary> /// <summary>
/// Empty result is a shared singleton — two calls that yield no matches return /// Empty result is a shared singleton — two calls that yield no matches return
/// the same object reference. /// the same object reference.