feat: add structured logging, Shouldly assertions, CPM, and project documentation
- Add Microsoft.Extensions.Logging + Serilog to NatsServer and NatsClient - Convert all test assertions from xUnit Assert to Shouldly - Add NSubstitute package for future mocking needs - Introduce Central Package Management via Directory.Packages.props - Add documentation_rules.md with style guide, generation/update rules, component map - Generate 10 documentation files across 5 component folders (GettingStarted, Protocol, Subscriptions, Server, Configuration/Operations) - Update CLAUDE.md with logging, testing, porting, agent model, CPM, and documentation guidance
This commit is contained in:
Joseph Doherty
242
Documentation/Subscriptions/SubList.md
Normal file
242
Documentation/Subscriptions/SubList.md
Normal file
@@ -0,0 +1,242 @@
|
||||
# SubList
|
||||
|
||||
`SubList` is the subscription routing trie. Every published message triggers a `Match()` call to find all interested subscribers. `SubList` stores subscriptions indexed by their subject tokens and returns a `SubListResult` containing both plain subscribers and queue groups.
|
||||
|
||||
Go reference: `golang/nats-server/server/sublist.go`
|
||||
|
||||
---
|
||||
|
||||
## Thread Safety
|
||||
|
||||
`SubList` uses a `ReaderWriterLockSlim` (`_lock`) with the following locking discipline:
|
||||
|
||||
| Operation | Lock |
|
||||
|-----------|------|
|
||||
| `Count` read | Read lock |
|
||||
| `Match()` — cache hit | Read lock only |
|
||||
| `Match()` — cache miss | Write lock (to update cache) |
|
||||
| `Insert()` | Write lock |
|
||||
| `Remove()` | Write lock |
|
||||
|
||||
Cache misses in `Match()` require a write lock because the cache must be updated after the trie traversal. To avoid a race between the read-lock check and the write-lock update, `Match()` uses double-checked locking: after acquiring the write lock, it checks the cache again before doing trie work.
|
||||
|
||||
---
|
||||
|
||||
## Trie Structure
|
||||
|
||||
The trie is built from two private classes, `TrieLevel` and `TrieNode`, nested inside `SubList`.
|
||||
|
||||
### `TrieLevel` and `TrieNode`
|
||||
|
||||
```csharp
|
||||
private sealed class TrieLevel
|
||||
{
|
||||
public readonly Dictionary<string, TrieNode> Nodes = new(StringComparer.Ordinal);
|
||||
public TrieNode? Pwc; // partial wildcard (*)
|
||||
public TrieNode? Fwc; // full wildcard (>)
|
||||
}
|
||||
|
||||
private sealed class TrieNode
|
||||
{
|
||||
public TrieLevel? Next;
|
||||
public readonly HashSet<Subscription> PlainSubs = [];
|
||||
public readonly Dictionary<string, HashSet<Subscription>> QueueSubs = new(StringComparer.Ordinal);
|
||||
|
||||
public bool IsEmpty => PlainSubs.Count == 0 && QueueSubs.Count == 0 &&
|
||||
(Next == null || (Next.Nodes.Count == 0 && Next.Pwc == null && Next.Fwc == null));
|
||||
}
|
||||
```
|
||||
|
||||
Each level in the trie represents one token position in a subject. A `TrieLevel` holds:
|
||||
|
||||
- `Nodes` — a dictionary keyed by literal token string, mapping to the `TrieNode` for that token. Uses `StringComparer.Ordinal` for performance.
|
||||
- `Pwc` — the node for the `*` wildcard at this level, or `null` if no `*` subscriptions exist at this depth.
|
||||
- `Fwc` — the node for the `>` wildcard at this level, or `null` if no `>` subscriptions exist at this depth.
|
||||
|
||||
A `TrieNode` sits at the boundary between two levels. It holds the subscriptions registered for subjects whose last token leads to this node:
|
||||
|
||||
- `PlainSubs` — a `HashSet<Subscription>` of plain (non-queue) subscribers.
|
||||
- `QueueSubs` — a dictionary from queue name to the set of members in that queue group. Uses `StringComparer.Ordinal`.
|
||||
- `Next` — the next `TrieLevel` for deeper token positions. `null` for leaf nodes.
|
||||
- `IsEmpty` — `true` when the node and all its descendants have no subscriptions. Used during `Remove()` to prune dead branches.
|
||||
|
||||
The trie root is a `TrieLevel` (`_root`) with no parent node.
|
||||
|
||||
### `TokenEnumerator`
|
||||
|
||||
`TokenEnumerator` is a `ref struct` that splits a subject string by `.` without allocating. It operates on a `ReadOnlySpan<char>` derived from the original string.
|
||||
|
||||
```csharp
|
||||
private ref struct TokenEnumerator
|
||||
{
|
||||
private ReadOnlySpan<char> _remaining;
|
||||
|
||||
public TokenEnumerator(string subject)
|
||||
{
|
||||
_remaining = subject.AsSpan();
|
||||
Current = default;
|
||||
}
|
||||
|
||||
public ReadOnlySpan<char> Current { get; private set; }
|
||||
|
||||
public TokenEnumerator GetEnumerator() => this;
|
||||
|
||||
public bool MoveNext()
|
||||
{
|
||||
if (_remaining.IsEmpty)
|
||||
return false;
|
||||
|
||||
int sep = _remaining.IndexOf(SubjectMatch.Sep);
|
||||
if (sep < 0)
|
||||
{
|
||||
Current = _remaining;
|
||||
_remaining = default;
|
||||
}
|
||||
else
|
||||
{
|
||||
Current = _remaining[..sep];
|
||||
_remaining = _remaining[(sep + 1)..];
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`TokenEnumerator` implements the `foreach` pattern directly (via `GetEnumerator()` returning `this`), so it can be used in `foreach` loops without boxing. `Insert()` uses it during trie traversal to avoid string allocations per token.
|
||||
|
||||
---
|
||||
|
||||
## Insert
|
||||
|
||||
`Insert(Subscription sub)` adds a subscription to the trie under a write lock.
|
||||
|
||||
The method walks the trie one token at a time using `TokenEnumerator`. For each token:
|
||||
- If the token is `*`, it creates or follows `level.Pwc`.
|
||||
- If the token is `>`, it creates or follows `level.Fwc` and sets `sawFwc = true` to reject further tokens.
|
||||
- Otherwise it creates or follows `level.Nodes[token]`.
|
||||
|
||||
At each step, `node.Next` is created if absent, and `level` advances to `node.Next`.
|
||||
|
||||
After all tokens are consumed, the subscription is added to the terminal node:
|
||||
- Plain subscription: `node.PlainSubs.Add(sub)`.
|
||||
- Queue subscription: `node.QueueSubs[sub.Queue].Add(sub)`, creating the inner `HashSet<Subscription>` if this is the first member of that group.
|
||||
|
||||
`_count` is incremented and `AddToCache` is called to update any cached results that would now include this subscription.
|
||||
|
||||
---
|
||||
|
||||
## Remove
|
||||
|
||||
`Remove(Subscription sub)` removes a subscription from the trie under a write lock.
|
||||
|
||||
The method walks the trie along the subscription's subject, recording the path as a `List<(TrieLevel, TrieNode, string token, bool isPwc, bool isFwc)>`. If any node along the path is missing, the method returns without error (the subscription was never inserted).
|
||||
|
||||
After locating the terminal node, the subscription is removed from `PlainSubs` or from the appropriate `QueueSubs` group. If the queue group becomes empty, its entry is removed from the dictionary.
|
||||
|
||||
If removal succeeds:
|
||||
- `_count` is decremented.
|
||||
- `RemoveFromCache` is called to invalidate affected cache entries.
|
||||
- The path list is walked backwards. At each step, if `node.IsEmpty` is `true`, the node is removed from its parent level (`Pwc = null`, `Fwc = null`, or `Nodes.Remove(token)`). This prunes dead branches so the trie does not accumulate empty nodes over time.
|
||||
|
||||
---
|
||||
|
||||
## Match
|
||||
|
||||
`Match(string subject)` is called for every published message. It returns a `SubListResult` containing all matching plain and queue subscriptions.
|
||||
|
||||
### Cache check and fallback
|
||||
|
||||
```csharp
|
||||
public SubListResult Match(string subject)
|
||||
{
|
||||
// Check cache under read lock first.
|
||||
_lock.EnterReadLock();
|
||||
try
|
||||
{
|
||||
if (_cache != null && _cache.TryGetValue(subject, out var cached))
|
||||
return cached;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_lock.ExitReadLock();
|
||||
}
|
||||
|
||||
// Cache miss -- tokenize and match under write lock (needed for cache update).
|
||||
var tokens = Tokenize(subject);
|
||||
if (tokens == null)
|
||||
return SubListResult.Empty;
|
||||
|
||||
_lock.EnterWriteLock();
|
||||
try
|
||||
{
|
||||
// Re-check cache after acquiring write lock.
|
||||
if (_cache != null && _cache.TryGetValue(subject, out var cached))
|
||||
return cached;
|
||||
|
||||
var plainSubs = new List<Subscription>();
|
||||
var queueSubs = new List<List<Subscription>>();
|
||||
|
||||
MatchLevel(_root, tokens, 0, plainSubs, queueSubs);
|
||||
...
|
||||
if (_cache != null)
|
||||
{
|
||||
_cache[subject] = result;
|
||||
if (_cache.Count > CacheMax) { /* sweep */ }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
finally { _lock.ExitWriteLock(); }
|
||||
}
|
||||
```
|
||||
|
||||
On a read-lock cache hit, `Match()` returns immediately with no trie traversal. On a miss, `Tokenize()` splits the subject before acquiring the write lock (subjects with empty tokens return `SubListResult.Empty` immediately). The write lock is then taken and the cache is checked again before invoking `MatchLevel`.
|
||||
|
||||
### `MatchLevel` traversal
|
||||
|
||||
`MatchLevel` is a recursive method that descends the trie matching tokens against the subject array. At each level, for each remaining token position:
|
||||
|
||||
1. If `level.Fwc` is set, all subscriptions from that node are added to the result. The `>` wildcard matches all remaining tokens, so no further recursion is needed for this branch.
|
||||
2. If `level.Pwc` is set, `MatchLevel` recurses with the next token index and `pwc.Next` as the new level. This handles `*` matching the current token.
|
||||
3. A literal dictionary lookup on `level.Nodes[tokens[i]]` advances the level pointer for the next iteration.
|
||||
|
||||
After all tokens are consumed, subscriptions from the final literal node and the final `*` position (if present at the last level) are added to the result. The `*` case at the last token requires explicit handling because the loop exits before the recursive call for `*` can execute.
|
||||
|
||||
`AddNodeToResults` flattens a node's `PlainSubs` into the accumulator list and merges its `QueueSubs` groups into the queue accumulator, combining groups by name across multiple matching nodes.
|
||||
|
||||
---
|
||||
|
||||
## Cache Strategy
|
||||
|
||||
The cache is a `Dictionary<string, SubListResult>` keyed by the literal published subject. All operations use `StringComparer.Ordinal`.
|
||||
|
||||
**Size limits:** The cache holds at most `CacheMax` (1024) entries. When `_cache.Count` exceeds this, a sweep removes entries until the count reaches `CacheSweep` (256). The sweep takes the first `count - 256` keys from the dictionary — no LRU ordering is maintained.
|
||||
|
||||
**`AddToCache`** is called from `Insert()` to keep cached results consistent after adding a subscription:
|
||||
- For a literal subscription subject, `AddToCache` does a direct lookup. If the exact key is in the cache, it creates a new `SubListResult` with the subscription appended and replaces the cached entry.
|
||||
- For a wildcard subscription subject, `AddToCache` scans all cached keys and updates any entry whose key is matched by `SubjectMatch.MatchLiteral(key, subject)`.
|
||||
|
||||
**`RemoveFromCache`** is called from `Remove()` to invalidate cached results after removing a subscription:
|
||||
- For a literal subscription subject, `RemoveFromCache` removes the exact cache key.
|
||||
- For a wildcard subscription subject, `RemoveFromCache` removes all cached keys matched by the pattern. Because it is difficult to reconstruct the correct result without a full trie traversal, invalidation is preferred over update.
|
||||
|
||||
The asymmetry between `AddToCache` (updates in place) and `RemoveFromCache` (invalidates) avoids a second trie traversal on removal at the cost of a cache miss on the next `Match()` for those keys.
|
||||
|
||||
---
|
||||
|
||||
## Disposal
|
||||
|
||||
`SubList` implements `IDisposable`. `Dispose()` releases the `ReaderWriterLockSlim`:
|
||||
|
||||
```csharp
|
||||
public void Dispose() => _lock.Dispose();
|
||||
```
|
||||
|
||||
`SubList` instances are owned by `NatsServer` and disposed during server shutdown.
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Subscriptions Overview](../Subscriptions/Overview.md)
|
||||
|
||||
<!-- Last verified against codebase: 2026-02-22 -->
|
||||
Reference in New Issue
Block a user