Improve source XML docs and refresh profiling artifacts

This captures the iterative CommentChecker cleanup plus updated snapshot/report outputs used to validate and benchmark the latest JetStream and transport work.
This commit is contained in:
Joseph Doherty
2026-03-14 03:13:17 -04:00
parent 56c773dc71
commit ba0d65317a
76 changed files with 3058 additions and 29987 deletions
+18 -8
View File
@@ -59,7 +59,7 @@ Benchmark run: 2026-03-13 America/Indiana/Indianapolis. Both servers ran on the
| Mode | Payload | Storage | Go msg/s | .NET msg/s | Ratio (.NET/Go) |
|------|---------|---------|----------|------------|-----------------|
| Synchronous | 16 B | Memory | 16,982 | 14,514 | 0.85x |
| Async (batch) | 128 B | File | 211,355 | 58,334 | 0.28x |
| Async (batch) | 128 B | File | 174,421 | 85,394 | 0.49x |
---
@@ -144,7 +144,7 @@ Benchmark run: 2026-03-13 America/Indiana/Indianapolis. Both servers ran on the
| Request/reply (single) | 0.89x | Close to parity |
| Request/reply (10Cx2S) | 0.86x | Close to parity |
| JetStream sync publish | 0.85x | Close to parity |
| JetStream async file publish | 0.28x | Storage-bound |
| JetStream async file publish | 0.49x | Improved after double-buffer + deferred fsync |
| JetStream ordered consume | 0.44x | Significant gap |
| JetStream durable fetch | 0.76x | Moderate gap |
| MQTT pub/sub | **1.32x** | .NET outperforms Go |
@@ -157,16 +157,26 @@ Benchmark run: 2026-03-13 America/Indiana/Indianapolis. Both servers ran on the
### Key Observations
1. **Multi pub/sub reached parity (1.01x)** after Round 10 pre-formatted MSG headers. Fan-out improved to 0.84x.
2. **TLS pub/sub shows a dramatic .NET advantage (4.70x)** — .NET's `SslStream` has significantly lower overhead in the bidirectional pub/sub path. TLS pub-only (ingest only) still favors Go at 0.39x, suggesting the advantage is in the read-and-deliver path.
3. **MQTT pub/sub remains a .NET strength at 1.32x.** Cross-protocol (NATS→MQTT) dropped to 0.71x — this benchmark shows high variance across runs.
4. **JetStream ordered consumer dropped to 0.44x** compared to earlier runs (0.62x). This test completes in <100ms and shows high variance.
5. **Single publisher 128B dropped to 0.37x** (from 0.62x with smaller message counts). With 500K messages, this benchmark runs long enough for Go's goroutine scheduler and buffer management to reach steady state, widening the gap. The 16B variant is stable at 0.74x.
6. **Request-reply latency stable** at 0.86x0.89x across all runs.
2. **JetStream async file publish improved to 0.49x** (from 0.28x) after Round 11 double-buffer + deferred fsync optimizations — a 75% improvement.
3. **TLS pub/sub shows a dramatic .NET advantage (4.70x)** — .NET's `SslStream` has significantly lower overhead in the bidirectional pub/sub path. TLS pub-only (ingest only) still favors Go at 0.39x, suggesting the advantage is in the read-and-deliver path.
4. **MQTT pub/sub remains a .NET strength at 1.32x.** Cross-protocol (NATS→MQTT) dropped to 0.71x — this benchmark shows high variance across runs.
5. **JetStream ordered consumer dropped to 0.44x** compared to earlier runs (0.62x). This test completes in <100ms and shows high variance.
6. **Single publisher 128B dropped to 0.37x** (from 0.62x with smaller message counts). With 500K messages, this benchmark runs long enough for Go's goroutine scheduler and buffer management to reach steady state, widening the gap. The 16B variant is stable at 0.74x.
7. **Request-reply latency stable** at 0.86x0.89x across all runs.
---
## Optimization History
### Round 11: JetStream FileStore Double-Buffer + Deferred Fsync
Two optimizations targeting the JetStream async file publish hot path (0.28x→0.49x, 75% improvement):
| # | Root Cause | Fix | Impact |
|---|-----------|-----|--------|
| 41 | **Lock contention between WriteAt and FlushPending**`MsgBlock.FlushPending()` held the write lock for the entire `RandomAccess.Write` call, blocking `WriteAt` (publish path) during disk I/O | Double-buffer: swap `_pendingBuf``_flushBuf` under write lock, then write old buffer to disk outside lock using separate `_flushLock`; publish path only blocked during buffer pointer swap, not disk I/O | Eliminates write-lock contention during disk I/O |
| 42 | **Synchronous fsync on publish path**`RotateBlock()` called `FlushToDisk()` which did `fsync` synchronously (1,557ms per profile), blocking the publish hot path for every block rotation | Deferred fsync: `RotateBlock` enqueues completed blocks into `ConcurrentQueue<MsgBlock> _needSyncBlocks`; background `FlushLoopAsync` drains the queue via `DrainSyncQueue()`, calling `Flush()` (fsync) off the publish path — matches Go's `needSync` flag + background goroutine pattern | Moves fsync entirely off the publish hot path |
### Round 10: Fan-Out Serial Path Optimization
Three optimizations making the serial fan-out path cheaper (fan-out 0.63x→0.84x, multi 0.65x→1.01x):
@@ -280,6 +290,6 @@ Additional fixes: SHA256 envelope bypass for unencrypted/uncompressed stores, RA
| Change | Expected Impact | Go Reference |
|--------|----------------|-------------|
| **Single publisher ingest path (0.37x at 128B)** | The pub-only path has the largest gap. Go's readLoop uses zero-copy buffer management with direct `[]byte` slicing; .NET parses into managed objects. Reducing allocations in the parser→ProcessMessage path would help. | Go: `client.go` readLoop, direct buffer slicing |
| **JetStream async file publish (0.28x)** | Storage-bound: FileStore AppendAsync bottleneck is synchronous `RandomAccess.Write` in flush loop and S2 compression overhead | Go: `filestore.go` uses `cache.buf`/`cache.idx` with mmap and goroutine-per-flush concurrency |
| **JetStream async file publish (0.49x)** | After double-buffer + deferred fsync, remaining gap is likely write coalescing and S2 compression overhead | Go: `filestore.go` uses `cache.buf`/`cache.idx` with mmap and goroutine-per-flush concurrency |
| **JetStream ordered consumer (0.44x)** | Pull consumer delivery pipeline has overhead in the fetch→deliver→ack cycle. The test completes in <100ms so numbers are noisy, but the gap is real. | Go: `consumer.go` delivery with direct buffer writes |
| **Write-loop / socket write overhead** | Fan-out (0.84x) and pub/sub (0.66x) gaps partly come from write-loop wakeup latency and socket write syscall overhead compared to Go's `writev()` | Go: `flushOutbound` uses `net.Buffers.WriteTo``writev()` with zero-copy buffer management |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+881 -11611
View File
File diff suppressed because it is too large Load Diff
+76 -18346
View File
File diff suppressed because it is too large Load Diff
@@ -33,6 +33,13 @@ public static partial class PermissionTemplates
/// Returns an empty list if any template resolves to no values (tag not found).
/// Returns a single-element list containing the original pattern if no templates are present.
/// </summary>
/// <param name="pattern">Template subject pattern to expand.</param>
/// <param name="name">User display name from JWT claims.</param>
/// <param name="subject">User public NKey subject from JWT claims.</param>
/// <param name="accountName">Account display name from account JWT.</param>
/// <param name="accountSubject">Account public NKey subject from account JWT.</param>
/// <param name="userTags">User tag set in <c>key:value</c> form.</param>
/// <param name="accountTags">Account tag set in <c>key:value</c> form.</param>
public static List<string> Expand(
string pattern,
string name, string subject,
@@ -71,6 +78,13 @@ public static partial class PermissionTemplates
/// Expands all patterns in a permission list, flattening multi-value expansions
/// into the result. Patterns that resolve to no values are omitted entirely.
/// </summary>
/// <param name="patterns">Permission subject patterns to expand.</param>
/// <param name="name">User display name from JWT claims.</param>
/// <param name="subject">User public NKey subject from JWT claims.</param>
/// <param name="accountName">Account display name from account JWT.</param>
/// <param name="accountSubject">Account public NKey subject from account JWT.</param>
/// <param name="userTags">User tag set in <c>key:value</c> form.</param>
/// <param name="accountTags">Account tag set in <c>key:value</c> form.</param>
public static List<string> ExpandAll(
IEnumerable<string> patterns,
string name, string subject,
@@ -65,12 +65,15 @@ public sealed class RemoteLeafOptions
/// Sets reconnect/connect delay for this remote.
/// Go reference: leafnode.go leafNodeCfg.setConnectDelay.
/// </summary>
/// <param name="delay">Delay before the next reconnect attempt to this remote leaf.</param>
public void SetConnectDelay(TimeSpan delay) => _connectDelay = delay;
/// <summary>
/// Starts or replaces the JetStream migration timer callback for this remote leaf.
/// Go reference: leafnode.go leafNodeCfg.migrateTimer.
/// </summary>
/// <param name="callback">Callback invoked when migration retry timing elapses.</param>
/// <param name="delay">Initial delay before invoking the migration callback.</param>
public void StartMigrateTimer(TimerCallback callback, TimeSpan delay)
{
ArgumentNullException.ThrowIfNull(callback);
@@ -93,6 +96,7 @@ public sealed class RemoteLeafOptions
/// Saves TLS hostname from URL for future SNI usage.
/// Go reference: leafnode.go leafNodeCfg.saveTLSHostname.
/// </summary>
/// <param name="url">Remote leaf URL that supplies the SNI host name.</param>
public void SaveTlsHostname(string url)
{
if (TryParseUrl(url, out var uri))
@@ -103,6 +107,7 @@ public sealed class RemoteLeafOptions
/// Saves username/password from URL user info for fallback auth.
/// Go reference: leafnode.go leafNodeCfg.saveUserPassword.
/// </summary>
/// <param name="url">Remote leaf URL containing optional user info credentials.</param>
public void SaveUserPassword(string url)
{
if (!TryParseUrl(url, out var uri) || string.IsNullOrEmpty(uri.UserInfo))
@@ -124,18 +129,25 @@ public sealed class RemoteLeafOptions
public sealed class LeafNodeOptions
{
/// <summary>Host/IP address where the leaf listener accepts incoming leaf connections.</summary>
public string Host { get; set; } = "0.0.0.0";
/// <summary>TCP port exposed for leaf node connections.</summary>
public int Port { get; set; }
// Auth for leaf listener
/// <summary>Optional username required for inbound leaf authentication.</summary>
public string? Username { get; set; }
/// <summary>Optional password required for inbound leaf authentication.</summary>
public string? Password { get; set; }
/// <summary>Maximum seconds a leaf connection has to complete authentication.</summary>
public double AuthTimeout { get; set; }
// Advertise address
/// <summary>Optional externally reachable leaf address advertised to peers.</summary>
public string? Advertise { get; set; }
// Per-subsystem write deadline
/// <summary>Write deadline applied to leaf network operations.</summary>
public TimeSpan WriteDeadline { get; set; }
/// <summary>
@@ -156,9 +168,13 @@ public sealed class LeafNodeOptions
/// </summary>
public string? JetStreamDomain { get; set; }
/// <summary>Subjects that this leaf cannot export to the remote account.</summary>
public List<string> DenyExports { get; set; } = [];
/// <summary>Subjects that this leaf cannot import from the remote account.</summary>
public List<string> DenyImports { get; set; } = [];
/// <summary>Subjects explicitly exported from this leaf to connected remotes.</summary>
public List<string> ExportSubjects { get; set; } = [];
/// <summary>Subjects explicitly imported from remote leaves into this server.</summary>
public List<string> ImportSubjects { get; set; } = [];
/// <summary>List of users for leaf listener authentication (from authorization.users).</summary>
@@ -15,10 +15,15 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
private readonly ConcurrentDictionary<string, HashSet<string>> _queueSubscriptions = new(StringComparer.Ordinal);
private Task? _loopTask;
/// <summary>Remote gateway server id learned during handshake.</summary>
public string? RemoteId { get; private set; }
/// <summary>Indicates whether this is an outbound (solicited) gateway connection.</summary>
public bool IsOutbound { get; internal set; }
/// <summary>Remote endpoint string for diagnostics and monitoring.</summary>
public string RemoteEndpoint => socket.RemoteEndPoint?.ToString() ?? Guid.NewGuid().ToString("N");
/// <summary>Callback invoked when remote A+/A- interest updates are received.</summary>
public Func<RemoteSubscription, Task>? RemoteSubscriptionReceived { get; set; }
/// <summary>Callback invoked when remote GMSG payloads are received.</summary>
public Func<GatewayMessage, Task>? MessageReceived { get; set; }
/// <summary>
@@ -31,6 +36,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// Adds a subject to the account-specific subscription set for this gateway connection.
/// Go: gateway.go — per-account subscription routing state on outbound connections.
/// </summary>
/// <param name="account">Account name for the subscription.</param>
/// <param name="subject">Subject to track.</param>
public void AddAccountSubscription(string account, string subject)
{
var subs = _accountSubscriptions.GetOrAdd(account, _ => new HashSet<string>(StringComparer.Ordinal));
@@ -40,6 +47,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Removes a subject from the account-specific subscription set for this gateway connection.
/// </summary>
/// <param name="account">Account name for the subscription.</param>
/// <param name="subject">Subject to untrack.</param>
public void RemoveAccountSubscription(string account, string subject)
{
if (_accountSubscriptions.TryGetValue(account, out var subs))
@@ -49,6 +58,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Returns a snapshot of all subjects tracked for the given account on this connection.
/// </summary>
/// <param name="account">Account name to query.</param>
/// <returns>Snapshot of tracked subjects.</returns>
public IReadOnlySet<string> GetAccountSubscriptions(string account)
{
if (_accountSubscriptions.TryGetValue(account, out var subs))
@@ -59,6 +70,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Returns the number of subjects tracked for the given account. Returns 0 for unknown accounts.
/// </summary>
/// <param name="account">Account name to query.</param>
/// <returns>Number of tracked subjects for the account.</returns>
public int AccountSubscriptionCount(string account)
{
if (_accountSubscriptions.TryGetValue(account, out var subs))
@@ -70,6 +83,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// Registers a queue group subscription for propagation to this gateway.
/// Go reference: gateway.go — sendQueueSubsToGateway.
/// </summary>
/// <param name="subject">Subject for the queue subscription.</param>
/// <param name="queueGroup">Queue group name.</param>
public void AddQueueSubscription(string subject, string queueGroup)
{
var groups = _queueSubscriptions.GetOrAdd(subject, _ => new HashSet<string>(StringComparer.Ordinal));
@@ -80,6 +95,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// Removes a queue group subscription from this gateway connection's tracking state.
/// Go reference: gateway.go — sendQueueSubsToGateway (removal path).
/// </summary>
/// <param name="subject">Subject for the queue subscription.</param>
/// <param name="queueGroup">Queue group name.</param>
public void RemoveQueueSubscription(string subject, string queueGroup)
{
if (_queueSubscriptions.TryGetValue(subject, out var groups))
@@ -89,6 +106,8 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Returns a snapshot of all queue group names registered for the given subject.
/// </summary>
/// <param name="subject">Subject to query.</param>
/// <returns>Snapshot of queue group names.</returns>
public IReadOnlySet<string> GetQueueGroups(string subject)
{
if (_queueSubscriptions.TryGetValue(subject, out var groups))
@@ -104,6 +123,9 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Returns true if the given subject/queueGroup pair is currently registered on this gateway connection.
/// </summary>
/// <param name="subject">Subject to query.</param>
/// <param name="queueGroup">Queue group name to query.</param>
/// <returns><see langword="true"/> when the pair is registered.</returns>
public bool HasQueueSubscription(string subject, string queueGroup)
{
if (!_queueSubscriptions.TryGetValue(subject, out var groups))
@@ -111,6 +133,11 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
lock (groups) return groups.Contains(queueGroup);
}
/// <summary>
/// Performs outbound gateway handshake by sending local id and reading remote id.
/// </summary>
/// <param name="serverId">Local server id.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformOutboundHandshakeAsync(string serverId, CancellationToken ct)
{
await WriteLineAsync($"GATEWAY {serverId}", ct);
@@ -118,6 +145,11 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
RemoteId = ParseHandshake(line);
}
/// <summary>
/// Performs inbound gateway handshake by reading remote id and sending local id.
/// </summary>
/// <param name="serverId">Local server id.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformInboundHandshakeAsync(string serverId, CancellationToken ct)
{
var line = await ReadLineAsync(ct);
@@ -125,6 +157,10 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync($"GATEWAY {serverId}", ct);
}
/// <summary>
/// Starts the background frame read loop for this connection.
/// </summary>
/// <param name="ct">Cancellation token controlling loop lifetime.</param>
public void StartLoop(CancellationToken ct)
{
if (_loopTask != null)
@@ -134,15 +170,42 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
_loopTask = Task.Run(() => ReadLoopAsync(linked.Token), linked.Token);
}
/// <summary>
/// Waits for the gateway read loop to exit.
/// </summary>
/// <param name="ct">Cancellation token for wait operation.</param>
/// <returns>A task that completes when loop exits.</returns>
public Task WaitUntilClosedAsync(CancellationToken ct)
=> _loopTask?.WaitAsync(ct) ?? Task.CompletedTask;
/// <summary>
/// Sends an A+ protocol line to advertise interest.
/// </summary>
/// <param name="account">Account for the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendAPlusAsync(string account, string subject, string? queue, CancellationToken ct)
=> WriteLineAsync(queue is { Length: > 0 } ? $"A+ {account} {subject} {queue}" : $"A+ {account} {subject}", ct);
/// <summary>
/// Sends an A- protocol line to remove advertised interest.
/// </summary>
/// <param name="account">Account for the interest update.</param>
/// <param name="subject">Subject being removed.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendAMinusAsync(string account, string subject, string? queue, CancellationToken ct)
=> WriteLineAsync(queue is { Length: > 0 } ? $"A- {account} {subject} {queue}" : $"A- {account} {subject}", ct);
/// <summary>
/// Sends a GMSG payload to the remote gateway when interest permits forwarding.
/// </summary>
/// <param name="account">Account associated with the message.</param>
/// <param name="subject">Subject being forwarded.</param>
/// <param name="replyTo">Optional reply subject.</param>
/// <param name="payload">Payload bytes.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendMessageAsync(string account, string subject, string? replyTo, ReadOnlyMemory<byte> payload, CancellationToken ct)
{
// Go: gateway.go:2900 (shouldForwardMsg) — check interest tracker before sending
@@ -166,6 +229,9 @@ public sealed class GatewayConnection(Socket socket) : IAsyncDisposable
}
}
/// <summary>
/// Disposes this gateway connection and stops background processing.
/// </summary>
public async ValueTask DisposeAsync()
{
await _closedCts.CancelAsync();
+49
View File
@@ -20,6 +20,8 @@ public static class ReplyMapper
/// Checks whether the subject starts with either gateway reply prefix:
/// <c>_GR_.</c> (current) or <c>$GR.</c> (legacy).
/// </summary>
/// <param name="subject">Subject to inspect.</param>
/// <returns><see langword="true"/> when the subject is gateway-routed.</returns>
public static bool HasGatewayReplyPrefix(string? subject)
=> IsGatewayRoutedSubject(subject, out _);
@@ -28,6 +30,9 @@ public static class ReplyMapper
/// old prefix (<c>$GR.</c>) was used.
/// Go reference: isGWRoutedSubjectAndIsOldPrefix.
/// </summary>
/// <param name="subject">Subject to inspect.</param>
/// <param name="isOldPrefix">Set to <see langword="true"/> when the legacy prefix is used.</param>
/// <returns><see langword="true"/> when the subject is gateway-routed.</returns>
public static bool IsGatewayRoutedSubject(string? subject, out bool isOldPrefix)
{
isOldPrefix = false;
@@ -51,6 +56,8 @@ public static class ReplyMapper
/// Go reference: gateway.go uses SHA-256 truncated to base-62; we use FNV-1a for speed
/// while maintaining determinism and good distribution.
/// </summary>
/// <param name="replyTo">Reply subject to hash.</param>
/// <returns>Non-negative deterministic hash value.</returns>
public static long ComputeReplyHash(string replyTo)
{
// FNV-1a 64-bit
@@ -72,6 +79,8 @@ public static class ReplyMapper
/// Computes the short (6-char) gateway hash used in modern gateway reply routing.
/// Go reference: getGWHash.
/// </summary>
/// <param name="gatewayName">Gateway name to hash.</param>
/// <returns>Lowercase 6-character hash token.</returns>
public static string ComputeGatewayHash(string gatewayName)
{
var digest = System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(gatewayName));
@@ -82,6 +91,8 @@ public static class ReplyMapper
/// Computes the short (4-char) legacy gateway hash used with old prefixes.
/// Go reference: getOldHash.
/// </summary>
/// <param name="gatewayName">Gateway name to hash.</param>
/// <returns>Lowercase 4-character hash token.</returns>
public static string ComputeOldGatewayHash(string gatewayName)
{
var digest = System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(gatewayName));
@@ -92,6 +103,10 @@ public static class ReplyMapper
/// Converts a reply subject to gateway form with an explicit hash segment.
/// Format: <c>_GR_.{clusterId}.{hash}.{originalReply}</c>.
/// </summary>
/// <param name="replyTo">Original reply subject.</param>
/// <param name="localClusterId">Local cluster identifier to embed.</param>
/// <param name="hash">Precomputed reply hash.</param>
/// <returns>Gateway-form reply subject, or original when null/empty.</returns>
public static string? ToGatewayReply(string? replyTo, string localClusterId, long hash)
{
if (string.IsNullOrWhiteSpace(replyTo))
@@ -104,6 +119,9 @@ public static class ReplyMapper
/// Converts a reply subject to gateway form, automatically computing the hash.
/// Format: <c>_GR_.{clusterId}.{hash}.{originalReply}</c>.
/// </summary>
/// <param name="replyTo">Original reply subject.</param>
/// <param name="localClusterId">Local cluster identifier to embed.</param>
/// <returns>Gateway-form reply subject, or original when null/empty.</returns>
public static string? ToGatewayReply(string? replyTo, string localClusterId)
{
if (string.IsNullOrWhiteSpace(replyTo))
@@ -119,6 +137,9 @@ public static class ReplyMapper
/// legacy format (<c>_GR_.{clusterId}.{originalReply}</c>).
/// Nested prefixes are unwrapped iteratively.
/// </summary>
/// <param name="gatewayReply">Gateway-form reply subject.</param>
/// <param name="restoredReply">Receives restored original reply subject on success.</param>
/// <returns><see langword="true"/> when restoration succeeds.</returns>
public static bool TryRestoreGatewayReply(string? gatewayReply, out string restoredReply)
{
restoredReply = string.Empty;
@@ -161,6 +182,9 @@ public static class ReplyMapper
/// Extracts the cluster ID from a gateway reply subject.
/// The cluster ID is the first segment after the <c>_GR_.</c> prefix.
/// </summary>
/// <param name="gatewayReply">Gateway-form reply subject.</param>
/// <param name="clusterId">Receives extracted cluster identifier on success.</param>
/// <returns><see langword="true"/> when extraction succeeds.</returns>
public static bool TryExtractClusterId(string? gatewayReply, out string clusterId)
{
clusterId = string.Empty;
@@ -181,6 +205,9 @@ public static class ReplyMapper
/// Extracts the hash from a gateway reply subject (new format only).
/// Returns false if the reply uses the legacy format without a hash.
/// </summary>
/// <param name="gatewayReply">Gateway-form reply subject.</param>
/// <param name="hash">Receives extracted hash on success.</param>
/// <returns><see langword="true"/> when extraction succeeds.</returns>
public static bool TryExtractHash(string? gatewayReply, out long hash)
{
hash = 0;
@@ -236,6 +263,11 @@ public sealed class ReplyMapCache
private long _hits;
private long _misses;
/// <summary>
/// Creates an LRU reply mapping cache with TTL expiration.
/// </summary>
/// <param name="capacity">Maximum number of entries to retain.</param>
/// <param name="ttlMs">Time-to-live for entries in milliseconds.</param>
public ReplyMapCache(int capacity = 4096, int ttlMs = 60_000)
{
_capacity = capacity;
@@ -243,10 +275,19 @@ public sealed class ReplyMapCache
_map = new Dictionary<string, LinkedListNode<CacheEntry>>(capacity, StringComparer.Ordinal);
}
/// <summary>Total cache hits since creation.</summary>
public long Hits => Interlocked.Read(ref _hits);
/// <summary>Total cache misses since creation.</summary>
public long Misses => Interlocked.Read(ref _misses);
/// <summary>Current number of entries in the cache.</summary>
public int Count { get { lock (_lock) return _map.Count; } }
/// <summary>
/// Attempts to get a cached mapping value.
/// </summary>
/// <param name="key">Cache lookup key.</param>
/// <param name="value">Resolved cached value when found and not expired.</param>
/// <returns><see langword="true"/> when an unexpired value exists.</returns>
public bool TryGet(string key, out string? value)
{
lock (_lock)
@@ -276,6 +317,11 @@ public sealed class ReplyMapCache
return false;
}
/// <summary>
/// Inserts or updates a cached mapping value.
/// </summary>
/// <param name="key">Cache key.</param>
/// <param name="value">Cache value.</param>
public void Set(string key, string value)
{
lock (_lock)
@@ -302,6 +348,9 @@ public sealed class ReplyMapCache
}
}
/// <summary>
/// Clears all cached mappings.
/// </summary>
public void Clear()
{
lock (_lock)
@@ -18,10 +18,17 @@ public static class GslErrors
/// </summary>
internal sealed class Level<T> where T : IEquatable<T>
{
/// <summary>Literal-token child nodes.</summary>
public Dictionary<string, Node<T>> Nodes { get; } = new();
/// <summary>Single-token wildcard child node (<c>*</c>).</summary>
public Node<T>? Pwc { get; set; } // partial wildcard '*'
/// <summary>Terminal wildcard child node (<c>&gt;</c>).</summary>
public Node<T>? Fwc { get; set; } // full wildcard '>'
/// <summary>
/// Returns the number of child node references at this level, including wildcard pointers.
/// </summary>
/// <returns>Child count for this level.</returns>
public int NumNodes()
{
var num = Nodes.Count;
@@ -34,6 +41,8 @@ internal sealed class Level<T> where T : IEquatable<T>
/// Prune an empty node from the tree.
/// Go reference: server/gsl/gsl.go pruneNode
/// </summary>
/// <param name="n">Node to prune.</param>
/// <param name="token">Token key used for literal node lookup.</param>
public void PruneNode(Node<T> n, string token)
{
if (ReferenceEquals(n, Fwc))
@@ -51,7 +60,9 @@ internal sealed class Level<T> where T : IEquatable<T>
/// </summary>
internal sealed class Node<T> where T : IEquatable<T>
{
/// <summary>Next trie level for descendant tokens.</summary>
public Level<T>? Next { get; set; }
/// <summary>Subscriptions stored on this node keyed by value.</summary>
public Dictionary<T, string> Subs { get; } = new(); // value -> subject
/// <summary>
@@ -107,6 +118,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// Insert adds a subscription into the sublist.
/// Go reference: server/gsl/gsl.go Insert
/// </summary>
/// <param name="subject">Subject to insert.</param>
/// <param name="value">Subscription payload/value.</param>
public void Insert(string subject, T value)
{
_lock.EnterWriteLock();
@@ -185,6 +198,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// Remove will remove a subscription.
/// Go reference: server/gsl/gsl.go Remove
/// </summary>
/// <param name="subject">Subject to remove.</param>
/// <param name="value">Subscription payload/value to remove.</param>
public void Remove(string subject, T value)
{
_lock.EnterWriteLock();
@@ -202,6 +217,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// Match will match all entries to the literal subject and invoke the callback for each.
/// Go reference: server/gsl/gsl.go Match
/// </summary>
/// <param name="subject">Literal subject to match.</param>
/// <param name="callback">Callback invoked for each matched value.</param>
public void Match(string subject, Action<T> callback)
{
MatchInternal(subject, callback, doLock: true);
@@ -211,6 +228,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// MatchBytes will match all entries to the literal subject (as bytes) and invoke the callback for each.
/// Go reference: server/gsl/gsl.go MatchBytes
/// </summary>
/// <param name="subject">Literal subject bytes to match.</param>
/// <param name="callback">Callback invoked for each matched value.</param>
public void MatchBytes(ReadOnlySpan<byte> subject, Action<T> callback)
{
// Convert bytes to string then delegate
@@ -222,6 +241,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// HasInterest will return whether or not there is any interest in the subject.
/// Go reference: server/gsl/gsl.go HasInterest
/// </summary>
/// <param name="subject">Literal subject to test.</param>
/// <returns><see langword="true"/> when any subscription matches.</returns>
public bool HasInterest(string subject)
{
return HasInterestInternal(subject, doLock: true, np: null);
@@ -231,6 +252,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// NumInterest will return the number of subs interested in the subject.
/// Go reference: server/gsl/gsl.go NumInterest
/// </summary>
/// <param name="subject">Literal subject to test.</param>
/// <returns>Number of matched subscriptions.</returns>
public int NumInterest(string subject)
{
var np = new int[1]; // use array to pass by reference
@@ -242,6 +265,8 @@ public class GenericSubjectList<T> where T : IEquatable<T>
/// HasInterestStartingIn is a helper for subject tree intersection.
/// Go reference: server/gsl/gsl.go HasInterestStartingIn
/// </summary>
/// <param name="subject">Subject prefix to test.</param>
/// <returns><see langword="true"/> when interest exists beneath the prefix.</returns>
public bool HasInterestStartingIn(string subject)
{
_lock.EnterReadLock();
@@ -602,8 +627,16 @@ public class GenericSubjectList<T> where T : IEquatable<T>
{
private readonly string _subject;
/// <summary>
/// Creates a split enumerable over a subject string.
/// </summary>
/// <param name="subject">Subject to tokenize.</param>
public SplitEnumerable(string subject) => _subject = subject;
/// <summary>
/// Creates an enumerator for token iteration.
/// </summary>
/// <returns>Tokenizer enumerator.</returns>
public SplitEnumerator GetEnumerator() => new(_subject);
}
@@ -613,6 +646,10 @@ public class GenericSubjectList<T> where T : IEquatable<T>
private int _start;
private bool _done;
/// <summary>
/// Creates a tokenizer enumerator over the provided subject.
/// </summary>
/// <param name="subject">Subject to tokenize.</param>
public SplitEnumerator(string subject)
{
_subject = subject;
@@ -621,8 +658,13 @@ public class GenericSubjectList<T> where T : IEquatable<T>
Current = default!;
}
/// <summary>Current token from the subject split iteration.</summary>
public string Current { get; private set; }
/// <summary>
/// Advances to the next token in the subject string.
/// </summary>
/// <returns><see langword="true"/> when another token is available.</returns>
public bool MoveNext()
{
if (_done) return false;
@@ -31,6 +31,9 @@ public class HashWheel
private long _lowest;
private ulong _count;
/// <summary>
/// Initializes an empty time hash wheel used for expiration scheduling.
/// </summary>
public HashWheel()
{
_wheel = new Slot?[WheelSize];
@@ -56,6 +59,8 @@ public class HashWheel
/// Schedules a new timer task. If the sequence already exists in the target slot,
/// its expiration is updated without incrementing the count.
/// </summary>
/// <param name="seq">Sequence identifier for the tracked item.</param>
/// <param name="expires">Absolute expiration timestamp in nanoseconds.</param>
// Go: Add server/thw/thw.go:79
public void Add(ulong seq, long expires)
{
@@ -88,6 +93,8 @@ public class HashWheel
/// Removes a timer task. Returns true if the task was found and removed,
/// false if the task was not found.
/// </summary>
/// <param name="seq">Sequence identifier for the tracked item.</param>
/// <param name="expires">Previously scheduled expiration timestamp in nanoseconds.</param>
// Go: Remove server/thw/thw.go:103
public bool Remove(ulong seq, long expires)
{
@@ -119,6 +126,9 @@ public class HashWheel
/// Updates the expiration time of an existing timer task by removing it from
/// the old slot and adding it to the new one.
/// </summary>
/// <param name="seq">Sequence identifier for the tracked item.</param>
/// <param name="oldExpires">Previous expiration timestamp in nanoseconds.</param>
/// <param name="newExpires">New expiration timestamp in nanoseconds.</param>
// Go: Update server/thw/thw.go:123
public void Update(ulong seq, long oldExpires, long newExpires)
{
@@ -131,6 +141,7 @@ public class HashWheel
/// expired entry's sequence and expiration time. If the callback returns true,
/// the entry is removed; if false, it remains for future expiration checks.
/// </summary>
/// <param name="callback">Callback invoked for each expired entry; return true to remove it.</param>
// Go: ExpireTasks server/thw/thw.go:133
public void ExpireTasks(Func<ulong, long, bool> callback)
{
@@ -144,6 +155,8 @@ public class HashWheel
/// Internal expiration method that accepts an explicit timestamp.
/// Used by tests that need deterministic time control.
/// </summary>
/// <param name="ts">Current timestamp in nanoseconds used as expiration cutoff.</param>
/// <param name="callback">Callback invoked for each expired entry; return true to remove it.</param>
// Go: expireTasks server/thw/thw.go:138
internal void ExpireTasksInternal(long ts, Func<ulong, long, bool> callback)
{
@@ -215,6 +228,7 @@ public class HashWheel
/// Returns the earliest expiration time if it is before the given time.
/// Returns <see cref="long.MaxValue"/> if no expirations exist before the specified time.
/// </summary>
/// <param name="before">Upper time bound in nanoseconds.</param>
// Go: GetNextExpiration server/thw/thw.go:182
public long GetNextExpiration(long before)
{
@@ -231,6 +245,7 @@ public class HashWheel
/// The high sequence number is included and will be returned on decode.
/// Format: [1 byte magic version][8 bytes entry count][8 bytes highSeq][varint expires, uvarint seq pairs...]
/// </summary>
/// <param name="highSeq">High watermark sequence stored alongside wheel state.</param>
// Go: Encode server/thw/thw.go:197
public byte[] Encode(ulong highSeq)
{
@@ -278,6 +293,7 @@ public class HashWheel
/// Decodes a binary-encoded snapshot and replaces the contents of this wheel.
/// Returns the high sequence number from the snapshot and the number of bytes consumed.
/// </summary>
/// <param name="buf">Encoded wheel snapshot buffer.</param>
// Go: Decode server/thw/thw.go:216
public (ulong HighSeq, int BytesRead) Decode(ReadOnlySpan<byte> buf)
{
@@ -412,9 +428,11 @@ public class HashWheel
internal sealed class Slot
{
// Go: slot.entries — map of sequence to expires.
/// <summary>Entries assigned to this wheel slot keyed by sequence identifier.</summary>
public Dictionary<ulong, long> Entries { get; } = new();
// Go: slot.lowest — lowest expiration time in this slot.
/// <summary>Earliest expiration timestamp present in this slot.</summary>
public long Lowest { get; set; } = long.MaxValue;
}
}
+36
View File
@@ -11,11 +11,17 @@ namespace NATS.Server;
/// </summary>
public sealed class InternalClient : INatsClient
{
/// <summary>Unique internal client identifier used in server-side bookkeeping.</summary>
public ulong Id { get; }
/// <summary>Internal client kind (SYSTEM, ACCOUNT, JETSTREAM, etc.).</summary>
public ClientKind Kind { get; }
/// <summary>Indicates this client is server-internal and not backed by a socket connection.</summary>
public bool IsInternal => Kind.IsInternal();
/// <summary>Account context associated with this internal client.</summary>
public Account? Account { get; }
/// <summary>Client options are not applicable for socketless internal clients.</summary>
public ClientOptions? ClientOpts => null;
/// <summary>Permission overrides are not used for internal clients.</summary>
public ClientPermissions? Permissions => null;
/// <summary>
@@ -26,6 +32,12 @@ public sealed class InternalClient : INatsClient
private readonly Dictionary<string, Subscription> _subs = new(StringComparer.Ordinal);
/// <summary>
/// Creates a lightweight internal client used for in-process message delivery.
/// </summary>
/// <param name="id">Server-assigned internal client identifier.</param>
/// <param name="kind">Client kind that must represent an internal role.</param>
/// <param name="account">Account context used for subscription accounting.</param>
public InternalClient(ulong id, ClientKind kind, Account account)
{
if (!kind.IsInternal())
@@ -36,12 +48,28 @@ public sealed class InternalClient : INatsClient
Account = account;
}
/// <summary>
/// Delivers a message to this internal client and flushes immediately through the callback path.
/// </summary>
/// <param name="subject">Message subject.</param>
/// <param name="sid">Subscription identifier receiving the message.</param>
/// <param name="replyTo">Optional reply subject for request-reply flows.</param>
/// <param name="headers">Serialized NATS headers payload.</param>
/// <param name="payload">Message payload bytes.</param>
public void SendMessage(string subject, string sid, string? replyTo,
ReadOnlyMemory<byte> headers, ReadOnlyMemory<byte> payload)
{
MessageCallback?.Invoke(subject, sid, replyTo, headers, payload);
}
/// <summary>
/// Delivers a message without explicit flush semantics; equivalent to <see cref="SendMessage"/>.
/// </summary>
/// <param name="subject">Message subject.</param>
/// <param name="sid">Subscription identifier receiving the message.</param>
/// <param name="replyTo">Optional reply subject for request-reply flows.</param>
/// <param name="headers">Serialized NATS headers payload.</param>
/// <param name="payload">Message payload bytes.</param>
public void SendMessageNoFlush(string subject, string sid, string? replyTo,
ReadOnlyMemory<byte> headers, ReadOnlyMemory<byte> payload)
{
@@ -49,20 +77,28 @@ public sealed class InternalClient : INatsClient
MessageCallback?.Invoke(subject, sid, replyTo, headers, payload);
}
/// <summary>Signals flush completion for interface compatibility; no-op for internal clients.</summary>
public void SignalFlush() { } // no-op for internal clients
/// <summary>Queues outbound bytes for interface compatibility; always succeeds for internal clients.</summary>
/// <param name="data">Serialized protocol bytes.</param>
public bool QueueOutbound(ReadOnlyMemory<byte> data) => true; // no-op for internal clients
/// <summary>Removes a subscription and updates account subscription counters.</summary>
/// <param name="sid">Subscription identifier to remove.</param>
public void RemoveSubscription(string sid)
{
if (_subs.Remove(sid))
Account?.DecrementSubscriptions();
}
/// <summary>Adds or replaces a subscription tracked by this internal client.</summary>
/// <param name="sub">Subscription to register.</param>
public void AddSubscription(Subscription sub)
{
_subs[sub.Sid] = sub;
}
/// <summary>Current subscription map keyed by subscription identifier.</summary>
public IReadOnlyDictionary<string, Subscription> Subscriptions => _subs;
}
@@ -17,6 +17,12 @@ public static class ConsumerApiHandlers
private const string UnpinPrefix = JetStreamApiSubjects.ConsumerUnpin;
private const string NextPrefix = JetStreamApiSubjects.ConsumerNext;
/// <summary>
/// Handles consumer create/update requests by parsing subject and config payload.
/// </summary>
/// <param name="subject">Consumer create API subject.</param>
/// <param name="payload">JSON payload containing consumer configuration.</param>
/// <param name="consumerManager">Consumer manager responsible for create/update operations.</param>
public static JetStreamApiResponse HandleCreate(string subject, ReadOnlySpan<byte> payload, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, CreatePrefix);
@@ -31,6 +37,11 @@ public static class ConsumerApiHandlers
return consumerManager.CreateOrUpdate(stream, config);
}
/// <summary>
/// Handles consumer info requests for a specific stream/durable pair.
/// </summary>
/// <param name="subject">Consumer info API subject.</param>
/// <param name="consumerManager">Consumer manager responsible for lookup operations.</param>
public static JetStreamApiResponse HandleInfo(string subject, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, InfoPrefix);
@@ -41,6 +52,11 @@ public static class ConsumerApiHandlers
return consumerManager.GetInfo(stream, durableName);
}
/// <summary>
/// Handles consumer delete requests for a specific stream/durable pair.
/// </summary>
/// <param name="subject">Consumer delete API subject.</param>
/// <param name="consumerManager">Consumer manager responsible for deletion.</param>
public static JetStreamApiResponse HandleDelete(string subject, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, DeletePrefix);
@@ -53,6 +69,12 @@ public static class ConsumerApiHandlers
: JetStreamApiResponse.NotFound(subject);
}
/// <summary>
/// Handles paginated consumer name listing requests for a stream.
/// </summary>
/// <param name="subject">Consumer names API subject.</param>
/// <param name="payload">JSON payload containing pagination offset.</param>
/// <param name="consumerManager">Consumer manager used to list names.</param>
public static JetStreamApiResponse HandleNames(string subject, ReadOnlySpan<byte> payload, ConsumerManager consumerManager)
{
var stream = ParseStreamSubject(subject, NamesPrefix);
@@ -70,6 +92,12 @@ public static class ConsumerApiHandlers
};
}
/// <summary>
/// Handles paginated consumer info listing requests for a stream.
/// </summary>
/// <param name="subject">Consumer list API subject.</param>
/// <param name="payload">JSON payload containing pagination offset.</param>
/// <param name="consumerManager">Consumer manager used to list consumer infos.</param>
public static JetStreamApiResponse HandleList(string subject, ReadOnlySpan<byte> payload, ConsumerManager consumerManager)
{
var stream = ParseStreamSubject(subject, ListPrefix);
@@ -105,6 +133,12 @@ public static class ConsumerApiHandlers
return 0;
}
/// <summary>
/// Handles pause/resume requests for a specific consumer.
/// </summary>
/// <param name="subject">Consumer pause API subject.</param>
/// <param name="payload">JSON payload containing pause settings.</param>
/// <param name="consumerManager">Consumer manager responsible for pause state.</param>
public static JetStreamApiResponse HandlePause(string subject, ReadOnlySpan<byte> payload, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, PausePrefix);
@@ -131,6 +165,11 @@ public static class ConsumerApiHandlers
consumerManager.GetPauseUntil(stream, durableName));
}
/// <summary>
/// Handles consumer reset requests.
/// </summary>
/// <param name="subject">Consumer reset API subject.</param>
/// <param name="consumerManager">Consumer manager responsible for reset.</param>
public static JetStreamApiResponse HandleReset(string subject, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, ResetPrefix);
@@ -143,6 +182,11 @@ public static class ConsumerApiHandlers
: JetStreamApiResponse.NotFound(subject);
}
/// <summary>
/// Handles consumer unpin requests for pinned priority consumers.
/// </summary>
/// <param name="subject">Consumer unpin API subject.</param>
/// <param name="consumerManager">Consumer manager responsible for pin state.</param>
public static JetStreamApiResponse HandleUnpin(string subject, ConsumerManager consumerManager)
{
var parsed = ParseSubject(subject, UnpinPrefix);
@@ -155,6 +199,13 @@ public static class ConsumerApiHandlers
: JetStreamApiResponse.NotFound(subject);
}
/// <summary>
/// Handles pull-consumer next batch requests and returns fetched messages.
/// </summary>
/// <param name="subject">Consumer next API subject.</param>
/// <param name="payload">JSON payload containing pull request options.</param>
/// <param name="consumerManager">Consumer manager responsible for fetch operations.</param>
/// <param name="streamManager">Stream manager used to resolve stream stores.</param>
public static JetStreamApiResponse HandleNext(string subject, ReadOnlySpan<byte> payload, ConsumerManager consumerManager, StreamManager streamManager)
{
var parsed = ParseSubject(subject, NextPrefix);
@@ -191,6 +242,10 @@ public static class ConsumerApiHandlers
/// <see cref="JetStreamMetaGroup.ProposeCreateConsumerValidatedAsync"/>.
/// Go reference: jetstream_cluster.go:8100 jsClusteredConsumerRequest.
/// </summary>
/// <param name="subject">Consumer create API subject.</param>
/// <param name="payload">Serialized consumer create request payload.</param>
/// <param name="metaGroup">Meta-group coordinator that validates leadership and proposes RAFT changes.</param>
/// <param name="ct">Cancellation token for RAFT proposal and validation operations.</param>
public static async Task<JetStreamApiResponse> HandleClusteredCreateAsync(
string subject,
byte[] payload,
@@ -230,6 +285,9 @@ public static class ConsumerApiHandlers
/// <see cref="JetStreamMetaGroup.ProposeDeleteConsumerValidatedAsync"/>.
/// Go reference: jetstream_cluster.go jsClusteredConsumerDeleteRequest.
/// </summary>
/// <param name="subject">Consumer delete API subject.</param>
/// <param name="metaGroup">Meta-group coordinator that validates leadership and proposes RAFT changes.</param>
/// <param name="ct">Cancellation token for RAFT proposal and validation operations.</param>
public static async Task<JetStreamApiResponse> HandleClusteredDeleteAsync(
string subject,
JetStreamMetaGroup metaGroup,
@@ -14,6 +14,10 @@ public interface ILeaderForwarder
/// Returns the leader's response, or null when forwarding is not available
/// (e.g. no route to leader) so the caller can fall back to a NotLeader error.
/// </summary>
/// <param name="subject">JetStream API subject identifying the requested operation.</param>
/// <param name="payload">Serialized request payload forwarded to the leader.</param>
/// <param name="leaderName">Current meta-group leader name used as forwarding target.</param>
/// <param name="ct">Cancellation token for caller-driven request cancellation.</param>
Task<JetStreamApiResponse?> ForwardAsync(
string subject,
ReadOnlyMemory<byte> payload,
@@ -36,6 +40,10 @@ public sealed class DefaultLeaderForwarder
/// </summary>
public TimeSpan Timeout { get; }
/// <summary>
/// Creates a default leader forwarder with an optional forwarding timeout.
/// </summary>
/// <param name="timeout">Leader forwarding timeout; defaults to five seconds when omitted.</param>
public DefaultLeaderForwarder(TimeSpan? timeout = null)
{
Timeout = timeout ?? TimeSpan.FromSeconds(5);
@@ -73,11 +81,21 @@ public sealed class JetStreamApiRouter
private readonly ILeaderForwarder? _forwarder;
private long _forwardedCount;
/// <summary>
/// Creates a router with default in-memory managers and no clustering metadata.
/// </summary>
public JetStreamApiRouter()
: this(new StreamManager(), new ConsumerManager(), null)
{
}
/// <summary>
/// Creates a router with explicit managers and optional cluster leader-forwarding dependencies.
/// </summary>
/// <param name="streamManager">Stream manager handling stream API operations.</param>
/// <param name="consumerManager">Consumer manager handling consumer API operations.</param>
/// <param name="metaGroup">Optional meta-group used for leader checks and leader identity.</param>
/// <param name="forwarder">Optional forwarder used to proxy leader-only requests.</param>
public JetStreamApiRouter(
StreamManager streamManager,
ConsumerManager consumerManager,
@@ -104,6 +122,7 @@ public sealed class JetStreamApiRouter
/// Read-only operations (Info, Names, List, MessageGet, Snapshot, DirectGet, Next) do not.
/// Go reference: jetstream_api.go:200-300.
/// </summary>
/// <param name="subject">JetStream API subject to classify as leader-only or local-safe.</param>
public static bool IsLeaderRequired(string subject)
{
// Stream mutating operations
@@ -165,6 +184,9 @@ public sealed class JetStreamApiRouter
/// Async callers should use <see cref="RouteAsync"/> which also attempts forwarding.
/// Go reference: jetstream_api.go — jsClusteredStreamXxxRequest helpers.
/// </summary>
/// <param name="subject">JetStream API subject being routed.</param>
/// <param name="payload">Serialized request payload.</param>
/// <param name="leaderName">Known current leader name returned in the not-leader response.</param>
public static JetStreamApiResponse ForwardToLeader(string subject, ReadOnlySpan<byte> payload, string leaderName)
{
_ = subject;
@@ -181,6 +203,9 @@ public sealed class JetStreamApiRouter
/// Read-only operations are always handled locally regardless of leadership.
/// Go reference: jetstream_api.go:200-300 — leader-forwarding path.
/// </summary>
/// <param name="subject">JetStream API subject to dispatch.</param>
/// <param name="payload">Request payload bytes for the API operation.</param>
/// <param name="ct">Cancellation token for asynchronous routing and forwarding.</param>
public async Task<JetStreamApiResponse> RouteAsync(
string subject,
ReadOnlyMemory<byte> payload,
@@ -219,6 +244,11 @@ public sealed class JetStreamApiRouter
return Route(subject, payload.Span);
}
/// <summary>
/// Routes a JetStream API request synchronously to local handlers or not-leader fallback.
/// </summary>
/// <param name="subject">JetStream API subject to dispatch.</param>
/// <param name="payload">Request payload bytes for handler execution.</param>
public JetStreamApiResponse Route(string subject, ReadOnlySpan<byte> payload)
{
// Go reference: jetstream_api.go:200-300 — leader check + forwarding.
@@ -35,7 +35,9 @@ public sealed class AckProcessor
private int _maxDeliver;
private readonly List<ulong> _exceededSequences = new();
/// <summary>Highest contiguous acknowledged consumer sequence.</summary>
public ulong AckFloor { get; private set; }
/// <summary>Number of sequences terminated with +TERM or delivery-limit exhaustion.</summary>
public int TerminatedCount { get; private set; }
/// <summary>
@@ -60,12 +62,21 @@ public sealed class AckProcessor
/// <summary>Policy applied when a sequence exceeds its max delivery count.</summary>
public DeliveryExceededPolicy ExceededPolicy { get; set; } = DeliveryExceededPolicy.Drop;
/// <summary>
/// Creates an ack processor with optional redelivery backoff schedule.
/// </summary>
/// <param name="backoffMs">Optional per-delivery backoff delays in milliseconds.</param>
public AckProcessor(int[]? backoffMs = null)
{
_backoffMs = backoffMs;
}
// Go: consumer.go — ConsumerConfig maxAckPending + RedeliveryTracker integration
/// <summary>
/// Creates an ack processor that integrates with a redelivery tracker.
/// </summary>
/// <param name="tracker">Redelivery tracker used for delivery metadata.</param>
/// <param name="maxAckPending">Maximum pending acknowledgements; 0 means unlimited.</param>
public AckProcessor(RedeliveryTracker tracker, int maxAckPending = 0)
{
_tracker = tracker;
@@ -74,6 +85,11 @@ public sealed class AckProcessor
_backoffMs = null;
}
/// <summary>
/// Registers a delivered sequence with an acknowledgement deadline.
/// </summary>
/// <param name="sequence">Consumer sequence delivered to a client.</param>
/// <param name="ackWaitMs">Ack wait timeout in milliseconds.</param>
public void Register(ulong sequence, int ackWaitMs)
{
if (sequence <= AckFloor)
@@ -92,6 +108,11 @@ public sealed class AckProcessor
}
// Go: consumer.go — register with deliver subject; ackWait comes from the tracker
/// <summary>
/// Registers a delivered sequence with tracker-provided ack wait and deliver subject.
/// </summary>
/// <param name="sequence">Consumer sequence delivered to a client.</param>
/// <param name="deliverSubject">Deliver subject associated with the sequence.</param>
public void Register(ulong sequence, string deliverSubject)
{
if (_tracker is null)
@@ -105,6 +126,10 @@ public sealed class AckProcessor
}
// Go: consumer.go — processAck without payload: plain +ACK, also notifies tracker
/// <summary>
/// Processes a plain acknowledgement for a sequence.
/// </summary>
/// <param name="seq">Acknowledged sequence.</param>
public void ProcessAck(ulong seq)
{
AckSequence(seq);
@@ -112,6 +137,11 @@ public sealed class AckProcessor
}
// Go: consumer.go — returns ack deadline for a pending sequence; MinValue if not tracked
/// <summary>
/// Gets the current acknowledgement deadline for a pending sequence.
/// </summary>
/// <param name="seq">Sequence to query.</param>
/// <returns>Deadline in UTC offset form, or <see cref="DateTimeOffset.MinValue"/> when unknown.</returns>
public DateTimeOffset GetDeadline(ulong seq)
{
if (_pending.TryGetValue(seq, out var state))
@@ -121,9 +151,18 @@ public sealed class AckProcessor
}
// Go: consumer.go — maxAckPending=0 means unlimited; otherwise cap pending registrations
/// <summary>
/// Indicates whether another pending sequence can be registered.
/// </summary>
/// <returns><see langword="true"/> when pending registrations are below max limits.</returns>
public bool CanRegister() => _maxAckPending <= 0 || _pending.Count < _maxAckPending;
// Go: consumer.go:2550 — parse ack type prefix from raw payload bytes
/// <summary>
/// Parses an ack payload and returns its ack type.
/// </summary>
/// <param name="data">Raw ack payload bytes.</param>
/// <returns>Detected ack type.</returns>
public static AckType ParseAckType(ReadOnlySpan<byte> data)
{
if (data.StartsWith("+ACK"u8))
@@ -137,6 +176,12 @@ public sealed class AckProcessor
return AckType.Unknown;
}
/// <summary>
/// Finds one pending sequence whose ack deadline has expired.
/// </summary>
/// <param name="sequence">Receives expired sequence when found.</param>
/// <param name="deliveries">Receives current delivery count for the sequence.</param>
/// <returns><see langword="true"/> when an expired sequence is found.</returns>
public bool TryGetExpired(out ulong sequence, out int deliveries)
{
foreach (var (seq, state) in _pending)
@@ -157,6 +202,11 @@ public sealed class AckProcessor
// Go: consumer.go:2550 (processAck)
// Dispatches to the appropriate ack handler based on ack type prefix.
// Empty or "+ACK" → ack single; "-NAK" → schedule redelivery; "+TERM" → terminate; "+WPI" → progress reset.
/// <summary>
/// Processes an ack payload and dispatches to ack/nak/term/progress handling.
/// </summary>
/// <param name="seq">Sequence associated with the ack payload.</param>
/// <param name="payload">Raw ack payload bytes.</param>
public void ProcessAck(ulong seq, ReadOnlySpan<byte> payload)
{
if (payload.IsEmpty || payload.SequenceEqual("+ACK"u8))
@@ -197,6 +247,10 @@ public sealed class AckProcessor
}
// Go: consumer.go — processAck for "+ACK": removes from pending and advances AckFloor when contiguous
/// <summary>
/// Acknowledges a sequence and advances ack floor when possible.
/// </summary>
/// <param name="seq">Sequence to acknowledge.</param>
public void AckSequence(ulong seq)
{
_pending.Remove(seq);
@@ -221,6 +275,11 @@ public sealed class AckProcessor
}
// Go: consumer.go — processNak: schedules redelivery with optional explicit delay or backoff array
/// <summary>
/// Processes a NAK and schedules redelivery.
/// </summary>
/// <param name="seq">Sequence to redeliver.</param>
/// <param name="delayMs">Optional explicit redelivery delay in milliseconds.</param>
public void ProcessNak(ulong seq, int delayMs = 0)
{
if (_terminated.Contains(seq))
@@ -249,6 +308,10 @@ public sealed class AckProcessor
}
// Go: consumer.go — processTerm: removes from pending permanently; sequence is never redelivered
/// <summary>
/// Processes a TERM ack, permanently terminating redelivery for the sequence.
/// </summary>
/// <param name="seq">Sequence to terminate.</param>
public void ProcessTerm(ulong seq)
{
if (_pending.Remove(seq))
@@ -259,6 +322,10 @@ public sealed class AckProcessor
}
// Go: consumer.go — processAckProgress (+WPI): resets ack deadline to original ackWait without bumping delivery count
/// <summary>
/// Processes an in-progress ack (+WPI) by extending the sequence deadline.
/// </summary>
/// <param name="seq">Sequence to extend.</param>
public void ProcessProgress(ulong seq)
{
if (!_pending.TryGetValue(seq, out var state))
@@ -268,6 +335,11 @@ public sealed class AckProcessor
_pending[seq] = state;
}
/// <summary>
/// Schedules the next redelivery deadline for a sequence.
/// </summary>
/// <param name="sequence">Sequence to reschedule.</param>
/// <param name="delayMs">Redelivery delay in milliseconds.</param>
public void ScheduleRedelivery(ulong sequence, int delayMs)
{
if (!_pending.TryGetValue(sequence, out var state))
@@ -289,6 +361,10 @@ public sealed class AckProcessor
_pending[sequence] = state;
}
/// <summary>
/// Drops a pending sequence without advancing ack floor.
/// </summary>
/// <param name="sequence">Sequence to remove from pending state.</param>
public void Drop(ulong sequence)
{
_pending.Remove(sequence);
@@ -312,6 +388,7 @@ public sealed class AckProcessor
/// Resets the ack floor to the specified value.
/// Used during consumer reset.
/// </summary>
/// <param name="floor">New ack floor value.</param>
public void SetAckFloor(ulong floor)
{
AckFloor = floor;
@@ -320,9 +397,15 @@ public sealed class AckProcessor
_pending.Remove(key);
}
/// <summary>Indicates whether there are pending unacked sequences.</summary>
public bool HasPending => _pending.Count > 0;
/// <summary>Current number of pending unacked sequences.</summary>
public int PendingCount => _pending.Count;
/// <summary>
/// Acknowledges all pending sequences up to and including the provided sequence.
/// </summary>
/// <param name="sequence">Highest sequence to acknowledge.</param>
public void AckAll(ulong sequence)
{
foreach (var key in _pending.Keys.Where(k => k <= sequence).ToArray())
@@ -359,7 +442,9 @@ public sealed class AckProcessor
private sealed class PendingState
{
/// <summary>Current acknowledgement deadline in UTC.</summary>
public DateTime DeadlineUtc { get; set; }
/// <summary>Current delivery attempt count.</summary>
public int Deliveries { get; set; }
}
}
@@ -19,6 +19,9 @@ public sealed class PriorityGroupManager
/// Register a consumer in a named priority group.
/// Lower <paramref name="priority"/> values indicate higher priority.
/// </summary>
/// <param name="groupName">Priority group name used to coordinate active consumer selection.</param>
/// <param name="consumerId">Consumer identifier to register in the group.</param>
/// <param name="priority">Priority rank where lower numbers are favored.</param>
public void Register(string groupName, string consumerId, int priority)
{
var group = _groups.GetOrAdd(groupName, _ => new PriorityGroup());
@@ -41,6 +44,8 @@ public sealed class PriorityGroupManager
/// <summary>
/// Remove a consumer from a named priority group.
/// </summary>
/// <param name="groupName">Priority group name.</param>
/// <param name="consumerId">Consumer identifier to remove.</param>
public void Unregister(string groupName, string consumerId)
{
if (!_groups.TryGetValue(groupName, out var group))
@@ -61,6 +66,7 @@ public sealed class PriorityGroupManager
/// in the named group, or <c>null</c> if the group is empty or does not exist.
/// When multiple consumers share the same lowest priority, the first registered wins.
/// </summary>
/// <param name="groupName">Priority group name.</param>
public string? GetActiveConsumer(string groupName)
{
if (!_groups.TryGetValue(groupName, out var group))
@@ -86,6 +92,8 @@ public sealed class PriorityGroupManager
/// Returns <c>true</c> if the given consumer is the current active consumer
/// (lowest priority number) in the named group.
/// </summary>
/// <param name="groupName">Priority group name.</param>
/// <param name="consumerId">Consumer identifier to validate.</param>
public bool IsActive(string groupName, string consumerId)
{
var active = GetActiveConsumer(groupName);
@@ -96,6 +104,8 @@ public sealed class PriorityGroupManager
/// Assign a new pin ID to the named group, replacing any existing pin.
/// Go reference: consumer.go (assignNewPinId).
/// </summary>
/// <param name="groupName">Priority group name.</param>
/// <param name="consumerId">Consumer being pinned as active (reserved for API parity).</param>
/// <returns>The newly generated 22-character pin ID.</returns>
public string AssignPinId(string groupName, string consumerId)
{
@@ -115,6 +125,8 @@ public sealed class PriorityGroupManager
/// Returns <c>true</c> if the group exists and its current pin ID equals <paramref name="pinId"/>.
/// Go reference: consumer.go (setPinnedTimer).
/// </summary>
/// <param name="groupName">Priority group name.</param>
/// <param name="pinId">Pin identifier to validate.</param>
public bool ValidatePinId(string groupName, string pinId)
{
if (!_groups.TryGetValue(groupName, out var group))
@@ -131,6 +143,7 @@ public sealed class PriorityGroupManager
/// Clear the current pin ID for the named group. No-op if the group does not exist.
/// Go reference: consumer.go (setPinnedTimer).
/// </summary>
/// <param name="groupName">Priority group name.</param>
public void UnassignPinId(string groupName)
{
if (!_groups.TryGetValue(groupName, out var group))
@@ -144,8 +157,11 @@ public sealed class PriorityGroupManager
private sealed class PriorityGroup
{
/// <summary>Synchronization gate for mutable group membership and pin state.</summary>
public object Lock { get; } = new();
/// <summary>Registered consumers participating in this priority group.</summary>
public List<PriorityMember> Members { get; } = [];
/// <summary>Current sticky pin identifier used for temporary assignment stability.</summary>
public string? CurrentPinId { get; set; }
}
@@ -25,6 +25,7 @@ public enum ConsumerSignal
public sealed class PushConsumerEngine
{
// Go: consumer.go — DeliverSubject routes push-mode messages (cfg.DeliverSubject)
/// <summary>Deliver subject used for push-based consumer message delivery.</summary>
public string DeliverSubject { get; private set; } = string.Empty;
private CancellationTokenSource? _cts;
@@ -83,6 +84,11 @@ public sealed class PushConsumerEngine
FlowControlPendingCount--;
}
/// <summary>
/// Enqueues data and optional control frames for a message selected for push delivery.
/// </summary>
/// <param name="consumer">Consumer receiving the queued frames.</param>
/// <param name="message">Stored stream message selected for delivery.</param>
public void Enqueue(ConsumerHandle consumer, StoredMessage message)
{
if (message.Sequence <= consumer.AckProcessor.AckFloor)
@@ -131,6 +137,12 @@ public sealed class PushConsumerEngine
// StartDeliveryLoop wires the background pump that drains PushFrames and calls
// sendMessage for each frame. The delegate matches the wire-level send signature used
// by NatsClient.SendMessage, mapped to an async ValueTask for testability.
/// <summary>
/// Starts the background delivery loop that drains push frames to the subscriber callback.
/// </summary>
/// <param name="consumer">Consumer whose queued frames are delivered.</param>
/// <param name="sendMessage">Callback that publishes a prepared frame to the target subscriber.</param>
/// <param name="ct">Cancellation token used to stop delivery.</param>
public void StartDeliveryLoop(
ConsumerHandle consumer,
Func<string, string, ReadOnlyMemory<byte>, ReadOnlyMemory<byte>, CancellationToken, ValueTask> sendMessage,
@@ -154,6 +166,9 @@ public sealed class PushConsumerEngine
}
}
/// <summary>
/// Stops the delivery loop and heartbeat timer for this push engine.
/// </summary>
public void StopDeliveryLoop()
{
StopIdleHeartbeatTimer();
@@ -166,6 +181,10 @@ public sealed class PushConsumerEngine
/// Starts the gather loop that polls the store for new messages.
/// Go reference: consumer.go:1400 loopAndGatherMsgs.
/// </summary>
/// <param name="consumer">Consumer whose delivery cursor is advanced by the gather loop.</param>
/// <param name="store">Stream store used to load pending and new messages.</param>
/// <param name="sendMessage">Callback used to emit selected messages downstream.</param>
/// <param name="ct">Cancellation token that stops gather processing.</param>
public void StartGatherLoop(
ConsumerHandle consumer,
IStreamStore store,
@@ -194,6 +213,7 @@ public sealed class PushConsumerEngine
/// Signals the gather loop to wake up and re-poll the store.
/// Go reference: consumer.go:1620 — channel send wakes the loop.
/// </summary>
/// <param name="signal">Reason code for waking the gather loop.</param>
public void Signal(ConsumerSignal signal)
{
_signalChannel?.Writer.TryWrite(signal);
@@ -203,6 +223,8 @@ public sealed class PushConsumerEngine
/// Public test accessor for the filter predicate. Production code uses
/// the private ShouldDeliver; this entry point avoids reflection in unit tests.
/// </summary>
/// <param name="config">Consumer filter configuration.</param>
/// <param name="subject">Candidate stream subject.</param>
public static bool ShouldDeliverPublic(ConsumerConfig config, string subject)
=> ShouldDeliver(config, subject);
@@ -462,10 +484,15 @@ public sealed class PushConsumerEngine
public sealed class PushFrame
{
/// <summary>Indicates this frame carries stream data.</summary>
public bool IsData { get; init; }
/// <summary>Indicates this frame is a flow-control marker.</summary>
public bool IsFlowControl { get; init; }
/// <summary>Indicates this frame is an idle-heartbeat marker.</summary>
public bool IsHeartbeat { get; init; }
/// <summary>Stored message payload for data frames; null for control frames.</summary>
public StoredMessage? Message { get; init; }
/// <summary>Earliest UTC time the frame may be emitted, used for rate limiting.</summary>
public DateTime AvailableAtUtc { get; init; } = DateTime.UtcNow;
/// <summary>
@@ -23,6 +23,10 @@ public sealed class RedeliveryTracker
private readonly long _ackWaitMs;
// Go: consumer.go:100 — BackOff []time.Duration in ConsumerConfig; empty falls back to ackWait
/// <summary>
/// Initializes redelivery tracking with integer backoff delays in milliseconds.
/// </summary>
/// <param name="backoffMs">Backoff schedule where index maps to delivery attempt count.</param>
public RedeliveryTracker(int[] backoffMs)
{
_backoffMs = backoffMs;
@@ -32,6 +36,12 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — ConsumerConfig maxDeliver + ackWait + backoff, new overload storing config fields
/// <summary>
/// Initializes redelivery tracking using max deliveries, default ack wait, and optional long backoff schedule.
/// </summary>
/// <param name="maxDeliveries">Maximum deliveries before a message is considered exhausted.</param>
/// <param name="ackWaitMs">Default ack-wait timeout in milliseconds when no backoff entry exists.</param>
/// <param name="backoffMs">Optional per-delivery backoff schedule in milliseconds.</param>
public RedeliveryTracker(int maxDeliveries, long ackWaitMs, long[]? backoffMs = null)
{
_backoffMs = [];
@@ -43,6 +53,12 @@ public sealed class RedeliveryTracker
// Go: consumer.go:5540 — trackPending records delivery count and schedules deadline
// using the backoff array indexed by (deliveryCount-1), clamped at last entry.
// Returns the UTC time at which the sequence next becomes eligible for redelivery.
/// <summary>
/// Schedules a sequence for redelivery based on delivery count and ack wait/backoff policy.
/// </summary>
/// <param name="seq">Stream sequence to track.</param>
/// <param name="deliveryCount">Current delivery attempt count.</param>
/// <param name="ackWaitMs">Ack wait fallback in milliseconds.</param>
public DateTime Schedule(ulong seq, int deliveryCount, int ackWaitMs = 0)
{
var delayMs = ResolveDelay(deliveryCount, ackWaitMs);
@@ -58,6 +74,11 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — schedule with an explicit deadline into the priority queue
/// <summary>
/// Schedules a sequence for redelivery at an explicit deadline.
/// </summary>
/// <param name="seq">Stream sequence to track.</param>
/// <param name="deadline">UTC deadline when the sequence becomes due.</param>
public void Schedule(ulong seq, DateTimeOffset deadline)
{
_deliveryCounts.TryAdd(seq, 0);
@@ -65,6 +86,9 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — rdq entries are dispatched once their deadline has passed
/// <summary>
/// Returns all tracked sequences that are currently due for redelivery.
/// </summary>
public IReadOnlyList<ulong> GetDue()
{
var now = DateTime.UtcNow;
@@ -84,6 +108,10 @@ public sealed class RedeliveryTracker
// Go: consumer.go — drain the rdq priority queue of all entries whose deadline <= now,
// returning them in deadline order (earliest first).
/// <summary>
/// Returns sequences due at or before the supplied timestamp in deadline order.
/// </summary>
/// <param name="now">Current UTC time used as the due cutoff.</param>
public IEnumerable<ulong> GetDue(DateTimeOffset now)
{
List<(ulong seq, DateTimeOffset deadline)>? dequeued = null;
@@ -123,6 +151,10 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — acking a sequence removes it from the pending redelivery set
/// <summary>
/// Removes a tracked sequence after acknowledgement.
/// </summary>
/// <param name="seq">Stream sequence to stop tracking.</param>
public void Acknowledge(ulong seq)
{
_entries.Remove(seq);
@@ -131,6 +163,11 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — maxdeliver check: drop sequence once delivery count exceeds max
/// <summary>
/// Indicates whether a sequence has reached the supplied max-deliver threshold.
/// </summary>
/// <param name="seq">Stream sequence to inspect.</param>
/// <param name="maxDeliver">Maximum allowed delivery count.</param>
public bool IsMaxDeliveries(ulong seq, int maxDeliver)
{
if (maxDeliver <= 0)
@@ -143,6 +180,10 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — maxdeliver check using the stored _maxDeliveries from new constructor
/// <summary>
/// Indicates whether a sequence has reached the configured max-deliver threshold.
/// </summary>
/// <param name="seq">Stream sequence to inspect.</param>
public bool IsMaxDeliveries(ulong seq)
{
if (_maxDeliveries <= 0)
@@ -153,6 +194,10 @@ public sealed class RedeliveryTracker
}
// Go: consumer.go — rdc map increment: track how many times a sequence has been delivered
/// <summary>
/// Increments delivery attempt count for a tracked sequence.
/// </summary>
/// <param name="seq">Stream sequence to increment.</param>
public void IncrementDeliveryCount(ulong seq)
{
_deliveryCounts[seq] = _deliveryCounts.TryGetValue(seq, out var count) ? count + 1 : 1;
@@ -160,6 +205,10 @@ public sealed class RedeliveryTracker
// Go: consumer.go — backoff delay lookup: index by deliveryCount, clamp to last entry,
// fall back to ackWait when no backoff array is configured.
/// <summary>
/// Returns the redelivery backoff delay for a delivery attempt.
/// </summary>
/// <param name="deliveryCount">Delivery attempt count (1-based).</param>
public long GetBackoffDelay(int deliveryCount)
{
if (_backoffMsLong is { Length: > 0 })
@@ -172,8 +221,13 @@ public sealed class RedeliveryTracker
return _ackWaitMs;
}
/// <summary>
/// Indicates whether a sequence is currently tracked for redelivery.
/// </summary>
/// <param name="seq">Stream sequence to check.</param>
public bool IsTracking(ulong seq) => _entries.ContainsKey(seq);
/// <summary>Total number of sequences currently tracked for redelivery.</summary>
public int TrackedCount => _entries.Count;
// Go: consumer.go — backoff index = min(deliveries-1, len(backoff)-1);
@@ -191,7 +245,9 @@ public sealed class RedeliveryTracker
private sealed class RedeliveryEntry
{
/// <summary>UTC deadline when this sequence should be considered due.</summary>
public DateTime DeadlineUtc { get; set; }
/// <summary>Number of delivery attempts made for this sequence.</summary>
public int DeliveryCount { get; set; }
}
}
@@ -6,9 +6,13 @@ namespace NATS.Server.JetStream;
/// </summary>
public sealed class JetStreamApiStats
{
/// <summary>Current API sampling level.</summary>
public int Level { get; set; }
/// <summary>Total JetStream API requests processed.</summary>
public ulong Total { get; set; }
/// <summary>Total JetStream API requests that returned an error.</summary>
public ulong Errors { get; set; }
/// <summary>Number of API requests currently in flight.</summary>
public int Inflight { get; set; }
}
@@ -18,10 +22,15 @@ public sealed class JetStreamApiStats
/// </summary>
public sealed class JetStreamTier
{
/// <summary>Name of the storage tier.</summary>
public string Name { get; set; } = string.Empty;
/// <summary>Current memory usage in bytes for this tier.</summary>
public long Memory { get; set; }
/// <summary>Current file-store usage in bytes for this tier.</summary>
public long Store { get; set; }
/// <summary>Number of streams in this tier.</summary>
public int Streams { get; set; }
/// <summary>Number of consumers in this tier.</summary>
public int Consumers { get; set; }
}
@@ -31,14 +40,23 @@ public sealed class JetStreamTier
/// </summary>
public sealed class JetStreamAccountLimits
{
/// <summary>Maximum memory bytes allowed for this account.</summary>
public long MaxMemory { get; set; }
/// <summary>Maximum file-store bytes allowed for this account.</summary>
public long MaxStore { get; set; }
/// <summary>Maximum number of streams allowed for this account.</summary>
public int MaxStreams { get; set; }
/// <summary>Maximum number of consumers allowed for this account.</summary>
public int MaxConsumers { get; set; }
/// <summary>Maximum unacknowledged messages allowed across consumers.</summary>
public int MaxAckPending { get; set; }
/// <summary>Maximum bytes per memory-backed stream.</summary>
public long MemoryMaxStreamBytes { get; set; }
/// <summary>Maximum bytes per file-backed stream.</summary>
public long StoreMaxStreamBytes { get; set; }
/// <summary>Indicates whether explicit max byte limits are required.</summary>
public bool MaxBytesRequired { get; set; }
/// <summary>Per-tier limits and usage details keyed by tier name.</summary>
public Dictionary<string, JetStreamTier> Tiers { get; set; } = new(StringComparer.Ordinal);
}
@@ -48,11 +66,18 @@ public sealed class JetStreamAccountLimits
/// </summary>
public sealed class JetStreamStats
{
/// <summary>Total memory bytes currently used by JetStream.</summary>
public long Memory { get; set; }
/// <summary>Total file-store bytes currently used by JetStream.</summary>
public long Store { get; set; }
/// <summary>Memory bytes reserved by configured account limits.</summary>
public long ReservedMemory { get; set; }
/// <summary>File-store bytes reserved by configured account limits.</summary>
public long ReservedStore { get; set; }
/// <summary>Number of accounts with JetStream enabled.</summary>
public int Accounts { get; set; }
/// <summary>Number of high-availability assets under JetStream management.</summary>
public int HaAssets { get; set; }
/// <summary>JetStream API usage counters.</summary>
public JetStreamApiStats Api { get; set; } = new();
}
@@ -96,6 +96,10 @@ public sealed class MirrorCoordinator : IAsyncDisposable
// Error state tracking
private string? _errorMessage;
/// <summary>
/// Initializes a mirror coordinator that applies origin stream messages to a local target store.
/// </summary>
/// <param name="targetStore">Local stream store that persists mirrored messages.</param>
public MirrorCoordinator(IStreamStore targetStore)
{
_targetStore = targetStore;
@@ -111,6 +115,8 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// This is the direct-call path used when the origin and mirror are in the same process.
/// Go reference: server/stream.go:2863-3014 (processInboundMirrorMsg)
/// </summary>
/// <param name="message">Origin stream message to replicate into the mirror stream.</param>
/// <param name="ct">Cancellation token used to abort replication during shutdown.</param>
public async Task OnOriginAppendAsync(StoredMessage message, CancellationToken ct)
{
// Go: sseq == mset.mirror.sseq+1 — normal in-order delivery
@@ -135,6 +141,7 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// Enqueues a message for processing by the background sync loop.
/// Used when messages arrive asynchronously (e.g., from a pull consumer on the origin).
/// </summary>
/// <param name="message">Origin message queued for asynchronous mirror processing.</param>
public bool TryEnqueue(StoredMessage message)
{
return _inbound.Writer.TryWrite(message);
@@ -163,6 +170,8 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// actively pulls batches from the origin.
/// Go reference: server/stream.go:3125-3400 (setupMirrorConsumer)
/// </summary>
/// <param name="originStore">Origin store queried for mirror catch-up and incremental sync.</param>
/// <param name="batchSize">Maximum messages applied per pull iteration.</param>
public void StartPullSyncLoop(IStreamStore originStore, int batchSize = DefaultBatchSize)
{
lock (_gate)
@@ -254,6 +263,7 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// Records the next received sequence number from the origin stream.
/// Sets gap state when a gap (skipped sequences) is detected.
/// </summary>
/// <param name="seq">Observed origin sequence number.</param>
public void RecordSourceSeq(ulong seq)
{
if (_expectedOriginSeq > 0 && seq > _expectedOriginSeq + 1)
@@ -270,6 +280,7 @@ public sealed class MirrorCoordinator : IAsyncDisposable
// -------------------------------------------------------------------------
/// <summary>Sets the coordinator into an error state with the given message.</summary>
/// <param name="message">Human-readable mirror synchronization failure reason.</param>
public void SetError(string message) => _errorMessage = message;
/// <summary>Clears the error state.</summary>
@@ -279,6 +290,7 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// Reports current health state for monitoring.
/// Go reference: server/stream.go:2739-2743 (mirrorInfo), 2698-2736 (sourceInfo)
/// </summary>
/// <param name="originLastSeq">Optional latest sequence from the origin stream for lag calculation.</param>
public MirrorHealthReport GetHealthReport(ulong? originLastSeq = null)
{
var lag = originLastSeq.HasValue && originLastSeq.Value > LastOriginSequence
@@ -301,6 +313,7 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// Returns a structured monitoring response for this mirror.
/// Go reference: server/stream.go:2739-2743 (mirrorInfo building StreamSourceInfo)
/// </summary>
/// <param name="streamName">Local mirror stream name exposed in monitoring responses.</param>
public MirrorInfoResponse GetMirrorInfo(string streamName)
{
var report = GetHealthReport();
@@ -313,6 +326,9 @@ public sealed class MirrorCoordinator : IAsyncDisposable
};
}
/// <summary>
/// Stops mirror synchronization and completes inbound processing resources.
/// </summary>
public async ValueTask DisposeAsync()
{
await StopAsync();
@@ -471,11 +487,17 @@ public sealed class MirrorCoordinator : IAsyncDisposable
/// </summary>
public sealed record MirrorHealthReport
{
/// <summary>Last origin stream sequence that has been persisted locally.</summary>
public ulong LastOriginSequence { get; init; }
/// <summary>UTC timestamp of the last successful mirror apply.</summary>
public DateTime LastSyncUtc { get; init; }
/// <summary>Difference between origin head sequence and mirrored sequence.</summary>
public ulong Lag { get; init; }
/// <summary>Count of consecutive synchronization failures.</summary>
public int ConsecutiveFailures { get; init; }
/// <summary>Whether the mirror sync loop is currently active.</summary>
public bool IsRunning { get; init; }
/// <summary>Whether sync activity appears stale based on heartbeat interval.</summary>
public bool IsStalled { get; init; }
}
@@ -102,6 +102,15 @@ public sealed class SourceCoordinator : IAsyncDisposable
// Used for delta computation during aggregation.
private readonly Dictionary<string, long> _sourceCounterValues = new(StringComparer.Ordinal);
/// <summary>
/// Initializes a coordinator that mirrors one source stream into the target stream store.
/// </summary>
/// <param name="targetStore">
/// Target stream persistence used to store mirrored messages and maintain target-side ordering.
/// </param>
/// <param name="sourceConfig">
/// Source stream replication contract, including filter, account, transform, and dedup settings.
/// </param>
public SourceCoordinator(IStreamStore targetStore, StreamSourceConfig sourceConfig)
{
_targetStore = targetStore;
@@ -159,6 +168,8 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// This is the direct-call path used when the origin and target are in the same process.
/// Go reference: server/stream.go:3860-4007 (processInboundSourceMsg)
/// </summary>
/// <param name="message">Source stream message candidate to replicate into the target stream.</param>
/// <param name="ct">Cancellation token that stops replication when mirror coordination is shutting down.</param>
public async Task OnOriginAppendAsync(StoredMessage message, CancellationToken ct)
{
// Account isolation: skip messages from different accounts.
@@ -223,6 +234,7 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// <summary>
/// Enqueues a message for processing by the background sync loop.
/// </summary>
/// <param name="message">Source message queued for asynchronous mirror processing.</param>
public bool TryEnqueue(StoredMessage message)
{
return _inbound.Writer.TryWrite(message);
@@ -248,6 +260,8 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// Starts a pull-based sync loop that actively fetches from the origin store.
/// Go reference: server/stream.go:3474-3720 (setupSourceConsumer + trySetupSourceConsumer)
/// </summary>
/// <param name="originStore">Origin store used as the authoritative source for backfill and catch-up reads.</param>
/// <param name="batchSize">Maximum number of origin messages processed per pull iteration.</param>
public void StartPullSyncLoop(IStreamStore originStore, int batchSize = DefaultBatchSize)
{
lock (_gate)
@@ -296,6 +310,7 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// Reports current health state for monitoring.
/// Go reference: server/stream.go:2687-2695 (sourcesInfo)
/// </summary>
/// <param name="originLastSeq">Optional current origin sequence used to compute real-time lag from monitoring data.</param>
public SourceHealthReport GetHealthReport(ulong? originLastSeq = null)
{
var lag = originLastSeq.HasValue && originLastSeq.Value > LastOriginSequence
@@ -335,6 +350,9 @@ public sealed class SourceCoordinator : IAsyncDisposable
};
}
/// <summary>
/// Stops source synchronization and completes the inbound queue used by the mirror worker.
/// </summary>
public async ValueTask DisposeAsync()
{
await StopAsync();
@@ -576,6 +594,7 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// Returns true if the given message ID is already present in the dedup window.
/// Go reference: server/stream.go duplicate window check
/// </summary>
/// <param name="msgId">Client-provided Nats-Msg-Id used to enforce at-most-once mirror replay.</param>
public bool IsDuplicate(string msgId)
{
PruneDedupWindowIfNeeded();
@@ -586,6 +605,7 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// Records a message ID in the dedup window with the current timestamp.
/// Go reference: server/stream.go duplicate window tracking
/// </summary>
/// <param name="msgId">Client-provided Nats-Msg-Id to mark as recently observed for this source.</param>
public void RecordMsgId(string msgId)
{
_dedupWindow[msgId] = DateTime.UtcNow;
@@ -597,6 +617,7 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// time-based pruning done by <see cref="PruneDedupWindowIfNeeded"/>.
/// Go reference: server/stream.go duplicate window pruning
/// </summary>
/// <param name="cutoff">UTC threshold; entries older than this instant are removed from dedup state.</param>
public void PruneDedupWindow(DateTimeOffset cutoff)
{
var cutoffDt = cutoff.UtcDateTime;
@@ -642,15 +663,25 @@ public sealed class SourceCoordinator : IAsyncDisposable
/// </summary>
public sealed record SourceHealthReport
{
/// <summary>Name of the origin stream being mirrored.</summary>
public string SourceName { get; init; } = string.Empty;
/// <summary>Optional filter that restricts which source subjects are mirrored.</summary>
public string? FilterSubject { get; init; }
/// <summary>Last origin sequence number that was persisted to the target stream.</summary>
public ulong LastOriginSequence { get; init; }
/// <summary>Timestamp of the most recent successful mirror write.</summary>
public DateTime LastSyncUtc { get; init; }
/// <summary>Difference between latest known origin sequence and last replicated sequence.</summary>
public ulong Lag { get; init; }
/// <summary>Count of consecutive sync failures since the last successful mirror operation.</summary>
public int ConsecutiveFailures { get; init; }
/// <summary>Whether the coordinator currently has an active sync loop.</summary>
public bool IsRunning { get; init; }
/// <summary>Whether the source is considered stalled based on heartbeat and last-sync time.</summary>
public bool IsStalled { get; init; }
/// <summary>Total number of source messages skipped because they did not match the configured filter.</summary>
public long FilteredOutCount { get; init; }
/// <summary>Total number of source messages skipped because their message IDs were already seen.</summary>
public long DeduplicatedCount { get; init; }
}
@@ -30,9 +30,13 @@ public sealed record SnapshotRestoreResult(
file sealed class TarMessageEntry
{
/// <summary>Original stream sequence used to restore message order.</summary>
public ulong Sequence { get; init; }
/// <summary>Subject that the message was originally stored under.</summary>
public string Subject { get; init; } = string.Empty;
/// <summary>Base64-encoded payload bytes stored in the snapshot archive.</summary>
public string Payload { get; init; } = string.Empty; // base-64
/// <summary>UTC timestamp encoded using ISO-8601 format.</summary>
public string Timestamp { get; init; } = string.Empty; // ISO-8601 UTC
}
@@ -46,9 +50,20 @@ public sealed class StreamSnapshotService
// Existing thin wrappers (kept for API compatibility)
// ──────────────────────────────────────────────────────────────────────
/// <summary>
/// Creates a store-native snapshot through the stream store implementation.
/// </summary>
/// <param name="stream">Stream whose persistent state is being snapshotted.</param>
/// <param name="ct">Cancellation token for snapshot creation.</param>
public ValueTask<byte[]> SnapshotAsync(StreamHandle stream, CancellationToken ct)
=> stream.Store.CreateSnapshotAsync(ct);
/// <summary>
/// Restores a store-native snapshot through the stream store implementation.
/// </summary>
/// <param name="stream">Stream receiving restored state.</param>
/// <param name="snapshot">Snapshot payload produced by the underlying store format.</param>
/// <param name="ct">Cancellation token for restore execution.</param>
public ValueTask RestoreAsync(StreamHandle stream, ReadOnlyMemory<byte> snapshot, CancellationToken ct)
=> stream.Store.RestoreSnapshotAsync(snapshot, ct);
@@ -66,6 +81,8 @@ public sealed class StreamSnapshotService
/// messages/000002.json
/// …
/// </summary>
/// <param name="stream">Stream to snapshot, including config and persisted messages.</param>
/// <param name="ct">Cancellation token for snapshot generation.</param>
public async Task<byte[]> CreateTarSnapshotAsync(StreamHandle stream, CancellationToken ct)
{
// Collect messages first (outside the TAR buffer so we hold no lock).
@@ -103,6 +120,9 @@ public sealed class StreamSnapshotService
/// Decompress a Snappy-compressed TAR archive, validate stream.json, and
/// replay all message entries back into the store.
/// </summary>
/// <param name="stream">Target stream that receives restored messages.</param>
/// <param name="snapshot">Snappy-compressed TAR snapshot bytes.</param>
/// <param name="ct">Cancellation token for restore execution.</param>
public async Task<SnapshotRestoreResult> RestoreTarSnapshotAsync(
StreamHandle stream,
ReadOnlyMemory<byte> snapshot,
@@ -174,6 +194,9 @@ public sealed class StreamSnapshotService
/// Same as <see cref="CreateTarSnapshotAsync"/> but cancels automatically if
/// the operation has not completed within <paramref name="deadline"/>.
/// </summary>
/// <param name="stream">Stream to snapshot.</param>
/// <param name="deadline">Maximum allowed runtime before cancellation.</param>
/// <param name="ct">Caller cancellation token linked with the deadline token.</param>
public async Task<byte[]> CreateTarSnapshotWithDeadlineAsync(
StreamHandle stream,
TimeSpan deadline,
@@ -39,6 +39,11 @@ public sealed class ConsumerFileStore : IConsumerStore
// Reference: golang/nats-server/server/errors.go
public static readonly Exception ErrNoAckPolicy = new InvalidOperationException("ErrNoAckPolicy");
/// <summary>
/// Creates a file-backed consumer state store and starts background flush processing.
/// </summary>
/// <param name="stateFile">Path to the on-disk consumer state file.</param>
/// <param name="cfg">Consumer configuration that drives ack and redelivery persistence behavior.</param>
public ConsumerFileStore(string stateFile, ConsumerConfig cfg)
{
_stateFile = stateFile;
@@ -63,6 +68,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.SetStarting — filestore.go:11660
/// <inheritdoc />
public void SetStarting(ulong sseq)
{
lock (_mu)
@@ -72,6 +78,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.UpdateStarting — filestore.go:11665
/// <inheritdoc />
public void UpdateStarting(ulong sseq)
{
lock (_mu)
@@ -81,6 +88,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.Reset — filestore.go:11670
/// <inheritdoc />
public void Reset(ulong sseq)
{
lock (_mu)
@@ -94,6 +102,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.HasState — filestore.go
/// <inheritdoc />
public bool HasState()
{
lock (_mu)
@@ -102,6 +111,7 @@ public sealed class ConsumerFileStore : IConsumerStore
// Go: consumerFileStore.UpdateDelivered — filestore.go:11700
// dseq=consumer delivery seq, sseq=stream seq, dc=delivery count, ts=Unix nanosec timestamp
/// <inheritdoc />
public void UpdateDelivered(ulong dseq, ulong sseq, ulong dc, long ts)
{
lock (_mu)
@@ -138,6 +148,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.UpdateAcks — filestore.go:11760
/// <inheritdoc />
public void UpdateAcks(ulong dseq, ulong sseq)
{
lock (_mu)
@@ -171,6 +182,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.Update — filestore.go
/// <inheritdoc />
public void Update(ConsumerState state)
{
lock (_mu)
@@ -181,6 +193,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.State — filestore.go:12103
/// <inheritdoc />
public ConsumerState State()
{
lock (_mu)
@@ -207,12 +220,14 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.BorrowState — filestore.go:12109
/// <inheritdoc />
public ConsumerState BorrowState()
{
lock (_mu) return _state;
}
// Go: consumerFileStore.EncodedState — filestore.go
/// <inheritdoc />
public byte[] EncodedState()
{
lock (_mu)
@@ -220,9 +235,11 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.Type — filestore.go:12099
/// <inheritdoc />
public StorageType Type() => StorageType.File;
// Go: consumerFileStore.Stop — filestore.go:12327
/// <inheritdoc />
public void Stop()
{
lock (_mu)
@@ -240,6 +257,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.Delete — filestore.go:12382
/// <inheritdoc />
public void Delete()
{
lock (_mu)
@@ -258,6 +276,7 @@ public sealed class ConsumerFileStore : IConsumerStore
}
// Go: consumerFileStore.StreamDelete — filestore.go:12387
/// <inheritdoc />
public void StreamDelete()
{
Delete();
+214 -5
View File
@@ -91,6 +91,12 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
private const int CoalesceMinimum = 16 * 1024; // 16KB — Go: filestore.go:328
private const int MaxFlushWaitMs = 8; // 8ms — Go: filestore.go:331
// Go: msgBlock.needSync — deferred fsync flag. Set when a block has unflushed
// data that needs to reach stable storage. The background flush loop handles
// the actual fsync outside the hot path, matching Go's behaviour.
// Reference: golang/nats-server/server/filestore.go:258 (needSync field).
private readonly ConcurrentQueue<MsgBlock> _needSyncBlocks = new();
// Go: filestore.go — generation counter for cache invalidation.
// Incremented on every write (Append/StoreRawMsg) and delete (Remove/Purge/Compact).
// NumFiltered caches results keyed by (filter, generation) so repeated calls for
@@ -99,15 +105,39 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
private ulong _generation;
private readonly Dictionary<string, (ulong Generation, ulong Count)> _numFilteredCache = new(StringComparer.Ordinal);
/// <summary>
/// Number of active message blocks currently managed by the store.
/// </summary>
public int BlockCount => _blocks.Count;
/// <summary>
/// Indicates whether recovery used the index manifest fast path at startup.
/// </summary>
public bool UsedIndexManifestOnStartup { get; private set; }
// IStreamStore cached state properties — O(1), maintained incrementally.
/// <summary>
/// Highest sequence watermark observed by this stream store.
/// </summary>
public ulong LastSeq => _last;
/// <summary>
/// Current number of live messages tracked by the stream.
/// </summary>
public ulong MessageCount => _messageCount;
/// <summary>
/// Total bytes of live message payload and headers tracked by the stream.
/// </summary>
public ulong TotalBytes => _totalBytes;
/// <inheritdoc />
ulong IStreamStore.FirstSeq => _messageCount == 0 ? (_first > 0 ? _first : 0UL) : _firstSeq;
/// <summary>
/// Creates a file-backed JetStream store and recovers persisted stream blocks.
/// </summary>
/// <param name="options">File store options controlling directory layout, limits, and transforms.</param>
public FileStore(FileStoreOptions options)
{
_options = options;
@@ -136,6 +166,12 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
_flushTask = Task.Run(() => FlushLoopAsync(_flushCts.Token));
}
/// <summary>
/// Appends a new message to the stream and returns the assigned sequence.
/// </summary>
/// <param name="subject">Publish subject for the appended message.</param>
/// <param name="payload">Message payload bytes.</param>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<ulong> AppendAsync(string subject, ReadOnlyMemory<byte> payload, CancellationToken ct)
{
if (_stopped)
@@ -198,11 +234,21 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.FromResult(_last);
}
/// <summary>
/// Loads a stored message by exact sequence.
/// </summary>
/// <param name="sequence">Sequence to load.</param>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<StoredMessage?> LoadAsync(ulong sequence, CancellationToken ct)
{
return ValueTask.FromResult(MaterializeMessage(sequence));
}
/// <summary>
/// Loads the most recent message for a subject.
/// </summary>
/// <param name="subject">Subject to query.</param>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<StoredMessage?> LoadLastBySubjectAsync(string subject, CancellationToken ct)
{
if (_lastSequenceBySubject.TryGetValue(subject, out var sequence))
@@ -214,6 +260,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.FromResult<StoredMessage?>(null);
}
/// <summary>
/// Lists all live messages in sequence order.
/// </summary>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<IReadOnlyList<StoredMessage>> ListAsync(CancellationToken ct)
{
var messages = _meta.Keys
@@ -225,6 +275,11 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.FromResult<IReadOnlyList<StoredMessage>>(messages);
}
/// <summary>
/// Removes a message by sequence when present.
/// </summary>
/// <param name="sequence">Sequence to remove.</param>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<bool> RemoveAsync(ulong sequence, CancellationToken ct)
{
if (!RemoveTrackedMessage(sequence, preserveHighWaterMark: false))
@@ -238,6 +293,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.FromResult(true);
}
/// <summary>
/// Purges all messages and block files for this stream store.
/// </summary>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask PurgeAsync(CancellationToken ct)
{
// Stop the background flush loop before disposing blocks to prevent
@@ -273,6 +332,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.CompletedTask;
}
/// <summary>
/// Creates a serialized snapshot of all live stream messages.
/// </summary>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<byte[]> CreateSnapshotAsync(CancellationToken ct)
{
var snapshot = _meta.Keys
@@ -295,6 +358,11 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.FromResult(JsonSerializer.SerializeToUtf8Bytes(snapshot));
}
/// <summary>
/// Restores stream messages from a previously created snapshot payload.
/// </summary>
/// <param name="snapshot">Serialized snapshot payload.</param>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask RestoreSnapshotAsync(ReadOnlyMemory<byte> snapshot, CancellationToken ct)
{
_meta.Clear();
@@ -347,6 +415,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return ValueTask.CompletedTask;
}
/// <summary>
/// Returns lightweight stream counters used by API responses.
/// </summary>
/// <param name="ct">Cancellation token for async callers.</param>
public ValueTask<ApiStreamState> GetStateAsync(CancellationToken ct)
{
return ValueTask.FromResult(new ApiStreamState
@@ -358,6 +430,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
});
}
/// <summary>
/// Trims oldest messages until live count is at or below the configured limit.
/// </summary>
/// <param name="maxMessages">Maximum live message count to retain.</param>
public void TrimToMaxMessages(ulong maxMessages)
{
var trimmed = false;
@@ -390,6 +466,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// this specific message; otherwise the stream's MaxAgeMs applies.
/// Reference: golang/nats-server/server/filestore.go:6790 (storeMsg).
/// </summary>
/// <param name="subject">Publish subject for the message.</param>
/// <param name="hdr">Optional protocol headers.</param>
/// <param name="msg">Message payload bytes.</param>
/// <param name="ttl">Optional per-message TTL in nanoseconds.</param>
public (ulong Seq, long Ts) StoreMsg(string subject, byte[]? hdr, byte[] msg, long ttl)
{
if (_stopped)
@@ -467,6 +547,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns the number of messages removed.
/// Reference: golang/nats-server/server/filestore.go — PurgeEx.
/// </summary>
/// <param name="subject">Subject filter; empty means all subjects.</param>
/// <param name="seq">Upper inclusive sequence boundary (0 means stream last sequence).</param>
/// <param name="keep">Number of newest matching messages to keep.</param>
public ulong PurgeEx(string subject, ulong seq, ulong keep)
{
// Go parity: empty subject with keep=0 and seq=0 is a full purge.
@@ -529,6 +612,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// and returns the count removed.
/// Reference: golang/nats-server/server/filestore.go — Compact.
/// </summary>
/// <param name="seq">Exclusive upper bound for removed sequences.</param>
public ulong Compact(ulong seq)
{
if (seq == 0)
@@ -566,6 +650,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// and updates the last sequence pointer.
/// Reference: golang/nats-server/server/filestore.go — Truncate.
/// </summary>
/// <param name="seq">Highest sequence to keep.</param>
public void Truncate(ulong seq)
{
if (seq == 0)
@@ -609,6 +694,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns <c>_last + 1</c> if no message exists at or after <paramref name="t"/>.
/// Reference: golang/nats-server/server/filestore.go — GetSeqFromTime.
/// </summary>
/// <param name="t">UTC timestamp used as lookup lower bound.</param>
public ulong GetSeqFromTime(DateTime t)
{
var utc = t.Kind == DateTimeKind.Utc ? t : t.ToUniversalTime();
@@ -635,6 +721,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// messages already cached in <c>_meta</c> are filtered directly.
/// Reference: golang/nats-server/server/filestore.go:3191 (FilteredState).
/// </summary>
/// <param name="seq">Starting sequence lower bound.</param>
/// <param name="subject">Optional subject filter.</param>
public SimpleState FilteredState(ulong seq, string subject)
{
// Fast path: binary-search to find the first block whose LastSequence >= seq,
@@ -712,6 +800,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// sets, which is deferred. This method is the extension point for that optimization.
/// Reference: golang/nats-server/server/filestore.go (block-level subject tracking).
/// </summary>
/// <param name="filter">Subject filter used by the scan.</param>
/// <param name="firstBlock">First block candidate considered by the scan.</param>
public static bool CheckSkipFirstBlock(string filter, MsgBlock firstBlock)
{
// Without per-block subject metadata we cannot skip based on subject alone.
@@ -732,6 +822,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <paramref name="filterSubject"/>. Supports NATS wildcard filters.
/// Reference: golang/nats-server/server/filestore.go — SubjectsState.
/// </summary>
/// <param name="filterSubject">Optional subject filter with wildcard support.</param>
public Dictionary<string, SimpleState> SubjectsState(string filterSubject)
{
var result = new Dictionary<string, SimpleState>(StringComparer.Ordinal);
@@ -771,6 +862,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <paramref name="filterSubject"/>. Supports NATS wildcard filters.
/// Reference: golang/nats-server/server/filestore.go — SubjectsTotals.
/// </summary>
/// <param name="filterSubject">Optional subject filter with wildcard support.</param>
public Dictionary<string, ulong> SubjectsTotals(string filterSubject)
{
var result = new Dictionary<string, ulong>(StringComparer.Ordinal);
@@ -834,6 +926,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <see cref="StreamState.Subjects"/> dictionary.
/// Reference: golang/nats-server/server/filestore.go — FastState.
/// </summary>
/// <param name="state">State object to populate.</param>
public void FastState(ref StreamState state)
{
state.Msgs = _messageCount;
@@ -1063,6 +1156,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// An empty or null filter counts all messages.
/// Reference: golang/nats-server/server/filestore.go — fss NumFiltered (subject-state cache).
/// </summary>
/// <param name="filter">Subject filter; null or empty counts all messages.</param>
public ulong NumFiltered(string filter)
{
var key = filter ?? string.Empty;
@@ -1083,6 +1177,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
return count;
}
/// <summary>
/// Asynchronously disposes this file store and flushes pending buffered writes.
/// </summary>
public async ValueTask DisposeAsync()
{
// Stop the background flush loop first to prevent it from accessing
@@ -1118,6 +1215,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Stops the store and deletes all persisted data (blocks, index files).
/// Reference: golang/nats-server/server/filestore.go — fileStore.Delete.
/// </summary>
/// <param name="inline">Reserved for parity with Go signature; currently ignored.</param>
public void Delete(bool inline = false)
{
Stop();
@@ -1154,11 +1252,14 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
// Flush any pending buffered writes before sealing the outgoing block.
_activeBlock?.FlushPending();
// Go: filestore.go:4499 (flushPendingMsgsLocked) — evict the outgoing block's
// write cache via WriteCacheManager before rotating to the new block.
// WriteCacheManager.EvictBlock flushes to disk then clears the cache.
// Evict from write cache manager (tracking only, no fsync).
if (_activeBlock is not null)
_writeCache.EvictBlock(_activeBlock.BlockId);
{
_writeCache.EvictBlockNoSync(_activeBlock.BlockId);
// Defer fsync to the background flush loop, matching Go's needSync pattern.
// Reference: golang/nats-server/server/filestore.go:7207-7266
_needSyncBlocks.Enqueue(_activeBlock);
}
// Clear the write cache on the outgoing active block — it is now sealed.
// This frees memory; future reads on sealed blocks go to disk.
@@ -1869,6 +1970,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns <c>true</c> if the sequence existed and was removed.
/// Reference: golang/nats-server/server/filestore.go — RemoveMsg.
/// </summary>
/// <param name="seq">Sequence to remove.</param>
public bool RemoveMsg(ulong seq)
{
if (!RemoveTrackedMessage(seq, preserveHighWaterMark: true))
@@ -1888,6 +1990,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns <c>true</c> if the sequence existed and was erased.
/// Reference: golang/nats-server/server/filestore.go:5890 (eraseMsg).
/// </summary>
/// <param name="seq">Sequence to erase.</param>
public bool EraseMsg(ulong seq)
{
if (!RemoveTrackedMessage(seq, preserveHighWaterMark: true))
@@ -1908,6 +2011,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns the skipped sequence number.
/// Reference: golang/nats-server/server/filestore.go — SkipMsg.
/// </summary>
/// <param name="seq">Requested sequence or 0 to auto-assign next sequence.</param>
public ulong SkipMsg(ulong seq)
{
// When seq is 0, auto-assign next sequence.
@@ -1946,6 +2050,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// (_last + 1); otherwise an <see cref="InvalidOperationException"/> is thrown
/// (Go: ErrSequenceMismatch).
/// </summary>
/// <param name="seq">Start sequence or 0 to use next available sequence.</param>
/// <param name="num">Number of contiguous sequence slots to reserve.</param>
public void SkipMsgs(ulong seq, ulong num)
{
if (seq != 0)
@@ -1973,6 +2079,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// future memory-pressure eviction removes entries from <c>_meta</c>.
/// Reference: golang/nats-server/server/filestore.go:8308 (LoadMsg).
/// </summary>
/// <param name="seq">Exact sequence to load.</param>
/// <param name="sm">Optional reusable output container.</param>
public StoreMsg LoadMsg(ulong seq, StoreMsg? sm)
{
var stored = MaterializeMessage(seq);
@@ -2044,6 +2152,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Throws <see cref="KeyNotFoundException"/> if no message exists on the subject.
/// Reference: golang/nats-server/server/filestore.go — LoadLastMsg.
/// </summary>
/// <param name="subject">Subject filter for selecting the latest message.</param>
/// <param name="sm">Optional reusable output container.</param>
public StoreMsg LoadLastMsg(string subject, StoreMsg? sm)
{
ulong? bestSeq = null;
@@ -2077,6 +2187,10 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// sequences skipped to reach it.
/// Reference: golang/nats-server/server/filestore.go — LoadNextMsg.
/// </summary>
/// <param name="filter">Subject filter with wildcard support.</param>
/// <param name="wc">Indicates whether caller precomputed filter as wildcard.</param>
/// <param name="start">Inclusive start sequence.</param>
/// <param name="sm">Optional reusable output container.</param>
public (StoreMsg Msg, ulong Skip) LoadNextMsg(string filter, bool wc, ulong start, StoreMsg? sm)
{
ulong? bestSeq = null;
@@ -2133,6 +2247,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <paramref name="maxAllowed"/> results.
/// Reference: golang/nats-server/server/filestore.go — MultiLastSeqs.
/// </summary>
/// <param name="filters">Subject filters used to select candidate subjects.</param>
/// <param name="maxSeq">Maximum allowed sequence in result set (0 means no cap).</param>
/// <param name="maxAllowed">Maximum number of results allowed before throwing.</param>
public ulong[] MultiLastSeqs(string[] filters, ulong maxSeq, int maxAllowed)
{
var lastPerSubject = new Dictionary<string, ulong>(StringComparer.Ordinal);
@@ -2166,6 +2283,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Throws <see cref="KeyNotFoundException"/> if the sequence does not exist.
/// Reference: golang/nats-server/server/filestore.go — SubjectForSeq.
/// </summary>
/// <param name="seq">Sequence to inspect.</param>
public string SubjectForSeq(ulong seq)
{
if (!_meta.TryGetValue(seq, out var meta))
@@ -2180,6 +2298,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Returns (total, validThrough) where validThrough is the last sequence checked.
/// Reference: golang/nats-server/server/filestore.go — NumPending.
/// </summary>
/// <param name="sseq">Starting sequence lower bound.</param>
/// <param name="filter">Optional subject filter.</param>
/// <param name="lastPerSubject">Whether to count only one newest message per subject.</param>
public (ulong Total, ulong ValidThrough) NumPending(ulong sseq, string filter, bool lastPerSubject)
{
var candidates = _meta
@@ -2220,6 +2341,13 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// sequence.</para>
/// Reference: golang/nats-server/server/filestore.go:6756 (storeRawMsg).
/// </summary>
/// <param name="subject">Publish subject for the replicated message.</param>
/// <param name="hdr">Optional protocol headers.</param>
/// <param name="msg">Message payload bytes.</param>
/// <param name="seq">Sequence assigned by replication source.</param>
/// <param name="ts">Nanosecond timestamp assigned by replication source.</param>
/// <param name="ttl">Optional per-message TTL in nanoseconds.</param>
/// <param name="discardNewCheck">Reserved parity flag for discard-new behavior.</param>
public void StoreRawMsg(string subject, byte[]? hdr, byte[] msg, ulong seq, long ts, long ttl, bool discardNewCheck)
{
if (_stopped)
@@ -2263,6 +2391,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Throws <see cref="KeyNotFoundException"/> if no such message exists.
/// Reference: golang/nats-server/server/filestore.go — LoadPrevMsg.
/// </summary>
/// <param name="start">Exclusive upper sequence bound.</param>
/// <param name="sm">Optional reusable output container.</param>
public StoreMsg LoadPrevMsg(ulong start, StoreMsg? sm)
{
if (start == 0)
@@ -2334,6 +2464,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// encoding will be added when the RAFT snapshot codec is implemented (Task 9).
/// Reference: golang/nats-server/server/filestore.go — EncodedStreamState.
/// </summary>
/// <param name="failed">Number of failed apply operations from consensus layer.</param>
public byte[] EncodedStreamState(ulong failed) => [];
/// <summary>
@@ -2341,6 +2472,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// MaxAge, etc.) to the store options.
/// Reference: golang/nats-server/server/filestore.go — UpdateConfig.
/// </summary>
/// <param name="cfg">Updated stream configuration.</param>
public void UpdateConfig(StreamConfig cfg)
{
_options.MaxMsgsPerSubject = cfg.MaxMsgsPer;
@@ -2405,6 +2537,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// matching the Go server's consumer directory layout.
/// Reference: golang/nats-server/server/filestore.go — newConsumerFileStore.
/// </summary>
/// <param name="name">Consumer durable name.</param>
/// <param name="created">Consumer creation timestamp.</param>
/// <param name="cfg">Consumer configuration snapshot.</param>
public IConsumerStore ConsumerStore(string name, DateTime created, ConsumerConfig cfg)
{
var consumerDir = Path.Combine(_options.Directory, "obs", name);
@@ -2428,6 +2563,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// </summary>
public async Task FlushAllPending()
{
DrainSyncQueue();
_activeBlock?.FlushPending();
_activeBlock?.Flush();
await WriteStreamStateAsync();
@@ -2445,7 +2581,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
while (!ct.IsCancellationRequested)
{
try { await _flushSignal.Reader.WaitToReadAsync(ct); }
catch (OperationCanceledException) { return; }
catch (OperationCanceledException) { break; }
_flushSignal.Reader.TryRead(out _);
var block = _activeBlock;
@@ -2465,6 +2601,26 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
}
block.FlushPending();
// Drain deferred sync queue — fsync sealed blocks outside the hot path.
// Go: filestore.go:7242-7266 — sync happens after releasing the block lock.
DrainSyncQueue();
}
// Final drain on shutdown to ensure all data reaches stable storage.
DrainSyncQueue();
}
/// <summary>
/// Fsyncs all blocks queued by <see cref="RotateBlock"/> for deferred sync.
/// Called from the background flush loop and on shutdown.
/// </summary>
private void DrainSyncQueue()
{
while (_needSyncBlocks.TryDequeue(out var syncBlock))
{
try { syncBlock.Flush(); }
catch (ObjectDisposedException) { /* Block was already disposed (e.g. purge). */ }
}
}
@@ -2528,18 +2684,45 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
private sealed record StreamStateSnapshot
{
/// <summary>
/// First live sequence at checkpoint time.
/// </summary>
public ulong FirstSeq { get; init; }
/// <summary>
/// Last seen sequence watermark at checkpoint time.
/// </summary>
public ulong LastSeq { get; init; }
/// <summary>
/// Number of live messages at checkpoint time.
/// </summary>
public ulong Messages { get; init; }
/// <summary>
/// Approximate bytes written across active blocks at checkpoint time.
/// </summary>
public ulong Bytes { get; init; }
}
private sealed class FileRecord
{
/// <summary>
/// Stream sequence for this snapshot record.
/// </summary>
public ulong Sequence { get; init; }
/// <summary>
/// Subject associated with the message.
/// </summary>
public string? Subject { get; init; }
/// <summary>
/// Optional base64-encoded protocol headers.
/// </summary>
public string? HeadersBase64 { get; init; }
/// <summary>
/// Base64-encoded persisted payload.
/// </summary>
public string? PayloadBase64 { get; init; }
/// <summary>
/// Original message timestamp in UTC.
/// </summary>
public DateTime TimestampUtc { get; init; }
}
@@ -2570,8 +2753,17 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <summary>Tracks per-block cache state.</summary>
internal sealed class CacheEntry
{
/// <summary>
/// Block identifier for this cache entry.
/// </summary>
public int BlockId { get; init; }
/// <summary>
/// Last write time in Environment.TickCount64 milliseconds.
/// </summary>
public long LastWriteTime { get; set; } // Environment.TickCount64 (ms)
/// <summary>
/// Approximate bytes currently buffered for this block.
/// </summary>
public long ApproximateBytes { get; set; }
}
@@ -2623,6 +2815,8 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Records a write to the specified block, updating the entry's timestamp and size.
/// Reference: golang/nats-server/server/filestore.go:6529 (lwts update on write).
/// </summary>
/// <param name="blockId">Block id receiving the write.</param>
/// <param name="bytes">Approximate bytes added to cache.</param>
public void TrackWrite(int blockId, long bytes)
{
var now = Environment.TickCount64;
@@ -2642,6 +2836,9 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Allows tests to simulate past writes without sleeping, avoiding timing
/// dependencies in TTL and size-cap eviction tests.
/// </summary>
/// <param name="blockId">Block id receiving the synthetic write.</param>
/// <param name="bytes">Approximate bytes added to cache.</param>
/// <param name="tickCount64Ms">Synthetic write timestamp in TickCount64 milliseconds.</param>
internal void TrackWriteAt(int blockId, long bytes, long tickCount64Ms)
{
_entries.AddOrUpdate(
@@ -2660,6 +2857,7 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// Called from <see cref="FileStore.RotateBlock"/> for the outgoing block.
/// Reference: golang/nats-server/server/filestore.go:4499 (flushPendingMsgsLocked on rotation).
/// </summary>
/// <param name="blockId">Block id to evict.</param>
public void EvictBlock(int blockId)
{
if (!_entries.TryRemove(blockId, out _))
@@ -2673,6 +2871,17 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
block.ClearCache();
}
/// <summary>
/// Removes the cache entry for the specified block without calling Flush (no fsync).
/// The caller is responsible for ensuring the block is synced later (e.g. via the
/// background flush loop's deferred sync queue).
/// Used by <see cref="FileStore.RotateBlock"/> to avoid synchronous fsync on the hot path.
/// </summary>
public void EvictBlockNoSync(int blockId)
{
_entries.TryRemove(blockId, out _);
}
/// <summary>
/// Flushes and clears the cache for all currently tracked blocks.
/// Reference: golang/nats-server/server/filestore.go:5499 (flushPendingMsgsLocked, all blocks).
@@ -4,43 +4,58 @@ namespace NATS.Server.JetStream.Storage;
public sealed class FileStoreOptions
{
/// <summary>Root directory where JetStream file store data is persisted.</summary>
public string Directory { get; set; } = string.Empty;
/// <summary>Block size used for stream data files in bytes.</summary>
public int BlockSizeBytes { get; set; } = 64 * 1024;
/// <summary>Manifest file name that tracks index metadata for stream blocks.</summary>
public string IndexManifestFileName { get; set; } = "index.manifest.json";
/// <summary>Maximum message age in milliseconds before retention eviction.</summary>
public int MaxAgeMs { get; set; }
// Go: StreamConfig.MaxBytes — maximum total bytes for the stream.
// Reference: golang/nats-server/server/filestore.go — maxBytes field.
/// <summary>Maximum total bytes retained by the stream across all subjects.</summary>
public long MaxBytes { get; set; }
// Go: StreamConfig.Discard — discard policy (Old or New).
// Reference: golang/nats-server/server/filestore.go — discardPolicy field.
/// <summary>Discard strategy applied when limits are reached.</summary>
public DiscardPolicy Discard { get; set; } = DiscardPolicy.Old;
// Legacy boolean compression / encryption flags (FSV1 envelope format).
// When set and the corresponding enum is left at its default (NoCompression /
// NoCipher), the legacy Deflate / XOR path is used for backward compatibility.
/// <summary>Enables legacy compression behavior for backward-compatible file envelopes.</summary>
public bool EnableCompression { get; set; }
/// <summary>Enables legacy encryption behavior for backward-compatible file envelopes.</summary>
public bool EnableEncryption { get; set; }
/// <summary>When enabled, verifies payload checksums during load and replay.</summary>
public bool EnablePayloadIntegrityChecks { get; set; } = true;
/// <summary>Raw key material used by encrypted file store modes.</summary>
public byte[]? EncryptionKey { get; set; }
// Go parity: StoreCompression / StoreCipher (filestore.go ~line 91-92).
// When Compression == S2Compression the S2/Snappy codec is used (FSV2 envelope).
// When Cipher != NoCipher an AEAD cipher is used instead of the legacy XOR.
// Enums are defined in AeadEncryptor.cs.
/// <summary>Compression algorithm used for new file store blocks.</summary>
public StoreCompression Compression { get; set; } = StoreCompression.NoCompression;
/// <summary>Cipher suite used for new file store blocks.</summary>
public StoreCipher Cipher { get; set; } = StoreCipher.NoCipher;
// Go: StreamConfig.MaxMsgsPer — maximum messages per subject (1 = keep last per subject).
// Reference: golang/nats-server/server/filestore.go — per-subject message limits.
/// <summary>Maximum retained message count per subject.</summary>
public int MaxMsgsPerSubject { get; set; }
// Go: filestore.go:4443 (setupWriteCache) — bounded write-cache settings.
// MaxCacheSize: total bytes across all cached blocks before eviction kicks in.
// CacheExpiry: TTL after which an idle block's cache is flushed and cleared.
// Reference: golang/nats-server/server/filestore.go:6220 (expireCacheLocked).
/// <summary>Upper bound for in-memory block cache usage before eviction.</summary>
public long MaxCacheSize { get; set; } = 64 * 1024 * 1024; // 64 MB default
/// <summary>Idle time after which cached block data is expired.</summary>
public TimeSpan CacheExpiry { get; set; } = TimeSpan.FromSeconds(2);
}
@@ -12,45 +12,69 @@ namespace NATS.Server.JetStream.Storage;
public interface IConsumerStore
{
// Go: ConsumerStore.SetStarting — initialise the starting stream sequence for a new consumer
/// <summary>Sets the initial stream sequence from which consumer delivery begins.</summary>
/// <param name="sseq">Starting stream sequence for this consumer.</param>
void SetStarting(ulong sseq);
// Go: ConsumerStore.UpdateStarting — update the starting sequence after a reset
/// <summary>Updates the persisted start sequence after consumer reconfiguration or replay reset.</summary>
/// <param name="sseq">New starting stream sequence.</param>
void UpdateStarting(ulong sseq);
// Go: ConsumerStore.Reset — reset state to a given stream sequence
/// <summary>Resets consumer progress and pending state to a specific stream sequence.</summary>
/// <param name="sseq">Stream sequence used as the reset baseline.</param>
void Reset(ulong sseq);
// Go: ConsumerStore.HasState — returns true if any persisted state exists
/// <summary>Indicates whether durable state for the consumer exists in storage.</summary>
bool HasState();
// Go: ConsumerStore.UpdateDelivered — record a new delivery (dseq=consumer seq, sseq=stream seq,
// dc=delivery count, ts=Unix nanosecond timestamp)
/// <summary>Records an attempted delivery so replay and redelivery bookkeeping remain durable.</summary>
/// <param name="dseq">Consumer delivery sequence number.</param>
/// <param name="sseq">Source stream sequence delivered to the consumer.</param>
/// <param name="dc">Delivery attempt count for this stream message.</param>
/// <param name="ts">Delivery timestamp in Unix nanoseconds.</param>
void UpdateDelivered(ulong dseq, ulong sseq, ulong dc, long ts);
// Go: ConsumerStore.UpdateAcks — record an acknowledgement (dseq=consumer seq, sseq=stream seq)
/// <summary>Persists acknowledgement progress so ack floors survive restart and failover.</summary>
/// <param name="dseq">Acknowledged consumer delivery sequence.</param>
/// <param name="sseq">Acknowledged source stream sequence.</param>
void UpdateAcks(ulong dseq, ulong sseq);
// Go: ConsumerStore.Update — overwrite the full consumer state in one call
/// <summary>Overwrites the full persisted consumer state snapshot.</summary>
/// <param name="state">Complete consumer state to persist.</param>
void Update(ConsumerState state);
// Go: ConsumerStore.State — return a snapshot of current consumer state
/// <summary>Returns a copy of current persisted consumer state.</summary>
ConsumerState State();
// Go: ConsumerStore.BorrowState — return state without copying (caller must not retain beyond call)
/// <summary>Returns a non-copied state view for short-lived internal access.</summary>
ConsumerState BorrowState();
// Go: ConsumerStore.EncodedState — return the binary-encoded state for replication
/// <summary>Returns binary-encoded consumer state for replication and snapshot transfer.</summary>
byte[] EncodedState();
// Go: ConsumerStore.Type — the storage type backing this store (File or Memory)
/// <summary>Returns the backing storage type used by this consumer store.</summary>
StorageType Type();
// Go: ConsumerStore.Stop — flush and close the store without deleting data
/// <summary>Flushes and closes the store while retaining persisted consumer state.</summary>
void Stop();
// Go: ConsumerStore.Delete — stop the store and delete all persisted state
/// <summary>Deletes all persisted consumer state and releases underlying resources.</summary>
void Delete();
// Go: ConsumerStore.StreamDelete — called when the parent stream is deleted
/// <summary>Handles parent stream deletion and cleans consumer persistence accordingly.</summary>
void StreamDelete();
}
@@ -21,9 +21,13 @@ public sealed class MemStore : IStreamStore
private sealed class SnapshotRecord
{
/// <summary>Stream sequence for the captured message.</summary>
public ulong Sequence { get; init; }
/// <summary>Published subject for the captured message.</summary>
public string Subject { get; init; } = string.Empty;
/// <summary>Base64-encoded payload bytes persisted in the snapshot.</summary>
public string PayloadBase64 { get; init; } = string.Empty;
/// <summary>Original message timestamp in UTC.</summary>
public DateTime TimestampUtc { get; init; }
}
@@ -39,6 +43,14 @@ public sealed class MemStore : IStreamStore
public readonly ulong Seq;
public readonly long Ts; // Unix nanoseconds
/// <summary>
/// Creates the in-memory representation for a single stream message.
/// </summary>
/// <param name="subj">Subject the message was published to.</param>
/// <param name="hdr">Optional NATS header bytes.</param>
/// <param name="data">Optional payload bytes.</param>
/// <param name="seq">Assigned stream sequence.</param>
/// <param name="ts">Publish timestamp in Unix nanoseconds.</param>
public Msg(string subj, byte[]? hdr, byte[]? data, ulong seq, long ts)
{
Subj = subj;
@@ -106,8 +118,15 @@ public sealed class MemStore : IStreamStore
// Constructor
// -------------------------------------------------------------------------
/// <summary>
/// Initializes an empty in-memory stream store with default limits.
/// </summary>
public MemStore() { }
/// <summary>
/// Initializes an in-memory stream store using stream retention and TTL configuration.
/// </summary>
/// <param name="cfg">Stream configuration used to seed limits and sequence watermark state.</param>
public MemStore(StreamConfig cfg)
{
_cfg = cfg;
@@ -123,9 +142,13 @@ public sealed class MemStore : IStreamStore
}
// IStreamStore cached state properties — O(1), maintained incrementally.
/// <summary>Gets the highest sequence assigned by this stream store.</summary>
public ulong LastSeq { get { lock (_gate) return _st.LastSeq; } }
/// <summary>Gets the current number of retained messages.</summary>
public ulong MessageCount { get { lock (_gate) return _st.Msgs; } }
/// <summary>Gets the total byte usage for retained messages.</summary>
public ulong TotalBytes { get { lock (_gate) return _st.Bytes; } }
/// <inheritdoc />
ulong IStreamStore.FirstSeq { get { lock (_gate) return _st.Msgs == 0 ? (_st.FirstSeq > 0 ? _st.FirstSeq : 0UL) : _st.FirstSeq; } }
// -------------------------------------------------------------------------
@@ -133,6 +156,13 @@ public sealed class MemStore : IStreamStore
// -------------------------------------------------------------------------
// Go: memStore.StoreMsg — async wrapper
/// <summary>
/// Appends a new message to the stream and returns the assigned sequence.
/// </summary>
/// <param name="subject">Subject used for stream indexing and filtering.</param>
/// <param name="payload">Message payload bytes.</param>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>The sequence assigned to the stored message.</returns>
public ValueTask<ulong> AppendAsync(string subject, ReadOnlyMemory<byte> payload, CancellationToken ct)
{
lock (_gate)
@@ -144,6 +174,12 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Loads a stored message by its stream sequence.
/// </summary>
/// <param name="sequence">Sequence to load.</param>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>The stored message, or <see langword="null"/> when the sequence is absent.</returns>
public ValueTask<StoredMessage?> LoadAsync(ulong sequence, CancellationToken ct)
{
lock (_gate)
@@ -160,6 +196,12 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Loads the most recent retained message for a concrete subject.
/// </summary>
/// <param name="subject">Subject to query.</param>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>The latest message for the subject, or <see langword="null"/> when none exists.</returns>
public ValueTask<StoredMessage?> LoadLastBySubjectAsync(string subject, CancellationToken ct)
{
lock (_gate)
@@ -178,6 +220,11 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Lists all retained messages in ascending sequence order.
/// </summary>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>Snapshot of retained messages.</returns>
public ValueTask<IReadOnlyList<StoredMessage>> ListAsync(CancellationToken ct)
{
lock (_gate)
@@ -196,6 +243,12 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Soft-deletes a message by sequence.
/// </summary>
/// <param name="sequence">Sequence to remove.</param>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns><see langword="true"/> when the message existed and was removed.</returns>
public ValueTask<bool> RemoveAsync(ulong sequence, CancellationToken ct)
{
lock (_gate)
@@ -204,6 +257,10 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Removes all retained messages while preserving next-sequence continuity.
/// </summary>
/// <param name="ct">Cancellation token reserved for API parity.</param>
public ValueTask PurgeAsync(CancellationToken ct)
{
lock (_gate)
@@ -213,6 +270,11 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Creates a JSON snapshot of retained messages for backup and restore workflows.
/// </summary>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>UTF-8 JSON payload representing stream messages.</returns>
public ValueTask<byte[]> CreateSnapshotAsync(CancellationToken ct)
{
lock (_gate)
@@ -232,6 +294,11 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Restores the stream from a previously captured snapshot payload.
/// </summary>
/// <param name="snapshot">Serialized snapshot bytes.</param>
/// <param name="ct">Cancellation token reserved for API parity.</param>
public ValueTask RestoreSnapshotAsync(ReadOnlyMemory<byte> snapshot, CancellationToken ct)
{
lock (_gate)
@@ -266,6 +333,11 @@ public sealed class MemStore : IStreamStore
}
}
/// <summary>
/// Returns API state counters used by JetStream management endpoints.
/// </summary>
/// <param name="ct">Cancellation token reserved for API parity.</param>
/// <returns>Current message, sequence, and byte counters.</returns>
public ValueTask<ApiStreamState> GetStateAsync(CancellationToken ct)
{
lock (_gate)
@@ -290,6 +362,7 @@ public sealed class MemStore : IStreamStore
// -------------------------------------------------------------------------
// Go: memStore.StoreMsg server/memstore.go:350
/// <inheritdoc />
(ulong Seq, long Ts) IStreamStore.StoreMsg(string subject, byte[]? hdr, byte[] msg, long ttl)
{
lock (_gate)
@@ -303,6 +376,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.StoreRawMsg server/memstore.go:329
/// <inheritdoc />
void IStreamStore.StoreRawMsg(string subject, byte[]? hdr, byte[] msg, ulong seq, long ts, long ttl, bool discardNewCheck)
{
lock (_gate)
@@ -312,6 +386,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.SkipMsg server/memstore.go:368
/// <inheritdoc />
ulong IStreamStore.SkipMsg(ulong seq)
{
lock (_gate)
@@ -336,6 +411,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.SkipMsgs server/memstore.go:395
/// <inheritdoc />
void IStreamStore.SkipMsgs(ulong seq, ulong num)
{
lock (_gate)
@@ -361,9 +437,11 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.FlushAllPending server/memstore.go:423 — no-op for in-memory store
/// <inheritdoc />
Task IStreamStore.FlushAllPending() => Task.CompletedTask;
// Go: memStore.LoadMsg server/memstore.go:1692
/// <inheritdoc />
StoreMsg IStreamStore.LoadMsg(ulong seq, StoreMsg? sm)
{
lock (_gate)
@@ -375,6 +453,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.LoadNextMsg server/memstore.go:1798
/// <inheritdoc />
(StoreMsg Msg, ulong Skip) IStreamStore.LoadNextMsg(string filter, bool wc, ulong start, StoreMsg? sm)
{
lock (_gate)
@@ -397,6 +476,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.LoadLastMsg server/memstore.go:1724
/// <inheritdoc />
StoreMsg IStreamStore.LoadLastMsg(string subject, StoreMsg? sm)
{
lock (_gate)
@@ -442,6 +522,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.LoadPrevMsg — walk backwards from start
/// <inheritdoc />
StoreMsg IStreamStore.LoadPrevMsg(ulong start, StoreMsg? sm)
{
lock (_gate)
@@ -457,6 +538,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.RemoveMsg — soft delete
/// <inheritdoc />
bool IStreamStore.RemoveMsg(ulong seq)
{
lock (_gate)
@@ -466,6 +548,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.EraseMsg — overwrite then remove
/// <inheritdoc />
bool IStreamStore.EraseMsg(ulong seq)
{
lock (_gate)
@@ -475,6 +558,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.Purge server/memstore.go:1471
/// <inheritdoc />
ulong IStreamStore.Purge()
{
lock (_gate)
@@ -484,6 +568,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.PurgeEx server/memstore.go:1422
/// <inheritdoc />
ulong IStreamStore.PurgeEx(string subject, ulong seq, ulong keep)
{
if (string.IsNullOrEmpty(subject) || subject == ">")
@@ -536,9 +621,11 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.Compact server/memstore.go:1509
/// <inheritdoc />
ulong IStreamStore.Compact(ulong seq) => CompactInternal(seq);
// Go: memStore.Truncate server/memstore.go:1618
/// <inheritdoc />
void IStreamStore.Truncate(ulong seq)
{
lock (_gate)
@@ -574,6 +661,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.GetSeqFromTime server/memstore.go:453
/// <inheritdoc />
ulong IStreamStore.GetSeqFromTime(DateTime t)
{
lock (_gate)
@@ -642,10 +730,12 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.FilteredState server/memstore.go:531
/// <inheritdoc />
SimpleState IStreamStore.FilteredState(ulong seq, string subject)
=> FilteredStateInternal(seq, subject);
// Go: memStore.SubjectsState server/memstore.go:748
/// <inheritdoc />
Dictionary<string, SimpleState> IStreamStore.SubjectsState(string filterSubject)
{
lock (_gate)
@@ -668,6 +758,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.SubjectsTotals server/memstore.go:881
/// <inheritdoc />
Dictionary<string, ulong> IStreamStore.SubjectsTotals(string filterSubject)
{
lock (_gate)
@@ -683,6 +774,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.AllLastSeqs server/memstore.go:780
/// <inheritdoc />
ulong[] IStreamStore.AllLastSeqs()
{
lock (_gate)
@@ -695,6 +787,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.MultiLastSeqs server/memstore.go:828
/// <inheritdoc />
ulong[] IStreamStore.MultiLastSeqs(string[] filters, ulong maxSeq, int maxAllowed)
{
lock (_gate)
@@ -739,6 +832,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.SubjectForSeq server/memstore.go:1678
/// <inheritdoc />
string IStreamStore.SubjectForSeq(ulong seq)
{
lock (_gate)
@@ -750,6 +844,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.NumPending server/memstore.go:913
/// <inheritdoc />
(ulong Total, ulong ValidThrough) IStreamStore.NumPending(ulong sseq, string filter, bool lastPerSubject)
{
lock (_gate)
@@ -760,6 +855,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.State server/memstore.go — full state
/// <inheritdoc />
StorageStreamState IStreamStore.State()
{
lock (_gate)
@@ -784,6 +880,7 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.FastState server/memstore.go — populate without deleted list
/// <inheritdoc />
void IStreamStore.FastState(ref StorageStreamState state)
{
lock (_gate)
@@ -800,9 +897,11 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.Type
/// <inheritdoc />
StorageType IStreamStore.Type() => StorageType.Memory;
// Go: memStore.UpdateConfig server/memstore.go:86
/// <inheritdoc />
void IStreamStore.UpdateConfig(StreamConfig cfg)
{
lock (_gate)
@@ -831,9 +930,11 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.Stop — no-op for in-memory store
/// <inheritdoc />
void IStreamStore.Stop() { }
// Go: memStore.Delete — clear everything
/// <inheritdoc />
void IStreamStore.Delete(bool inline)
{
lock (_gate)
@@ -849,11 +950,14 @@ public sealed class MemStore : IStreamStore
}
// Go: memStore.ResetState
/// <inheritdoc />
void IStreamStore.ResetState() { }
// EncodedStreamState, ConsumerStore — not needed for MemStore tests
/// <inheritdoc />
byte[] IStreamStore.EncodedStreamState(ulong failed) => [];
/// <inheritdoc />
IConsumerStore IStreamStore.ConsumerStore(string name, DateTime created, ConsumerConfig cfg)
=> throw new NotSupportedException("MemStore does not implement ConsumerStore.");
@@ -861,6 +965,10 @@ public sealed class MemStore : IStreamStore
// TrimToMaxMessages — legacy helper used by existing async tests
// -------------------------------------------------------------------------
/// <summary>
/// Trims oldest messages until the stream contains at most <paramref name="maxMessages"/> entries.
/// </summary>
/// <param name="maxMessages">Maximum retained message count.</param>
public void TrimToMaxMessages(ulong maxMessages)
{
lock (_gate)
@@ -1232,6 +1340,9 @@ public sealed class MemStore : IStreamStore
/// <paramref name="filter"/> at or after <paramref name="start"/>. Called with
/// <c>_gate</c> already held.
/// </summary>
/// <param name="filter">Subject wildcard filter used for matching.</param>
/// <param name="start">Minimum sequence to include.</param>
/// <returns>First and last matching sequence with a found flag.</returns>
internal (ulong First, ulong Last, bool Found) NextWildcardMatchLocked(string filter, ulong start)
{
ulong first = _st.LastSeq, last = 0;
@@ -1256,6 +1367,9 @@ public sealed class MemStore : IStreamStore
/// equals <paramref name="filter"/> at or after <paramref name="start"/>. Called
/// with <c>_gate</c> already held.
/// </summary>
/// <param name="filter">Literal subject to match.</param>
/// <param name="start">Minimum sequence to include.</param>
/// <returns>First and last matching sequence with a found flag.</returns>
internal (ulong First, ulong Last, bool Found) NextLiteralMatchLocked(string filter, ulong start)
{
if (!_fss.TryGetValue(filter, out var ss)) return (0, 0, false);
@@ -53,6 +53,7 @@ public sealed class MessageRecord
/// <summary>
/// Encodes a <see cref="MessageRecord"/> to its binary wire format.
/// </summary>
/// <param name="record">Record to encode.</param>
/// <returns>The encoded byte array.</returns>
public static byte[] Encode(MessageRecord record)
{
@@ -66,6 +67,10 @@ public sealed class MessageRecord
/// <summary>
/// Computes the encoded byte size of a record without allocating.
/// </summary>
/// <param name="subject">Subject for the record.</param>
/// <param name="headers">Header bytes for the record.</param>
/// <param name="payload">Payload bytes for the record.</param>
/// <returns>Total encoded byte size.</returns>
public static int MeasureEncodedSize(string subject, ReadOnlySpan<byte> headers, ReadOnlySpan<byte> payload)
{
var subjectByteCount = Encoding.UTF8.GetByteCount(subject);
@@ -81,6 +86,15 @@ public sealed class MessageRecord
/// Go equivalent: writeMsgRecordLocked writes directly into cache.buf.
/// Returns the number of bytes written.
/// </summary>
/// <param name="buffer">Target buffer that receives encoded bytes.</param>
/// <param name="bufOffset">Starting offset in <paramref name="buffer"/>.</param>
/// <param name="sequence">Stream sequence to encode.</param>
/// <param name="subject">Subject to encode.</param>
/// <param name="headers">Header bytes to encode.</param>
/// <param name="payload">Payload bytes to encode.</param>
/// <param name="timestamp">Publish timestamp in Unix nanoseconds.</param>
/// <param name="deleted">Whether to mark the record as deleted.</param>
/// <returns>Number of bytes written to <paramref name="buffer"/>.</returns>
public static int EncodeTo(
byte[] buffer, int bufOffset,
ulong sequence, string subject,
@@ -507,6 +507,7 @@ public sealed class MsgBlock : IDisposable
/// This mirrors Go's SkipMsg tombstone behaviour.
/// Reference: golang/nats-server/server/filestore.go — SkipMsg.
/// </summary>
/// <param name="sequence">Sequence number to reserve as a deleted skip record.</param>
public void WriteSkip(ulong sequence)
{
_lock.EnterWriteLock();
@@ -647,6 +648,8 @@ public sealed class MsgBlock : IDisposable
/// Returns true if the given sequence number has been soft-deleted in this block.
/// Reference: golang/nats-server/server/filestore.go — dmap (deleted map) lookup.
/// </summary>
/// <param name="sequence">Sequence number to test.</param>
/// <returns><see langword="true"/> when the sequence is marked deleted.</returns>
public bool IsDeleted(ulong sequence)
{
_lock.EnterReadLock();
@@ -21,6 +21,8 @@ internal static class S2Codec
/// Returns the compressed bytes, which may be longer than the input for
/// very small payloads (Snappy does not guarantee compression for tiny inputs).
/// </summary>
/// <param name="data">Uncompressed payload bytes.</param>
/// <returns>Compressed payload bytes.</returns>
public static byte[] Compress(ReadOnlySpan<byte> data)
{
if (data.IsEmpty)
@@ -32,6 +34,8 @@ internal static class S2Codec
/// <summary>
/// Decompresses Snappy-compressed <paramref name="data"/>.
/// </summary>
/// <param name="data">Compressed payload bytes.</param>
/// <returns>Decompressed payload bytes.</returns>
/// <exception cref="InvalidDataException">If the data is not valid Snappy.</exception>
public static byte[] Decompress(ReadOnlySpan<byte> data)
{
@@ -45,6 +49,9 @@ internal static class S2Codec
/// Compresses only the body portion of <paramref name="data"/>, leaving the
/// last <paramref name="checksumSize"/> bytes uncompressed (appended verbatim).
/// </summary>
/// <param name="data">Body plus trailing checksum bytes.</param>
/// <param name="checksumSize">Number of trailing checksum bytes to keep raw.</param>
/// <returns>Compressed body with raw trailing checksum bytes appended.</returns>
/// <remarks>
/// In the Go FileStore the trailing bytes of a stored record can be a raw
/// checksum that is not part of the compressed payload. This helper mirrors
@@ -82,6 +89,9 @@ internal static class S2Codec
/// Decompresses only the body portion of <paramref name="data"/>, treating
/// the last <paramref name="checksumSize"/> bytes as a raw (uncompressed) checksum.
/// </summary>
/// <param name="data">Compressed body plus trailing checksum bytes.</param>
/// <param name="checksumSize">Number of trailing checksum bytes kept raw.</param>
/// <returns>Decompressed body with original trailing checksum bytes appended.</returns>
public static byte[] DecompressWithTrailingChecksum(ReadOnlySpan<byte> data, int checksumSize)
{
if (checksumSize < 0)
@@ -48,6 +48,8 @@ internal sealed class SequenceSet : IEnumerable<ulong>
/// Returns <c>true</c> if the sequence was not already present.
/// Reference: golang/nats-server/server/avl/seqset.go:44 (Insert).
/// </summary>
/// <param name="seq">Sequence to add.</param>
/// <returns><see langword="true"/> when the sequence was newly added.</returns>
public bool Add(ulong seq)
{
// Strategy: find the position where seq belongs (binary search by Start),
@@ -122,6 +124,8 @@ internal sealed class SequenceSet : IEnumerable<ulong>
/// Returns <c>true</c> if the sequence was present.
/// Reference: golang/nats-server/server/avl/seqset.go:80 (Delete).
/// </summary>
/// <param name="seq">Sequence to remove.</param>
/// <returns><see langword="true"/> when the sequence existed in the set.</returns>
public bool Remove(ulong seq)
{
// Binary search for the range that contains seq.
@@ -170,6 +174,8 @@ internal sealed class SequenceSet : IEnumerable<ulong>
/// Binary search: O(log R) where R is the number of distinct ranges.
/// Reference: golang/nats-server/server/avl/seqset.go:52 (Exists).
/// </summary>
/// <param name="seq">Sequence to test for membership.</param>
/// <returns><see langword="true"/> when the set contains <paramref name="seq"/>.</returns>
public bool Contains(ulong seq)
{
var lo = 0;
@@ -225,6 +231,7 @@ internal sealed class SequenceSet : IEnumerable<ulong>
}
}
/// <inheritdoc />
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
=> GetEnumerator();
}
@@ -10,18 +10,23 @@ namespace NATS.Server.JetStream.Storage;
public sealed class StoreMsg
{
// Go: StoreMsg.subj
/// <summary>Subject associated with this stored message.</summary>
public string Subject { get; set; } = string.Empty;
// Go: StoreMsg.hdr — NATS message headers (optional)
/// <summary>Optional encoded header bytes.</summary>
public byte[]? Header { get; set; }
// Go: StoreMsg.msg — message body
/// <summary>Optional message payload bytes.</summary>
public byte[]? Data { get; set; }
// Go: StoreMsg.seq — stream sequence number
/// <summary>Stream sequence number.</summary>
public ulong Sequence { get; set; }
// Go: StoreMsg.ts — wall-clock timestamp in Unix nanoseconds
/// <summary>Publish timestamp in Unix nanoseconds.</summary>
public long Timestamp { get; set; }
/// <summary>
@@ -2,12 +2,19 @@ namespace NATS.Server.JetStream.Storage;
public sealed class StoredMessage
{
/// <summary>Stream sequence assigned to this message.</summary>
public ulong Sequence { get; init; }
/// <summary>Subject the message was published to.</summary>
public string Subject { get; init; } = string.Empty;
/// <summary>Message payload bytes.</summary>
public ReadOnlyMemory<byte> Payload { get; init; }
/// <summary>Raw protocol header bytes used for header parsing and replay.</summary>
internal ReadOnlyMemory<byte> RawHeaders { get; init; }
/// <summary>Message timestamp in UTC.</summary>
public DateTime TimestampUtc { get; init; } = DateTime.UtcNow;
/// <summary>Optional account name associated with the message.</summary>
public string? Account { get; init; }
/// <summary>Indicates whether the message has been redelivered.</summary>
public bool Redelivered { get; init; }
/// <summary>
@@ -20,6 +27,10 @@ public sealed class StoredMessage
/// </summary>
public string? MsgId => Headers is not null && Headers.TryGetValue("Nats-Msg-Id", out var id) ? id : null;
/// <summary>
/// Converts this message to a compact index representation.
/// </summary>
/// <returns>Message index used by listing and lookup operations.</returns>
internal StoredMessageIndex ToIndex()
=> new(Sequence, Subject, Payload.Length, TimestampUtc);
}
@@ -9,39 +9,51 @@ namespace NATS.Server.JetStream.Storage;
public record struct StreamState
{
// Go: StreamState.Msgs — total number of messages in the stream
/// <summary>Total number of retained messages in the stream.</summary>
public ulong Msgs { get; set; }
// Go: StreamState.Bytes — total bytes stored
/// <summary>Total bytes consumed by retained messages.</summary>
public ulong Bytes { get; set; }
// Go: StreamState.FirstSeq — sequence number of the oldest message
/// <summary>Sequence number of the oldest retained message.</summary>
public ulong FirstSeq { get; set; }
// Go: StreamState.FirstTime — wall-clock time of the oldest message
/// <summary>Timestamp of the oldest retained message.</summary>
public DateTime FirstTime { get; set; }
// Go: StreamState.LastSeq — sequence number of the newest message
/// <summary>Sequence number of the newest retained message.</summary>
public ulong LastSeq { get; set; }
// Go: StreamState.LastTime — wall-clock time of the newest message
/// <summary>Timestamp of the newest retained message.</summary>
public DateTime LastTime { get; set; }
// Go: StreamState.NumSubjects — count of distinct subjects in the stream
/// <summary>Count of distinct subjects currently represented in the stream.</summary>
public int NumSubjects { get; set; }
// Go: StreamState.Subjects — per-subject message counts (populated on demand)
/// <summary>Optional per-subject retained message totals.</summary>
public Dictionary<string, ulong>? Subjects { get; set; }
// Go: StreamState.NumDeleted — number of interior gaps (deleted sequences)
/// <summary>Count of deleted interior sequences currently tracked.</summary>
public int NumDeleted { get; set; }
// Go: StreamState.Deleted — explicit list of deleted sequences (populated on demand)
/// <summary>Optional list of deleted interior sequence numbers.</summary>
public ulong[]? Deleted { get; set; }
// Go: StreamState.Lost (LostStreamData) — sequences that were lost due to storage corruption
/// <summary>Optional corruption/loss metadata for this stream.</summary>
public LostStreamData? Lost { get; set; }
// Go: StreamState.Consumers — number of consumers attached to the stream
/// <summary>Number of consumers currently attached to the stream.</summary>
public int Consumers { get; set; }
}
@@ -53,9 +65,11 @@ public record struct StreamState
public sealed class LostStreamData
{
// Go: LostStreamData.Msgs — sequences of lost messages
/// <summary>Sequences that were lost due to storage corruption.</summary>
public ulong[]? Msgs { get; set; }
// Go: LostStreamData.Bytes — total bytes of lost data
/// <summary>Total bytes estimated as lost.</summary>
public ulong Bytes { get; set; }
}
@@ -68,11 +82,14 @@ public sealed class LostStreamData
public record struct SimpleState
{
// Go: SimpleState.Msgs — number of messages matching the filter
/// <summary>Count of matching retained messages.</summary>
public ulong Msgs { get; set; }
// Go: SimpleState.First — first sequence number matching the filter
/// <summary>First sequence that matches the filter.</summary>
public ulong First { get; set; }
// Go: SimpleState.Last — last sequence number matching the filter
/// <summary>Last sequence that matches the filter.</summary>
public ulong Last { get; set; }
}
@@ -20,9 +20,13 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
private string? _remoteCluster;
private Task? _loopTask;
/// <summary>Remote server identifier learned from LEAF handshake.</summary>
public string? RemoteId { get; internal set; }
/// <summary>Remote endpoint string for diagnostics and monitoring.</summary>
public string RemoteEndpoint => socket.RemoteEndPoint?.ToString() ?? Guid.NewGuid().ToString("N");
/// <summary>Callback invoked when remote LS+/LS- interest updates are received.</summary>
public Func<RemoteSubscription, Task>? RemoteSubscriptionReceived { get; set; }
/// <summary>Callback invoked when remote LMSG payloads are received.</summary>
public Func<LeafMessage, Task>? MessageReceived { get; set; }
/// <summary>
@@ -97,6 +101,8 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
/// permissions as synced. Passing null for either list clears that list.
/// Go reference: leafnode.go — sendPermsAndAccountInfo.
/// </summary>
/// <param name="publishAllow">Subjects this leaf is allowed to publish to.</param>
/// <param name="subscribeAllow">Subjects this leaf is allowed to subscribe to.</param>
public void SetPermissions(IEnumerable<string>? publishAllow, IEnumerable<string>? subscribeAllow)
{
AllowedPublishSubjects.Clear();
@@ -111,6 +117,11 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
PermsSynced = true;
}
/// <summary>
/// Performs the outbound LEAF handshake for a solicited connection.
/// </summary>
/// <param name="serverId">Local server identifier to advertise.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformOutboundHandshakeAsync(string serverId, CancellationToken ct)
{
var handshakeLine = BuildHandshakeLine(serverId);
@@ -119,6 +130,11 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
ParseHandshakeResponse(line);
}
/// <summary>
/// Performs the inbound LEAF handshake for an accepted connection.
/// </summary>
/// <param name="serverId">Local server identifier to advertise.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformInboundHandshakeAsync(string serverId, CancellationToken ct)
{
var line = await ReadLineAsync(ct);
@@ -127,6 +143,10 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync(handshakeLine, ct);
}
/// <summary>
/// Starts the background read loop for this leaf connection.
/// </summary>
/// <param name="ct">Cancellation token controlling the loop lifetime.</param>
public void StartLoop(CancellationToken ct)
{
if (_loopTask != null)
@@ -136,12 +156,32 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
_loopTask = Task.Run(() => ReadLoopAsync(linked.Token), linked.Token);
}
/// <summary>
/// Waits until the leaf read loop exits.
/// </summary>
/// <param name="ct">Cancellation token used while waiting.</param>
/// <returns>A task that completes when the loop is closed.</returns>
public Task WaitUntilClosedAsync(CancellationToken ct)
=> _loopTask?.WaitAsync(ct) ?? Task.CompletedTask;
/// <summary>
/// Sends LS+ interest for a subject, optionally with queue group.
/// </summary>
/// <param name="account">Account for the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group name.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendLsPlusAsync(string account, string subject, string? queue, CancellationToken ct)
=> SendLsPlusAsync(account, subject, queue, queueWeight: 0, ct);
/// <summary>
/// Sends LS+ interest for a subject with optional queue group and weight.
/// </summary>
/// <param name="account">Account for the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group name.</param>
/// <param name="queueWeight">Queue weight to advertise when queue is present.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendLsPlusAsync(string account, string subject, string? queue, int queueWeight, CancellationToken ct)
{
string frame;
@@ -155,6 +195,13 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
return WriteLineAsync(frame, ct);
}
/// <summary>
/// Sends LS- interest removal for a subject.
/// </summary>
/// <param name="account">Account for the interest update.</param>
/// <param name="subject">Subject being removed.</param>
/// <param name="queue">Optional queue group name.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendLsMinusAsync(string account, string subject, string? queue, CancellationToken ct)
=> WriteLineAsync(queue is { Length: > 0 } ? $"LS- {account} {subject} {queue}" : $"LS- {account} {subject}", ct);
@@ -162,6 +209,8 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
/// Sends a CONNECT protocol line with JSON payload for solicited leaf links.
/// Go reference: leafnode.go sendLeafConnect.
/// </summary>
/// <param name="connectInfo">Leaf CONNECT payload to serialize.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public Task SendLeafConnectAsync(LeafConnectInfo connectInfo, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(connectInfo);
@@ -169,6 +218,14 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
return WriteLineAsync($"CONNECT {json}", ct);
}
/// <summary>
/// Sends an LMSG frame to the remote leaf connection.
/// </summary>
/// <param name="account">Account associated with the message.</param>
/// <param name="subject">Subject to deliver.</param>
/// <param name="replyTo">Optional reply subject.</param>
/// <param name="payload">Payload bytes.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendMessageAsync(string account, string subject, string? replyTo, ReadOnlyMemory<byte> payload, CancellationToken ct)
{
var reply = string.IsNullOrEmpty(replyTo) ? "-" : replyTo;
@@ -188,6 +245,9 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
}
}
/// <summary>
/// Disposes this leaf connection and stops background processing.
/// </summary>
public async ValueTask DisposeAsync()
{
await _closedCts.CancelAsync();
@@ -206,16 +266,22 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
return $"LEAF {serverId}";
}
/// <summary>Indicates whether this is a solicited leaf connection.</summary>
public bool IsSolicitedLeafNode() => IsSolicited;
/// <summary>Indicates whether this leaf is operating in spoke mode.</summary>
public bool IsSpokeLeafNode() => IsSpoke;
/// <summary>Indicates whether this leaf is operating in hub mode.</summary>
public bool IsHubLeafNode() => !IsSpoke;
/// <summary>Indicates whether this leaf is isolated from hub propagation.</summary>
public bool IsIsolatedLeafNode() => Isolated;
/// <summary>Returns the remote cluster name if advertised by the peer.</summary>
public string? RemoteCluster() => _remoteCluster;
/// <summary>
/// Applies connect delay only when this is a solicited leaf connection.
/// Go reference: leafnode.go setLeafConnectDelayIfSoliciting.
/// </summary>
/// <param name="delay">Reconnect delay to apply.</param>
public void SetLeafConnectDelayIfSoliciting(TimeSpan delay)
{
if (IsSolicited)
@@ -226,6 +292,7 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
/// Handles remote ERR protocol for leaf links and applies reconnect delay hints.
/// Go reference: leafnode.go leafProcessErr.
/// </summary>
/// <param name="errStr">Error text received from the remote leaf peer.</param>
public void LeafProcessErr(string errStr)
{
if (string.IsNullOrWhiteSpace(errStr))
@@ -254,12 +321,15 @@ public sealed class LeafConnection(Socket socket) : IAsyncDisposable
/// Handles subscription permission violations.
/// Go reference: leafnode.go leafSubPermViolation.
/// </summary>
/// <param name="subj">Subject that triggered the violation.</param>
public void LeafSubPermViolation(string subj) => LeafPermViolation(pub: false, subj);
/// <summary>
/// Handles publish/subscribe permission violations.
/// Go reference: leafnode.go leafPermViolation.
/// </summary>
/// <param name="pub"><see langword="true"/> for publish violations, otherwise subscribe.</param>
/// <param name="subj">Subject that triggered the violation.</param>
public void LeafPermViolation(bool pub, string subj)
=> SetLeafConnectDelayIfSoliciting(LeafNodeManager.LeafNodeReconnectAfterPermViolation);
@@ -26,8 +26,11 @@ public sealed class WebSocketStreamAdapter : Stream
}
// Stream capability overrides
/// <inheritdoc />
public override bool CanRead => true;
/// <inheritdoc />
public override bool CanWrite => true;
/// <inheritdoc />
public override bool CanSeek => false;
// Telemetry properties
@@ -37,12 +40,7 @@ public sealed class WebSocketStreamAdapter : Stream
public int MessagesRead { get; private set; }
public int MessagesWritten { get; private set; }
/// <summary>
/// Reads data from the WebSocket into <paramref name="buffer"/>.
/// If the internal read buffer has buffered data from a previous message,
/// that is served first. Otherwise a new WebSocket message is received.
/// Go reference: client.go wsRead.
/// </summary>
/// <inheritdoc />
public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken ct)
{
ObjectDisposedException.ThrowIf(_disposed, this);
@@ -160,10 +158,7 @@ public sealed class WebSocketStreamAdapter : Stream
}
}
/// <summary>
/// Sends <paramref name="buffer"/> as a single binary WebSocket message.
/// Go reference: client.go wsWrite.
/// </summary>
/// <inheritdoc />
public override async Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken ct)
{
ObjectDisposedException.ThrowIf(_disposed, this);
@@ -193,7 +188,9 @@ public sealed class WebSocketStreamAdapter : Stream
public override Task FlushAsync(CancellationToken ct) => Task.CompletedTask;
// Not-supported synchronous and seeking members
/// <inheritdoc />
public override long Length => throw new NotSupportedException();
/// <inheritdoc />
public override long Position
{
get => throw new NotSupportedException();
+32
View File
@@ -47,11 +47,15 @@ public sealed class MqttConnection : IAsyncDisposable
/// </summary>
public MqttNatsClientAdapter? Adapter { get; private set; }
/// <summary>MQTT client identifier currently bound to this connection.</summary>
public string ClientId => _clientId;
/// <summary>
/// Creates a connection from a TcpClient (standard accept path).
/// </summary>
/// <param name="client">Accepted TCP client transport.</param>
/// <param name="listener">Owning MQTT listener and session coordinator.</param>
/// <param name="useBinaryProtocol">Whether to run MQTT binary protocol mode.</param>
public MqttConnection(TcpClient client, MqttListener listener, bool useBinaryProtocol = true)
{
_tcpClient = client;
@@ -64,6 +68,9 @@ public sealed class MqttConnection : IAsyncDisposable
/// <summary>
/// Creates a connection from an arbitrary Stream (for TLS wrapping or testing).
/// </summary>
/// <param name="stream">Input/output transport stream for this connection.</param>
/// <param name="listener">Owning MQTT listener and session coordinator.</param>
/// <param name="useBinaryProtocol">Whether to run MQTT binary protocol mode.</param>
public MqttConnection(Stream stream, MqttListener listener, bool useBinaryProtocol = true)
{
_stream = stream;
@@ -76,6 +83,10 @@ public sealed class MqttConnection : IAsyncDisposable
/// Creates a connection from a Stream with a TLS client certificate.
/// Used by the accept loop after TLS handshake completes.
/// </summary>
/// <param name="stream">Input/output transport stream for this connection.</param>
/// <param name="listener">Owning MQTT listener and session coordinator.</param>
/// <param name="useBinaryProtocol">Whether to run MQTT binary protocol mode.</param>
/// <param name="clientCert">Peer certificate captured during TLS handshake.</param>
public MqttConnection(Stream stream, MqttListener listener, bool useBinaryProtocol, X509Certificate2? clientCert)
{
_stream = stream;
@@ -85,6 +96,10 @@ public sealed class MqttConnection : IAsyncDisposable
_isPlainSocket = false; // TLS-wrapped stream
}
/// <summary>
/// Runs the MQTT connection loop until cancellation or disconnect.
/// </summary>
/// <param name="ct">Cancellation token controlling connection lifetime.</param>
public async Task RunAsync(CancellationToken ct)
{
if (_useBinaryProtocol)
@@ -492,6 +507,12 @@ public sealed class MqttConnection : IAsyncDisposable
/// <summary>
/// Sends a binary MQTT PUBLISH packet to this connection (for message delivery).
/// </summary>
/// <param name="topic">Topic to publish to the client.</param>
/// <param name="payload">Message payload bytes.</param>
/// <param name="qos">MQTT QoS level for delivery.</param>
/// <param name="retain">Whether the retained flag should be set on the packet.</param>
/// <param name="packetId">Packet identifier for QoS flows that require acknowledgements.</param>
/// <param name="ct">Cancellation token for the send operation.</param>
public async Task SendBinaryPublishAsync(string topic, ReadOnlyMemory<byte> payload, byte qos,
bool retain, ushort packetId, CancellationToken ct)
{
@@ -503,6 +524,9 @@ public sealed class MqttConnection : IAsyncDisposable
/// Sends a message to the connection. Used by the listener for fan-out delivery.
/// In binary mode, sends a PUBLISH packet; in text mode, sends a text line.
/// </summary>
/// <param name="topic">Destination topic.</param>
/// <param name="payload">UTF-8 payload text.</param>
/// <param name="ct">Cancellation token for the send operation.</param>
public Task SendMessageAsync(string topic, string payload, CancellationToken ct)
{
if (_useBinaryProtocol)
@@ -519,6 +543,11 @@ public sealed class MqttConnection : IAsyncDisposable
/// Zero-allocation hot path — formats the packet directly into the buffer.
/// Called synchronously from the NATS delivery path (DeliverMessage).
/// </summary>
/// <param name="topicUtf8">Destination topic encoded as UTF-8 bytes.</param>
/// <param name="payload">Message payload bytes.</param>
/// <param name="qos">MQTT QoS level for the packet.</param>
/// <param name="retain">Whether to set the retain flag.</param>
/// <param name="packetId">Packet identifier used for QoS handshakes.</param>
public void EnqueuePublishNoFlush(ReadOnlySpan<byte> topicUtf8, ReadOnlyMemory<byte> payload,
byte qos = 0, bool retain = false, ushort packetId = 0)
{
@@ -606,6 +635,9 @@ public sealed class MqttConnection : IAsyncDisposable
catch (ObjectDisposedException) { }
}
/// <summary>
/// Disposes connection resources and unregisters listener/adapter state.
/// </summary>
public async ValueTask DisposeAsync()
{
// Clean up adapter subscriptions and unregister from listener
@@ -25,6 +25,11 @@ public sealed class MqttConsumerManager
private readonly ConsumerManager _consumerManager;
private readonly ConcurrentDictionary<string, MqttConsumerBinding> _bindings = new(StringComparer.Ordinal);
/// <summary>
/// Creates an MQTT consumer manager backed by JetStream stream and consumer managers.
/// </summary>
/// <param name="streamManager">Stream manager used to resolve MQTT backing streams.</param>
/// <param name="consumerManager">Consumer manager used to create and delete durable consumers.</param>
public MqttConsumerManager(StreamManager streamManager, ConsumerManager consumerManager)
{
_streamManager = streamManager;
@@ -37,6 +42,11 @@ public sealed class MqttConsumerManager
/// Returns the binding, or null if creation failed.
/// Go reference: server/mqtt.go mqttProcessSub consumer creation.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="natsSubject">NATS subject mapped from the MQTT topic filter.</param>
/// <param name="qos">Requested MQTT QoS level.</param>
/// <param name="maxAckPending">Maximum number of unacknowledged deliveries.</param>
/// <returns>Created consumer binding, or <see langword="null"/> on failure.</returns>
public MqttConsumerBinding? CreateSubscriptionConsumer(string clientId, string natsSubject, int qos, int maxAckPending)
{
var durableName = $"$MQTT_{clientId}_{natsSubject.Replace('.', '_').Replace('*', 'W').Replace('>', 'G')}";
@@ -65,6 +75,8 @@ public sealed class MqttConsumerManager
/// Removes the JetStream consumer for an MQTT subscription.
/// Called on UNSUBSCRIBE or clean session disconnect.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="natsSubject">NATS subject mapped from the MQTT topic filter.</param>
public void RemoveSubscriptionConsumer(string clientId, string natsSubject)
{
var key = $"{clientId}:{natsSubject}";
@@ -77,6 +89,7 @@ public sealed class MqttConsumerManager
/// <summary>
/// Removes all consumers for a client. Called on clean session disconnect.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
public void RemoveAllConsumers(string clientId)
{
var prefix = $"{clientId}:";
@@ -93,6 +106,9 @@ public sealed class MqttConsumerManager
/// <summary>
/// Gets the binding for a subscription, or null if none exists.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="natsSubject">NATS subject mapped from the MQTT topic filter.</param>
/// <returns>Consumer binding for the subscription, or <see langword="null"/>.</returns>
public MqttConsumerBinding? GetBinding(string clientId, string natsSubject)
{
return _bindings.TryGetValue($"{clientId}:{natsSubject}", out var binding) ? binding : null;
@@ -101,6 +117,8 @@ public sealed class MqttConsumerManager
/// <summary>
/// Gets all bindings for a client (for session persistence).
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <returns>Per-subscription bindings keyed by NATS subject.</returns>
public IReadOnlyDictionary<string, MqttConsumerBinding> GetClientBindings(string clientId)
{
var prefix = $"{clientId}:";
@@ -113,6 +131,9 @@ public sealed class MqttConsumerManager
/// Publishes a message to the $MQTT_msgs stream for QoS delivery.
/// Returns the sequence number, or 0 if publish failed.
/// </summary>
/// <param name="natsSubject">NATS subject mapped from MQTT topic.</param>
/// <param name="payload">Message payload bytes.</param>
/// <returns>Stored sequence number, or <c>0</c> if publish failed.</returns>
public ulong PublishToStream(string natsSubject, ReadOnlyMemory<byte> payload)
{
var subject = $"{MqttProtocolConstants.StreamSubjectPrefix}{natsSubject}";
@@ -129,6 +150,8 @@ public sealed class MqttConsumerManager
/// Acknowledges a message in the stream by removing it (for interest-based retention).
/// Called when PUBACK is received for QoS 1.
/// </summary>
/// <param name="sequence">Stream sequence to acknowledge and remove.</param>
/// <returns><see langword="true"/> when the sequence was removed.</returns>
public bool AcknowledgeMessage(ulong sequence)
{
if (_streamManager.TryGet(MqttProtocolConstants.StreamName, out var handle))
@@ -142,6 +165,9 @@ public sealed class MqttConsumerManager
/// <summary>
/// Loads a message from the $MQTT_msgs stream by sequence.
/// </summary>
/// <param name="sequence">Sequence to load.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Stored message, or <see langword="null"/> if not found.</returns>
public async ValueTask<StoredMessage?> LoadMessageAsync(ulong sequence, CancellationToken ct = default)
{
if (_streamManager.TryGet(MqttProtocolConstants.StreamName, out var handle))
@@ -156,6 +182,10 @@ public sealed class MqttConsumerManager
/// Stores a QoS 2 incoming message for deduplication.
/// Returns the sequence number, or 0 if failed.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="packetId">MQTT packet identifier for QoS 2 flow.</param>
/// <param name="payload">Incoming payload bytes.</param>
/// <returns>Stored sequence number, or <c>0</c> if store failed.</returns>
public ulong StoreQoS2Incoming(string clientId, ushort packetId, ReadOnlyMemory<byte> payload)
{
var subject = $"{MqttProtocolConstants.QoS2IncomingMsgsStreamSubjectPrefix}{clientId}.{packetId}";
@@ -170,6 +200,10 @@ public sealed class MqttConsumerManager
/// <summary>
/// Loads a QoS 2 incoming message for delivery on PUBREL.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="packetId">MQTT packet identifier for QoS 2 flow.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Stored QoS 2 message, or <see langword="null"/> when missing.</returns>
public async ValueTask<StoredMessage?> LoadQoS2IncomingAsync(string clientId, ushort packetId, CancellationToken ct = default)
{
var subject = $"{MqttProtocolConstants.QoS2IncomingMsgsStreamSubjectPrefix}{clientId}.{packetId}";
@@ -184,6 +218,10 @@ public sealed class MqttConsumerManager
/// <summary>
/// Removes a QoS 2 incoming message after PUBCOMP.
/// </summary>
/// <param name="clientId">MQTT client identifier.</param>
/// <param name="packetId">MQTT packet identifier for QoS 2 flow.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns><see langword="true"/> when a stored QoS 2 message was removed.</returns>
public async ValueTask<bool> RemoveQoS2IncomingAsync(string clientId, ushort packetId, CancellationToken ct = default)
{
var subject = $"{MqttProtocolConstants.QoS2IncomingMsgsStreamSubjectPrefix}{clientId}.{packetId}";
@@ -12,6 +12,10 @@ public sealed class MqttFlowController : IDisposable
private readonly ConcurrentDictionary<string, SubscriptionFlowState> _subscriptions = new(StringComparer.Ordinal);
private int _defaultMaxAckPending;
/// <summary>
/// Initializes MQTT flow control with the default per-subscription outstanding ack limit.
/// </summary>
/// <param name="defaultMaxAckPending">Default max number of in-flight QoS 1/2 publishes per subscription.</param>
public MqttFlowController(int defaultMaxAckPending = 1024)
{
_defaultMaxAckPending = defaultMaxAckPending;
@@ -24,6 +28,8 @@ public sealed class MqttFlowController : IDisposable
/// Tries to acquire a slot for sending a QoS message on the given subscription.
/// Returns true if a slot was acquired, false if the limit would be exceeded.
/// </summary>
/// <param name="subscriptionId">Subscription identifier used for per-subscription flow tracking.</param>
/// <param name="ct">Cancellation token for the semaphore wait operation.</param>
public async ValueTask<bool> TryAcquireAsync(string subscriptionId, CancellationToken ct = default)
{
var state = GetOrCreate(subscriptionId);
@@ -33,6 +39,8 @@ public sealed class MqttFlowController : IDisposable
/// <summary>
/// Waits for a slot to become available. Blocks until one is released or cancelled.
/// </summary>
/// <param name="subscriptionId">Subscription identifier used for per-subscription flow tracking.</param>
/// <param name="ct">Cancellation token for the semaphore wait operation.</param>
public async ValueTask AcquireAsync(string subscriptionId, CancellationToken ct = default)
{
var state = GetOrCreate(subscriptionId);
@@ -43,6 +51,7 @@ public sealed class MqttFlowController : IDisposable
/// Releases a slot after receiving PUBACK/PUBCOMP.
/// If the semaphore is already at max (duplicate or spurious ack), the release is a no-op.
/// </summary>
/// <param name="subscriptionId">Subscription whose pending count should be decremented.</param>
public void Release(string subscriptionId)
{
if (_subscriptions.TryGetValue(subscriptionId, out var state))
@@ -57,6 +66,7 @@ public sealed class MqttFlowController : IDisposable
/// <summary>
/// Returns the current pending count for a subscription.
/// </summary>
/// <param name="subscriptionId">Subscription identifier to inspect.</param>
public int GetPendingCount(string subscriptionId)
{
if (!_subscriptions.TryGetValue(subscriptionId, out var state))
@@ -67,6 +77,7 @@ public sealed class MqttFlowController : IDisposable
/// <summary>
/// Updates the MaxAckPending limit (e.g., on config reload).
/// </summary>
/// <param name="newLimit">New default in-flight limit for subscriptions created after the update.</param>
public void UpdateLimit(int newLimit)
{
_defaultMaxAckPending = newLimit;
@@ -77,6 +88,7 @@ public sealed class MqttFlowController : IDisposable
/// Used to pause JetStream consumer delivery when the limit is reached.
/// Go reference: server/mqtt.go mqttMaxAckPending flow control.
/// </summary>
/// <param name="subscriptionId">Subscription identifier to evaluate.</param>
public bool IsAtCapacity(string subscriptionId)
{
if (!_subscriptions.TryGetValue(subscriptionId, out var state))
@@ -87,6 +99,7 @@ public sealed class MqttFlowController : IDisposable
/// <summary>
/// Removes tracking for a subscription.
/// </summary>
/// <param name="subscriptionId">Subscription identifier to remove from flow-control tracking.</param>
public void RemoveSubscription(string subscriptionId)
{
if (_subscriptions.TryRemove(subscriptionId, out var state))
@@ -96,6 +109,9 @@ public sealed class MqttFlowController : IDisposable
/// <summary>Number of tracked subscriptions.</summary>
public int SubscriptionCount => _subscriptions.Count;
/// <summary>
/// Disposes all semaphore resources tracked for MQTT subscriptions.
/// </summary>
public void Dispose()
{
foreach (var kvp in _subscriptions)
@@ -114,7 +130,9 @@ public sealed class MqttFlowController : IDisposable
private sealed class SubscriptionFlowState
{
/// <summary>Configured maximum pending QoS acknowledgements for this subscription.</summary>
public int MaxAckPending { get; init; }
/// <summary>Semaphore that enforces pending message capacity for this subscription.</summary>
public required SemaphoreSlim Semaphore { get; init; }
}
}
@@ -18,14 +18,25 @@ public sealed class MqttNatsClientAdapter : INatsClient
private readonly MqttConnection _connection;
private readonly Dictionary<string, Subscription> _subs = new(StringComparer.Ordinal);
/// <summary>Server-assigned client identifier for routing/monitoring.</summary>
public ulong Id { get; }
/// <summary>Client kind exposed to the NATS routing layer.</summary>
public ClientKind Kind => ClientKind.Client;
/// <summary>Account currently associated with this MQTT client.</summary>
public Account? Account { get; set; }
/// <summary>CONNECT options are not exposed for MQTT adapter clients.</summary>
public ClientOptions? ClientOpts => null;
/// <summary>Resolved permissions for this adapter client.</summary>
public ClientPermissions? Permissions { get; set; }
/// <summary>MQTT client identifier from the underlying connection.</summary>
public string MqttClientId => _connection.ClientId;
/// <summary>
/// Creates an adapter that exposes an MQTT connection as an <see cref="INatsClient"/>.
/// </summary>
/// <param name="connection">Underlying MQTT connection.</param>
/// <param name="id">Server-assigned adapter/client id.</param>
public MqttNatsClientAdapter(MqttConnection connection, ulong id)
{
_connection = connection;
@@ -36,6 +47,11 @@ public sealed class MqttNatsClientAdapter : INatsClient
/// Delivers a NATS message to this MQTT client by translating the NATS subject
/// to an MQTT topic and enqueueing a PUBLISH packet into the direct buffer.
/// </summary>
/// <param name="subject">NATS subject being delivered.</param>
/// <param name="sid">Subscription id on this client.</param>
/// <param name="replyTo">Optional reply subject.</param>
/// <param name="headers">Encoded NATS headers.</param>
/// <param name="payload">Message payload bytes.</param>
public void SendMessage(string subject, string sid, string? replyTo,
ReadOnlyMemory<byte> headers, ReadOnlyMemory<byte> payload)
{
@@ -47,6 +63,11 @@ public sealed class MqttNatsClientAdapter : INatsClient
/// Enqueues an MQTT PUBLISH into the connection's direct buffer without flushing.
/// Uses cached topic bytes to avoid re-encoding. Zero allocation on the hot path.
/// </summary>
/// <param name="subject">NATS subject being delivered.</param>
/// <param name="sid">Subscription id on this client.</param>
/// <param name="replyTo">Optional reply subject.</param>
/// <param name="headers">Encoded NATS headers.</param>
/// <param name="payload">Message payload bytes.</param>
public void SendMessageNoFlush(string subject, string sid, string? replyTo,
ReadOnlyMemory<byte> headers, ReadOnlyMemory<byte> payload)
{
@@ -62,12 +83,21 @@ public sealed class MqttNatsClientAdapter : INatsClient
_connection.SignalMqttFlush();
}
/// <summary>
/// Queues raw outbound bytes. No-op for MQTT adapter clients.
/// </summary>
/// <param name="data">Raw protocol bytes.</param>
/// <returns>Always <see langword="true"/>.</returns>
public bool QueueOutbound(ReadOnlyMemory<byte> data)
{
// No-op for MQTT — binary framing, not raw NATS protocol bytes
return true;
}
/// <summary>
/// Removes a subscription by id and unregisters it from the account sublist.
/// </summary>
/// <param name="sid">Subscription id to remove.</param>
public void RemoveSubscription(string sid)
{
if (_subs.Remove(sid, out var sub))
@@ -81,6 +111,10 @@ public sealed class MqttNatsClientAdapter : INatsClient
/// Creates a NATS subscription for an MQTT topic filter and inserts it into
/// the account's SubList so NATS messages are delivered to this MQTT client.
/// </summary>
/// <param name="natsSubject">Mapped NATS subject to subscribe to.</param>
/// <param name="sid">Subscription id for tracking/removal.</param>
/// <param name="queue">Optional queue group.</param>
/// <returns>The created subscription instance.</returns>
public Subscription AddSubscription(string natsSubject, string sid, string? queue = null)
{
// Pre-warm topic bytes cache for this subject to avoid cache miss on first message.
@@ -114,5 +148,6 @@ public sealed class MqttNatsClientAdapter : INatsClient
_subs.Clear();
}
/// <summary>Current subscriptions keyed by subscription id.</summary>
public IReadOnlyDictionary<string, Subscription> Subscriptions => _subs;
}
+25
View File
@@ -6,8 +6,11 @@ namespace NATS.Server.Mqtt;
/// </summary>
public sealed class MqttJsa
{
/// <summary>Account that owns the MQTT JetStream operations.</summary>
public string AccountName { get; set; } = string.Empty;
/// <summary>Reply subject prefix used for MQTT JetStream API calls.</summary>
public string ReplyPrefix { get; set; } = string.Empty;
/// <summary>Optional JetStream domain for cross-domain routing.</summary>
public string? Domain { get; set; }
}
@@ -17,8 +20,11 @@ public sealed class MqttJsa
/// </summary>
public sealed class MqttJsPubMsg
{
/// <summary>Target NATS subject for the publish operation.</summary>
public string Subject { get; set; } = string.Empty;
/// <summary>Published payload bytes.</summary>
public byte[] Payload { get; set; } = [];
/// <summary>Optional reply subject for request/reply semantics.</summary>
public string? ReplyTo { get; set; }
}
@@ -28,7 +34,9 @@ public sealed class MqttJsPubMsg
/// </summary>
public sealed class MqttRetMsgDel
{
/// <summary>MQTT topic whose retained message should be removed.</summary>
public string Topic { get; set; } = string.Empty;
/// <summary>JetStream sequence of the retained message record.</summary>
public ulong Sequence { get; set; }
}
@@ -38,8 +46,11 @@ public sealed class MqttRetMsgDel
/// </summary>
public sealed class MqttPersistedSession
{
/// <summary>MQTT client identifier for the persisted session.</summary>
public string ClientId { get; set; } = string.Empty;
/// <summary>Last issued packet identifier for this session.</summary>
public int LastPacketId { get; set; }
/// <summary>Maximum number of unacknowledged QoS deliveries allowed.</summary>
public int MaxAckPending { get; set; }
}
@@ -49,7 +60,9 @@ public sealed class MqttPersistedSession
/// </summary>
public sealed class MqttRetainedMessageRef
{
/// <summary>JetStream sequence containing the retained MQTT payload.</summary>
public ulong StreamSequence { get; set; }
/// <summary>NATS subject mapped from the retained MQTT topic.</summary>
public string Subject { get; set; } = string.Empty;
}
@@ -59,10 +72,15 @@ public sealed class MqttRetainedMessageRef
/// </summary>
public sealed class MqttSub
{
/// <summary>MQTT topic filter for this subscription.</summary>
public string Filter { get; set; } = string.Empty;
/// <summary>Requested MQTT QoS level.</summary>
public byte Qos { get; set; }
/// <summary>Optional JetStream durable consumer name.</summary>
public string? JsDur { get; set; }
/// <summary>Indicates whether this is a permanent subscription.</summary>
public bool Prm { get; set; }
/// <summary>Reserved flag kept for Go protocol parity.</summary>
public bool Reserved { get; set; }
}
@@ -72,8 +90,11 @@ public sealed class MqttSub
/// </summary>
public sealed class MqttFilter
{
/// <summary>Original MQTT topic filter.</summary>
public string Filter { get; set; } = string.Empty;
/// <summary>QoS level attached to the filter.</summary>
public byte Qos { get; set; }
/// <summary>Parsed token optimization hint used for dispatch lookups.</summary>
public string? TopicToken { get; set; }
}
@@ -83,8 +104,12 @@ public sealed class MqttFilter
/// </summary>
public sealed class MqttParsedPublishNatsHeader
{
/// <summary>Subject extracted from MQTT publish headers, when present.</summary>
public string? Subject { get; set; }
/// <summary>Mapped subject after account/topic translation.</summary>
public string? Mapped { get; set; }
/// <summary>Indicates the packet represents a PUBLISH flow.</summary>
public bool IsPublish { get; set; }
/// <summary>Indicates the packet represents a PUBREL flow.</summary>
public bool IsPubRel { get; set; }
}
+37
View File
@@ -53,6 +53,8 @@ public sealed class MqttRetainedStore
/// An empty payload clears the retained message.
/// Go reference: server/mqtt.go mqttHandleRetainedMsg.
/// </summary>
/// <param name="topic">MQTT topic for the retained message.</param>
/// <param name="payload">Retained payload bytes; empty clears retained state.</param>
public void SetRetained(string topic, ReadOnlyMemory<byte> payload)
{
if (payload.IsEmpty)
@@ -69,6 +71,8 @@ public sealed class MqttRetainedStore
/// <summary>
/// Gets the retained message payload for a topic, or null if none.
/// </summary>
/// <param name="topic">MQTT topic to query.</param>
/// <returns>Retained payload, or <see langword="null"/> if none exists.</returns>
public ReadOnlyMemory<byte>? GetRetained(string topic)
{
if (_retained.TryGetValue(topic, out var payload))
@@ -82,6 +86,8 @@ public sealed class MqttRetainedStore
/// Supports '+' (single-level) and '#' (multi-level) wildcards.
/// Go reference: server/mqtt.go mqttGetRetainedMessages ~line 1650.
/// </summary>
/// <param name="filter">MQTT topic filter.</param>
/// <returns>Matching retained messages.</returns>
public IReadOnlyList<MqttRetainedMessage> GetMatchingRetained(string filter)
{
var results = new List<MqttRetainedMessage>();
@@ -100,6 +106,9 @@ public sealed class MqttRetainedStore
/// Returns the number of messages delivered.
/// Go reference: server/mqtt.go mqttGetRetainedMessages / mqttHandleRetainedMsg ~line 1650.
/// </summary>
/// <param name="topicFilter">MQTT topic filter for retained delivery.</param>
/// <param name="deliver">Callback invoked for each matching retained message.</param>
/// <returns>Number of retained messages delivered.</returns>
public int DeliverRetainedOnSubscribe(string topicFilter, Action<string, byte[], byte, bool> deliver)
{
var matches = GetMatchingRetained(topicFilter);
@@ -114,6 +123,9 @@ public sealed class MqttRetainedStore
/// Empty payload = tombstone (delete retained).
/// Go reference: server/mqtt.go mqttHandleRetainedMsg with JetStream.
/// </summary>
/// <param name="topic">MQTT topic for the retained message.</param>
/// <param name="payload">Retained payload bytes; empty clears retained state.</param>
/// <param name="ct">Cancellation token.</param>
public async Task SetRetainedAsync(string topic, ReadOnlyMemory<byte> payload, CancellationToken ct = default)
{
SetRetained(topic, payload);
@@ -138,6 +150,9 @@ public sealed class MqttRetainedStore
/// Gets the retained message, checking backing store if not in memory.
/// Returns null if the topic was explicitly cleared in this session.
/// </summary>
/// <param name="topic">MQTT topic to query.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Retained payload bytes, or <see langword="null"/> if none exists.</returns>
public async Task<byte[]?> GetRetainedAsync(string topic, CancellationToken ct = default)
{
var mem = GetRetained(topic);
@@ -163,6 +178,9 @@ public sealed class MqttRetainedStore
/// Matches an MQTT topic against a filter pattern.
/// '+' matches exactly one level, '#' matches zero or more levels (must be last).
/// </summary>
/// <param name="topic">Concrete MQTT topic.</param>
/// <param name="filter">MQTT filter pattern.</param>
/// <returns><see langword="true"/> when the topic matches the filter.</returns>
internal static bool MqttTopicMatch(string topic, string filter)
{
var topicLevels = topic.Split('/');
@@ -209,7 +227,9 @@ public enum MqttQos2State
/// </summary>
internal sealed class MqttQos2Flow
{
/// <summary>Current QoS 2 handshake state for the packet.</summary>
public MqttQos2State State { get; set; }
/// <summary>UTC timestamp when the flow was created.</summary>
public DateTime StartedAtUtc { get; init; }
}
@@ -239,6 +259,8 @@ public sealed class MqttQos2StateMachine
/// Begins a new QoS 2 flow for the given packet ID.
/// Returns false if a flow for this packet ID already exists (duplicate publish).
/// </summary>
/// <param name="packetId">MQTT packet identifier for the flow.</param>
/// <returns><see langword="true"/> when a new flow was created.</returns>
public bool BeginPublish(ushort packetId)
{
var flow = new MqttQos2Flow
@@ -254,6 +276,8 @@ public sealed class MqttQos2StateMachine
/// Processes a PUBREC for the given packet ID.
/// Returns false if the flow is not in the expected state.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the state transition succeeded.</returns>
public bool ProcessPubRec(ushort packetId)
{
if (!_flows.TryGetValue(packetId, out var flow))
@@ -270,6 +294,8 @@ public sealed class MqttQos2StateMachine
/// Processes a PUBREL for the given packet ID.
/// Returns false if the flow is not in the expected state.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the state transition succeeded.</returns>
public bool ProcessPubRel(ushort packetId)
{
if (!_flows.TryGetValue(packetId, out var flow))
@@ -287,6 +313,8 @@ public sealed class MqttQos2StateMachine
/// Returns false if the flow is not in the expected state.
/// Removes the flow on completion.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the flow completed and was removed.</returns>
public bool ProcessPubComp(ushort packetId)
{
if (!_flows.TryGetValue(packetId, out var flow))
@@ -303,6 +331,8 @@ public sealed class MqttQos2StateMachine
/// <summary>
/// Gets the current state for a packet ID, or null if no flow exists.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns>Current state, or <see langword="null"/> if no flow exists.</returns>
public MqttQos2State? GetState(ushort packetId)
{
if (_flows.TryGetValue(packetId, out var flow))
@@ -331,6 +361,7 @@ public sealed class MqttQos2StateMachine
/// <summary>
/// Removes a flow (e.g., after timeout cleanup).
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
public void RemoveFlow(ushort packetId) =>
_flows.TryRemove(packetId, out _);
@@ -339,6 +370,8 @@ public sealed class MqttQos2StateMachine
/// Alias for <see cref="ProcessPubRec"/> — transitions AwaitingPubRec → AwaitingPubRel.
/// Returns false if the flow is not in the expected state.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the state transition succeeded.</returns>
public bool RegisterPubRec(ushort packetId) => ProcessPubRec(packetId);
/// <summary>
@@ -346,6 +379,8 @@ public sealed class MqttQos2StateMachine
/// Alias for <see cref="ProcessPubRel"/> — transitions AwaitingPubRel → AwaitingPubComp.
/// Returns false if the flow is not in the expected state.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the state transition succeeded.</returns>
public bool RegisterPubRel(ushort packetId) => ProcessPubRel(packetId);
/// <summary>
@@ -353,5 +388,7 @@ public sealed class MqttQos2StateMachine
/// Alias for <see cref="ProcessPubComp"/> — transitions AwaitingPubComp → Complete and removes the flow.
/// Returns false if the flow is not in the expected state.
/// </summary>
/// <param name="packetId">MQTT packet identifier.</param>
/// <returns><see langword="true"/> when the flow completed.</returns>
public bool CompletePubComp(ushort packetId) => ProcessPubComp(packetId);
}
+26
View File
@@ -8,39 +8,65 @@ namespace NATS.Server;
public sealed class MqttOptions
{
// Network
/// <summary>Host interface for the MQTT listener.</summary>
public string Host { get; set; } = "";
/// <summary>Port for the MQTT listener.</summary>
public int Port { get; set; }
// Auth override (MQTT-specific, separate from global auth)
/// <summary>Default user to apply when MQTT clients connect without credentials.</summary>
public string? NoAuthUser { get; set; }
/// <summary>Optional username required for MQTT authentication.</summary>
public string? Username { get; set; }
/// <summary>Optional password required for MQTT authentication.</summary>
public string? Password { get; set; }
/// <summary>Optional bearer token accepted for MQTT authentication.</summary>
public string? Token { get; set; }
/// <summary>Authentication timeout in seconds for MQTT CONNECT processing.</summary>
public double AuthTimeout { get; set; }
// TLS
/// <summary>Path to the server certificate used for MQTT TLS.</summary>
public string? TlsCert { get; set; }
/// <summary>Path to the private key used for MQTT TLS.</summary>
public string? TlsKey { get; set; }
/// <summary>Path to the CA certificate bundle used to validate peer certificates.</summary>
public string? TlsCaCert { get; set; }
/// <summary>Enables client certificate verification for MQTT TLS connections.</summary>
public bool TlsVerify { get; set; }
/// <summary>TLS handshake timeout in seconds for MQTT clients.</summary>
public double TlsTimeout { get; set; } = 2.0;
/// <summary>Enables TLS certificate subject mapping to users.</summary>
public bool TlsMap { get; set; }
/// <summary>Set of pinned client certificate fingerprints allowed for MQTT connections.</summary>
public HashSet<string>? TlsPinnedCerts { get; set; }
// JetStream integration
/// <summary>JetStream domain used by MQTT-backed streams and consumers.</summary>
public string? JsDomain { get; set; }
/// <summary>Replica count for MQTT-created JetStream streams.</summary>
public int StreamReplicas { get; set; }
/// <summary>Replica count for MQTT-created JetStream consumers.</summary>
public int ConsumerReplicas { get; set; }
/// <summary>Stores MQTT JetStream consumer state in memory when enabled.</summary>
public bool ConsumerMemoryStorage { get; set; }
/// <summary>Idle timeout after which inactive MQTT consumers are cleaned up.</summary>
public TimeSpan ConsumerInactiveThreshold { get; set; }
// QoS
/// <summary>Maximum time to wait for QoS acknowledgements before redelivery.</summary>
public TimeSpan AckWait { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>Maximum number of outstanding unacknowledged QoS messages per consumer.</summary>
public ushort MaxAckPending { get; set; }
/// <summary>Timeout for internal JetStream API requests made by MQTT components.</summary>
public TimeSpan JsApiTimeout { get; set; } = TimeSpan.FromSeconds(5);
/// <summary>Enables durable MQTT session persistence across reconnects.</summary>
public bool SessionPersistence { get; set; } = true;
/// <summary>Time-to-live for persisted MQTT session state.</summary>
public TimeSpan SessionTtl { get; set; } = TimeSpan.FromHours(1);
/// <summary>Enables sending PUBACK for QoS 1 publishes.</summary>
public bool Qos1PubAck { get; set; } = true;
/// <summary>Indicates whether MQTT TLS is configured with both certificate and key.</summary>
public bool HasTls => TlsCert != null && TlsKey != null;
}
+1 -3
View File
@@ -340,9 +340,7 @@ public sealed class NatsClient : INatsClient, IDisposable
return ClientConnectionType.Nats;
}
/// <summary>
/// Returns a compact connection identity string for diagnostics.
/// </summary>
/// <inheritdoc />
public override string ToString()
{
var endpoint = RemoteIp is null ? "unknown" : $"{RemoteIp}:{RemotePort}";
+1 -3
View File
@@ -3283,9 +3283,7 @@ public sealed class NatsServer : IMessageRouter, ISubListAccess, IDisposable
_options.SystemAccount = newOpts.SystemAccount;
}
/// <summary>
/// Returns a compact server identity string for diagnostics.
/// </summary>
/// <inheritdoc />
public override string ToString()
=> $"NatsServer(ServerId={ServerId}, Name={ServerName}, Addr={Addr()}, Clients={ClientCount})";
+47
View File
@@ -21,16 +21,30 @@ public enum CommandType
public readonly struct ParsedCommand
{
/// <summary>Parsed command type used by server dispatch.</summary>
public CommandType Type { get; init; }
/// <summary>Original protocol operation token (for example, <c>PUB</c> or <c>SUB</c>).</summary>
public string? Operation { get; init; }
/// <summary>Command subject when the operation carries one.</summary>
public string? Subject { get; init; }
/// <summary>Optional reply subject used for request-reply flows.</summary>
public string? ReplyTo { get; init; }
/// <summary>Queue group name for queue subscriptions.</summary>
public string? Queue { get; init; }
/// <summary>Subscription identifier supplied by the client.</summary>
public string? Sid { get; init; }
/// <summary>Maximum message count for <c>UNSUB</c>; <c>-1</c> when not specified.</summary>
public int MaxMessages { get; init; }
/// <summary>Header byte size for <c>HPUB</c>; <c>-1</c> for non-header payloads.</summary>
public int HeaderSize { get; init; }
/// <summary>Payload bytes associated with this command, when applicable.</summary>
public ReadOnlyMemory<byte> Payload { get; init; }
/// <summary>
/// Creates a command without payload fields for control-line only operations.
/// </summary>
/// <param name="type">Command type to emit.</param>
/// <param name="operation">Operation token used for tracing and diagnostics.</param>
public static ParsedCommand Simple(CommandType type, string operation) =>
new() { Type = type, Operation = operation, MaxMessages = -1 };
}
@@ -39,6 +53,7 @@ public sealed class NatsParser
{
private static ReadOnlySpan<byte> CrLfBytes => "\r\n"u8;
private ILogger? _logger;
/// <summary>Optional protocol logger used to trace inbound operations.</summary>
public ILogger? Logger { set => _logger = value; }
// State for split-packet payload reading
@@ -50,6 +65,11 @@ public sealed class NatsParser
private CommandType _pendingType;
private string _pendingOperation = string.Empty;
/// <summary>
/// Initializes a parser for the NATS text protocol control and payload frames.
/// </summary>
/// <param name="maxPayload">Reserved for payload policy compatibility; max payload is enforced at connection level.</param>
/// <param name="logger">Optional logger for protocol trace output.</param>
public NatsParser(int maxPayload = NatsProtocol.MaxPayloadSize, ILogger? logger = null)
{
_logger = logger;
@@ -65,6 +85,11 @@ public sealed class NatsParser
_logger.LogTrace("<<- {Op} {Arg}", op, Encoding.ASCII.GetString(arg));
}
/// <summary>
/// Attempts to parse one protocol command from the current receive buffer.
/// </summary>
/// <param name="buffer">Receive buffer; advanced past the parsed command on success.</param>
/// <param name="command">Materialized parsed command when parsing succeeds.</param>
public bool TryParse(ref ReadOnlySequence<byte> buffer, out ParsedCommand command)
{
command = default;
@@ -76,6 +101,11 @@ public sealed class NatsParser
return true;
}
/// <summary>
/// Attempts to parse one protocol command and return a non-materialized payload view.
/// </summary>
/// <param name="buffer">Receive buffer; advanced past the parsed command on success.</param>
/// <param name="command">Parsed command view that can be materialized by the caller.</param>
internal bool TryParseView(ref ReadOnlySequence<byte> buffer, out ParsedCommandView command)
{
command = default;
@@ -213,6 +243,12 @@ public sealed class NatsParser
}
// Go reference: parser.go protoSnippet(start, max, buf).
/// <summary>
/// Builds an ASCII-safe snippet used in protocol violation diagnostics.
/// </summary>
/// <param name="start">Start index in the source buffer.</param>
/// <param name="max">Maximum number of bytes to include in the snippet.</param>
/// <param name="buffer">Source protocol bytes.</param>
internal static string ProtoSnippet(int start, int max, ReadOnlySpan<byte> buffer)
{
if (start >= buffer.Length)
@@ -229,6 +265,10 @@ public sealed class NatsParser
return JsonSerializer.Serialize(Encoding.ASCII.GetString(slice));
}
/// <summary>
/// Builds a protocol snippet using the default snippet length.
/// </summary>
/// <param name="buffer">Source protocol bytes.</param>
internal static string ProtoSnippet(ReadOnlySpan<byte> buffer) =>
ProtoSnippet(0, NatsProtocol.ProtoSnippetSize, buffer);
@@ -468,6 +508,7 @@ public sealed class NatsParser
/// <summary>
/// Parse a decimal integer from ASCII bytes. Returns -1 on error.
/// </summary>
/// <param name="data">ASCII digit span containing a non-negative integer.</param>
internal static int ParseSize(Span<byte> data)
{
if (data.Length == 0 || data.Length > 9)
@@ -487,6 +528,8 @@ public sealed class NatsParser
/// Split by spaces/tabs into argument ranges. Returns the number of arguments found.
/// Uses Span&lt;Range&gt; for zero-allocation argument splitting.
/// </summary>
/// <param name="data">Control-line argument bytes.</param>
/// <param name="ranges">Destination range buffer populated with argument spans.</param>
internal static int SplitArgs(Span<byte> data, Span<Range> ranges)
{
int count = 0;
@@ -525,6 +568,10 @@ public sealed class NatsParser
public class ProtocolViolationException : Exception
{
/// <summary>
/// Initializes an exception for malformed or protocol-incompatible client frames.
/// </summary>
/// <param name="message">Human-readable protocol violation details.</param>
public ProtocolViolationException(string message)
: base(message)
{
+28
View File
@@ -10,13 +10,19 @@ namespace NATS.Server.Protocol;
/// </summary>
public sealed class ProxyAddress
{
/// <summary>Source IP address reported by PROXY protocol.</summary>
public required IPAddress SrcIp { get; init; }
/// <summary>Source TCP port reported by PROXY protocol.</summary>
public required ushort SrcPort { get; init; }
/// <summary>Destination IP address reported by PROXY protocol.</summary>
public required IPAddress DstIp { get; init; }
/// <summary>Destination TCP port reported by PROXY protocol.</summary>
public required ushort DstPort { get; init; }
/// <summary>Network family derived from source address (<c>tcp4</c> or <c>tcp6</c>).</summary>
public string Network => SrcIp.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork ? "tcp4" : "tcp6";
/// <inheritdoc />
public override string ToString() =>
SrcIp.AddressFamily == System.Net.Sockets.AddressFamily.InterNetworkV6
? $"[{SrcIp}]:{SrcPort}"
@@ -36,7 +42,9 @@ public enum ProxyParseResultKind
public sealed class ProxyParseResult
{
/// <summary>Result kind indicating proxy or local passthrough.</summary>
public required ProxyParseResultKind Kind { get; init; }
/// <summary>Parsed address payload when <see cref="Kind"/> is <see cref="ProxyParseResultKind.Proxy"/>.</summary>
public ProxyAddress? Address { get; init; }
}
@@ -88,6 +96,8 @@ public static class ProxyProtocolParser
/// entire header (up to the CRLF for v1, or the full fixed+address block for v2).
/// Throws <see cref="ProxyProtocolException"/> for malformed input.
/// </summary>
/// <param name="data">Raw bytes containing a complete PROXY header.</param>
/// <returns>Parsed PROXY result including kind and optional address.</returns>
public static ProxyParseResult Parse(ReadOnlySpan<byte> data)
{
if (data.Length < 6)
@@ -114,6 +124,8 @@ public static class ProxyProtocolParser
/// Expects the "PROXY " prefix (6 bytes) to have already been stripped.
/// Reference: readProxyProtoV1Header (client_proxyproto.go:134)
/// </summary>
/// <param name="afterPrefix">Bytes following the stripped <c>PROXY </c> prefix.</param>
/// <returns>Parsed PROXY result including kind and optional address.</returns>
public static ProxyParseResult ParseV1(ReadOnlySpan<byte> afterPrefix)
{
if (afterPrefix.Length > V1MaxLineLen - 6)
@@ -187,6 +199,8 @@ public static class ProxyProtocolParser
/// Parses a full PROXY protocol v2 binary header including signature.
/// Reference: readProxyProtoV2Header / parseProxyProtoV2Header (client_proxyproto.go:274)
/// </summary>
/// <param name="data">Raw bytes containing full PROXY v2 header.</param>
/// <returns>Parsed PROXY result including kind and optional address.</returns>
public static ProxyParseResult ParseV2(ReadOnlySpan<byte> data)
{
if (data.Length < V2HeaderSize)
@@ -205,6 +219,8 @@ public static class ProxyProtocolParser
/// 12-byte signature, then the variable-length address block.
/// Reference: parseProxyProtoV2Header (client_proxyproto.go:301)
/// </summary>
/// <param name="header">Bytes immediately after v2 signature.</param>
/// <returns>Parsed PROXY result including kind and optional address.</returns>
public static ProxyParseResult ParseV2AfterSig(ReadOnlySpan<byte> header)
{
if (header.Length < 4)
@@ -290,6 +306,12 @@ public static class ProxyProtocolParser
// -------------------------------------------------------------------------
/// <summary>Builds a valid PROXY v2 binary header for the given parameters.</summary>
/// <param name="srcIp">Source IP address.</param>
/// <param name="dstIp">Destination IP address.</param>
/// <param name="srcPort">Source TCP port.</param>
/// <param name="dstPort">Destination TCP port.</param>
/// <param name="isIPv6">Set to <see langword="true"/> to emit IPv6 header family.</param>
/// <returns>Encoded PROXY v2 header bytes.</returns>
public static byte[] BuildV2Header(
string srcIp, string dstIp, ushort srcPort, ushort dstPort, bool isIPv6 = false)
{
@@ -339,6 +361,12 @@ public static class ProxyProtocolParser
}
/// <summary>Builds a PROXY v1 text header.</summary>
/// <param name="protocol">Protocol token (e.g. <c>TCP4</c>, <c>TCP6</c>, <c>UNKNOWN</c>).</param>
/// <param name="srcIp">Source IP address.</param>
/// <param name="dstIp">Destination IP address.</param>
/// <param name="srcPort">Source TCP port.</param>
/// <param name="dstPort">Destination TCP port.</param>
/// <returns>Encoded PROXY v1 header bytes.</returns>
public static byte[] BuildV1Header(
string protocol, string srcIp, string dstIp, ushort srcPort, ushort dstPort)
{
+25
View File
@@ -73,6 +73,11 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go: server/raft.go:2854-2916 (sendAppendEntry / sendAppendEntryLocked)
/// </summary>
/// <param name="leaderId">Leader node id sending append entries.</param>
/// <param name="followerIds">Follower node ids targeted for replication.</param>
/// <param name="entry">Log entry to replicate.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Per-follower append dispatch results.</returns>
public Task<IReadOnlyList<AppendResult>> AppendEntriesAsync(
string leaderId,
IReadOnlyList<string> followerIds,
@@ -115,6 +120,11 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go: server/raft.go:3594-3630 (requestVote / sendVoteRequest)
/// </summary>
/// <param name="candidateId">Candidate node id requesting votes.</param>
/// <param name="voterId">Target voter node id.</param>
/// <param name="request">Vote request payload.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Vote response placeholder for transport dispatch.</returns>
public Task<VoteResponse> RequestVoteAsync(
string candidateId,
string voterId,
@@ -149,6 +159,10 @@ public sealed class NatsRaftTransport : IRaftTransport
/// Go: server/raft.go:3247 (buildSnapshotAppendEntry),
/// raft.go:2168 — raftCatchupReply = "$NRG.CR.%s"
/// </summary>
/// <param name="leaderId">Leader node id sending snapshot.</param>
/// <param name="followerId">Follower node id receiving snapshot.</param>
/// <param name="snapshot">Snapshot payload to install.</param>
/// <param name="ct">Cancellation token.</param>
public Task InstallSnapshotAsync(
string leaderId,
string followerId,
@@ -179,6 +193,7 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go: server/raft.go:949 — ForwardProposal → n.sendq.push to n.psubj
/// </summary>
/// <param name="entry">Serialized proposal bytes.</param>
public void ForwardProposal(ReadOnlyMemory<byte> entry)
{
var proposalSubject = RaftSubjects.Proposal(_groupId);
@@ -192,6 +207,7 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go: server/raft.go:986 — ProposeRemovePeer → n.sendq.push to n.rpsubj
/// </summary>
/// <param name="peer">Peer id to remove from the group.</param>
public void ProposeRemovePeer(string peer)
{
var removePeerSubject = RaftSubjects.RemovePeer(_groupId);
@@ -209,6 +225,10 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go reference: raft.go sendTimeoutNow
/// </summary>
/// <param name="leaderId">Leader node id issuing TimeoutNow.</param>
/// <param name="targetId">Target follower node id.</param>
/// <param name="term">Leader term for the request.</param>
/// <param name="ct">Cancellation token.</param>
public Task SendTimeoutNowAsync(string leaderId, string targetId, ulong term, CancellationToken ct)
{
_ = targetId;
@@ -226,6 +246,11 @@ public sealed class NatsRaftTransport : IRaftTransport
///
/// Go reference: raft.go — leader sends empty AppendEntries to confirm quorum for reads.
/// </summary>
/// <param name="leaderId">Leader node id sending heartbeats.</param>
/// <param name="followerIds">Follower node ids targeted by heartbeats.</param>
/// <param name="term">Leader term used in heartbeat payloads.</param>
/// <param name="onAck">Callback invoked per follower dispatch.</param>
/// <param name="ct">Cancellation token.</param>
public Task SendHeartbeatAsync(
string leaderId,
IReadOnlyList<string> followerIds,
+213
View File
@@ -55,41 +55,110 @@ public sealed class RaftNode : IDisposable
// Pre-vote: Go NATS server does not implement pre-vote (RFC 5849 §9.6). Skipped for parity.
/// <summary>
/// Unique node identifier in this RAFT group.
/// </summary>
public string Id { get; }
/// <summary>
/// Logical RAFT group name used for shared consensus state.
/// </summary>
public string GroupName { get; }
/// <summary>
/// UTC timestamp when this node instance was created.
/// </summary>
public DateTime CreatedUtc => _createdUtc;
/// <summary>
/// Current RAFT term tracked by this node.
/// </summary>
public int Term => TermState.CurrentTerm;
/// <summary>
/// Indicates whether this node is currently leader.
/// </summary>
public bool IsLeader => Role == RaftRole.Leader;
/// <summary>
/// UTC time when this node last became leader.
/// </summary>
public DateTime? LeaderSince => _leaderSinceUtc;
/// <summary>
/// Current leader id for this group, or empty when unknown.
/// </summary>
public string GroupLeader => _groupLeader;
/// <summary>
/// True when no leader is currently known.
/// </summary>
public bool Leaderless => string.IsNullOrEmpty(_groupLeader);
/// <summary>
/// True once any leader has previously been observed.
/// </summary>
public bool HadPreviousLeader => _hadPreviousLeader;
/// <summary>
/// Current RAFT role for this node.
/// </summary>
public RaftRole Role { get; private set; } = RaftRole.Follower;
/// <summary>
/// Indicates observer mode (non-voting) status.
/// </summary>
public bool IsObserver => _observerMode;
/// <summary>
/// Indicates whether this node has been deleted.
/// </summary>
public bool IsDeleted => _isDeleted;
/// <summary>
/// Active member ids in the current configuration.
/// </summary>
public IReadOnlyCollection<string> Members => _members;
/// <summary>
/// Durable term and vote state.
/// </summary>
public RaftTermState TermState { get; } = new();
/// <summary>
/// Highest applied log index.
/// </summary>
public long AppliedIndex { get; set; }
/// <summary>
/// In-memory RAFT log for this node.
/// </summary>
public RaftLog Log { get; private set; } = new();
// B1: Commit tracking
// Go reference: raft.go:150-160 (applied/processed fields), raft.go:2100-2150 (ApplyQ)
/// <summary>
/// Highest committed index acknowledged by quorum.
/// </summary>
public long CommitIndex { get; private set; }
/// <summary>
/// Highest index processed by the state machine.
/// </summary>
public long ProcessedIndex { get; private set; }
/// <summary>
/// Queue of committed entries awaiting apply.
/// </summary>
public CommitQueue<RaftLogEntry> CommitQueue { get; } = new();
// B2: Election timeout configuration (milliseconds)
/// <summary>
/// Minimum election timeout jitter bound in milliseconds.
/// </summary>
public int ElectionTimeoutMinMs { get; set; } = 150;
/// <summary>
/// Maximum election timeout jitter bound in milliseconds.
/// </summary>
public int ElectionTimeoutMaxMs { get; set; } = 300;
// B6: Pre-vote protocol
// Go reference: raft.go:1600-1700 (pre-vote logic)
// When enabled, a node first conducts a pre-vote round before starting a real election.
// This prevents partitioned nodes from disrupting the cluster by incrementing terms.
/// <summary>
/// Enables pre-vote rounds before real elections.
/// </summary>
public bool PreVoteEnabled { get; set; } = true;
// B4: True while a membership change log entry is pending quorum.
// Go reference: raft.go:961-1019 single-change invariant.
/// <summary>
/// True while a membership-change entry is pending commit.
/// </summary>
public bool MembershipChangeInProgress => Interlocked.Read(ref _membershipChangeIndex) > 0;
/// <summary>
@@ -124,6 +193,15 @@ public sealed class RaftNode : IDisposable
// Go reference: raft.go resetElectionTimeout (uses rand.Int63n for jitter)
private Random _random;
/// <summary>
/// Creates a RAFT node with optional transport and persistence wiring.
/// </summary>
/// <param name="id">Node id.</param>
/// <param name="transport">Transport for peer RPCs.</param>
/// <param name="persistDirectory">Optional persistence directory for term/log metadata.</param>
/// <param name="compactionOptions">Optional log compaction policy.</param>
/// <param name="random">Optional random source for deterministic timeout tests.</param>
/// <param name="group">Optional RAFT group name.</param>
public RaftNode(string id, IRaftTransport? transport = null, string? persistDirectory = null,
CompactionOptions? compactionOptions = null, Random? random = null, string? group = null)
{
@@ -138,6 +216,10 @@ public sealed class RaftNode : IDisposable
_random = random ?? Random.Shared;
}
/// <summary>
/// Configures known cluster peers and resets membership tracking from that set.
/// </summary>
/// <param name="peers">Peers participating in the same RAFT group.</param>
public void ConfigureCluster(IEnumerable<RaftNode> peers)
{
var configuredPeers = peers as ICollection<RaftNode> ?? peers.ToList();
@@ -163,10 +245,22 @@ public sealed class RaftNode : IDisposable
_clusterSize = Math.Max(configuredPeers.Count, 1);
}
/// <summary>
/// Adds a member id to the active configuration.
/// </summary>
/// <param name="memberId">Member id to add.</param>
public void AddMember(string memberId) => _members.Add(memberId);
/// <summary>
/// Removes a member id from the active configuration.
/// </summary>
/// <param name="memberId">Member id to remove.</param>
public void RemoveMember(string memberId) => _members.Remove(memberId);
/// <summary>
/// Starts a new election term and attempts leadership with self-vote.
/// </summary>
/// <param name="clusterSize">Current cluster size used for quorum math.</param>
public void StartElection(int clusterSize)
{
_groupLeader = NoLeader;
@@ -178,6 +272,11 @@ public sealed class RaftNode : IDisposable
TryBecomeLeader(clusterSize);
}
/// <summary>
/// Evaluates and records a vote request for the supplied candidate and term.
/// </summary>
/// <param name="term">Candidate term.</param>
/// <param name="candidateId">Candidate id requesting vote.</param>
public VoteResponse GrantVote(int term, string candidateId = "")
{
if (term < TermState.CurrentTerm)
@@ -199,6 +298,11 @@ public sealed class RaftNode : IDisposable
return new VoteResponse { Granted = true };
}
/// <summary>
/// Processes leader heartbeat and refreshes follower election timeout state.
/// </summary>
/// <param name="term">Leader term from heartbeat.</param>
/// <param name="fromPeerId">Optional sending leader id.</param>
public void ReceiveHeartbeat(int term, string? fromPeerId = null)
{
if (term < TermState.CurrentTerm)
@@ -224,6 +328,11 @@ public sealed class RaftNode : IDisposable
}
}
/// <summary>
/// Processes a vote response while campaigning and checks leader transition.
/// </summary>
/// <param name="response">Vote response from a peer.</param>
/// <param name="clusterSize">Cluster size used for quorum math.</param>
public void ReceiveVote(VoteResponse response, int clusterSize = 3)
{
if (!response.Granted)
@@ -354,6 +463,11 @@ public sealed class RaftNode : IDisposable
"The leader may be partitioned.");
}
/// <summary>
/// Proposes a single command and waits for quorum replication.
/// </summary>
/// <param name="command">Command payload to append to the RAFT log.</param>
/// <param name="ct">Cancellation token for replication work.</param>
public async ValueTask<long> ProposeAsync(string command, CancellationToken ct)
{
if (Role != RaftRole.Leader)
@@ -415,6 +529,8 @@ public sealed class RaftNode : IDisposable
/// Proposes a batch of commands in order and returns their resulting indexes.
/// Go reference: raft.go ProposeMulti.
/// </summary>
/// <param name="commands">Commands to propose sequentially.</param>
/// <param name="ct">Cancellation token for replication work.</param>
public async ValueTask<IReadOnlyList<long>> ProposeMultiAsync(IEnumerable<string> commands, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(commands);
@@ -429,6 +545,10 @@ public sealed class RaftNode : IDisposable
return indexes;
}
/// <summary>
/// Marks an index applied and returns approximate applied entry and byte counts.
/// </summary>
/// <param name="index">Applied index boundary.</param>
public (long Entries, long Bytes) Applied(long index)
{
MarkProcessed(index);
@@ -448,6 +568,8 @@ public sealed class RaftNode : IDisposable
/// After the entry reaches quorum the peer is added to _members.
/// Go reference: raft.go:961-990 (proposeAddPeer).
/// </summary>
/// <param name="peerId">Peer id to add.</param>
/// <param name="ct">Cancellation token for replication work.</param>
public async ValueTask<long> ProposeAddPeerAsync(string peerId, CancellationToken ct)
{
if (Role != RaftRole.Leader)
@@ -496,6 +618,8 @@ public sealed class RaftNode : IDisposable
/// Only the leader may propose; only one membership change may be in flight at a time.
/// Go reference: raft.go:992-1019 (proposeRemovePeer).
/// </summary>
/// <param name="peerId">Peer id to remove.</param>
/// <param name="ct">Cancellation token for replication work.</param>
public async ValueTask<long> ProposeRemovePeerAsync(string peerId, CancellationToken ct)
{
if (Role != RaftRole.Leader)
@@ -545,6 +669,8 @@ public sealed class RaftNode : IDisposable
/// are replicated to all nodes that participate in either configuration.
/// Go reference: raft.go Section 4 (joint consensus).
/// </summary>
/// <param name="cold">Old voter configuration.</param>
/// <param name="cnew">New voter configuration.</param>
public void BeginJointConsensus(IReadOnlyCollection<string> cold, IReadOnlyCollection<string> cnew)
{
_jointOldMembers = new HashSet<string>(cold, StringComparer.Ordinal);
@@ -579,6 +705,8 @@ public sealed class RaftNode : IDisposable
/// Returns false when not in joint consensus.
/// Go reference: raft.go Section 4 — joint config quorum calculation.
/// </summary>
/// <param name="coldVoters">Acknowledging voters from old configuration.</param>
/// <param name="cnewVoters">Acknowledging voters from new configuration.</param>
public bool CalculateJointQuorum(
IReadOnlyCollection<string> coldVoters,
IReadOnlyCollection<string> cnewVoters)
@@ -600,6 +728,7 @@ public sealed class RaftNode : IDisposable
/// do not need to be replayed on restart.
/// Go reference: raft.go CreateSnapshotCheckpoint.
/// </summary>
/// <param name="ct">Cancellation token for snapshot persistence.</param>
public async Task<RaftSnapshot> CreateSnapshotCheckpointAsync(CancellationToken ct)
{
var snapshot = new RaftSnapshot
@@ -618,6 +747,8 @@ public sealed class RaftNode : IDisposable
/// apply pipeline, discards pending entries, then fast-forwards to the snapshot state.
/// Go reference: raft.go DrainAndReplaySnapshot.
/// </summary>
/// <param name="snapshot">Snapshot to install.</param>
/// <param name="ct">Cancellation token for snapshot persistence.</param>
public async Task DrainAndReplaySnapshotAsync(RaftSnapshot snapshot, CancellationToken ct)
{
// Drain any pending commit-queue entries that are now superseded by the snapshot
@@ -745,17 +876,27 @@ public sealed class RaftNode : IDisposable
/// Marks the given index as processed by the state machine.
/// Go reference: raft.go applied/processed tracking.
/// </summary>
/// <param name="index">Processed index boundary.</param>
public void MarkProcessed(long index)
{
if (index > ProcessedIndex)
ProcessedIndex = index;
}
/// <summary>
/// Appends a replicated entry received from leader into local log state.
/// </summary>
/// <param name="entry">Replicated log entry.</param>
public void ReceiveReplicatedEntry(RaftLogEntry entry)
{
Log.AppendReplicated(entry);
}
/// <summary>
/// Validates term and appends a leader entry while refreshing election timeout.
/// </summary>
/// <param name="entry">Replicated entry from leader.</param>
/// <param name="ct">Cancellation token.</param>
public Task TryAppendFromLeaderAsync(RaftLogEntry entry, CancellationToken ct)
{
_ = ct;
@@ -769,6 +910,10 @@ public sealed class RaftNode : IDisposable
return Task.CompletedTask;
}
/// <summary>
/// Creates and persists a snapshot at the current applied index.
/// </summary>
/// <param name="ct">Cancellation token for snapshot persistence.</param>
public async Task<RaftSnapshot> CreateSnapshotAsync(CancellationToken ct)
{
var snapshot = new RaftSnapshot
@@ -780,6 +925,11 @@ public sealed class RaftNode : IDisposable
return snapshot;
}
/// <summary>
/// Installs a snapshot as local log baseline and persists snapshot metadata.
/// </summary>
/// <param name="snapshot">Snapshot to install.</param>
/// <param name="ct">Cancellation token for snapshot persistence.</param>
public Task InstallSnapshotAsync(RaftSnapshot snapshot, CancellationToken ct)
{
Log.ReplaceWithSnapshot(snapshot);
@@ -787,6 +937,9 @@ public sealed class RaftNode : IDisposable
return _snapshotStore.SaveAsync(snapshot, ct);
}
/// <summary>
/// Forces this node to step down to follower state.
/// </summary>
public void RequestStepDown()
{
Role = RaftRole.Follower;
@@ -796,12 +949,18 @@ public sealed class RaftNode : IDisposable
_leaderSinceUtc = null;
}
/// <summary>
/// Returns current log, commit, and applied progress counters.
/// </summary>
public (long Index, long Commit, long Applied) Progress()
{
var index = Log.Entries.Count > 0 ? Log.Entries[^1].Index : Log.BaseIndex;
return (index, CommitIndex, AppliedIndex);
}
/// <summary>
/// Returns approximate in-memory log size in entries and UTF-8 command bytes.
/// </summary>
public (long Entries, long Bytes) Size()
{
var entries = (long)Log.Entries.Count;
@@ -809,9 +968,16 @@ public sealed class RaftNode : IDisposable
return (entries, bytes);
}
/// <summary>
/// Returns configured cluster size fallbacking to member count.
/// </summary>
public int ClusterSize()
=> _clusterSize > 0 ? _clusterSize : Math.Max(_members.Count, 1);
/// <summary>
/// Adjusts bootstrap cluster size before any leader has been observed.
/// </summary>
/// <param name="clusterSize">Desired bootstrap cluster size.</param>
public bool AdjustBootClusterSize(int clusterSize)
{
if (!Leaderless || HadPreviousLeader)
@@ -821,6 +987,10 @@ public sealed class RaftNode : IDisposable
return true;
}
/// <summary>
/// Adjusts active cluster size when this node is current leader.
/// </summary>
/// <param name="clusterSize">Desired cluster size.</param>
public bool AdjustClusterSize(int clusterSize)
{
if (!IsLeader)
@@ -830,6 +1000,10 @@ public sealed class RaftNode : IDisposable
return true;
}
/// <summary>
/// Enables or disables observer (non-voting) mode.
/// </summary>
/// <param name="enabled">True to enable observer mode.</param>
public void SetObserver(bool enabled)
=> _observerMode = enabled;
@@ -853,6 +1027,9 @@ public sealed class RaftNode : IDisposable
return TimeSpan.FromMilliseconds(ms);
}
/// <summary>
/// Returns randomized timeout used for campaign pacing and tests.
/// </summary>
public TimeSpan RandomizedCampaignTimeout()
{
var min = (int)MinCampaignTimeoutDefault.TotalMilliseconds;
@@ -876,6 +1053,7 @@ public sealed class RaftNode : IDisposable
/// an election campaign is triggered automatically.
/// Go reference: raft.go:1500-1550 (campaign logic).
/// </summary>
/// <param name="ct">Cancellation token that stops timer callbacks.</param>
public void StartElectionTimer(CancellationToken ct = default)
{
_electionTimerCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
@@ -918,6 +1096,8 @@ public sealed class RaftNode : IDisposable
///
/// Go reference: raft.go stepDown (leadership transfer variant) / sendTimeoutNow.
/// </summary>
/// <param name="targetId">Peer id to transfer leadership to.</param>
/// <param name="ct">Cancellation token for transfer workflow.</param>
public async Task<bool> TransferLeadershipAsync(string targetId, CancellationToken ct)
{
if (Role != RaftRole.Leader)
@@ -965,6 +1145,7 @@ public sealed class RaftNode : IDisposable
///
/// Go reference: raft.go processTimeoutNow — triggers immediate campaign.
/// </summary>
/// <param name="term">Leader term attached to TimeoutNow.</param>
public void ReceiveTimeoutNow(ulong term)
{
// Accept the sender's term if it's higher.
@@ -1007,6 +1188,7 @@ public sealed class RaftNode : IDisposable
/// Checks if this node's log is current (within one election timeout of the leader).
/// Go reference: raft.go isCurrent check.
/// </summary>
/// <param name="electionTimeout">Timeout window used for currentness check.</param>
public bool IsCurrent(TimeSpan electionTimeout)
{
// A leader is always current
@@ -1020,6 +1202,7 @@ public sealed class RaftNode : IDisposable
/// <summary>
/// Overall health check: node is active and peers are responsive.
/// </summary>
/// <param name="healthThreshold">Maximum age of peer contact considered healthy.</param>
public bool IsHealthy(TimeSpan healthThreshold)
{
if (Role == RaftRole.Leader)
@@ -1044,6 +1227,10 @@ public sealed class RaftNode : IDisposable
/// Pre-votes do NOT change any persistent state (no term increment, no votedFor change).
/// Go reference: raft.go:1600-1700 (pre-vote logic).
/// </summary>
/// <param name="term">Candidate term.</param>
/// <param name="lastTerm">Candidate log last term.</param>
/// <param name="lastIndex">Candidate log last index.</param>
/// <param name="candidateId">Candidate id requesting pre-vote.</param>
public bool RequestPreVote(ulong term, ulong lastTerm, ulong lastIndex, string candidateId)
{
_ = candidateId; // used for logging in production; not needed for correctness
@@ -1126,6 +1313,9 @@ public sealed class RaftNode : IDisposable
}
}
/// <summary>
/// Stops elections and signals waiters that this node is stopped.
/// </summary>
public void Stop()
{
Role = RaftRole.Follower;
@@ -1135,11 +1325,17 @@ public sealed class RaftNode : IDisposable
_stopSignal.TrySetResult();
}
/// <summary>
/// Blocks until stop signal has been observed.
/// </summary>
public void WaitForStop()
{
_stopSignal.Task.GetAwaiter().GetResult();
}
/// <summary>
/// Stops the node and deletes persisted RAFT state from disk.
/// </summary>
public void Delete()
{
Stop();
@@ -1152,6 +1348,10 @@ public sealed class RaftNode : IDisposable
Directory.Delete(_persistDirectory, recursive: true);
}
/// <summary>
/// Persists log, applied index, and durable term/vote metadata to storage.
/// </summary>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PersistAsync(CancellationToken ct)
{
var dir = _persistDirectory ?? Path.Combine(Path.GetTempPath(), "natsdotnet-raft", Id);
@@ -1172,6 +1372,10 @@ public sealed class RaftNode : IDisposable
ct);
}
/// <summary>
/// Loads previously persisted log and durable term/vote state from storage.
/// </summary>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task LoadPersistedStateAsync(CancellationToken ct)
{
var dir = _persistDirectory ?? Path.Combine(Path.GetTempPath(), "natsdotnet-raft", Id);
@@ -1207,10 +1411,19 @@ public sealed class RaftNode : IDisposable
/// <summary>Durable term + vote metadata written alongside the log.</summary>
private sealed class RaftMetaState
{
/// <summary>
/// Persisted current term.
/// </summary>
public int CurrentTerm { get; set; }
/// <summary>
/// Persisted voted-for candidate id for current term.
/// </summary>
public string? VotedFor { get; set; }
}
/// <summary>
/// Disposes the node by stopping timers and signaling shutdown.
/// </summary>
public void Dispose()
{
Stop();
+75
View File
@@ -2,8 +2,31 @@ namespace NATS.Server.Raft;
public interface IRaftTransport
{
/// <summary>
/// Replicates a log entry from leader to a set of followers.
/// </summary>
/// <param name="leaderId">Leader node id sending the append.</param>
/// <param name="followerIds">Follower node ids targeted for replication.</param>
/// <param name="entry">Log entry to replicate.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Per-follower append results.</returns>
Task<IReadOnlyList<AppendResult>> AppendEntriesAsync(string leaderId, IReadOnlyList<string> followerIds, RaftLogEntry entry, CancellationToken ct);
/// <summary>
/// Sends a vote request from a candidate to a voter.
/// </summary>
/// <param name="candidateId">Candidate node id requesting the vote.</param>
/// <param name="voterId">Target voter node id.</param>
/// <param name="request">Vote request payload.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Vote response from the target voter.</returns>
Task<VoteResponse> RequestVoteAsync(string candidateId, string voterId, VoteRequest request, CancellationToken ct);
/// <summary>
/// Installs a snapshot from leader to follower.
/// </summary>
/// <param name="leaderId">Leader node id sending the snapshot.</param>
/// <param name="followerId">Follower node id receiving the snapshot.</param>
/// <param name="snapshot">Snapshot payload.</param>
/// <param name="ct">Cancellation token.</param>
Task InstallSnapshotAsync(string leaderId, string followerId, RaftSnapshot snapshot, CancellationToken ct);
/// <summary>
@@ -11,6 +34,10 @@ public interface IRaftTransport
/// an election and bypass its election timer. Used for leadership transfer.
/// Go reference: raft.go sendTimeoutNow
/// </summary>
/// <param name="leaderId">Leader node id issuing TimeoutNow.</param>
/// <param name="targetId">Target follower node id.</param>
/// <param name="term">Leader term to include in the request.</param>
/// <param name="ct">Cancellation token.</param>
Task SendTimeoutNowAsync(string leaderId, string targetId, ulong term, CancellationToken ct);
/// <summary>
@@ -20,6 +47,11 @@ public interface IRaftTransport
/// serving a linearizable read.
/// Go reference: raft.go — leader sends AppendEntries (empty) to confirm quorum for reads.
/// </summary>
/// <param name="leaderId">Leader node id sending heartbeats.</param>
/// <param name="followerIds">Follower node ids to contact.</param>
/// <param name="term">Leader term for heartbeat messages.</param>
/// <param name="onAck">Callback invoked for each acknowledged follower id.</param>
/// <param name="ct">Cancellation token.</param>
Task SendHeartbeatAsync(string leaderId, IReadOnlyList<string> followerIds, int term, Action<string> onAck, CancellationToken ct);
}
@@ -27,11 +59,23 @@ public sealed class InMemoryRaftTransport : IRaftTransport
{
private readonly Dictionary<string, RaftNode> _nodes = new(StringComparer.Ordinal);
/// <summary>
/// Registers a node in the in-memory transport registry.
/// </summary>
/// <param name="node">Node instance to register.</param>
public void Register(RaftNode node)
{
_nodes[node.Id] = node;
}
/// <summary>
/// Replicates a log entry to registered follower nodes.
/// </summary>
/// <param name="leaderId">Leader node id sending the append.</param>
/// <param name="followerIds">Follower node ids targeted for replication.</param>
/// <param name="entry">Log entry to replicate.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Per-follower append results.</returns>
public Task<IReadOnlyList<AppendResult>> AppendEntriesAsync(string leaderId, IReadOnlyList<string> followerIds, RaftLogEntry entry, CancellationToken ct)
{
var results = new List<AppendResult>(followerIds.Count);
@@ -51,6 +95,14 @@ public sealed class InMemoryRaftTransport : IRaftTransport
return Task.FromResult<IReadOnlyList<AppendResult>>(results);
}
/// <summary>
/// Sends a vote request to a registered voter node.
/// </summary>
/// <param name="candidateId">Candidate node id requesting a vote.</param>
/// <param name="voterId">Voter node id receiving the request.</param>
/// <param name="request">Vote request payload.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>Vote response from the voter or a rejected response if unreachable.</returns>
public Task<VoteResponse> RequestVoteAsync(string candidateId, string voterId, VoteRequest request, CancellationToken ct)
{
if (_nodes.TryGetValue(voterId, out var node))
@@ -59,6 +111,13 @@ public sealed class InMemoryRaftTransport : IRaftTransport
return Task.FromResult(new VoteResponse { Granted = false });
}
/// <summary>
/// Installs a snapshot on a registered follower node.
/// </summary>
/// <param name="leaderId">Leader node id sending the snapshot.</param>
/// <param name="followerId">Follower node id receiving the snapshot.</param>
/// <param name="snapshot">Snapshot payload.</param>
/// <param name="ct">Cancellation token.</param>
public async Task InstallSnapshotAsync(string leaderId, string followerId, RaftSnapshot snapshot, CancellationToken ct)
{
_ = leaderId;
@@ -66,6 +125,13 @@ public sealed class InMemoryRaftTransport : IRaftTransport
await node.InstallSnapshotAsync(snapshot, ct);
}
/// <summary>
/// Delivers heartbeat notifications to follower nodes.
/// </summary>
/// <param name="leaderId">Leader node id sending heartbeats.</param>
/// <param name="followerIds">Follower node ids to notify.</param>
/// <param name="term">Leader term for heartbeat messages.</param>
/// <param name="ct">Cancellation token.</param>
public async Task AppendHeartbeatAsync(string leaderId, IReadOnlyList<string> followerIds, int term, CancellationToken ct)
{
_ = leaderId;
@@ -85,6 +151,11 @@ public sealed class InMemoryRaftTransport : IRaftTransport
/// Unreachable followers (not registered in the transport) produce no acknowledgement.
/// Go reference: raft.go — leader sends empty AppendEntries to confirm quorum for reads.
/// </summary>
/// <param name="leaderId">Leader node id sending heartbeats.</param>
/// <param name="followerIds">Follower node ids to contact.</param>
/// <param name="term">Leader term for heartbeat messages.</param>
/// <param name="onAck">Callback invoked for each acknowledged follower id.</param>
/// <param name="ct">Cancellation token.</param>
public Task SendHeartbeatAsync(string leaderId, IReadOnlyList<string> followerIds, int term, Action<string> onAck, CancellationToken ct)
{
_ = leaderId;
@@ -106,6 +177,10 @@ public sealed class InMemoryRaftTransport : IRaftTransport
/// If the target is not registered (simulating an unreachable peer), does nothing.
/// Go reference: raft.go sendTimeoutNow / processTimeoutNow
/// </summary>
/// <param name="leaderId">Leader node id issuing TimeoutNow.</param>
/// <param name="targetId">Target follower node id.</param>
/// <param name="term">Leader term for the TimeoutNow request.</param>
/// <param name="ct">Cancellation token.</param>
public Task SendTimeoutNowAsync(string leaderId, string targetId, ulong term, CancellationToken ct)
{
_ = leaderId;
+15
View File
@@ -106,6 +106,7 @@ public readonly record struct RaftVoteRequestWire(
/// if the span is not exactly 32 bytes.
/// Go: server/raft.go:4571-4583 — decodeVoteRequest()
/// </summary>
/// <param name="msg">Raw VoteRequest payload received from the RAFT transport.</param>
public static RaftVoteRequestWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != RaftWireConstants.VoteRequestLen)
@@ -156,6 +157,7 @@ public readonly record struct RaftVoteResponseWire(
/// if the span is not exactly 17 bytes.
/// Go: server/raft.go:4753-4762 — decodeVoteResponse()
/// </summary>
/// <param name="msg">Raw VoteResponse payload received from a peer.</param>
public static RaftVoteResponseWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != RaftWireConstants.VoteResponseLen)
@@ -252,6 +254,7 @@ public readonly record struct RaftAppendEntryWire(
/// if the buffer is shorter than the minimum header length or malformed.
/// Go: server/raft.go:2714-2746 — decodeAppendEntry()
/// </summary>
/// <param name="msg">Raw AppendEntry payload containing header and entry data.</param>
public static RaftAppendEntryWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length < RaftWireConstants.AppendEntryBaseLen)
@@ -340,6 +343,7 @@ public readonly record struct RaftAppendEntryResponseWire(
/// if the span is not exactly 25 bytes.
/// Go: server/raft.go:2799-2817 — decodeAppendEntryResponse()
/// </summary>
/// <param name="msg">Raw AppendEntryResponse payload from a follower.</param>
public static RaftAppendEntryResponseWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != RaftWireConstants.AppendEntryResponseLen)
@@ -387,6 +391,7 @@ public readonly record struct RaftPreVoteRequestWire(
/// Decodes a PreVoteRequest from a span. Throws <see cref="ArgumentException"/>
/// if the span is not exactly 32 bytes.
/// </summary>
/// <param name="msg">Raw PreVoteRequest payload received from a candidate.</param>
public static RaftPreVoteRequestWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != RaftWireConstants.VoteRequestLen)
@@ -429,6 +434,7 @@ public readonly record struct RaftPreVoteResponseWire(
/// Decodes a PreVoteResponse from a span. Throws <see cref="ArgumentException"/>
/// if the span is not exactly 17 bytes.
/// </summary>
/// <param name="msg">Raw PreVoteResponse payload received from a peer.</param>
public static RaftPreVoteResponseWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != RaftWireConstants.VoteResponseLen)
@@ -475,6 +481,7 @@ public readonly record struct RaftTimeoutNowWire(ulong Term, string LeaderId)
/// Decodes a TimeoutNow message from a span. Throws <see cref="ArgumentException"/>
/// if the span is not exactly 16 bytes.
/// </summary>
/// <param name="msg">Raw TimeoutNow payload sent by the current leader.</param>
public static RaftTimeoutNowWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length != MessageLen)
@@ -538,6 +545,7 @@ public readonly record struct RaftInstallSnapshotChunkWire(
/// Throws <see cref="ArgumentException"/> when the buffer is shorter than
/// the fixed <see cref="HeaderLen"/> bytes.
/// </summary>
/// <param name="msg">Raw InstallSnapshot chunk payload including fixed header and chunk data.</param>
public static RaftInstallSnapshotChunkWire Decode(ReadOnlySpan<byte> msg)
{
if (msg.Length < HeaderLen)
@@ -566,6 +574,8 @@ internal static class RaftWireHelpers
/// copy(buf[:idLen], id) semantics).
/// Go: server/raft.go:2693 — copy(buf[:idLen], ae.leader)
/// </summary>
/// <param name="dest">Destination span containing an ID field slot in a RAFT wire frame.</param>
/// <param name="id">Peer/leader identifier to encode.</param>
public static void WriteId(Span<byte> dest, string id)
{
// Zero-fill the 8-byte slot first.
@@ -580,6 +590,7 @@ internal static class RaftWireHelpers
/// that zero-padded IDs decode back to their original string.
/// Go: server/raft.go:4581 — string(copyBytes(msg[24:24+idLen]))
/// </summary>
/// <param name="src">Source span containing an encoded fixed-width ID field.</param>
public static string ReadId(ReadOnlySpan<byte> src)
{
var idBytes = src[..RaftWireConstants.IdLen];
@@ -594,6 +605,8 @@ internal static class RaftWireHelpers
/// number of bytes written (1-10).
/// Go: server/raft.go:2682 — binary.PutUvarint(_lterm[:], ae.lterm)
/// </summary>
/// <param name="buf">Destination buffer where the uvarint bytes are written.</param>
/// <param name="value">Unsigned integer value to encode.</param>
public static int WriteUvarint(Span<byte> buf, ulong value)
{
var pos = 0;
@@ -611,6 +624,8 @@ internal static class RaftWireHelpers
/// and returns the number of bytes consumed (0 on overflow or empty input).
/// Go: server/raft.go:2740 — binary.Uvarint(msg[ri:])
/// </summary>
/// <param name="buf">Source buffer containing uvarint-encoded data.</param>
/// <param name="value">Decoded unsigned integer output.</param>
public static int ReadUvarint(ReadOnlySpan<byte> buf, out ulong value)
{
value = 0;
+107
View File
@@ -17,7 +17,9 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
private readonly CancellationTokenSource _closedCts = new();
private Task? _frameLoopTask;
/// <summary>Remote server id learned during ROUTE handshake.</summary>
public string? RemoteServerId { get; private set; }
/// <summary>Remote endpoint string for diagnostics and monitoring.</summary>
public string RemoteEndpoint => _socket.RemoteEndPoint?.ToString() ?? Guid.NewGuid().ToString("N");
/// <summary>
@@ -61,6 +63,9 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
/// either side is 0 for backward compatibility with peers that do not support pooling.
/// Go reference: server/route.go negotiateRoutePool.
/// </summary>
/// <param name="localPoolSize">Local configured route pool size.</param>
/// <param name="remotePoolSize">Remote configured route pool size.</param>
/// <returns>Negotiated pool size.</returns>
public static int NegotiatePoolSize(int localPoolSize, int remotePoolSize)
{
if (localPoolSize == 0 || remotePoolSize == 0)
@@ -72,14 +77,22 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
/// <summary>
/// Applies the result of pool size negotiation to this connection.
/// </summary>
/// <param name="negotiatedPoolSize">Negotiated pool size.</param>
internal void SetNegotiatedPoolSize(int negotiatedPoolSize)
{
NegotiatedPoolSize = negotiatedPoolSize;
}
/// <summary>Callback invoked when remote RS/LS interest updates are received.</summary>
public Func<RemoteSubscription, Task>? RemoteSubscriptionReceived { get; set; }
/// <summary>Callback invoked when remote RMSG payloads are received.</summary>
public Func<RouteMessage, Task>? RoutedMessageReceived { get; set; }
/// <summary>
/// Performs outbound ROUTE handshake by sending local id and reading remote id.
/// </summary>
/// <param name="serverId">Local server id.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformOutboundHandshakeAsync(string serverId, CancellationToken ct)
{
await WriteLineAsync($"ROUTE {serverId}", ct);
@@ -87,6 +100,11 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
RemoteServerId = ParseHandshake(line);
}
/// <summary>
/// Performs inbound ROUTE handshake by reading remote id and sending local id.
/// </summary>
/// <param name="serverId">Local server id.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task PerformInboundHandshakeAsync(string serverId, CancellationToken ct)
{
var line = await ReadLineAsync(ct);
@@ -94,6 +112,10 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync($"ROUTE {serverId}", ct);
}
/// <summary>
/// Starts the background route frame read loop.
/// </summary>
/// <param name="ct">Cancellation token controlling loop lifetime.</param>
public void StartFrameLoop(CancellationToken ct)
{
if (_frameLoopTask != null)
@@ -103,9 +125,24 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
_frameLoopTask = Task.Run(() => ReadFramesAsync(linked.Token), linked.Token);
}
/// <summary>
/// Sends an RS+ protocol line for route interest propagation.
/// </summary>
/// <param name="account">Account associated with the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRsPlusAsync(string account, string subject, string? queue, CancellationToken ct)
=> await SendRsPlusAsync(account, subject, queue, queueWeight: 0, ct);
/// <summary>
/// Sends an RS+ protocol line with optional queue weight.
/// </summary>
/// <param name="account">Account associated with the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="queueWeight">Queue weight to advertise when queue is present.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRsPlusAsync(string account, string subject, string? queue, int queueWeight, CancellationToken ct)
{
string frame;
@@ -119,6 +156,13 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync(frame, ct);
}
/// <summary>
/// Sends an RS- protocol line to remove route interest.
/// </summary>
/// <param name="account">Account associated with the interest update.</param>
/// <param name="subject">Subject being removed.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRsMinusAsync(string account, string subject, string? queue, CancellationToken ct)
{
var frame = queue is { Length: > 0 }
@@ -127,6 +171,13 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync(frame, ct);
}
/// <summary>
/// Sends an LS+ protocol line for leaf interest propagation over route links.
/// </summary>
/// <param name="account">Account associated with the interest update.</param>
/// <param name="subject">Subject being added.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendLsPlusAsync(string account, string subject, string? queue, CancellationToken ct)
{
var frame = queue is { Length: > 0 }
@@ -135,6 +186,13 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync(frame, ct);
}
/// <summary>
/// Sends an LS- protocol line for leaf interest removal over route links.
/// </summary>
/// <param name="account">Account associated with the interest update.</param>
/// <param name="subject">Subject being removed.</param>
/// <param name="queue">Optional queue group.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendLsMinusAsync(string account, string subject, string? queue, CancellationToken ct)
{
var frame = queue is { Length: > 0 }
@@ -143,6 +201,11 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await WriteLineAsync(frame, ct);
}
/// <summary>
/// Sends a batch of RS+ protocol lines for non-removal subscriptions.
/// </summary>
/// <param name="subscriptions">Subscriptions to encode and send.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRouteSubProtosAsync(IEnumerable<RemoteSubscription> subscriptions, CancellationToken ct)
{
var protos = new List<string>();
@@ -162,6 +225,11 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await SendRouteSubOrUnSubProtosAsync(protos, ct);
}
/// <summary>
/// Sends a batch of RS- protocol lines for subscription removals.
/// </summary>
/// <param name="subscriptions">Subscriptions to encode and send as removals.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRouteUnSubProtosAsync(IEnumerable<RemoteSubscription> subscriptions, CancellationToken ct)
{
var protos = new List<string>();
@@ -176,6 +244,11 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await SendRouteSubOrUnSubProtosAsync(protos, ct);
}
/// <summary>
/// Sends a pre-built batch of route subscription protocol lines.
/// </summary>
/// <param name="protocols">Protocol lines to send.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRouteSubOrUnSubProtosAsync(IEnumerable<string> protocols, CancellationToken ct)
{
var sb = new StringBuilder();
@@ -202,6 +275,14 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
}
}
/// <summary>
/// Sends an RMSG payload frame to the remote route.
/// </summary>
/// <param name="account">Account associated with the message.</param>
/// <param name="subject">Subject being routed.</param>
/// <param name="replyTo">Optional reply subject.</param>
/// <param name="payload">Payload bytes.</param>
/// <param name="ct">Cancellation token for I/O operations.</param>
public async Task SendRmsgAsync(string account, string subject, string? replyTo, ReadOnlyMemory<byte> payload, CancellationToken ct)
{
var replyToken = string.IsNullOrEmpty(replyTo) ? "-" : replyTo;
@@ -221,6 +302,10 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
}
}
/// <summary>
/// Waits until the route frame loop exits.
/// </summary>
/// <param name="ct">Cancellation token for wait operation.</param>
public async Task WaitUntilClosedAsync(CancellationToken ct)
{
if (_frameLoopTask == null)
@@ -230,6 +315,9 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
await _frameLoopTask.WaitAsync(linked.Token);
}
/// <summary>
/// Disposes this route connection and stops background processing.
/// </summary>
public async ValueTask DisposeAsync()
{
await _closedCts.CancelAsync();
@@ -413,6 +501,14 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
return true;
}
/// <summary>
/// Parses a remote RS-/LS- protocol line into account/subject/queue components.
/// </summary>
/// <param name="line">Raw protocol line.</param>
/// <param name="account">Receives parsed account on success.</param>
/// <param name="subject">Receives parsed subject on success.</param>
/// <param name="queue">Receives parsed queue group on success.</param>
/// <returns><see langword="true"/> when parsing succeeds.</returns>
internal static bool TryParseRemoteUnsub(string line, out string account, out string subject, out string? queue)
{
account = "$G";
@@ -426,6 +522,10 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
return TryParseAccountScopedInterest(parts, out account, out subject, out queue);
}
/// <summary>
/// Indicates whether this route was solicited outbound by this server.
/// </summary>
/// <returns><see langword="true"/> when route is solicited.</returns>
public bool IsSolicitedRoute()
=> IsSolicited;
@@ -434,6 +534,13 @@ public sealed class RouteConnection(Socket socket) : IAsyncDisposable
|| token.Contains('*', StringComparison.Ordinal)
|| token.Contains('>', StringComparison.Ordinal);
/// <summary>
/// Builds the CONNECT payload JSON used during route handshake.
/// </summary>
/// <param name="serverId">Local server id and advertised name.</param>
/// <param name="accounts">Optional exported account list.</param>
/// <param name="topologySnapshot">Optional topology snapshot string.</param>
/// <returns>Serialized CONNECT JSON payload.</returns>
public static string BuildConnectInfoJson(string serverId, IEnumerable<string>? accounts, string? topologySnapshot)
{
var payload = new
@@ -6,6 +6,11 @@ public static class SubjectMatch
public const char Fwc = '>'; // full wildcard
public const char Sep = '.'; // token separator
/// <summary>
/// Validates a subject expression for subscription indexing and routing.
/// </summary>
/// <param name="subject">Subject text to validate.</param>
/// <returns><see langword="true"/> when the subject follows token and wildcard placement rules.</returns>
public static bool IsValidSubject(string subject)
{
if (string.IsNullOrEmpty(subject))
@@ -47,6 +52,11 @@ public static class SubjectMatch
return true;
}
/// <summary>
/// Determines whether a subject contains only literal tokens.
/// </summary>
/// <param name="subject">Subject to inspect.</param>
/// <returns><see langword="true"/> when no wildcard token is present.</returns>
public static bool IsLiteral(string subject)
{
for (int i = 0; i < subject.Length; i++)
@@ -64,18 +74,32 @@ public static class SubjectMatch
return true;
}
/// <summary>
/// Validates a publish subject, which must be syntactically valid and wildcard-free.
/// </summary>
/// <param name="subject">Publish subject to validate.</param>
public static bool IsValidPublishSubject(string subject)
{
return IsValidSubject(subject) && IsLiteral(subject);
}
/// <summary>
/// Checks whether the subject contains wildcard tokens.
/// </summary>
/// <param name="subject">Subject expression to evaluate.</param>
public static bool SubjectHasWildcard(string subject) => !IsLiteral(subject);
/// <summary>
/// Validates a literal subject used by publish and direct lookup paths.
/// </summary>
/// <param name="subject">Literal subject candidate.</param>
public static bool IsValidLiteralSubject(string subject) => IsValidPublishSubject(subject);
/// <summary>
/// Match a literal subject against a pattern that may contain wildcards.
/// </summary>
/// <param name="literal">Concrete subject from a published message.</param>
/// <param name="pattern">Subscription pattern that may include <c>*</c> and <c>&gt;</c>.</param>
public static bool MatchLiteral(string literal, string pattern)
{
int li = 0, pi = 0;
@@ -119,6 +143,7 @@ public static class SubjectMatch
}
/// <summary>Count dot-delimited tokens. Empty string returns 0.</summary>
/// <param name="subject">Subject string to tokenize.</param>
public static int NumTokens(string subject)
{
if (string.IsNullOrEmpty(subject))
@@ -133,6 +158,8 @@ public static class SubjectMatch
}
/// <summary>Return the 0-based nth token as a span. Returns empty if out of range.</summary>
/// <param name="subject">Subject containing dot-delimited tokens.</param>
/// <param name="index">Zero-based token index.</param>
public static ReadOnlySpan<char> TokenAt(string subject, int index)
{
if (string.IsNullOrEmpty(subject))
@@ -160,6 +187,8 @@ public static class SubjectMatch
/// Determines if two subject patterns (possibly containing wildcards) can both
/// match the same literal subject. Reference: Go sublist.go SubjectsCollide.
/// </summary>
/// <param name="subj1">First subject pattern.</param>
/// <param name="subj2">Second subject pattern.</param>
public static bool SubjectsCollide(string subj1, string subj2)
{
if (subj1 == subj2)
@@ -202,20 +231,40 @@ public static class SubjectMatch
// Go reference: sublist.go SubjectMatchesFilter / subjectIsSubsetMatch / isSubsetMatch / isSubsetMatchTokenized.
// This is used by JetStream stores to evaluate subject filters with wildcard semantics.
/// <summary>
/// Evaluates whether a stream subject satisfies a JetStream filter expression.
/// </summary>
/// <param name="subject">Subject from a stored or incoming message.</param>
/// <param name="filter">Configured filter expression.</param>
public static bool SubjectMatchesFilter(string subject, string filter) => SubjectIsSubsetMatch(subject, filter);
/// <summary>
/// Determines if the left-hand subject pattern is covered by the test pattern.
/// </summary>
/// <param name="subject">Subject pattern being tested.</param>
/// <param name="test">Pattern used as the subset constraint.</param>
public static bool SubjectIsSubsetMatch(string subject, string test)
{
var subjectTokens = TokenizeSubject(subject);
return IsSubsetMatch(subjectTokens, test);
}
/// <summary>
/// Determines subset matching for a tokenized subject against a test expression.
/// </summary>
/// <param name="tokens">Tokenized subject to evaluate.</param>
/// <param name="test">Test expression with optional wildcards.</param>
public static bool IsSubsetMatch(string[] tokens, string test)
{
var testTokens = TokenizeSubject(test);
return IsSubsetMatchTokenized(tokens, testTokens);
}
/// <summary>
/// Performs token-level subset matching between two subject expressions.
/// </summary>
/// <param name="tokens">Tokenized subject expression.</param>
/// <param name="test">Tokenized test expression.</param>
public static bool IsSubsetMatchTokenized(IReadOnlyList<string> tokens, IReadOnlyList<string> test)
{
for (var i = 0; i < test.Count; i++)
@@ -249,6 +298,11 @@ public static class SubjectMatch
return tokens.Count == test.Count;
}
/// <summary>
/// Compares a token span to a candidate token using ordinal semantics.
/// </summary>
/// <param name="token">Token span from a subject.</param>
/// <param name="candidate">Candidate token text.</param>
internal static bool TokenEquals(ReadOnlySpan<char> token, string candidate)
=> token.SequenceEqual(candidate);
@@ -266,6 +320,8 @@ public static class SubjectMatch
/// <summary>
/// Validates subject. When checkRunes is true, also rejects null bytes.
/// </summary>
/// <param name="subject">Subject text to validate.</param>
/// <param name="checkRunes">Whether to reject invalid rune content such as null bytes.</param>
public static bool IsValidSubject(string subject, bool checkRunes)
{
if (!IsValidSubject(subject))
+32
View File
@@ -39,6 +39,11 @@ public static class StatusAssertionMaps
[2] = StatusAssertion.Unknown,
};
/// <summary>
/// Converts numeric status assertion values to their canonical string form.
/// </summary>
/// <param name="sa">Numeric status assertion value.</param>
/// <returns>Status assertion string (defaults to <c>unknown</c>).</returns>
public static string GetStatusAssertionStr(int sa)
{
var value = StatusAssertionIntToVal.TryGetValue(sa, out var mapped)
@@ -50,6 +55,7 @@ public static class StatusAssertionMaps
public sealed class StatusAssertionJsonConverter : JsonConverter<StatusAssertion>
{
/// <inheritdoc />
public override StatusAssertion Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
if (reader.TokenType == JsonTokenType.String)
@@ -70,6 +76,7 @@ public sealed class StatusAssertionJsonConverter : JsonConverter<StatusAssertion
return StatusAssertion.Unknown;
}
/// <inheritdoc />
public override void Write(Utf8JsonWriter writer, StatusAssertion value, JsonSerializerOptions options)
{
if (!StatusAssertionMaps.StatusAssertionValToStr.TryGetValue(value, out var str))
@@ -80,29 +87,38 @@ public sealed class StatusAssertionJsonConverter : JsonConverter<StatusAssertion
public sealed class ChainLink
{
/// <summary>Leaf certificate for chain/OCSP evaluation.</summary>
public X509Certificate2? Leaf { get; set; }
/// <summary>Issuer certificate corresponding to <see cref="Leaf"/>.</summary>
public X509Certificate2? Issuer { get; set; }
/// <summary>Discovered HTTP(S) OCSP responder endpoints for the leaf certificate.</summary>
public IReadOnlyList<Uri>? OCSPWebEndpoints { get; set; }
}
public sealed class OcspResponseInfo
{
/// <summary>Time at which OCSP response status was known to be correct.</summary>
public DateTime ThisUpdate { get; init; }
/// <summary>Optional time after which responder no longer vouches for response status.</summary>
public DateTime? NextUpdate { get; init; }
}
public sealed class CertInfo
{
[JsonPropertyName("subject")]
/// <summary>Subject distinguished name.</summary>
public string Subject { get; init; } = string.Empty;
[JsonPropertyName("issuer")]
/// <summary>Issuer distinguished name.</summary>
public string Issuer { get; init; } = string.Empty;
[JsonPropertyName("fingerprint")]
/// <summary>Certificate fingerprint string.</summary>
public string Fingerprint { get; init; } = string.Empty;
[JsonPropertyName("raw")]
/// <summary>Raw DER certificate bytes.</summary>
public byte[] Raw { get; init; } = [];
}
@@ -112,16 +128,32 @@ public sealed class OCSPPeerConfig
public static readonly TimeSpan DefaultOCSPResponderTimeout = TimeSpan.FromSeconds(2);
public static readonly TimeSpan DefaultTTLUnsetNextUpdate = TimeSpan.FromHours(1);
/// <summary>Enables OCSP peer verification.</summary>
public bool Verify { get; set; }
/// <summary>Responder timeout in seconds.</summary>
public double Timeout { get; set; } = DefaultOCSPResponderTimeout.TotalSeconds;
/// <summary>Allowed clock skew in seconds when validating response times.</summary>
public double ClockSkew { get; set; } = DefaultAllowedClockSkew.TotalSeconds;
/// <summary>When true, OCSP failures are reported as warnings only.</summary>
public bool WarnOnly { get; set; }
/// <summary>When true, unknown OCSP status is treated as success.</summary>
public bool UnknownIsGood { get; set; }
/// <summary>When true, allows handshake when CA responder is unreachable.</summary>
public bool AllowWhenCAUnreachable { get; set; }
/// <summary>Fallback TTL in seconds when OCSP NextUpdate is absent.</summary>
public double TTLUnsetNextUpdate { get; set; } = DefaultTTLUnsetNextUpdate.TotalSeconds;
/// <summary>
/// Creates a default OCSP peer configuration instance.
/// </summary>
/// <returns>Default-configured <see cref="OCSPPeerConfig"/>.</returns>
public static OCSPPeerConfig NewOCSPPeerConfig() => new();
/// <summary>
/// Parses OCSP peer configuration values from configuration map entries.
/// </summary>
/// <param name="values">Configuration key/value pairs.</param>
/// <returns>Parsed OCSP peer configuration.</returns>
public static OCSPPeerConfig Parse(IReadOnlyDictionary<string, object?> values)
{
var cfg = NewOCSPPeerConfig();
+5
View File
@@ -59,10 +59,15 @@ public sealed class PeekableStream : Stream
public override Task FlushAsync(CancellationToken ct) => _inner.FlushAsync(ct);
// Required Stream overrides
/// <inheritdoc />
public override bool CanRead => _inner.CanRead;
/// <inheritdoc />
public override bool CanSeek => false;
/// <inheritdoc />
public override bool CanWrite => _inner.CanWrite;
/// <inheritdoc />
public override long Length => throw new NotSupportedException();
/// <inheritdoc />
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
+66
View File
@@ -13,6 +13,12 @@ public static class TlsHelper
private const string OcspAccessMethodOid = "1.3.6.1.5.5.7.48.1";
private const string OcspSigningEkuOid = "1.3.6.1.5.5.7.3.9";
/// <summary>
/// Loads the server certificate, optionally combining with a PEM private key.
/// </summary>
/// <param name="certPath">Path to certificate file.</param>
/// <param name="keyPath">Optional path to private key PEM.</param>
/// <returns>Loaded X509 certificate.</returns>
public static X509Certificate2 LoadCertificate(string certPath, string? keyPath)
{
if (keyPath != null)
@@ -20,6 +26,11 @@ public static class TlsHelper
return X509CertificateLoader.LoadCertificateFromFile(certPath);
}
/// <summary>
/// Loads one or more CA certificates from a PEM file.
/// </summary>
/// <param name="caPath">Path to PEM CA certificate bundle.</param>
/// <returns>Loaded CA certificates.</returns>
public static X509Certificate2Collection LoadCaCertificates(string caPath)
{
var pem = File.ReadAllText(caPath);
@@ -30,6 +41,8 @@ public static class TlsHelper
/// Parses one or more PEM blocks and requires all blocks to be CERTIFICATE.
/// Mirrors Go parseCertPEM behavior by rejecting unexpected block types.
/// </summary>
/// <param name="pemData">PEM text containing certificate blocks.</param>
/// <returns>Parsed certificate collection.</returns>
public static X509Certificate2Collection ParseCertPem(string pemData)
{
if (string.IsNullOrWhiteSpace(pemData))
@@ -66,6 +79,11 @@ public static class TlsHelper
return certs;
}
/// <summary>
/// Builds server TLS authentication options from configured NATS TLS settings.
/// </summary>
/// <param name="opts">Server options containing TLS and OCSP settings.</param>
/// <returns>Configured SSL server authentication options.</returns>
public static SslServerAuthenticationOptions BuildServerAuthOptions(NatsOptions opts)
{
var cert = LoadCertificate(opts.TlsCert!, opts.TlsKey);
@@ -117,6 +135,9 @@ public static class TlsHelper
/// When <paramref name="offline"/> is false the runtime will contact the
/// certificate's OCSP responder to obtain a fresh stapled response.
/// </summary>
/// <param name="opts">Server options containing TLS and OCSP settings.</param>
/// <param name="offline">When <see langword="true"/>, avoids online OCSP fetch while building context.</param>
/// <returns>Certificate context for SSL stream usage, or <see langword="null"/>.</returns>
public static SslStreamCertificateContext? BuildCertificateContext(NatsOptions opts, bool offline = false)
{
if (!opts.HasTls) return null;
@@ -130,6 +151,11 @@ public static class TlsHelper
return SslStreamCertificateContext.Create(cert, chain, offline: offline);
}
/// <summary>
/// Computes the SHA-256 hash of certificate SubjectPublicKeyInfo bytes.
/// </summary>
/// <param name="cert">Certificate to hash.</param>
/// <returns>Lowercase hexadecimal hash string.</returns>
public static string GetCertificateHash(X509Certificate2 cert)
{
var spki = cert.PublicKey.ExportSubjectPublicKeyInfo();
@@ -137,12 +163,22 @@ public static class TlsHelper
return Convert.ToHexStringLower(hash);
}
/// <summary>
/// Generates a base64-encoded SHA-256 fingerprint for the raw certificate.
/// </summary>
/// <param name="cert">Certificate to fingerprint.</param>
/// <returns>Base64 fingerprint string.</returns>
public static string GenerateFingerprint(X509Certificate2 cert)
{
var hash = SHA256.HashData(cert.RawData);
return Convert.ToBase64String(hash);
}
/// <summary>
/// Filters a URI set to valid absolute HTTP(S) endpoints.
/// </summary>
/// <param name="uris">Candidate URI strings.</param>
/// <returns>Validated HTTP(S) endpoint list.</returns>
public static IReadOnlyList<Uri> GetWebEndpoints(IEnumerable<string> uris)
{
var urls = new List<Uri>();
@@ -159,16 +195,32 @@ public static class TlsHelper
return urls;
}
/// <summary>
/// Returns certificate subject distinguished-name text.
/// </summary>
/// <param name="cert">Certificate to inspect.</param>
/// <returns>Subject DN string, or empty when unavailable.</returns>
public static string GetSubjectDNForm(X509Certificate2? cert)
{
return cert?.SubjectName.Name ?? string.Empty;
}
/// <summary>
/// Returns certificate issuer distinguished-name text.
/// </summary>
/// <param name="cert">Certificate to inspect.</param>
/// <returns>Issuer DN string, or empty when unavailable.</returns>
public static string GetIssuerDNForm(X509Certificate2? cert)
{
return cert?.IssuerName.Name ?? string.Empty;
}
/// <summary>
/// Checks whether a certificate matches the configured pinned-cert set.
/// </summary>
/// <param name="cert">Certificate to test.</param>
/// <param name="pinned">Pinned certificate hash set.</param>
/// <returns><see langword="true"/> when certificate hash is pinned.</returns>
public static bool MatchesPinnedCert(X509Certificate2 cert, HashSet<string> pinned)
{
var hash = GetCertificateHash(cert);
@@ -179,6 +231,8 @@ public static class TlsHelper
/// Checks if a chain link is eligible for OCSP validation by ensuring the leaf
/// certificate includes at least one valid HTTP(S) OCSP AIA endpoint.
/// </summary>
/// <param name="link">Chain link containing leaf certificate metadata.</param>
/// <returns><see langword="true"/> when OCSP checking can be performed.</returns>
public static bool CertOCSPEligible(ChainLink? link)
{
if (link?.Leaf is null)
@@ -202,6 +256,9 @@ public static class TlsHelper
/// <summary>
/// Returns the positional issuer certificate for a leaf in a verified chain.
/// </summary>
/// <param name="chain">Verified certificate chain.</param>
/// <param name="leafPos">Index of leaf certificate in chain.</param>
/// <returns>Issuer certificate, or <see langword="null"/> when unavailable.</returns>
public static X509Certificate2? GetLeafIssuerCert(IReadOnlyList<X509Certificate2>? chain, int leafPos)
{
if (chain is null || chain.Count == 0 || leafPos < 0 || leafPos >= chain.Count - 1)
@@ -213,6 +270,9 @@ public static class TlsHelper
/// Equivalent to Go certstore.GetLeafIssuer: verifies the leaf against the
/// supplied trust root and returns the first issuer in the verified chain.
/// </summary>
/// <param name="leaf">Leaf certificate.</param>
/// <param name="trustedRoot">Trusted root certificate.</param>
/// <returns>Issuer certificate from built chain, or <see langword="null"/>.</returns>
public static X509Certificate2? GetLeafIssuer(X509Certificate2 leaf, X509Certificate2 trustedRoot)
{
using var chain = new X509Chain();
@@ -230,6 +290,9 @@ public static class TlsHelper
/// <summary>
/// Checks OCSP response currency semantics with clock skew and fallback TTL.
/// </summary>
/// <param name="response">Parsed OCSP response timestamps.</param>
/// <param name="opts">OCSP peer-validation options.</param>
/// <returns><see langword="true"/> when response timestamps are currently valid.</returns>
public static bool OcspResponseCurrent(OcspResponseInfo response, OCSPPeerConfig opts)
{
var skew = TimeSpan.FromSeconds(opts.ClockSkew);
@@ -261,6 +324,9 @@ public static class TlsHelper
/// Validates OCSP delegated signer semantics. Direct issuer signatures are valid;
/// delegated certificates must include id-kp-OCSPSigning EKU.
/// </summary>
/// <param name="issuer">Issuer certificate.</param>
/// <param name="responderCertificate">Optional delegated OCSP responder certificate.</param>
/// <returns><see langword="true"/> when delegation/signing semantics are valid.</returns>
public static bool ValidDelegationCheck(X509Certificate2? issuer, X509Certificate2? responderCertificate)
{
if (issuer is null)
+28
View File
@@ -20,9 +20,20 @@ public sealed class WsConnection : Stream
private readonly object _writeLock = new();
private readonly List<ControlFrameAction> _pendingControlWrites = [];
/// <summary>Indicates whether a close frame has been received from the remote peer.</summary>
public bool CloseReceived => _readInfo.CloseReceived;
/// <summary>WebSocket close status code received from the remote peer.</summary>
public int CloseStatus => _readInfo.CloseStatus;
/// <summary>
/// Initializes a WebSocket framed stream wrapper over an existing transport stream.
/// </summary>
/// <param name="inner">Underlying network stream.</param>
/// <param name="compress">Whether outbound frames may use per-message compression.</param>
/// <param name="maskRead">Whether inbound frames are expected to be masked.</param>
/// <param name="maskWrite">Whether outbound frames should be masked.</param>
/// <param name="browser">Whether browser framing constraints should be applied.</param>
/// <param name="noCompFrag">Whether compressed payloads should avoid browser fragmentation logic.</param>
public WsConnection(Stream inner, bool compress, bool maskRead, bool maskWrite, bool browser, bool noCompFrag)
{
_inner = inner;
@@ -34,6 +45,7 @@ public sealed class WsConnection : Stream
_readInfo = new WsReadInfo(expectMask: maskRead);
}
/// <inheritdoc />
public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken ct = default)
{
// Drain any buffered decoded payloads first
@@ -73,6 +85,7 @@ public sealed class WsConnection : Stream
}
}
/// <inheritdoc />
public override async ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken ct = default)
{
var data = buffer.Span;
@@ -145,6 +158,8 @@ public sealed class WsConnection : Stream
/// <summary>
/// Sends a WebSocket close frame.
/// </summary>
/// <param name="reason">Server close reason mapped to WebSocket close status and reason text.</param>
/// <param name="ct">Cancellation token for asynchronous close frame transmission.</param>
public async Task SendCloseAsync(ClientClosedReason reason, CancellationToken ct = default)
{
var status = WsFrameWriter.MapCloseStatus(reason);
@@ -175,18 +190,30 @@ public sealed class WsConnection : Stream
}
// Stream abstract members
/// <inheritdoc />
public override bool CanRead => true;
/// <inheritdoc />
public override bool CanWrite => true;
/// <inheritdoc />
public override bool CanSeek => false;
/// <inheritdoc />
public override long Length => throw new NotSupportedException();
/// <inheritdoc />
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
/// <inheritdoc />
public override void Flush() => _inner.Flush();
/// <inheritdoc />
public override Task FlushAsync(CancellationToken ct) => _inner.FlushAsync(ct);
/// <inheritdoc />
public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException("Use ReadAsync");
/// <inheritdoc />
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException("Use WriteAsync");
/// <inheritdoc />
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
/// <inheritdoc />
public override void SetLength(long value) => throw new NotSupportedException();
/// <inheritdoc />
protected override void Dispose(bool disposing)
{
if (disposing)
@@ -194,6 +221,7 @@ public sealed class WsConnection : Stream
base.Dispose(disposing);
}
/// <inheritdoc />
public override async ValueTask DisposeAsync()
{
await _inner.DisposeAsync();
@@ -14,6 +14,11 @@ public static class WsFrameWriter
/// Creates a complete frame header for a single-frame message (first=true, final=true).
/// Returns (header bytes, mask key or null).
/// </summary>
/// <param name="useMasking">Whether to include a masking key and mask payload bytes.</param>
/// <param name="compressed">Whether to set RSV1 for permessage-deflate.</param>
/// <param name="opcode">WebSocket opcode for the frame.</param>
/// <param name="payloadLength">Payload length in bytes.</param>
/// <returns>Encoded header bytes and optional mask key.</returns>
public static (byte[] header, byte[]? key) CreateFrameHeader(
bool useMasking, bool compressed, int opcode, int payloadLength)
{
@@ -27,6 +32,14 @@ public static class WsFrameWriter
/// Fills a pre-allocated frame header buffer.
/// Returns (bytes written, mask key or null).
/// </summary>
/// <param name="fh">Destination buffer for header bytes.</param>
/// <param name="useMasking">Whether to include masking metadata.</param>
/// <param name="first">Whether this is the first frame in a message.</param>
/// <param name="final">Whether this is the final frame in a message.</param>
/// <param name="compressed">Whether to set RSV1 for permessage-deflate.</param>
/// <param name="opcode">WebSocket opcode for the frame.</param>
/// <param name="payloadLength">Payload length in bytes.</param>
/// <returns>Number of bytes written and optional mask key.</returns>
public static (int written, byte[]? key) FillFrameHeader(
Span<byte> fh, bool useMasking, bool first, bool final, bool compressed, int opcode, int payloadLength)
{
@@ -74,6 +87,8 @@ public static class WsFrameWriter
/// <summary>
/// XOR masks a buffer with a 4-byte key. Applies in-place.
/// </summary>
/// <param name="key">Four-byte masking key.</param>
/// <param name="buf">Buffer to mask in place.</param>
public static void MaskBuf(ReadOnlySpan<byte> key, Span<byte> buf)
{
for (int i = 0; i < buf.Length; i++)
@@ -83,6 +98,8 @@ public static class WsFrameWriter
/// <summary>
/// XOR masks multiple contiguous buffers as if they were one.
/// </summary>
/// <param name="key">Four-byte masking key.</param>
/// <param name="bufs">Buffers to mask as a contiguous payload sequence.</param>
public static void MaskBufs(ReadOnlySpan<byte> key, List<byte[]> bufs)
{
int pos = 0;
@@ -100,6 +117,9 @@ public static class WsFrameWriter
/// Creates a close message payload: 2-byte status code + optional UTF-8 body.
/// Body truncated to fit MaxControlPayloadSize with "..." suffix.
/// </summary>
/// <param name="status">WebSocket close status code.</param>
/// <param name="body">Optional close reason text.</param>
/// <returns>Encoded close payload bytes.</returns>
public static byte[] CreateCloseMessage(int status, string body)
{
var bodyBytes = Encoding.UTF8.GetBytes(body);
@@ -128,6 +148,10 @@ public static class WsFrameWriter
/// <summary>
/// Builds a complete control frame (header + payload, optional masking).
/// </summary>
/// <param name="opcode">Control frame opcode.</param>
/// <param name="payload">Control frame payload bytes.</param>
/// <param name="useMasking">Whether to apply client-style masking.</param>
/// <returns>Complete encoded control frame.</returns>
public static byte[] BuildControlFrame(int opcode, ReadOnlySpan<byte> payload, bool useMasking)
{
int headerSize = 2 + (useMasking ? 4 : 0);
@@ -149,6 +173,8 @@ public static class WsFrameWriter
/// Maps a ClientClosedReason to a WebSocket close status code.
/// Matches Go wsEnqueueCloseMessage in websocket.go lines 668-694.
/// </summary>
/// <param name="reason">Connection close reason.</param>
/// <returns>WebSocket close status code.</returns>
public static int MapCloseStatus(ClientClosedReason reason) => reason switch
{
ClientClosedReason.ClientClosed => WsConstants.CloseStatusNormalClosure,
@@ -16,7 +16,7 @@ public class OrderedConsumerTests(JetStreamServerPairFixture fixture, ITestOutpu
public async Task JSOrderedConsumer_Throughput()
{
const int payloadSize = 128;
const int messageCount = 25_000;
const int messageCount = 200_000;
BenchmarkResult? dotnetResult = null;
try