natsdotnet/Documentation/GettingStarted/Architecture.md

Architecture

This document describes the overall architecture of the NATS .NET server — its layers, component responsibilities, message flow, and the mapping from the Go reference implementation.

Project Overview

This project is a port of the NATS server (golang/nats-server/) to .NET 10 / C#. The Go source in golang/nats-server/server/ is the authoritative reference.

Current scope: core pub/sub with wildcard subject matching and queue groups; authentication (username/password, token, NKey, JWT, TLS client certificate mapping); TLS transport; WebSocket transport; config file parsing with hot reload; clustering via routes (in-process subscription propagation and message routing); gateway and leaf node managers (bootstrapped, protocol stubs); JetStream (streams, consumers, file and memory storage, RAFT consensus); and HTTP monitoring endpoints (/varz, /connz, /routez, /jsz, etc.).

---

Solution Layout

NatsDotNet.slnx
src/
  NATS.Server/              # Core server library — no executable entry point
    Auth/                   # Auth mechanisms: username/password, token, NKey, JWT, TLS mapping
    Configuration/          # Config file lexer/parser, ClusterOptions, JetStreamOptions, etc.
    Events/                 # Internal event system (connect/disconnect advisory subjects)
    Gateways/               # GatewayManager, GatewayConnection (inter-cluster bridge)
    Imports/                # Account import/export maps, service latency tracking
    JetStream/              # Streams, consumers, storage, API routing, RAFT meta-group
    LeafNodes/              # LeafNodeManager, LeafConnection (hub-and-spoke topology)
    Monitoring/             # HTTP monitoring server: /varz, /connz, /jsz, /subsz
    Protocol/               # NatsParser state machine, NatsProtocol constants and wire helpers
    Raft/                   # RaftNode, RaftLog, RaftReplicator, snapshot support
    Routes/                 # RouteManager, RouteConnection (full-mesh cluster routes)
    Subscriptions/          # SubList trie, SubjectMatch, Subscription, SubListResult
    Tls/                    # TLS handshake wrapper, OCSP stapling, TlsRateLimiter
    WebSocket/              # WsUpgrade, WsConnection, frame writer and compression
    NatsClient.cs           # Per-connection client: I/O pipeline, command dispatch, sub tracking
    NatsServer.cs           # Server orchestrator: accept loop, client registry, message routing
    NatsOptions.cs          # Top-level configuration model
  NATS.Server.Host/         # Console application — wires logging, parses CLI args, starts server
tests/
  NATS.Server.Tests/        # xUnit test project — 92 .cs test files covering all subsystems

NATS.Server depends only on Microsoft.Extensions.Logging.Abstractions. All Serilog wiring is in NATS.Server.Host. This keeps the core library testable without a console host.

---

Layers

The implementation is organized bottom-up: each layer depends only on the layers below it.

NatsServer          (orchestrator: accept loop, client registry, message routing)
    └── NatsClient  (per-connection: I/O pipeline, command dispatch, subscription tracking)
            └── NatsParser      (protocol state machine: bytes → ParsedCommand)
            └── SubList         (trie: subject → matching subscriptions)
                    └── SubjectMatch    (validation and wildcard matching primitives)

SubjectMatch and SubList

SubjectMatch (Subscriptions/SubjectMatch.cs) provides the primitive operations used throughout the subscription system:

• IsValidSubject(string) — rejects empty tokens, whitespace, tokens after >
• IsLiteral(string) — returns false if the subject contains a bare * or > wildcard token
• IsValidPublishSubject(string) — combines both; publish subjects must be literal
• MatchLiteral(string literal, string pattern) — token-by-token matching for cache maintenance

SubList (Subscriptions/SubList.cs) is a trie-based subscription store. Each trie level (TrieLevel) holds a Dictionary<string, TrieNode> for literal tokens plus dedicated Pwc and Fwc pointers for * and > wildcards. Each TrieNode holds PlainSubs (a HashSet<Subscription>) and QueueSubs (a Dictionary<string, HashSet<Subscription>> keyed by queue group name).

SubList.Match(string subject) checks an in-memory cache first, then falls back to a recursive trie walk (MatchLevel) if there is a cache miss. The result is stored as a SubListResult — an immutable snapshot containing PlainSubs (array) and QueueSubs (jagged array, one sub-array per group).

The cache holds up to 1,024 entries. When that limit is exceeded, 256 entries are swept. Wildcard subscriptions invalidate all matching cache keys on insert and removal; literal subscriptions invalidate only their own key.

NatsParser

NatsParser (Protocol/NatsParser.cs) is a stateless-per-invocation parser that operates on ReadOnlySequence<byte> from System.IO.Pipelines. The public entry point is:

public bool TryParse(ref ReadOnlySequence<byte> buffer, out ParsedCommand command)

It reads one control line at a time (up to NatsProtocol.MaxControlLineSize = 4,096 bytes), identifies the command by the first two lowercased bytes, then dispatches to a per-command parse method. Commands with payloads (PUB, HPUB) set _awaitingPayload = true and return on the next call via TryReadPayload once enough bytes arrive. This handles TCP fragmentation without buffering the entire payload before parsing begins.

ParsedCommand is a readonly struct — zero heap allocation for control-only commands (PING, PONG, SUB, UNSUB, CONNECT). Payload commands allocate a byte[] for the payload body.

Command dispatch in NatsClient.DispatchCommandAsync covers: Connect, Ping/Pong, Sub, Unsub, Pub, HPub.

NatsClient

NatsClient (NatsClient.cs) handles a single TCP connection. On RunAsync, it sends the initial INFO frame and then starts two concurrent tasks: FillPipeAsync (socket → PipeWriter) and ProcessCommandsAsync (PipeReader → parser → dispatch). The tasks share a Pipe from System.IO.Pipelines. Either task completing (EOF, cancellation, or error) causes RunAsync to return, which triggers cleanup via Router.RemoveClient(this).

Key fields:

public sealed class NatsClient : INatsClient, IDisposable
{
    private readonly Socket _socket;
    private readonly Stream _stream;           // plain NetworkStream or TlsConnectionWrapper
    private readonly NatsParser _parser;
    private readonly Channel<ReadOnlyMemory<byte>> _outbound = Channel.CreateBounded<ReadOnlyMemory<byte>>(
        new BoundedChannelOptions(8192) { SingleReader = true, FullMode = BoundedChannelFullMode.Wait });
    private long _pendingBytes;                // bytes queued but not yet written
    private readonly ClientFlagHolder _flags = new();  // ConnectReceived, TraceMode, etc.
    private readonly Dictionary<string, Subscription> _subs = new();

    public ulong Id { get; }
    public ClientKind Kind { get; }            // CLIENT, ROUTER, LEAF, SYSTEM
    public Account? Account { get; private set; }
}

Write serialization uses a bounded Channel<ReadOnlyMemory<byte>>(8192) (_outbound). All outbound message deliveries enqueue a pre-encoded frame into this channel. A dedicated write loop drains the channel sequentially, preventing interleaved writes from concurrent message deliveries. A _pendingBytes counter tracks bytes queued but not yet written, enabling slow-consumer detection and back-pressure enforcement.

Subscription state is a Dictionary<string, Subscription> keyed by SID. This dictionary is accessed only from the single processing task, so no locking is needed. SUB inserts into this dictionary and into SubList; UNSUB either sets MaxMessages for auto-unsubscribe or immediately removes from both.

NatsClient exposes two interfaces to NatsServer:

public interface IMessageRouter
{
    void ProcessMessage(string subject, string? replyTo,
        ReadOnlyMemory<byte> headers, ReadOnlyMemory<byte> payload, NatsClient sender);
    void RemoveClient(NatsClient client);
}

public interface ISubListAccess
{
    SubList SubList { get; }
}

NatsServer implements both. NatsClient.Router is set to the server instance immediately after construction.

NatsServer

NatsServer (NatsServer.cs) owns the TCP listener, the shared SubList, and the client registry. Each accepted connection gets a unique clientId (incremented via Interlocked.Increment), a scoped logger, and a NatsClient instance registered in _clients. RunClientAsync is fired as a detached task — the accept loop does not await it.

Key fields:

public sealed class NatsServer : IMessageRouter, ISubListAccess, IDisposable
{
    // Client registry
    private readonly ConcurrentDictionary<ulong, NatsClient> _clients = new();
    private readonly ConcurrentQueue<ClosedClient> _closedClients = new();
    private ulong _nextClientId;
    private int _activeClientCount;

    // Account system
    private readonly ConcurrentDictionary<string, Account> _accounts = new(StringComparer.Ordinal);
    private readonly Account _globalAccount;
    private readonly Account _systemAccount;
    private AuthService _authService;

    // Subsystem managers
    private readonly RouteManager? _routeManager;
    private readonly GatewayManager? _gatewayManager;
    private readonly LeafNodeManager? _leafNodeManager;
    private readonly JetStreamService? _jetStreamService;
    private MonitorServer? _monitorServer;

    // TLS / transport
    private readonly SslServerAuthenticationOptions? _sslOptions;
    private readonly TlsRateLimiter? _tlsRateLimiter;
    private Socket? _listener;
    private Socket? _wsListener;

    // Shutdown coordination
    private readonly CancellationTokenSource _quitCts = new();
    private readonly TaskCompletionSource _shutdownComplete = new(TaskCreationOptions.RunContinuationsAsynchronously);
    private int _shutdown;
    private int _lameDuck;
}

Message delivery happens in ProcessMessage:

1. Call _subList.Match(subject) to get a SubListResult.
2. Iterate result.PlainSubs — deliver to each subscriber, skipping the sender if echo is false.
3. Iterate result.QueueSubs — for each group, use round-robin (modulo Interlocked.Increment) to pick one member to receive the message.

Delivery via SendMessageAsync is fire-and-forget (not awaited) to avoid blocking the publishing client's processing task while waiting for slow subscribers.

---

Message Flow

Client sends: PUB orders.new 12\r\nhello world\r\n

1. FillPipeAsync      reads bytes from socket → PipeWriter
2. ProcessCommandsAsync reads from PipeReader
3. NatsParser.TryParse  parses control line "PUB orders.new 12\r\n"
                         sets _awaitingPayload = true
                         on next call: reads payload + \r\n trailer
                         returns ParsedCommand { Type=Pub, Subject="orders.new", Payload=[...] }
4. DispatchCommandAsync  calls ProcessPub(cmd)
5. ProcessPub            calls Router.ProcessMessage("orders.new", null, default, payload, this)
6. NatsServer.ProcessMessage
       → _subList.Match("orders.new")
           returns SubListResult { PlainSubs=[sub1, sub2], QueueSubs=[[sub3, sub4]] }
       → DeliverMessage(sub1, ...)   → sub1.Client.SendMessageAsync(...)
       → DeliverMessage(sub2, ...)   → sub2.Client.SendMessageAsync(...)
       → round-robin pick from [sub3, sub4], e.g. sub3
       → DeliverMessage(sub3, ...)   → sub3.Client.SendMessageAsync(...)
7. SendMessageAsync  enqueues encoded MSG frame into _outbound channel; write loop drains to socket

---

Go Reference Mapping

Go source	.NET source
server/sublist.go	src/NATS.Server/Subscriptions/SubList.cs
server/parser.go	src/NATS.Server/Protocol/NatsParser.cs
server/client.go	src/NATS.Server/NatsClient.cs
server/server.go	src/NATS.Server/NatsServer.cs
server/opts.go	src/NATS.Server/NatsOptions.cs + src/NATS.Server/Configuration/
server/auth.go	src/NATS.Server/Auth/AuthService.cs
server/route.go	src/NATS.Server/Routes/RouteManager.cs
server/gateway.go	src/NATS.Server/Gateways/GatewayManager.cs
server/leafnode.go	src/NATS.Server/LeafNodes/LeafNodeManager.cs
server/jetstream.go	src/NATS.Server/JetStream/JetStreamService.cs
server/stream.go	src/NATS.Server/JetStream/StreamManager.cs (via JetStreamService)
server/consumer.go	src/NATS.Server/JetStream/ConsumerManager.cs
server/raft.go	src/NATS.Server/Raft/RaftNode.cs
server/monitor.go	src/NATS.Server/Monitoring/MonitorServer.cs

The Go sublist.go uses atomic generation counters to invalidate a result cache. The .NET SubList uses a different strategy: it maintains the cache under ReaderWriterLockSlim and does targeted invalidation at insert/remove time, avoiding the need for generation counters.

The Go client.go uses goroutines for readLoop and writeLoop. The .NET equivalent uses async Task with System.IO.Pipelines: FillPipeAsync (writer side) and ProcessCommandsAsync (reader side).

---

Key .NET Design Choices

Concern	.NET choice	Reason
I/O buffering	System.IO.Pipelines (Pipe, PipeReader, PipeWriter)	Zero-copy buffer management; backpressure built in
SubList thread safety	ReaderWriterLockSlim	Multiple concurrent readers (match), exclusive writers (insert/remove)
Client registry	ConcurrentDictionary<ulong, NatsClient>	Lock-free concurrent access from accept loop and cleanup tasks
Write serialization	Channel<ReadOnlyMemory<byte>>(8192) bounded queue per client with _pendingBytes slow-consumer tracking	Sequential drain by a single writer task prevents interleaved MSG frames; bounded capacity enables back-pressure
Concurrency	async/await + Task	Maps Go goroutines to .NET task-based async; no dedicated threads per connection
Protocol constants	NatsProtocol static class	Pre-encoded byte arrays (PongBytes, CrLf, etc.) avoid per-call allocations

---

Related Documentation

• Setup
• Protocol Overview
• Subscriptions Overview
• Server Overview
• Configuration Overview