docs: add Authentication, Clustering, JetStream, Monitoring overviews; update existing docs
New files: - Documentation/Authentication/Overview.md — all 7 auth mechanisms with real source snippets (NKey/JWT/username-password/token/TLS mapping), nonce generation, account system, permissions, JWT permission templates - Documentation/Clustering/Overview.md — route TCP handshake, in-process subscription propagation, gateway/leaf node stubs, honest gaps list - Documentation/JetStream/Overview.md — API surface (4 handled subjects), streams, consumers, storage (MemStore/FileStore), in-process RAFT, mirror/source, gaps list - Documentation/Monitoring/Overview.md — all 12 endpoints with real field tables, Go compatibility notes Updated files: - GettingStarted/Architecture.md — 14-subdirectory tree, real NatsClient/NatsServer field snippets, 9 new Go reference rows, Channel write queue design choice - GettingStarted/Setup.md — xUnit 3, 100 test files grouped by area - Operations/Overview.md — 99 test files, accurate Program.cs snippet, limitations section renamed to "Known Gaps vs Go Reference" with 7 real gaps - Server/Overview.md — grouped fields, TLS/WS accept path, lame-duck mode, POSIX signals - Configuration/Overview.md — 14 subsystem option tables, 24-row CLI table, LogOverrides - Server/Client.md — Channel write queue, 4-task RunAsync, CommandMatrix, real fields All docs verified against codebase 2026-02-23; 713 tests pass.
This commit is contained in:
Joseph Doherty
@@ -6,7 +6,7 @@ This document describes the overall architecture of the NATS .NET server — its
|
||||
|
||||
This project is a port of the [NATS server](https://github.com/nats-io/nats-server) (`golang/nats-server/`) to .NET 10 / C#. The Go source in `golang/nats-server/server/` is the authoritative reference.
|
||||
|
||||
Current scope: base publish-subscribe server with wildcard subject matching and queue groups. Authentication, clustering (routes, gateways, leaf nodes), JetStream, and HTTP monitoring are not yet implemented.
|
||||
Current scope: core pub/sub with wildcard subject matching and queue groups; authentication (username/password, token, NKey, JWT, TLS client certificate mapping); TLS transport; WebSocket transport; config file parsing with hot reload; clustering via routes (in-process subscription propagation and message routing); gateway and leaf node managers (bootstrapped, protocol stubs); JetStream (streams, consumers, file and memory storage, RAFT consensus); and HTTP monitoring endpoints (`/varz`, `/connz`, `/routez`, `/jsz`, etc.).
|
||||
|
||||
---
|
||||
|
||||
@@ -15,10 +15,27 @@ Current scope: base publish-subscribe server with wildcard subject matching and
|
||||
```
|
||||
NatsDotNet.slnx
|
||||
src/
|
||||
NATS.Server/ # Core server library — no executable entry point
|
||||
NATS.Server.Host/ # Console application — wires logging, parses CLI args, starts server
|
||||
NATS.Server/ # Core server library — no executable entry point
|
||||
Auth/ # Auth mechanisms: username/password, token, NKey, JWT, TLS mapping
|
||||
Configuration/ # Config file lexer/parser, ClusterOptions, JetStreamOptions, etc.
|
||||
Events/ # Internal event system (connect/disconnect advisory subjects)
|
||||
Gateways/ # GatewayManager, GatewayConnection (inter-cluster bridge)
|
||||
Imports/ # Account import/export maps, service latency tracking
|
||||
JetStream/ # Streams, consumers, storage, API routing, RAFT meta-group
|
||||
LeafNodes/ # LeafNodeManager, LeafConnection (hub-and-spoke topology)
|
||||
Monitoring/ # HTTP monitoring server: /varz, /connz, /jsz, /subsz
|
||||
Protocol/ # NatsParser state machine, NatsProtocol constants and wire helpers
|
||||
Raft/ # RaftNode, RaftLog, RaftReplicator, snapshot support
|
||||
Routes/ # RouteManager, RouteConnection (full-mesh cluster routes)
|
||||
Subscriptions/ # SubList trie, SubjectMatch, Subscription, SubListResult
|
||||
Tls/ # TLS handshake wrapper, OCSP stapling, TlsRateLimiter
|
||||
WebSocket/ # WsUpgrade, WsConnection, frame writer and compression
|
||||
NatsClient.cs # Per-connection client: I/O pipeline, command dispatch, sub tracking
|
||||
NatsServer.cs # Server orchestrator: accept loop, client registry, message routing
|
||||
NatsOptions.cs # Top-level configuration model
|
||||
NATS.Server.Host/ # Console application — wires logging, parses CLI args, starts server
|
||||
tests/
|
||||
NATS.Server.Tests/ # xUnit test project — unit and integration tests
|
||||
NATS.Server.Tests/ # xUnit test project — 92 .cs test files covering all subsystems
|
||||
```
|
||||
|
||||
`NATS.Server` depends only on `Microsoft.Extensions.Logging.Abstractions`. All Serilog wiring is in `NATS.Server.Host`. This keeps the core library testable without a console host.
|
||||
@@ -68,16 +85,29 @@ Command dispatch in `NatsClient.DispatchCommandAsync` covers: `Connect`, `Ping`/
|
||||
|
||||
### NatsClient
|
||||
|
||||
`NatsClient` (`NatsClient.cs`) handles a single TCP connection. On `RunAsync`, it sends the initial `INFO` frame and then starts two concurrent tasks:
|
||||
`NatsClient` (`NatsClient.cs`) handles a single TCP connection. On `RunAsync`, it sends the initial `INFO` frame and then starts two concurrent tasks: `FillPipeAsync` (socket → `PipeWriter`) and `ProcessCommandsAsync` (`PipeReader` → parser → dispatch). The tasks share a `Pipe` from `System.IO.Pipelines`. Either task completing (EOF, cancellation, or error) causes `RunAsync` to return, which triggers cleanup via `Router.RemoveClient(this)`.
|
||||
|
||||
Key fields:
|
||||
|
||||
```csharp
|
||||
var fillTask = FillPipeAsync(pipe.Writer, ct); // socket → PipeWriter
|
||||
var processTask = ProcessCommandsAsync(pipe.Reader, ct); // PipeReader → parser → dispatch
|
||||
public sealed class NatsClient : INatsClient, IDisposable
|
||||
{
|
||||
private readonly Socket _socket;
|
||||
private readonly Stream _stream; // plain NetworkStream or TlsConnectionWrapper
|
||||
private readonly NatsParser _parser;
|
||||
private readonly Channel<ReadOnlyMemory<byte>> _outbound = Channel.CreateBounded<ReadOnlyMemory<byte>>(
|
||||
new BoundedChannelOptions(8192) { SingleReader = true, FullMode = BoundedChannelFullMode.Wait });
|
||||
private long _pendingBytes; // bytes queued but not yet written
|
||||
private readonly ClientFlagHolder _flags = new(); // ConnectReceived, TraceMode, etc.
|
||||
private readonly Dictionary<string, Subscription> _subs = new();
|
||||
|
||||
public ulong Id { get; }
|
||||
public ClientKind Kind { get; } // CLIENT, ROUTER, LEAF, SYSTEM
|
||||
public Account? Account { get; private set; }
|
||||
}
|
||||
```
|
||||
|
||||
`FillPipeAsync` reads from the `NetworkStream` into a `PipeWriter` in 4,096-byte chunks. `ProcessCommandsAsync` reads from the `PipeReader`, calls `NatsParser.TryParse` in a loop, and dispatches each `ParsedCommand`. The tasks share a `Pipe` instance from `System.IO.Pipelines`. Either task completing (EOF, cancellation, or error) causes `RunAsync` to return, which triggers cleanup via `Router.RemoveClient(this)`.
|
||||
|
||||
Write serialization uses a `SemaphoreSlim(1,1)` (`_writeLock`). All outbound writes (`SendMessageAsync`, `WriteAsync`) acquire this lock before touching the `NetworkStream`, preventing interleaved writes from concurrent message deliveries.
|
||||
Write serialization uses a bounded `Channel<ReadOnlyMemory<byte>>(8192)` (`_outbound`). All outbound message deliveries enqueue a pre-encoded frame into this channel. A dedicated write loop drains the channel sequentially, preventing interleaved writes from concurrent message deliveries. A `_pendingBytes` counter tracks bytes queued but not yet written, enabling slow-consumer detection and back-pressure enforcement.
|
||||
|
||||
Subscription state is a `Dictionary<string, Subscription>` keyed by SID. This dictionary is accessed only from the single processing task, so no locking is needed. `SUB` inserts into this dictionary and into `SubList`; `UNSUB` either sets `MaxMessages` for auto-unsubscribe or immediately removes from both.
|
||||
|
||||
@@ -101,27 +131,46 @@ public interface ISubListAccess
|
||||
|
||||
### NatsServer
|
||||
|
||||
`NatsServer` (`NatsServer.cs`) owns the TCP listener, the shared `SubList`, and the client registry. Its `StartAsync` method runs the accept loop:
|
||||
`NatsServer` (`NatsServer.cs`) owns the TCP listener, the shared `SubList`, and the client registry. Each accepted connection gets a unique `clientId` (incremented via `Interlocked.Increment`), a scoped logger, and a `NatsClient` instance registered in `_clients`. `RunClientAsync` is fired as a detached task — the accept loop does not await it.
|
||||
|
||||
Key fields:
|
||||
|
||||
```csharp
|
||||
public async Task StartAsync(CancellationToken ct)
|
||||
public sealed class NatsServer : IMessageRouter, ISubListAccess, IDisposable
|
||||
{
|
||||
_listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
|
||||
_listener.Bind(new IPEndPoint(
|
||||
_options.Host == "0.0.0.0" ? IPAddress.Any : IPAddress.Parse(_options.Host),
|
||||
_options.Port));
|
||||
_listener.Listen(128);
|
||||
// ...
|
||||
while (!ct.IsCancellationRequested)
|
||||
{
|
||||
var socket = await _listener.AcceptAsync(ct);
|
||||
// create NatsClient, fire-and-forget RunClientAsync
|
||||
}
|
||||
// Client registry
|
||||
private readonly ConcurrentDictionary<ulong, NatsClient> _clients = new();
|
||||
private readonly ConcurrentQueue<ClosedClient> _closedClients = new();
|
||||
private ulong _nextClientId;
|
||||
private int _activeClientCount;
|
||||
|
||||
// Account system
|
||||
private readonly ConcurrentDictionary<string, Account> _accounts = new(StringComparer.Ordinal);
|
||||
private readonly Account _globalAccount;
|
||||
private readonly Account _systemAccount;
|
||||
private AuthService _authService;
|
||||
|
||||
// Subsystem managers
|
||||
private readonly RouteManager? _routeManager;
|
||||
private readonly GatewayManager? _gatewayManager;
|
||||
private readonly LeafNodeManager? _leafNodeManager;
|
||||
private readonly JetStreamService? _jetStreamService;
|
||||
private MonitorServer? _monitorServer;
|
||||
|
||||
// TLS / transport
|
||||
private readonly SslServerAuthenticationOptions? _sslOptions;
|
||||
private readonly TlsRateLimiter? _tlsRateLimiter;
|
||||
private Socket? _listener;
|
||||
private Socket? _wsListener;
|
||||
|
||||
// Shutdown coordination
|
||||
private readonly CancellationTokenSource _quitCts = new();
|
||||
private readonly TaskCompletionSource _shutdownComplete = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
private int _shutdown;
|
||||
private int _lameDuck;
|
||||
}
|
||||
```
|
||||
|
||||
Each accepted connection gets a unique `clientId` (incremented via `Interlocked.Increment`), a scoped logger, and a `NatsClient` instance registered in `_clients` (`ConcurrentDictionary<ulong, NatsClient>`). `RunClientAsync` is fired as a detached task — the accept loop does not await it.
|
||||
|
||||
Message delivery happens in `ProcessMessage`:
|
||||
|
||||
1. Call `_subList.Match(subject)` to get a `SubListResult`.
|
||||
@@ -152,7 +201,7 @@ Client sends: PUB orders.new 12\r\nhello world\r\n
|
||||
→ DeliverMessage(sub2, ...) → sub2.Client.SendMessageAsync(...)
|
||||
→ round-robin pick from [sub3, sub4], e.g. sub3
|
||||
→ DeliverMessage(sub3, ...) → sub3.Client.SendMessageAsync(...)
|
||||
7. SendMessageAsync acquires _writeLock, writes MSG frame to socket
|
||||
7. SendMessageAsync enqueues encoded MSG frame into _outbound channel; write loop drains to socket
|
||||
```
|
||||
|
||||
---
|
||||
@@ -165,7 +214,16 @@ Client sends: PUB orders.new 12\r\nhello world\r\n
|
||||
| `server/parser.go` | `src/NATS.Server/Protocol/NatsParser.cs` |
|
||||
| `server/client.go` | `src/NATS.Server/NatsClient.cs` |
|
||||
| `server/server.go` | `src/NATS.Server/NatsServer.cs` |
|
||||
| `server/opts.go` | `src/NATS.Server/NatsOptions.cs` |
|
||||
| `server/opts.go` | `src/NATS.Server/NatsOptions.cs` + `src/NATS.Server/Configuration/` |
|
||||
| `server/auth.go` | `src/NATS.Server/Auth/AuthService.cs` |
|
||||
| `server/route.go` | `src/NATS.Server/Routes/RouteManager.cs` |
|
||||
| `server/gateway.go` | `src/NATS.Server/Gateways/GatewayManager.cs` |
|
||||
| `server/leafnode.go` | `src/NATS.Server/LeafNodes/LeafNodeManager.cs` |
|
||||
| `server/jetstream.go` | `src/NATS.Server/JetStream/JetStreamService.cs` |
|
||||
| `server/stream.go` | `src/NATS.Server/JetStream/StreamManager.cs` (via `JetStreamService`) |
|
||||
| `server/consumer.go` | `src/NATS.Server/JetStream/ConsumerManager.cs` |
|
||||
| `server/raft.go` | `src/NATS.Server/Raft/RaftNode.cs` |
|
||||
| `server/monitor.go` | `src/NATS.Server/Monitoring/MonitorServer.cs` |
|
||||
|
||||
The Go `sublist.go` uses atomic generation counters to invalidate a result cache. The .NET `SubList` uses a different strategy: it maintains the cache under `ReaderWriterLockSlim` and does targeted invalidation at insert/remove time, avoiding the need for generation counters.
|
||||
|
||||
@@ -180,7 +238,7 @@ The Go `client.go` uses goroutines for `readLoop` and `writeLoop`. The .NET equi
|
||||
| I/O buffering | `System.IO.Pipelines` (`Pipe`, `PipeReader`, `PipeWriter`) | Zero-copy buffer management; backpressure built in |
|
||||
| SubList thread safety | `ReaderWriterLockSlim` | Multiple concurrent readers (match), exclusive writers (insert/remove) |
|
||||
| Client registry | `ConcurrentDictionary<ulong, NatsClient>` | Lock-free concurrent access from accept loop and cleanup tasks |
|
||||
| Write serialization | `SemaphoreSlim(1,1)` per client | Prevents interleaved MSG frames from concurrent deliveries |
|
||||
| Write serialization | `Channel<ReadOnlyMemory<byte>>(8192)` bounded queue per client with `_pendingBytes` slow-consumer tracking | Sequential drain by a single writer task prevents interleaved MSG frames; bounded capacity enables back-pressure |
|
||||
| Concurrency | `async/await` + `Task` | Maps Go goroutines to .NET task-based async; no dedicated threads per connection |
|
||||
| Protocol constants | `NatsProtocol` static class | Pre-encoded byte arrays (`PongBytes`, `CrLf`, etc.) avoid per-call allocations |
|
||||
|
||||
@@ -194,4 +252,4 @@ The Go `client.go` uses goroutines for `readLoop` and `writeLoop`. The .NET equi
|
||||
- [Server Overview](../Server/Overview.md)
|
||||
- [Configuration Overview](../Configuration/Overview.md)
|
||||
|
||||
<!-- Last verified against codebase: 2026-02-22 -->
|
||||
<!-- Last verified against codebase: 2026-02-23 -->
|
||||
|
||||
Reference in New Issue
Block a user