37575dc41c
Side-by-side performance benchmarks using NATS.Client.Core against both servers on ephemeral ports. Includes core pub/sub, request/reply latency, and JetStream throughput tests with comparison output and benchmarks_comparison.md results. Also fixes timestamp flakiness in StoreInterfaceTests by using explicit timestamps.
213 lines
12 KiB
Markdown
213 lines
12 KiB
Markdown
# AGENTS.md
|
|
|
|
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
This project ports the [NATS server](https://github.com/nats-io/nats-server) from Go to .NET 10 / C#. The Go reference implementation lives in `golang/nats-server/`. The .NET port lives at the repository root.
|
|
|
|
NATS is a high-performance publish-subscribe messaging system. It supports wildcards (`*` single token, `>` multi-token), queue groups for load balancing, request-reply, clustering (full-mesh routes, gateways, leaf nodes), and persistent streaming via JetStream.
|
|
|
|
## Build & Test Commands
|
|
|
|
The solution file is `NatsDotNet.slnx`.
|
|
|
|
```bash
|
|
# Build the solution
|
|
dotnet build
|
|
|
|
# Run all tests
|
|
dotnet test
|
|
|
|
# Run tests with verbose output
|
|
dotnet test -v normal
|
|
|
|
# Run a single test project
|
|
dotnet test tests/NATS.Server.Tests
|
|
|
|
# Run a specific test project
|
|
dotnet test tests/NATS.Server.Core.Tests
|
|
dotnet test tests/NATS.Server.JetStream.Tests
|
|
|
|
# Run a specific test by name
|
|
dotnet test tests/NATS.Server.Core.Tests --filter "FullyQualifiedName~TestName"
|
|
|
|
# Run the NATS server (default port 4222)
|
|
dotnet run --project src/NATS.Server.Host
|
|
|
|
# Run the NATS server on a custom port
|
|
dotnet run --project src/NATS.Server.Host -- -p 14222
|
|
|
|
# Clean and rebuild
|
|
dotnet clean && dotnet build
|
|
```
|
|
|
|
## .NET Project Structure
|
|
|
|
```
|
|
NatsDotNet.slnx # Solution file
|
|
src/
|
|
NATS.Server/ # Core server library
|
|
NatsServer.cs # Server: listener, accept loop, shutdown
|
|
NatsClient.cs # Per-connection client: read/write loops, sub tracking
|
|
NatsOptions.cs # Server configuration (port, host, etc.)
|
|
Protocol/
|
|
NatsParser.cs # Protocol state machine (PUB, SUB, UNSUB, etc.)
|
|
NatsProtocol.cs # Wire-level protocol writing (INFO, MSG, PING/PONG)
|
|
Subscriptions/
|
|
SubjectMatch.cs # Subject validation and wildcard matching
|
|
SubList.cs # Trie-based subscription list with caching
|
|
SubListResult.cs # Match result container (plain subs + queue groups)
|
|
Subscription.cs # Subscription model (subject, sid, queue, client)
|
|
NATS.Server.Host/ # Executable host app
|
|
Program.cs # Entry point, CLI arg parsing (-p port)
|
|
tests/
|
|
NATS.Server.TestUtilities/ # Shared helpers, fixtures, parity tools (class library)
|
|
NATS.Server.Core.Tests/ # Client, server, parser, config, subscriptions, protocol
|
|
NATS.Server.Auth.Tests/ # Auth, accounts, permissions, JWT, NKeys
|
|
NATS.Server.JetStream.Tests/ # JetStream API, streams, consumers, storage, cluster
|
|
NATS.Server.Raft.Tests/ # RAFT consensus
|
|
NATS.Server.Clustering.Tests/ # Routes, cluster topology, inter-server protocol
|
|
NATS.Server.Gateways.Tests/ # Gateway connections, interest modes
|
|
NATS.Server.LeafNodes.Tests/ # Leaf node connections, hub-spoke
|
|
NATS.Server.Mqtt.Tests/ # MQTT protocol bridge
|
|
NATS.Server.Monitoring.Tests/ # Monitor endpoints, events, system events
|
|
NATS.Server.Transport.Tests/ # WebSocket, TLS, OCSP, IO
|
|
NATS.E2E.Tests/ # End-to-end tests using NATS.Client.Core NuGet
|
|
```
|
|
|
|
## Go Reference Commands
|
|
|
|
```bash
|
|
# Build the Go reference server
|
|
cd golang/nats-server && go build
|
|
|
|
# Run Go tests for a specific area
|
|
cd golang/nats-server && go test -v -run TestName ./server/ -count=1 -timeout=30m
|
|
|
|
# Run all Go server tests (slow, ~30min)
|
|
cd golang/nats-server && go test -v ./server/ -count=1 -timeout=30m
|
|
```
|
|
|
|
## Architecture: NATS Server (Reference)
|
|
|
|
The Go source in `golang/nats-server/server/` is the authoritative reference. Key files by subsystem:
|
|
|
|
### Core Message Path
|
|
- **`server.go`** — Server struct, startup lifecycle (`NewServer` → `Run` → `WaitForShutdown`), listener management
|
|
- **`client.go`** (6700 lines) — Connection handling, `readLoop`/`writeLoop` goroutines, per-client subscription tracking, dynamic buffer sizing (512→65536 bytes), client types: `CLIENT`, `ROUTER`, `GATEWAY`, `LEAF`, `SYSTEM`
|
|
- **`parser.go`** — Protocol state machine. Text protocol: `PUB`, `SUB`, `UNSUB`, `CONNECT`, `INFO`, `PING/PONG`, `MSG`. Extended: `HPUB/HMSG` (headers), `RPUB/RMSG` (routes). Control line limit: 4096 bytes. Default max payload: 1MB.
|
|
- **`sublist.go`** — Trie-based subject matcher with wildcard support. Nodes have `psubs` (plain), `qsubs` (queue groups), special pointers for `*` and `>` wildcards. Results are cached with atomic generation IDs for invalidation.
|
|
|
|
### Authentication & Accounts
|
|
- **`auth.go`** — Auth mechanisms: username/password, token, NKeys (Ed25519), JWT, external auth callout, LDAP
|
|
- **`accounts.go`** (137KB) — Multi-tenant account isolation. Each account has its own `Sublist`, client set, and subject namespace. Supports exports/imports between accounts, service latency tracking.
|
|
- **`jwt.go`**, **`nkey.go`** — JWT claims parsing and NKey validation
|
|
|
|
### Clustering
|
|
- **`route.go`** — Full-mesh cluster routes. Route pooling (default 3 connections per peer). Account-specific dedicated routes. Protocol: `RS+`/`RS-` for subscribe propagation, `RMSG` for routed messages.
|
|
- **`gateway.go`** (103KB) — Inter-cluster bridges. Interest-only mode optimizes traffic. Reply subject mapping (`_GR_.` prefix) avoids cross-cluster conflicts.
|
|
- **`leafnode.go`** — Hub-and-spoke topology for edge deployments. Only subscribed subjects shared with hub. Loop detection via `$LDS.` prefix.
|
|
|
|
### JetStream (Persistence)
|
|
- **`jetstream.go`** — Orchestration, API subject handlers (`$JS.API.*`)
|
|
- **`stream.go`** (8000 lines) — Stream lifecycle, retention policies (Limits, Interest, WorkQueue), subject transforms, mirroring/sourcing
|
|
- **`consumer.go`** — Stateful readers. Push vs pull delivery. Ack policies: None, All, Explicit. Redelivery tracking, priority groups.
|
|
- **`filestore.go`** (337KB) — Block-based persistent storage with S2 compression, encryption (ChaCha20/AES-GCM), indexing
|
|
- **`memstore.go`** — In-memory storage with hash-wheel TTL expiration
|
|
- **`raft.go`** — RAFT consensus for clustered JetStream. Meta-cluster for metadata, per-stream/consumer RAFT groups.
|
|
|
|
### Configuration & Monitoring
|
|
- **`opts.go`** — CLI flags + config file loading. CLI overrides config. Supports hot reload on signal.
|
|
- **`monitor.go`** — HTTP endpoints: `/varz`, `/connz`, `/routez`, `/gatewayz`, `/jsz`, `/healthz`
|
|
- **`conf/`** — Config file parser (custom format with includes)
|
|
|
|
### Internal Data Structures
|
|
- **`server/avl/`** — AVL tree for sparse sequence sets (ack tracking)
|
|
- **`server/stree/`** — Subject tree for per-subject state in streams
|
|
- **`server/gsl/`** — Generic subject list, optimized trie
|
|
- **`server/thw/`** — Time hash wheel for efficient TTL expiration
|
|
|
|
## Key Porting Considerations
|
|
|
|
**Concurrency model:** Go uses goroutines (one per connection readLoop + writeLoop). Map to async/await with `Task`-based I/O. Use `Channel<T>` or `Pipe` for producer-consumer patterns where Go uses channels.
|
|
|
|
**Locking:** Go `sync.RWMutex` maps to `ReaderWriterLockSlim`. Go `sync.Map` maps to `ConcurrentDictionary`. Go `atomic` operations map to `Interlocked` or `volatile`.
|
|
|
|
**Subject matching:** The `Sublist` trie is performance-critical. Every published message triggers a `Match()` call. Cache invalidation uses atomic generation counters.
|
|
|
|
**Protocol parsing:** The parser is a byte-by-byte state machine. In .NET, use `System.IO.Pipelines` for zero-copy parsing with `ReadOnlySequence<byte>`.
|
|
|
|
**Buffer management:** Go uses `[]byte` slices with pooling. Map to `ArrayPool<byte>` and `Memory<T>`/`Span<T>`.
|
|
|
|
**Compression:** NATS uses S2 (Snappy variant) for route/gateway compression. Use an equivalent .NET S2 library or IronSnappy.
|
|
|
|
**Ports:** Client=4222, Cluster=6222, Monitoring=8222, Leaf=5222, Gateway=7222.
|
|
|
|
## Message Flow Summary
|
|
|
|
```
|
|
Client PUB → parser → permission check → Sublist.Match() →
|
|
├─ Local subscribers: MSG to each (queue subs: pick one per group)
|
|
├─ Cluster routes: RMSG to peers (who deliver to their locals)
|
|
├─ Gateways: forward to interested remote clusters
|
|
└─ JetStream: if subject matches a stream, store + deliver to consumers
|
|
```
|
|
|
|
## NuGet Package Management
|
|
|
|
This solution uses **Central Package Management (CPM)** via `Directory.Packages.props` at the repo root. All package versions are defined centrally there.
|
|
|
|
- In `.csproj` files, use `<PackageReference Include="Foo" />` **without** a `Version` attribute
|
|
- To add a new package: add a `<PackageVersion>` entry in `Directory.Packages.props`, then reference it without version in the project's csproj
|
|
- To update a version: change it only in `Directory.Packages.props` — all projects pick it up automatically
|
|
- Never specify `Version` on `<PackageReference>` in individual csproj files
|
|
|
|
## Logging
|
|
|
|
Use **Microsoft.Extensions.Logging** (`ILogger<T>`) for all logging throughout the server. Wire up **Serilog** as the logging provider in the host application.
|
|
|
|
- Inject `ILogger<T>` via constructor in all components (NatsServer, NatsClient, etc.)
|
|
- Use **Serilog.Context.LogContext** to push contextual properties (client ID, remote endpoint, subscription subject) so they appear on all log entries within that scope
|
|
- Use structured logging with message templates: `logger.LogInformation("Client {ClientId} subscribed to {Subject}", id, subject)` — never string interpolation
|
|
- Log levels: `Trace` for protocol bytes, `Debug` for per-message flow, `Information` for lifecycle events (connect/disconnect), `Warning` for protocol violations, `Error` for unexpected failures
|
|
|
|
## Testing
|
|
|
|
- **xUnit 3** for test framework
|
|
- **Shouldly** for assertions — use `value.ShouldBe(expected)`, `action.ShouldThrow<T>()`, etc. Do NOT use `Assert.*` from xUnit
|
|
- **NSubstitute** for mocking/substitution when needed
|
|
- Do **NOT** use FluentAssertions or Moq — these are explicitly excluded
|
|
- Test project uses global `using Shouldly;`
|
|
|
|
## Porting Guidelines
|
|
|
|
- Use modern .NET 10 / C# 14 best practices (primary constructors, collection expressions, `field` keyword where stable, file-scoped namespaces, raw string literals, etc.)
|
|
- Prefer `readonly record struct` for small value types over mutable structs
|
|
- Use `required` properties and `init` setters for initialization-only state
|
|
- Use pattern matching and switch expressions where they improve clarity
|
|
- Prefer `System.Text.Json` source generators for JSON serialization
|
|
- Use `ValueTask` where appropriate for hot-path async methods
|
|
|
|
## Agent Model Guidance
|
|
|
|
- **Sonnet** (`model: "sonnet"`) — use for simpler implementation tasks: straightforward file modifications, adding packages, converting assertions, boilerplate code
|
|
- **Opus** (default) — use for complex tasks, architectural decisions, design work, tricky protocol logic, and code review
|
|
- **Parallel subagents** — use where tasks are independent and don't touch the same files (e.g., converting test files in parallel, adding packages while updating docs)
|
|
|
|
## Documentation
|
|
|
|
Follow the documentation rules in [`documentation_rules.md`](documentation_rules.md) for all project documentation. Key points:
|
|
|
|
- Documentation lives in `Documentation/` with component subfolders (Protocol, Subscriptions, Server, Configuration, Operations)
|
|
- Use `PascalCase.md` file names, always specify language on code blocks, use real code snippets (not invented examples)
|
|
- Update documentation when code changes — see the trigger rules and component map in the rules file
|
|
- Technical and direct tone, explain "why" not just "what", present tense
|
|
|
|
## Conventions
|
|
|
|
- Reference the Go implementation file and line when porting a subsystem
|
|
- Maintain protocol compatibility — the .NET server must interoperate with existing NATS clients and Go servers in a cluster
|
|
- Use the same configuration file format as the Go server (parsed by `conf/` package)
|
|
- Match the Go server's monitoring JSON response shapes for tooling compatibility
|