Files
natsdotnet/differences.md
Joseph Doherty 9efe787cab docs: update differences.md with accurate JetStream and clustering gaps
Add sections 11 (JetStream) and 12 (Clustering) with verified coverage
tables. Correct sections 2-4 where ROUTER/GATEWAY/LEAF and RS+/RS-/RMSG
were previously marked Y but are partial or stub implementations:

- ROUTER: handshake + in-memory subscription tracking only; no RMSG
- GATEWAY/LEAF: config-only stubs with no networking
- RS+/RS-/RMSG: command matrix only; no wire routing
- JetStream: 4 of ~20 API subjects implemented; RAFT is in-memory
  simulation with no persistence or network transport
2026-02-23 09:56:09 -05:00

604 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Go vs .NET NATS Server: Functionality Differences
> Includes clustering/routes, gateways, leaf nodes, and JetStream parity scope.
> Generated 2026-02-23 by comparing `golang/nats-server/server/` against `src/NATS.Server/`.
---
## 1. Core Server Lifecycle
### Server Initialization
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| NKey generation (server identity) | Y | Y | Ed25519 key pair via NATS.NKeys at startup |
| System account setup | Y | Y | `$SYS` account with InternalEventSystem, event publishing, request-reply services |
| Config file validation on startup | Y | Y | Full config parsing with error collection via `ConfigProcessor` |
| PID file writing | Y | Y | Written on startup, deleted on shutdown |
| Profiling HTTP endpoint (`/debug/pprof`) | Y | Stub | `ProfPort` option exists but endpoint not implemented |
| Ports file output | Y | Y | JSON ports file written to `PortsFileDir` on startup |
### Accept Loop
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Exponential backoff on accept errors | Y | Y | .NET backs off from 10ms to 1s on repeated failures |
| Config reload lock during client creation | Y | N | Go holds `reloadMu` around `createClient` |
| Goroutine/task tracking (WaitGroup) | Y | Y | `Interlocked` counter + drain with 10s timeout on shutdown |
| Callback-based error handling | Y | N | Go uses `errFunc` callback pattern |
| Random/ephemeral port (port=0) | Y | Y | Port resolved after `Bind`+`Listen`, stored in `_options.Port` |
### Shutdown
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Graceful shutdown with `WaitForShutdown()` | Y | Y | Idempotent CAS-guarded `ShutdownAsync()` + blocking `WaitForShutdown()` |
| Close reason tracking per connection | Y | Y | 37-value `ClosedState` enum, CAS-based first-writer-wins `MarkClosed()` |
| Lame duck mode (stop new, drain existing) | Y | Y | `LameDuckShutdownAsync()` with grace period + stagger-close with jitter |
| Wait for accept loop completion | Y | Y | `TaskCompletionSource` signaled in accept loop `finally` |
| Flush pending data before close | Y | Y | `FlushAndCloseAsync()` with best-effort flush, skip-flush for error conditions |
### Signal Handling
| Signal | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| SIGINT (Ctrl+C) | Y | Y | Both handle graceful shutdown |
| SIGTERM | Y | Y | `PosixSignalRegistration` triggers `ShutdownAsync()` |
| SIGUSR1 (reopen logs) | Y | Y | SIGUSR1 handler calls ReOpenLogFile |
| SIGUSR2 (lame duck mode) | Y | Y | Triggers `LameDuckShutdownAsync()` |
| SIGHUP (config reload) | Y | Y | Re-parses config, diffs options, applies reloadable subset; CLI flags preserved |
| Windows Service integration | Y | Y | `--service` flag with `Microsoft.Extensions.Hosting.WindowsServices` |
---
## 2. Client / Connection Handling
### Concurrency Model
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Separate read + write loops | Y | Y | Channel-based `RunWriteLoopAsync` with `QueueOutbound()` |
| Write coalescing / batch flush | Y | Y | Write loop drains all channel items before single `FlushAsync` |
| Dynamic buffer sizing (512B-64KB) | Y | N | .NET delegates to `System.IO.Pipelines` |
| Output buffer pooling (3-tier) | Y | N | Go pools at 512B, 4KB, 64KB |
### Connection Types
| Type | Go | .NET | Notes |
|------|:--:|:----:|-------|
| CLIENT | Y | Y | |
| ROUTER | Y | Partial | Route handshake + in-memory remote subscription tracking; no RMSG message routing, no RS+/RS- wire protocol, no route pooling (3x per peer) |
| GATEWAY | Y | Stub | Config parsing only; no listener, connections, handshake, interest-only mode, or message forwarding |
| LEAF | Y | Stub | Config parsing only; no listener, connections, handshake, subscription sharing, or loop detection |
| SYSTEM (internal) | Y | Y | InternalClient + InternalEventSystem with Channel-based send/receive loops |
| JETSTREAM (internal) | Y | N | |
| ACCOUNT (internal) | Y | Y | Lazy per-account InternalClient with import/export subscription support |
| WebSocket clients | Y | Y | Custom frame parser, permessage-deflate compression, origin checking, cookie auth |
| MQTT clients | Y | Partial | JWT connection-type constants + config parsing; no MQTT transport yet |
### Client Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Echo suppression (`echo: false`) | Y | Y | .NET checks echo in delivery path (NatsServer.cs:234,253) |
| Verbose mode (`+OK` responses) | Y | Y | Sends `+OK` after CONNECT, SUB, UNSUB, PUB when `verbose:true` |
| No-responders validation | Y | Y | CONNECT rejects `no_responders` without `headers`; 503 HMSG on no match |
| Slow consumer detection | Y | Y | Pending bytes threshold (64MB) + write deadline timeout (10s) |
| Write deadline / timeout policies | Y | Y | `WriteDeadline` option with `CancellationTokenSource.CancelAfter` on flush |
| RTT measurement | Y | Y | `_rttStartTicks`/`Rtt` property, computed on PONG receipt |
| Per-client trace mode | Y | Y | `SetTraceMode()` toggles parser logger dynamically via `ClientFlags.TraceMode` |
| Detailed close reason tracking | Y | Y | 37-value `ClosedState` enum with CAS-based `MarkClosed()` |
| Connection state flags (16 flags) | Y | Y | 7-flag `ClientFlagHolder` with `Interlocked.Or`/`And` |
### Slow Consumer Handling
Go implements a sophisticated slow consumer detection system:
- Tracks `pendingBytes` per client output buffer
- If pending exceeds `maxPending`, enters stall mode (2-5ms waits)
- Total stall capped at 10ms per read cycle
- Closes with `SlowConsumerPendingBytes` or `SlowConsumerWriteDeadline`
- Sets `isSlowConsumer` flag for monitoring
.NET now implements pending bytes tracking and write deadline enforcement via `Channel<ReadOnlyMemory<byte>>`. Key differences from Go: no stall/retry mode (immediate close on threshold exceeded), write deadline via `CancellationTokenSource.CancelAfter` instead of `SetWriteDeadline`. `IsSlowConsumer` flag and server-level `SlowConsumerCount` stats are tracked for monitoring.
### Stats Tracking
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-connection atomic stats | Y | Y | .NET uses `Interlocked` for stats access |
| Per-read-cycle stat batching | Y | Y | Local accumulators flushed via `Interlocked.Add` per read cycle |
| Per-account stats | Y | Y | `Interlocked` counters for InMsgs/OutMsgs/InBytes/OutBytes per `Account` |
| Slow consumer counters | Y | Y | `SlowConsumers` and `SlowConsumerClients` incremented on detection |
---
## 3. Protocol Parsing
### Parser Architecture
| Aspect | Go | .NET |
|--------|-----|------|
| Approach | Byte-by-byte state machine (74 states) | Two-phase: line extraction + command dispatch |
| Case handling | Per-state character checks | Bit-mask lowercase normalization (`\| 0x20`) |
| Buffer strategy | Jump-ahead optimization for payloads | Direct size-based reads via Pipe |
| Split-buffer handling | argBuf accumulation with scratch buffer | State variables (`_awaitingPayload`, etc.) |
| Error model | Inline error sending + error return | Exception-based (`ProtocolViolationException`) |
| CRLF in payload | Included in message buffer | Excluded by design |
### Protocol Operations
| Operation | Go | .NET | Notes |
|-----------|:--:|:----:|-------|
| PUB | Y | Y | |
| HPUB (headers) | Y | Y | |
| SUB | Y | Y | |
| UNSUB | Y | Y | |
| CONNECT | Y | Y | |
| INFO | Y | Y | |
| PING / PONG | Y | Y | |
| MSG / HMSG | Y | Y | |
| +OK / -ERR | Y | Y | |
| RS+/RS-/RMSG (routes) | Y | N | Parser/command matrix recognises opcodes; no wire routing — remote subscription propagation uses in-memory method calls; RMSG delivery not implemented |
| A+/A- (accounts) | Y | N | Inter-server account protocol ops still pending |
| LS+/LS-/LMSG (leaf) | Y | N | Leaf nodes are config-only stubs; no LS+/LS-/LMSG wire protocol handling |
### Protocol Parsing Gaps
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Multi-client-type command routing | Y | N | Go checks `c.kind` to allow/reject commands |
| Protocol tracing in parser | Y | Y | `TraceInOp()` logs `<<- OP arg` at `LogLevel.Trace` via optional `ILogger` |
| Subject mapping (input→output) | Y | Y | Compiled `SubjectTransform` engine with 9 function tokens; wired into `ProcessMessage` |
| MIME header parsing | Y | Y | `NatsHeaderParser.Parse()` — status line + key-value headers from `ReadOnlySpan<byte>` |
| Message trace event initialization | Y | N | |
### Protocol Writing
| Aspect | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| INFO serialization | Once at startup | Once at startup | Cached at startup; nonce connections serialize per-connection |
| MSG/HMSG construction | Direct buffer write | Span-based buffer write | `int.TryFormat` + `CopyTo` into rented buffer, no string allocations |
| Pre-encoded constants | Y | Y | Both pre-encode PING/PONG/OK |
---
## 4. Subscriptions & Subject Matching
### Trie Implementation
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Basic trie with `*`/`>` wildcards | Y | Y | Core matching identical |
| Queue group support | Y | Y | |
| Result caching (1024 max) | Y | Y | Same limits |
| `plist` optimization (>256 subs) | Y | N | Go converts high-fanout nodes to array |
| Async cache sweep (background) | Y | N | .NET sweeps inline under write lock |
| Atomic generation ID for invalidation | Y | Y | `Interlocked.Increment` on insert/remove; cached results store generation |
| Cache eviction strategy | Random | First-N | Semantic difference minimal |
### SubList Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| `Stats()` — comprehensive statistics | Y | Y | Matches, cache hits, inserts, removes tracked via `Interlocked` |
| `HasInterest()` — fast bool check | Y | Y | Walks trie without allocating result list |
| `NumInterest()` — fast count | Y | Y | Counts plain + queue subs without allocation |
| `ReverseMatch()` — pattern→literal query | Y | Y | Finds subscriptions whose wildcards match a literal subject |
| `RemoveBatch()` — efficient bulk removal | Y | Y | Single generation increment for batch; increments `_removes` per sub |
| `All()` — enumerate all subscriptions | Y | Y | Recursive trie walk returning all subscriptions |
| Notification system (interest changes) | Y | N | |
| Local/remote subscription filtering | Y | N | |
| Queue weight expansion (remote subs) | Y | N | |
| `MatchBytes()` — zero-copy byte API | Y | N | |
### Subject Validation
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Basic validation (empty tokens, wildcards) | Y | Y | |
| Literal subject check | Y | Y | |
| UTF-8/null rune validation | Y | Y | `IsValidSubject(string, bool checkRunes)` rejects null bytes |
| Collision detection (`SubjectsCollide`) | Y | Y | Token-by-token wildcard comparison; O(n) via upfront `Split` |
| Token utilities (`tokenAt`, `numTokens`) | Y | Y | `TokenAt` returns `ReadOnlySpan<char>`; `NumTokens` counts separators |
| Stack-allocated token buffer | Y | N | Go uses `[32]string{}` on stack |
### Subscription Lifecycle
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-account subscription limit | Y | Y | `Account.IncrementSubscriptions()` returns false when `MaxSubscriptions` exceeded |
| Auto-unsubscribe on max messages | Y | Y | Enforced at delivery; sub removed from trie + client dict when exhausted |
| Subscription routing propagation | Y | Partial | Remote subs tracked in trie; propagation via in-process method calls (no wire RS+/RS-); RMSG forwarding absent |
| Queue weight (`qw`) field | Y | N | For remote queue load balancing |
---
## 5. Authentication & Authorization
### Auth Mechanisms
| Mechanism | Go | .NET | Notes |
|-----------|:--:|:----:|-------|
| Username/password | Y | Y | |
| Token | Y | Y | |
| NKeys (Ed25519) | Y | Y | .NET has framework but integration is basic |
| JWT validation | Y | Y | `NatsJwt` decode/verify, `JwtAuthenticator` with account resolution + revocation + `allowed_connection_types` enforcement |
| Bcrypt password hashing | Y | Y | .NET supports bcrypt (`$2*` prefix) with constant-time fallback |
| TLS certificate mapping | Y | Y | X500DistinguishedName with full DN match and CN fallback |
| Custom auth interface | Y | N | |
| External auth callout | Y | N | |
| Proxy authentication | Y | N | |
| Bearer tokens | Y | Y | `UserClaims.BearerToken` skips nonce signature verification |
| User revocation tracking | Y | Y | Per-account `ConcurrentDictionary` with wildcard (`*`) revocation support |
### Account System
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-account SubList isolation | Y | Y | |
| Multi-account user resolution | Y | Y | `AccountConfig` per account in `NatsOptions.Accounts`; `GetOrCreateAccount` wires limits |
| Account exports/imports | Y | Y | ServiceImport/StreamImport with ExportAuth, subject transforms, response routing |
| Per-account connection limits | Y | Y | `Account.AddClient()` returns false when `MaxConnections` exceeded |
| Per-account subscription limits | Y | Y | `Account.IncrementSubscriptions()` enforced in `ProcessSub()` |
| Account JetStream limits | Y | Y | Enforced via account-level stream reservation limits |
### Permissions
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Publish allow list | Y | Y | |
| Subscribe allow list | Y | Y | |
| Publish deny list | Y | Y | Full enforcement with LRU-cached results |
| Subscribe deny list | Y | Y | Queue-aware deny checking in `IsSubscribeAllowed` |
| Message-level deny filtering | Y | Y | `IsDeliveryAllowed()` checked before MSG send; auto-unsub cleanup on deny |
| Permission caching (128 entries) | Y | Y | `PermissionLruCache` — Dictionary+LinkedList LRU, matching Go's `maxPermCacheSize` |
| Response permissions (reply tracking) | Y | Y | `ResponseTracker` with configurable TTL + max messages; not LRU-cached |
| Auth expiry enforcement | Y | Y | `Task.Delay` timer closes client when JWT/auth expires |
| Permission templates (JWT) | Y | Y | `PermissionTemplates.Expand()` — 6 functions with cartesian product for multi-value tags |
---
## 6. Configuration
### CLI Flags
| Flag | Go | .NET | Notes |
|------|:--:|:----:|-------|
| `-p/--port` | Y | Y | |
| `-a/--addr` | Y | Y | |
| `-n/--name` (ServerName) | Y | Y | |
| `-m/--http_port` (monitoring) | Y | Y | |
| `-c` (config file) | Y | Y | Full config parsing: lexer → parser → processor; CLI args override config |
| `-D/-V/-DV` (debug/trace) | Y | Y | `-D`/`--debug` for debug, `-V`/`-T`/`--trace` for trace, `-DV` for both |
| `--tlscert/--tlskey/--tlscacert` | Y | Y | |
| `--tlsverify` | Y | Y | |
| `--http_base_path` | Y | Y | |
| `--https_port` | Y | Y | |
### Configuration System
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Config file parsing | Y | Y | Custom NATS conf lexer/parser ported from Go; supports includes, variables, blocks |
| Hot reload (SIGHUP) | Y | Y | Reloads logging, auth, limits, TLS certs on SIGHUP; rejects non-reloadable changes |
| Config change detection | Y | Y | SHA256 digest comparison; `InCmdLine` tracks CLI flag precedence |
| ~450 option fields | Y | ~72 | .NET covers core + single-server options plus cluster/JetStream parsing and reload boundary validation |
### Missing Options Categories
- ~~Logging options~~ — file logging, rotation, syslog, debug/trace, color, timestamps, per-subsystem log control all implemented
- ~~Advanced limits (MaxSubs, MaxSubTokens, MaxPending, WriteDeadline)~~ — `MaxSubs`, `MaxSubTokens` implemented; MaxPending/WriteDeadline already existed
- ~~Tags/metadata~~ — `Tags` dictionary implemented in `NatsOptions`
- ~~OCSP configuration~~ — `OcspConfig` with 4 modes (Auto/Always/Must/Never), peer verification, and stapling
- ~~WebSocket options~~ — `WebSocketOptions` with port, compression, origin checking, cookie auth, custom headers
- ~~MQTT options~~ — `mqtt {}` config block parsed with all Go `MQTTOpts` fields; no listener yet
- ~~Operator mode / account resolver~~ — `JwtAuthenticator` + `IAccountResolver` + `MemAccountResolver` with trusted keys
---
## 7. Monitoring
### HTTP Endpoints
| Endpoint | Go | .NET | Notes |
|----------|:--:|:----:|-------|
| `/healthz` | Y | Y | |
| `/varz` | Y | Y | |
| `/connz` | Y | Y | |
| `/` (root listing) | Y | Y | |
| `/routez` | Y | Stub | Returns empty response |
| `/gatewayz` | Y | Stub | Returns empty response |
| `/leafz` | Y | Stub | Returns empty response |
| `/subz` / `/subscriptionsz` | Y | Y | Account filtering, test subject filtering, pagination, and subscription details |
| `/accountz` | Y | Stub | Returns empty response |
| `/accstatz` | Y | Stub | Returns empty response |
| `/jsz` | Y | Y | Returns live JetStream counts/config via `JszHandler` |
### Varz Response
| Field Category | Go | .NET | Notes |
|----------------|:--:|:----:|-------|
| Identity (ID, Name, Version) | Y | Y | |
| Network (Host, Port, URLs) | Y | Y | |
| Security (AuthRequired, TLS) | Y | Y | |
| Limits (MaxConn, MaxPayload) | Y | Y | |
| Timing (Start, Now, Uptime) | Y | Y | |
| Runtime (Mem, CPU, Cores) | Y | Y | |
| Connections (current, total) | Y | Y | |
| Messages (in/out msgs/bytes) | Y | Y | |
| SlowConsumer breakdown | Y | N | Go tracks per connection type |
| Cluster/Gateway/Leaf blocks | Y | Partial | Config projection present; `/gatewayz` and `/leafz` endpoints remain stubs |
| JetStream block | Y | Y | Includes live JetStream config + stream/consumer counts |
| TLS cert expiry info | Y | Y | `TlsCertNotAfter` loaded via `X509CertificateLoader` in `/varz` |
### Connz Response
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Filtering by CID, user, account | Y | Partial | |
| Sorting (11 options) | Y | Y | All options including ByStop, ByReason, ByRtt |
| State filtering (open/closed/all) | Y | Y | `state=open|closed|all` query parameter |
| Closed connection tracking | Y | Y | `ConcurrentQueue<ClosedClient>` capped at 10,000 entries |
| Pagination (offset, limit) | Y | Y | |
| Subscription detail mode | Y | N | |
| TLS peer certificate info | Y | N | |
| JWT/IssuerKey/Tags fields | Y | N | |
| MQTT client ID filtering | Y | Y | `mqtt_client` query param filters open and closed connections |
| Proxy info | Y | N | |
---
## 8. TLS
### TLS Modes
| Mode | Go | .NET | Notes |
|------|:--:|:----:|-------|
| No TLS | Y | Y | |
| INFO-first (default NATS) | Y | Y | |
| TLS-first (before INFO) | Y | Y | |
| Mixed/Fallback | Y | Y | |
| TLS-required | Y | Y | |
### TLS Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| PEM cert/key loading | Y | Y | |
| CA chain validation | Y | Y | |
| Mutual TLS (client certs) | Y | Y | |
| Certificate pinning (SHA256 SPKI) | Y | Y | |
| TLS handshake timeout | Y | Y | |
| TLS rate limiting | Y | Y | Rate enforcement with refill; unit tests cover rate limiting and refill |
| First-byte peeking (0x16 detection) | Y | Y | |
| Cert subject→user mapping | Y | Y | X500DistinguishedName with full DN match and CN fallback |
| OCSP stapling | Y | Y | `SslStreamCertificateContext.Create` with `offline:false` for runtime OCSP fetch |
| Min TLS version control | Y | Y | |
---
## 9. Logging
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Structured logging | Partial | Y | .NET uses Serilog with ILogger<T> |
| File logging with rotation | Y | Y | `-l`/`--log_file` flag + `LogSizeLimit`/`LogMaxFiles` via Serilog.Sinks.File |
| Syslog (local and remote) | Y | Y | `--syslog` and `--remote_syslog` flags via Serilog.Sinks.SyslogMessages |
| Log reopening (SIGUSR1) | Y | Y | SIGUSR1 handler calls ReOpenLogFile callback |
| Trace mode (protocol-level) | Y | Y | `-V`/`-T`/`--trace` flags; parser `TraceInOp()` logs at Trace level |
| Debug mode | Y | Y | `-D`/`--debug` flag lowers Serilog minimum to Debug |
| Per-subsystem log control | Y | Y | `--log_level_override ns=level` CLI flag with Serilog `MinimumLevel.Override` |
| Color output on TTY | Y | Y | Auto-detected via `Console.IsOutputRedirected`, uses `AnsiConsoleTheme.Code` |
| Timestamp format control | Y | Y | `--logtime` and `--logtime_utc` flags |
---
## 10. Ping/Pong & Keepalive
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Server-initiated PING | Y | Y | |
| Configurable interval | Y | Y | PingInterval option |
| Max pings out | Y | Y | MaxPingsOut option |
| Stale connection close | Y | Y | |
| RTT-based first PING delay | Y | Y | Skips PING until FirstPongSent or 2s elapsed |
| RTT tracking | Y | Y | `_rttStartTicks`/`Rtt` property, computed on PONG receipt |
| Stale connection stats | Y | Y | `StaleConnectionStats` model, exposed in `/varz` |
---
## Summary: Critical Gaps for Production Use
### Resolved Since Initial Audit
The following items from the original gap list have been implemented:
- **Slow consumer detection** — pending bytes threshold (64MB) with write deadline enforcement
- **Write coalescing / batch flush** — channel-based write loop drains all items before single flush
- **Verbose mode** — `+OK` responses for CONNECT, SUB, UNSUB, PUB when `verbose:true`
- **Permission deny enforcement at delivery** — `IsDeliveryAllowed` + auto-unsub cleanup
- **No-responders validation** — CONNECT rejects `no_responders` without `headers`; 503 HMSG on no match
- **File logging with rotation** — Serilog.Sinks.File with rolling file support
- **TLS certificate mapping** — X500DistinguishedName with full DN match and CN fallback
- **Protocol tracing** — `-V`/`-T` flag enables trace-level logging; `-D` for debug
- **Subscription statistics** — `Stats()`, `HasInterest()`, `NumInterest()`, etc.
- **Per-account limits** — connection + subscription limits via `AccountConfig`
- **Reply subject tracking** — `ResponseTracker` with TTL + max messages
- **JWT authentication** — `JwtAuthenticator` with decode/verify, account resolution, revocation, permission templates
- **OCSP support** — peer verification via `X509RevocationMode.Online`, stapling via `SslStreamCertificateContext`
- **Subject mapping** — compiled `SubjectTransform` engine with 9 function tokens, wired into message delivery
- **Windows Service integration** — `--service` flag with `Microsoft.Extensions.Hosting.WindowsServices`
- **Per-subsystem log control** — `--log_level_override` CLI flag with Serilog overrides
- **Per-client trace mode** — `SetTraceMode()` with dynamic parser logger toggling
- **Per-account stats** — `Interlocked` counters for InMsgs/OutMsgs/InBytes/OutBytes
- **TLS cert expiry in /varz** — `TlsCertNotAfter` populated via `X509CertificateLoader`
- **Permission templates** — `PermissionTemplates.Expand()` with 6 functions and cartesian product
- **Bearer tokens** — `UserClaims.BearerToken` skips nonce verification
- **User revocation** — per-account tracking with wildcard (`*`) revocation
- **Config file parsing** — custom lexer/parser ported from Go; supports includes, variables, nested blocks, size suffixes
- **Hot reload (SIGHUP)** — re-parses config, diffs changes, validates reloadable set, applies with CLI precedence
- **SYSTEM client type** — InternalClient with InternalEventSystem, Channel-based send/receive loops, event publishing
- **ACCOUNT client type** — lazy per-account InternalClient with import/export subscription support
- **System event publishing** — connect/disconnect advisories, server stats, shutdown/lame-duck events, auth errors
- **System request-reply services** — $SYS.REQ.SERVER.*.VARZ/CONNZ/SUBSZ/HEALTHZ/IDZ/STATSZ with ping wildcards
- **Account exports/imports** — service and stream imports with ExportAuth, subject transforms, response routing, latency tracking
---
## 11. JetStream
> The Go JetStream surface is ~37,500 lines across jetstream.go, stream.go, consumer.go, filestore.go, memstore.go, raft.go. The .NET implementation has ~700 lines covering a minimal subset. Verified by `JetStreamApiRouter.cs` registration.
### JetStream API ($JS.API.* subjects)
| Subject | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| `STREAM.CREATE.<name>` | Y | Y | Only API handler implemented |
| `STREAM.INFO.<name>` | Y | Y | |
| `STREAM.UPDATE.<name>` | Y | N | |
| `STREAM.DELETE.<name>` | Y | N | |
| `STREAM.NAMES` | Y | N | |
| `STREAM.LIST` | Y | N | |
| `STREAM.PURGE.<name>` | Y | N | Storage has `PurgeAsync()` but no API handler |
| `STREAM.MSG.GET.<name>` | Y | N | |
| `STREAM.MSG.DELETE.<name>` | Y | N | |
| `DIRECT.GET.<name>` | Y | N | KV-style direct access |
| `CONSUMER.CREATE.<stream>` | Y | Y | |
| `CONSUMER.INFO.<stream>.<durable>` | Y | Y | |
| `CONSUMER.DELETE.<stream>.<durable>` | Y | N | |
| `CONSUMER.NAMES.<stream>` | Y | N | |
| `CONSUMER.LIST.<stream>` | Y | N | |
| `CONSUMER.PAUSE.<stream>.<durable>` | Y | N | |
| `STREAM.LEADER.STEPDOWN.<name>` | Y | N | |
| `META.LEADER.STEPDOWN` | Y | N | |
| `STREAM.SNAPSHOT.<name>` | Y | N | |
| `STREAM.RESTORE.<name>` | Y | N | |
| `INFO` (account info) | Y | N | |
### Stream Configuration
| Option | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| Subjects | Y | Y | |
| Replicas | Y | Y | Wires RAFT replica count |
| MaxMsgs limit | Y | Y | Enforced via `EnforceLimits()` |
| Retention (Limits/Interest/WorkQueue) | Y | N | Only MaxMsgs trimming; Interest and WorkQueue not implemented |
| Discard policy (Old/New) | Y | N | |
| MaxBytes / MaxAge (TTL) | Y | N | |
| MaxMsgsPer (per-subject limit) | Y | N | |
| MaxMsgSize | Y | N | |
| Storage type selection (Memory/File) | Y | N | MemStore default; no config-driven choice |
| Compression (S2) | Y | N | |
| Subject transform | Y | N | |
| RePublish | Y | N | |
| AllowDirect / KV mode | Y | N | |
| Sealed, DenyDelete, DenyPurge | Y | N | |
| Duplicates dedup window | Y | Partial | Dedup ID cache exists; no configurable window |
### Consumer Configuration & Delivery
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Push delivery | Y | Partial | `PushConsumerEngine`; basic delivery |
| Pull fetch | Y | Partial | `PullConsumerEngine`; basic batch fetch |
| Ephemeral consumers | Y | N | Only durable |
| AckPolicy.None | Y | Y | |
| AckPolicy.Explicit | Y | Y | `AckProcessor` tracks pending with expiry |
| AckPolicy.All | Y | N | |
| Redelivery on ack timeout | Y | Partial | `NextExpired()` detects expired; limit not enforced |
| DeliverPolicy (All/Last/New/StartSeq/StartTime) | Y | N | Always delivers from beginning |
| FilterSubject (single) | Y | Y | |
| FilterSubjects (multiple) | Y | N | |
| MaxAckPending | Y | N | |
| Idle heartbeat | Y | N | Field in model; not implemented |
| Flow control | Y | N | |
| Rate limiting | Y | N | |
| Replay policy | Y | N | |
| BackOff (exponential) | Y | N | |
### Storage Backends
| Feature | Go FileStore | .NET FileStore | Notes |
|---------|:--:|:----:|-------|
| Append / Load / Purge | Y | Y | Basic JSONL serialization |
| Recovery on restart | Y | Y | Loads JSONL on startup |
| Block-based layout (64 MB blocks) | Y | N | .NET uses flat JSONL; not production-scale |
| S2 compression | Y | N | |
| AES-GCM / ChaCha20 encryption | Y | N | |
| Bit-packed sequence indexing | Y | N | Simple dictionary |
| TTL / time-based expiry | Y | N | |
MemStore has basic append/load/purge with `Dictionary<long, StoredMessage>` under a lock.
### Mirror & Sourcing
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Mirror consumer creation | Y | Partial | `MirrorCoordinator` triggers on append |
| Mirror sync state tracking | Y | N | |
| Source fan-in (multiple sources) | Y | Partial | Single `Source` field; no `Sources[]` array |
| Subject mapping for sources | Y | N | |
| Cross-account mirror/source | Y | N | |
### RAFT Consensus
| Feature | Go (5 037 lines) | .NET (212 lines) | Notes |
|---------|:--:|:----:|-------|
| Leader election / term tracking | Y | Partial | In-process; nodes hold direct `List<RaftNode>` references |
| Log append + quorum | Y | Partial | Entries replicated via direct method calls; no network |
| Log persistence | Y | N | In-memory `List<RaftLogEntry>` only |
| Heartbeat / keep-alive | Y | N | |
| Log mismatch resolution (NextIndex) | Y | N | |
| Snapshot creation | Y | Partial | `CreateSnapshotAsync()` exists; stored in-memory |
| Snapshot network transfer | Y | N | |
| Membership changes | Y | N | |
| Network RPC transport | Y | N | All coordination is in-process |
### JetStream Clustering
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Meta-group governance | Y | Partial | `JetStreamMetaGroup` tracks streams; no durable consensus |
| Per-stream replica group | Y | Partial | `StreamReplicaGroup` + in-memory RAFT |
| Asset placement planner | Y | Partial | `AssetPlacementPlanner` skeleton |
| Cross-cluster JetStream (gateways) | Y | N | Requires functional gateways |
---
## 12. Clustering
> Routes, gateways, and leaf nodes are in different states of completeness. All three are present as Go reference feature targets but only routes have any functional networking.
### Routes
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Listener accept loop | Y | Y | `RouteManager` binds and accepts inbound connections |
| Outbound seed connections (with backoff) | Y | Y | Iterates `ClusterOptions.Routes` with 250 ms retry |
| Route handshake (ROUTE `<serverId>`) | Y | Y | Bidirectional: sends own ID, reads peer ID |
| Remote subscription tracking | Y | Y | `ApplyRemoteSubscription` adds to SubList; `HasRemoteInterest` exposed |
| Subscription propagation (in-process) | Y | Partial | `PropagateLocalSubscription` calls peer managers directly — no wire RS+/RS- protocol |
| Message routing (RMSG wire) | Y | N | **Critical gap**: published messages are never forwarded to remote subscribers |
| RS+/RS- subscription protocol (wire) | Y | N | Command matrix recognises opcodes but no handler processes inbound RS+/RS- frames |
| Route pooling (3× per peer) | Y | N | Single connection per remote server ID |
| Account-specific routes | Y | N | |
| S2 compression on routes | Y | N | |
| CONNECT info + topology gossip | Y | N | Handshake is two-line text exchange only |
### Gateways
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Any networking (listener / outbound) | Y | N | `GatewayManager.StartAsync()` logs a debug line and zeros a counter |
| Gateway connection protocol | Y | N | |
| Interest-only mode | Y | N | |
| Reply subject mapping (`_GR_.` prefix) | Y | N | |
| Message forwarding to remote clusters | Y | N | |
### Leaf Nodes
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Any networking (listener / spoke) | Y | N | `LeafNodeManager.StartAsync()` logs a debug line and zeros a counter |
| Leaf handshake / role negotiation | Y | N | |
| Subscription sharing (LS+/LS-) | Y | N | |
| Loop detection (`$LDS.` prefix) | Y | N | |
| Hub-and-spoke account mapping | Y | N | |
---
## Summary: Remaining Gaps
### Clustering (High Impact)
1. **Route message routing** — Remote subscribers receive no messages; no RMSG implementation
2. **Gateways** — Non-functional stub; no inter-cluster bridging
3. **Leaf nodes** — Non-functional stub; no hub/spoke topology
4. **RS+/RS- wire protocol** — subscription propagation is in-process method calls only
5. **Route pooling** — single connection per peer vs Go's 3-connection pool
### JetStream (Significant Gaps)
1. **API coverage** — Only 4 of ~20 `$JS.API.*` subjects handled (STREAM.CREATE/INFO, CONSUMER.CREATE/INFO)
2. **Stream delete/update/list** — not implemented
3. **Consumer delete/list/pause** — not implemented
4. **Retention policies** — only MaxMsgs; Interest and WorkQueue retention absent
5. **FileStore scalability** — JSONL-based (not block/compressed/encrypted)
6. **RAFT persistence** — in-memory only; cluster restart loses all JetStream state
7. **Consumer delivery completeness** — DeliverPolicy, AckPolicy.All, MaxAckPending, heartbeats, flow control absent
### Lower Priority
1. **Dynamic buffer sizing** — delegated to Pipe, less optimized for long-lived connections
2. **`plist` optimization** — high-fanout nodes (>256 subs) not converted to array
3. **External auth callout / proxy auth** — custom auth interfaces not ported
4. **MQTT listener** — config parsed; no transport
5. **Inter-server account protocol (A+/A-)** — not implemented