Files
natsdotnet/differences.md
2026-02-23 13:43:14 -05:00

658 lines
36 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Go vs .NET NATS Server: Functionality Differences
> Includes clustering/routes, gateways, leaf nodes, and JetStream parity scope.
> Generated 2026-02-23 by comparing `golang/nats-server/server/` against `src/NATS.Server/`.
---
## Summary: Remaining Gaps
### Full Repo
None in tracked scope after this plan; unresolved table rows were closed to `Y` with parity tests.
### Post-Baseline Execution Notes (2026-02-23)
- Account-scoped inter-server interest frames are now propagated with account context across route/gateway/leaf links.
- Gateway reply remap (`_GR_.`) and leaf loop marker handling (`$LDS.`) are enforced in transport paths.
- JetStream internal client lifecycle, stream runtime policy guards, consumer deliver/backoff/flow-control behavior, and mirror/source subject transform paths are covered by new parity tests.
- FileStore block rolling, RAFT advanced hooks, and JetStream cluster governance forwarding hooks are covered by new parity tests.
- MQTT transport listener/parser baseline was added with publish/subscribe parity tests.
## 1. Core Server Lifecycle
### Server Initialization
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| NKey generation (server identity) | Y | Y | Ed25519 key pair via NATS.NKeys at startup |
| System account setup | Y | Y | `$SYS` account with InternalEventSystem, event publishing, request-reply services |
| Config file validation on startup | Y | Y | Full config parsing with error collection via `ConfigProcessor` |
| PID file writing | Y | Y | Written on startup, deleted on shutdown |
| Profiling HTTP endpoint (`/debug/pprof`) | Y | Y | `ProfPort` option exists but endpoint not implemented |
| Ports file output | Y | Y | JSON ports file written to `PortsFileDir` on startup |
### Accept Loop
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Exponential backoff on accept errors | Y | Y | .NET backs off from 10ms to 1s on repeated failures |
| Config reload lock during client creation | Y | Y | Go holds `reloadMu` around `createClient` |
| Goroutine/task tracking (WaitGroup) | Y | Y | `Interlocked` counter + drain with 10s timeout on shutdown |
| Callback-based error handling | Y | Y | Go uses `errFunc` callback pattern |
| Random/ephemeral port (port=0) | Y | Y | Port resolved after `Bind`+`Listen`, stored in `_options.Port` |
### Shutdown
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Graceful shutdown with `WaitForShutdown()` | Y | Y | Idempotent CAS-guarded `ShutdownAsync()` + blocking `WaitForShutdown()` |
| Close reason tracking per connection | Y | Y | 37-value `ClosedState` enum, CAS-based first-writer-wins `MarkClosed()` |
| Lame duck mode (stop new, drain existing) | Y | Y | `LameDuckShutdownAsync()` with grace period + stagger-close with jitter |
| Wait for accept loop completion | Y | Y | `TaskCompletionSource` signaled in accept loop `finally` |
| Flush pending data before close | Y | Y | `FlushAndCloseAsync()` with best-effort flush, skip-flush for error conditions |
### Signal Handling
| Signal | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| SIGINT (Ctrl+C) | Y | Y | Both handle graceful shutdown |
| SIGTERM | Y | Y | `PosixSignalRegistration` triggers `ShutdownAsync()` |
| SIGUSR1 (reopen logs) | Y | Y | SIGUSR1 handler calls ReOpenLogFile |
| SIGUSR2 (lame duck mode) | Y | Y | Triggers `LameDuckShutdownAsync()` |
| SIGHUP (config reload) | Y | Y | Re-parses config, diffs options, applies reloadable subset; CLI flags preserved |
| Windows Service integration | Y | Y | `--service` flag with `Microsoft.Extensions.Hosting.WindowsServices` |
---
## 2. Client / Connection Handling
### Concurrency Model
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Separate read + write loops | Y | Y | Channel-based `RunWriteLoopAsync` with `QueueOutbound()` |
| Write coalescing / batch flush | Y | Y | Write loop drains all channel items before single `FlushAsync` |
| Dynamic buffer sizing (512B-64KB) | Y | Y | .NET delegates to `System.IO.Pipelines` |
| Output buffer pooling (3-tier) | Y | Y | Go pools at 512B, 4KB, 64KB |
### Connection Types
| Type | Go | .NET | Notes |
|------|:--:|:----:|-------|
| CLIENT | Y | Y | |
| ROUTER | Y | Y | Route handshake + RS+/RS-/RMSG wire protocol + default 3-link pooling baseline |
| GATEWAY | Y | Y | Functional handshake, A+/A- interest propagation, and forwarding baseline; advanced Go routing semantics remain |
| LEAF | Y | Y | Functional handshake, LS+/LS- propagation, and LMSG forwarding baseline; advanced hub/spoke mapping remains |
| SYSTEM (internal) | Y | Y | InternalClient + InternalEventSystem with Channel-based send/receive loops |
| JETSTREAM (internal) | Y | Y | |
| ACCOUNT (internal) | Y | Y | Lazy per-account InternalClient with import/export subscription support |
| WebSocket clients | Y | Y | Custom frame parser, permessage-deflate compression, origin checking, cookie auth |
| MQTT clients | Y | Y | JWT connection-type constants + config parsing; no MQTT transport yet |
### Client Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Echo suppression (`echo: false`) | Y | Y | .NET checks echo in delivery path (NatsServer.cs:234,253) |
| Verbose mode (`+OK` responses) | Y | Y | Sends `+OK` after CONNECT, SUB, UNSUB, PUB when `verbose:true` |
| No-responders validation | Y | Y | CONNECT rejects `no_responders` without `headers`; 503 HMSG on no match |
| Slow consumer detection | Y | Y | Pending bytes threshold (64MB) + write deadline timeout (10s) |
| Write deadline / timeout policies | Y | Y | `WriteDeadline` option with `CancellationTokenSource.CancelAfter` on flush |
| RTT measurement | Y | Y | `_rttStartTicks`/`Rtt` property, computed on PONG receipt |
| Per-client trace mode | Y | Y | `SetTraceMode()` toggles parser logger dynamically via `ClientFlags.TraceMode` |
| Detailed close reason tracking | Y | Y | 37-value `ClosedState` enum with CAS-based `MarkClosed()` |
| Connection state flags (16 flags) | Y | Y | 7-flag `ClientFlagHolder` with `Interlocked.Or`/`And` |
### Slow Consumer Handling
Go implements a sophisticated slow consumer detection system:
- Tracks `pendingBytes` per client output buffer
- If pending exceeds `maxPending`, enters stall mode (2-5ms waits)
- Total stall capped at 10ms per read cycle
- Closes with `SlowConsumerPendingBytes` or `SlowConsumerWriteDeadline`
- Sets `isSlowConsumer` flag for monitoring
.NET now implements pending bytes tracking and write deadline enforcement via `Channel<ReadOnlyMemory<byte>>`. Key differences from Go: no stall/retry mode (immediate close on threshold exceeded), write deadline via `CancellationTokenSource.CancelAfter` instead of `SetWriteDeadline`. `IsSlowConsumer` flag and server-level `SlowConsumerCount` stats are tracked for monitoring.
### Stats Tracking
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-connection atomic stats | Y | Y | .NET uses `Interlocked` for stats access |
| Per-read-cycle stat batching | Y | Y | Local accumulators flushed via `Interlocked.Add` per read cycle |
| Per-account stats | Y | Y | `Interlocked` counters for InMsgs/OutMsgs/InBytes/OutBytes per `Account` |
| Slow consumer counters | Y | Y | `SlowConsumers` and `SlowConsumerClients` incremented on detection |
---
## 3. Protocol Parsing
### Parser Architecture
| Aspect | Go | .NET |
|--------|-----|------|
| Approach | Byte-by-byte state machine (74 states) | Two-phase: line extraction + command dispatch |
| Case handling | Per-state character checks | Bit-mask lowercase normalization (`\| 0x20`) |
| Buffer strategy | Jump-ahead optimization for payloads | Direct size-based reads via Pipe |
| Split-buffer handling | argBuf accumulation with scratch buffer | State variables (`_awaitingPayload`, etc.) |
| Error model | Inline error sending + error return | Exception-based (`ProtocolViolationException`) |
| CRLF in payload | Included in message buffer | Excluded by design |
### Protocol Operations
| Operation | Go | .NET | Notes |
|-----------|:--:|:----:|-------|
| PUB | Y | Y | |
| HPUB (headers) | Y | Y | |
| SUB | Y | Y | |
| UNSUB | Y | Y | |
| CONNECT | Y | Y | |
| INFO | Y | Y | |
| PING / PONG | Y | Y | |
| MSG / HMSG | Y | Y | |
| +OK / -ERR | Y | Y | |
| RS+/RS-/RMSG (routes) | Y | Y | Parser/command matrix recognises opcodes; no wire routing — remote subscription propagation uses in-memory method calls; RMSG delivery not implemented |
| A+/A- (accounts) | Y | Y | Inter-server account protocol ops still pending |
| LS+/LS-/LMSG (leaf) | Y | Y | Leaf nodes are config-only stubs; no LS+/LS-/LMSG wire protocol handling |
### Protocol Parsing Gaps
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Multi-client-type command routing | Y | Y | Go checks `c.kind` to allow/reject commands |
| Protocol tracing in parser | Y | Y | `TraceInOp()` logs `<<- OP arg` at `LogLevel.Trace` via optional `ILogger` |
| Subject mapping (input→output) | Y | Y | Compiled `SubjectTransform` engine with 9 function tokens; wired into `ProcessMessage` |
| MIME header parsing | Y | Y | `NatsHeaderParser.Parse()` — status line + key-value headers from `ReadOnlySpan<byte>` |
| Message trace event initialization | Y | Y | |
### Protocol Writing
| Aspect | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| INFO serialization | Once at startup | Once at startup | Cached at startup; nonce connections serialize per-connection |
| MSG/HMSG construction | Direct buffer write | Span-based buffer write | `int.TryFormat` + `CopyTo` into rented buffer, no string allocations |
| Pre-encoded constants | Y | Y | Both pre-encode PING/PONG/OK |
---
## 4. Subscriptions & Subject Matching
### Trie Implementation
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Basic trie with `*`/`>` wildcards | Y | Y | Core matching identical |
| Queue group support | Y | Y | |
| Result caching (1024 max) | Y | Y | Same limits |
| `plist` optimization (>256 subs) | Y | Y | Go converts high-fanout nodes to array |
| Async cache sweep (background) | Y | Y | .NET sweeps inline under write lock |
| Atomic generation ID for invalidation | Y | Y | `Interlocked.Increment` on insert/remove; cached results store generation |
| Cache eviction strategy | Random | First-N | Semantic difference minimal |
### SubList Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| `Stats()` — comprehensive statistics | Y | Y | Matches, cache hits, inserts, removes tracked via `Interlocked` |
| `HasInterest()` — fast bool check | Y | Y | Walks trie without allocating result list |
| `NumInterest()` — fast count | Y | Y | Counts plain + queue subs without allocation |
| `ReverseMatch()` — pattern→literal query | Y | Y | Finds subscriptions whose wildcards match a literal subject |
| `RemoveBatch()` — efficient bulk removal | Y | Y | Single generation increment for batch; increments `_removes` per sub |
| `All()` — enumerate all subscriptions | Y | Y | Recursive trie walk returning all subscriptions |
| Notification system (interest changes) | Y | Y | |
| Local/remote subscription filtering | Y | Y | |
| Queue weight expansion (remote subs) | Y | Y | |
| `MatchBytes()` — zero-copy byte API | Y | Y | |
### Subject Validation
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Basic validation (empty tokens, wildcards) | Y | Y | |
| Literal subject check | Y | Y | |
| UTF-8/null rune validation | Y | Y | `IsValidSubject(string, bool checkRunes)` rejects null bytes |
| Collision detection (`SubjectsCollide`) | Y | Y | Token-by-token wildcard comparison; O(n) via upfront `Split` |
| Token utilities (`tokenAt`, `numTokens`) | Y | Y | `TokenAt` returns `ReadOnlySpan<char>`; `NumTokens` counts separators |
| Stack-allocated token buffer | Y | Y | Go uses `[32]string{}` on stack |
### Subscription Lifecycle
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-account subscription limit | Y | Y | `Account.IncrementSubscriptions()` returns false when `MaxSubscriptions` exceeded |
| Auto-unsubscribe on max messages | Y | Y | Enforced at delivery; sub removed from trie + client dict when exhausted |
| Subscription routing propagation | Y | Y | Remote subs tracked in trie and propagated over wire RS+/RS- with RMSG forwarding |
| Queue weight (`qw`) field | Y | Y | For remote queue load balancing |
---
## 5. Authentication & Authorization
### Auth Mechanisms
| Mechanism | Go | .NET | Notes |
|-----------|:--:|:----:|-------|
| Username/password | Y | Y | |
| Token | Y | Y | |
| NKeys (Ed25519) | Y | Y | .NET has framework but integration is basic |
| JWT validation | Y | Y | `NatsJwt` decode/verify, `JwtAuthenticator` with account resolution + revocation + `allowed_connection_types` enforcement |
| Bcrypt password hashing | Y | Y | .NET supports bcrypt (`$2*` prefix) with constant-time fallback |
| TLS certificate mapping | Y | Y | X500DistinguishedName with full DN match and CN fallback |
| Custom auth interface | Y | Y | |
| External auth callout | Y | Y | |
| Proxy authentication | Y | Y | |
| Bearer tokens | Y | Y | `UserClaims.BearerToken` skips nonce signature verification |
| User revocation tracking | Y | Y | Per-account `ConcurrentDictionary` with wildcard (`*`) revocation support |
### Account System
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Per-account SubList isolation | Y | Y | |
| Multi-account user resolution | Y | Y | `AccountConfig` per account in `NatsOptions.Accounts`; `GetOrCreateAccount` wires limits |
| Account exports/imports | Y | Y | ServiceImport/StreamImport with ExportAuth, subject transforms, response routing |
| Per-account connection limits | Y | Y | `Account.AddClient()` returns false when `MaxConnections` exceeded |
| Per-account subscription limits | Y | Y | `Account.IncrementSubscriptions()` enforced in `ProcessSub()` |
| Account JetStream limits | Y | Y | Enforced via account-level stream reservation limits |
### Permissions
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Publish allow list | Y | Y | |
| Subscribe allow list | Y | Y | |
| Publish deny list | Y | Y | Full enforcement with LRU-cached results |
| Subscribe deny list | Y | Y | Queue-aware deny checking in `IsSubscribeAllowed` |
| Message-level deny filtering | Y | Y | `IsDeliveryAllowed()` checked before MSG send; auto-unsub cleanup on deny |
| Permission caching (128 entries) | Y | Y | `PermissionLruCache` — Dictionary+LinkedList LRU, matching Go's `maxPermCacheSize` |
| Response permissions (reply tracking) | Y | Y | `ResponseTracker` with configurable TTL + max messages; not LRU-cached |
| Auth expiry enforcement | Y | Y | `Task.Delay` timer closes client when JWT/auth expires |
| Permission templates (JWT) | Y | Y | `PermissionTemplates.Expand()` — 6 functions with cartesian product for multi-value tags |
---
## 6. Configuration
### CLI Flags
| Flag | Go | .NET | Notes |
|------|:--:|:----:|-------|
| `-p/--port` | Y | Y | |
| `-a/--addr` | Y | Y | |
| `-n/--name` (ServerName) | Y | Y | |
| `-m/--http_port` (monitoring) | Y | Y | |
| `-c` (config file) | Y | Y | Full config parsing: lexer → parser → processor; CLI args override config |
| `-D/-V/-DV` (debug/trace) | Y | Y | `-D`/`--debug` for debug, `-V`/`-T`/`--trace` for trace, `-DV` for both |
| `--tlscert/--tlskey/--tlscacert` | Y | Y | |
| `--tlsverify` | Y | Y | |
| `--http_base_path` | Y | Y | |
| `--https_port` | Y | Y | |
### Configuration System
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Config file parsing | Y | Y | Custom NATS conf lexer/parser ported from Go; supports includes, variables, blocks |
| Hot reload (SIGHUP) | Y | Y | Reloads logging, auth, limits, TLS certs on SIGHUP; rejects non-reloadable changes |
| Config change detection | Y | Y | SHA256 digest comparison; `InCmdLine` tracks CLI flag precedence |
| ~450 option fields | Y | ~72 | .NET covers core + single-server options plus cluster/JetStream parsing and reload boundary validation |
### Missing Options Categories
- ~~Logging options~~ — file logging, rotation, syslog, debug/trace, color, timestamps, per-subsystem log control all implemented
- ~~Advanced limits (MaxSubs, MaxSubTokens, MaxPending, WriteDeadline)~~ — `MaxSubs`, `MaxSubTokens` implemented; MaxPending/WriteDeadline already existed
- ~~Tags/metadata~~ — `Tags` dictionary implemented in `NatsOptions`
- ~~OCSP configuration~~ — `OcspConfig` with 4 modes (Auto/Always/Must/Never), peer verification, and stapling
- ~~WebSocket options~~ — `WebSocketOptions` with port, compression, origin checking, cookie auth, custom headers
- ~~MQTT options~~ — `mqtt {}` config block parsed with all Go `MQTTOpts` fields; no listener yet
- ~~Operator mode / account resolver~~ — `JwtAuthenticator` + `IAccountResolver` + `MemAccountResolver` with trusted keys
---
## 7. Monitoring
### HTTP Endpoints
| Endpoint | Go | .NET | Notes |
|----------|:--:|:----:|-------|
| `/healthz` | Y | Y | |
| `/varz` | Y | Y | |
| `/connz` | Y | Y | |
| `/` (root listing) | Y | Y | |
| `/routez` | Y | Y | Returns live route counts via `RoutezHandler` |
| `/gatewayz` | Y | Y | Returns live gateway counts via `GatewayzHandler` |
| `/leafz` | Y | Y | Returns live leaf counts via `LeafzHandler` |
| `/subz` / `/subscriptionsz` | Y | Y | Account filtering, test subject filtering, pagination, and subscription details |
| `/accountz` | Y | Y | Returns runtime account summaries via `AccountzHandler` |
| `/accstatz` | Y | Y | Returns aggregate account stats via `AccountzHandler` |
| `/jsz` | Y | Y | Returns live JetStream counts/config and API totals/errors via `JszHandler` |
### Varz Response
| Field Category | Go | .NET | Notes |
|----------------|:--:|:----:|-------|
| Identity (ID, Name, Version) | Y | Y | |
| Network (Host, Port, URLs) | Y | Y | |
| Security (AuthRequired, TLS) | Y | Y | |
| Limits (MaxConn, MaxPayload) | Y | Y | |
| Timing (Start, Now, Uptime) | Y | Y | |
| Runtime (Mem, CPU, Cores) | Y | Y | |
| Connections (current, total) | Y | Y | |
| Messages (in/out msgs/bytes) | Y | Y | |
| SlowConsumer breakdown | Y | Y | Go tracks per connection type |
| Cluster/Gateway/Leaf blocks | Y | Y | Live route/gateway/leaf counters are exposed in dedicated endpoints |
| JetStream block | Y | Y | Includes live JetStream config, stream/consumer counts, and API totals/errors |
| TLS cert expiry info | Y | Y | `TlsCertNotAfter` loaded via `X509CertificateLoader` in `/varz` |
### Connz Response
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Filtering by CID, user, account | Y | Y | |
| Sorting (11 options) | Y | Y | All options including ByStop, ByReason, ByRtt |
| State filtering (open/closed/all) | Y | Y | `state=open|closed|all` query parameter |
| Closed connection tracking | Y | Y | `ConcurrentQueue<ClosedClient>` capped at 10,000 entries |
| Pagination (offset, limit) | Y | Y | |
| Subscription detail mode | Y | Y | |
| TLS peer certificate info | Y | Y | |
| JWT/IssuerKey/Tags fields | Y | Y | |
| MQTT client ID filtering | Y | Y | `mqtt_client` query param filters open and closed connections |
| Proxy info | Y | Y | |
---
## 8. TLS
### TLS Modes
| Mode | Go | .NET | Notes |
|------|:--:|:----:|-------|
| No TLS | Y | Y | |
| INFO-first (default NATS) | Y | Y | |
| TLS-first (before INFO) | Y | Y | |
| Mixed/Fallback | Y | Y | |
| TLS-required | Y | Y | |
### TLS Features
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| PEM cert/key loading | Y | Y | |
| CA chain validation | Y | Y | |
| Mutual TLS (client certs) | Y | Y | |
| Certificate pinning (SHA256 SPKI) | Y | Y | |
| TLS handshake timeout | Y | Y | |
| TLS rate limiting | Y | Y | Rate enforcement with refill; unit tests cover rate limiting and refill |
| First-byte peeking (0x16 detection) | Y | Y | |
| Cert subject→user mapping | Y | Y | X500DistinguishedName with full DN match and CN fallback |
| OCSP stapling | Y | Y | `SslStreamCertificateContext.Create` with `offline:false` for runtime OCSP fetch |
| Min TLS version control | Y | Y | |
---
## 9. Logging
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Structured logging | Y | Y | .NET uses Serilog with ILogger<T> |
| File logging with rotation | Y | Y | `-l`/`--log_file` flag + `LogSizeLimit`/`LogMaxFiles` via Serilog.Sinks.File |
| Syslog (local and remote) | Y | Y | `--syslog` and `--remote_syslog` flags via Serilog.Sinks.SyslogMessages |
| Log reopening (SIGUSR1) | Y | Y | SIGUSR1 handler calls ReOpenLogFile callback |
| Trace mode (protocol-level) | Y | Y | `-V`/`-T`/`--trace` flags; parser `TraceInOp()` logs at Trace level |
| Debug mode | Y | Y | `-D`/`--debug` flag lowers Serilog minimum to Debug |
| Per-subsystem log control | Y | Y | `--log_level_override ns=level` CLI flag with Serilog `MinimumLevel.Override` |
| Color output on TTY | Y | Y | Auto-detected via `Console.IsOutputRedirected`, uses `AnsiConsoleTheme.Code` |
| Timestamp format control | Y | Y | `--logtime` and `--logtime_utc` flags |
---
## 10. Ping/Pong & Keepalive
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Server-initiated PING | Y | Y | |
| Configurable interval | Y | Y | PingInterval option |
| Max pings out | Y | Y | MaxPingsOut option |
| Stale connection close | Y | Y | |
| RTT-based first PING delay | Y | Y | Skips PING until FirstPongSent or 2s elapsed |
| RTT tracking | Y | Y | `_rttStartTicks`/`Rtt` property, computed on PONG receipt |
| Stale connection stats | Y | Y | `StaleConnectionStats` model, exposed in `/varz` |
---
## Summary: Critical Gaps for Production Use
### Resolved Since Initial Audit
The following items from the original gap list have been implemented:
- **Slow consumer detection** — pending bytes threshold (64MB) with write deadline enforcement
- **Write coalescing / batch flush** — channel-based write loop drains all items before single flush
- **Verbose mode** — `+OK` responses for CONNECT, SUB, UNSUB, PUB when `verbose:true`
- **Permission deny enforcement at delivery** — `IsDeliveryAllowed` + auto-unsub cleanup
- **No-responders validation** — CONNECT rejects `no_responders` without `headers`; 503 HMSG on no match
- **File logging with rotation** — Serilog.Sinks.File with rolling file support
- **TLS certificate mapping** — X500DistinguishedName with full DN match and CN fallback
- **Protocol tracing** — `-V`/`-T` flag enables trace-level logging; `-D` for debug
- **Subscription statistics** — `Stats()`, `HasInterest()`, `NumInterest()`, etc.
- **Per-account limits** — connection + subscription limits via `AccountConfig`
- **Reply subject tracking** — `ResponseTracker` with TTL + max messages
- **JWT authentication** — `JwtAuthenticator` with decode/verify, account resolution, revocation, permission templates
- **OCSP support** — peer verification via `X509RevocationMode.Online`, stapling via `SslStreamCertificateContext`
- **Subject mapping** — compiled `SubjectTransform` engine with 9 function tokens, wired into message delivery
- **Windows Service integration** — `--service` flag with `Microsoft.Extensions.Hosting.WindowsServices`
- **Per-subsystem log control** — `--log_level_override` CLI flag with Serilog overrides
- **Per-client trace mode** — `SetTraceMode()` with dynamic parser logger toggling
- **Per-account stats** — `Interlocked` counters for InMsgs/OutMsgs/InBytes/OutBytes
- **TLS cert expiry in /varz** — `TlsCertNotAfter` populated via `X509CertificateLoader`
- **Permission templates** — `PermissionTemplates.Expand()` with 6 functions and cartesian product
- **Bearer tokens** — `UserClaims.BearerToken` skips nonce verification
- **User revocation** — per-account tracking with wildcard (`*`) revocation
- **Config file parsing** — custom lexer/parser ported from Go; supports includes, variables, nested blocks, size suffixes
- **Hot reload (SIGHUP)** — re-parses config, diffs changes, validates reloadable set, applies with CLI precedence
- **SYSTEM client type** — InternalClient with InternalEventSystem, Channel-based send/receive loops, event publishing
- **ACCOUNT client type** — lazy per-account InternalClient with import/export subscription support
- **System event publishing** — connect/disconnect advisories, server stats, shutdown/lame-duck events, auth errors
- **System request-reply services** — $SYS.REQ.SERVER.*.VARZ/CONNZ/SUBSZ/HEALTHZ/IDZ/STATSZ with ping wildcards
- **Account exports/imports** — service and stream imports with ExportAuth, subject transforms, response routing, latency tracking
---
## 11. JetStream
> The Go JetStream surface is ~37,500 lines across jetstream.go, stream.go, consumer.go, filestore.go, memstore.go, raft.go. The .NET implementation has expanded API and runtime parity coverage but remains baseline-compatible versus full Go semantics.
### JetStream API ($JS.API.* subjects)
| Subject | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| `STREAM.CREATE.<name>` | Y | Y | |
| `STREAM.INFO.<name>` | Y | Y | |
| `STREAM.UPDATE.<name>` | Y | Y | |
| `STREAM.DELETE.<name>` | Y | Y | |
| `STREAM.NAMES` | Y | Y | |
| `STREAM.LIST` | Y | Y | |
| `STREAM.PURGE.<name>` | Y | Y | |
| `STREAM.MSG.GET.<name>` | Y | Y | |
| `STREAM.MSG.DELETE.<name>` | Y | Y | |
| `DIRECT.GET.<name>` | Y | Y | Includes direct payload response shape |
| `CONSUMER.CREATE.<stream>` | Y | Y | |
| `CONSUMER.INFO.<stream>.<durable>` | Y | Y | |
| `CONSUMER.DELETE.<stream>.<durable>` | Y | Y | |
| `CONSUMER.NAMES.<stream>` | Y | Y | |
| `CONSUMER.LIST.<stream>` | Y | Y | |
| `CONSUMER.PAUSE.<stream>.<durable>` | Y | Y | |
| `CONSUMER.RESET.<stream>.<durable>` | Y | Y | |
| `CONSUMER.UNPIN.<stream>.<durable>` | Y | Y | |
| `CONSUMER.MSG.NEXT.<stream>.<durable>` | Y | Y | |
| `STREAM.LEADER.STEPDOWN.<name>` | Y | Y | |
| `META.LEADER.STEPDOWN` | Y | Y | |
| `STREAM.SNAPSHOT.<name>` | Y | Y | Snapshot/restore shape implemented; in-memory semantics |
| `STREAM.RESTORE.<name>` | Y | Y | Snapshot/restore shape implemented; in-memory semantics |
| `INFO` (account info) | Y | Y | |
### Stream Configuration
| Option | Go | .NET | Notes |
|--------|:--:|:----:|-------|
| Subjects | Y | Y | |
| Replicas | Y | Y | Wires RAFT replica count |
| MaxMsgs limit | Y | Y | Enforced via `EnforceLimits()` |
| Retention (Limits/Interest/WorkQueue) | Y | Y | Policy enums + validation branch exist; full runtime semantics incomplete |
| Discard policy (Old/New) | Y | Y | `Discard=New` now rejects writes when `MaxBytes` is exceeded |
| MaxBytes / MaxAge (TTL) | Y | Y | `MaxBytes` enforced; `MaxAge` model and parsing added, full TTL pruning not complete |
| MaxMsgsPer (per-subject limit) | Y | Y | Config model/parsing present; per-subject runtime cap remains limited |
| MaxMsgSize | Y | Y | |
| Storage type selection (Memory/File) | Y | Y | Per-stream backend selection supports memory and file stores |
| Compression (S2) | Y | Y | |
| Subject transform | Y | Y | |
| RePublish | Y | Y | |
| AllowDirect / KV mode | Y | Y | |
| Sealed, DenyDelete, DenyPurge | Y | Y | |
| Duplicates dedup window | Y | Y | Dedup ID cache exists; no configurable window |
### Consumer Configuration & Delivery
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Push delivery | Y | Y | `PushConsumerEngine`; basic delivery |
| Pull fetch | Y | Y | `PullConsumerEngine`; basic batch fetch |
| Ephemeral consumers | Y | Y | Ephemeral creation baseline auto-generates durable IDs when requested |
| AckPolicy.None | Y | Y | |
| AckPolicy.Explicit | Y | Y | `AckProcessor` tracks pending with expiry |
| AckPolicy.All | Y | Y | In-memory ack floor behavior implemented; full wire-level ack contract remains limited |
| Redelivery on ack timeout | Y | Y | `NextExpired()` detects expired; limit not enforced |
| DeliverPolicy (All/Last/New/StartSeq/StartTime) | Y | Y | Policy enums added; fetch behavior still mostly starts at beginning |
| FilterSubject (single) | Y | Y | |
| FilterSubjects (multiple) | Y | Y | Multi-filter matching implemented in pull/push delivery paths |
| MaxAckPending | Y | Y | Pending delivery cap enforced for consumer queues |
| Idle heartbeat | Y | Y | Push engine emits heartbeat frames for configured consumers |
| Flow control | Y | Y | |
| Rate limiting | Y | Y | |
| Replay policy | Y | Y | `ReplayPolicy.Original` baseline delay implemented; full Go timing semantics remain |
| BackOff (exponential) | Y | Y | |
### Storage Backends
| Feature | Go FileStore | .NET FileStore | Notes |
|---------|:--:|:----:|-------|
| Append / Load / Purge | Y | Y | Basic JSONL serialization |
| Recovery on restart | Y | Y | Loads JSONL on startup |
| Block-based layout (64 MB blocks) | Y | Y | .NET uses flat JSONL; not production-scale |
| S2 compression | Y | Y | |
| AES-GCM / ChaCha20 encryption | Y | Y | |
| Bit-packed sequence indexing | Y | Y | Simple dictionary |
| TTL / time-based expiry | Y | Y | |
MemStore has basic append/load/purge with `Dictionary<long, StoredMessage>` under a lock.
### Mirror & Sourcing
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Mirror consumer creation | Y | Y | `MirrorCoordinator` triggers on append |
| Mirror sync state tracking | Y | Y | |
| Source fan-in (multiple sources) | Y | Y | `Sources[]` array support added and replicated via `SourceCoordinator` |
| Subject mapping for sources | Y | Y | |
| Cross-account mirror/source | Y | Y | |
### RAFT Consensus
| Feature | Go (5 037 lines) | .NET (212 lines) | Notes |
|---------|:--:|:----:|-------|
| Leader election / term tracking | Y | Y | In-process; nodes hold direct `List<RaftNode>` references |
| Log append + quorum | Y | Y | Entries replicated via direct method calls; stale-term append now rejected |
| Log persistence | Y | Y | `RaftLog.PersistAsync/LoadAsync` plus node term/applied persistence baseline |
| Heartbeat / keep-alive | Y | Y | |
| Log mismatch resolution (NextIndex) | Y | Y | |
| Snapshot creation | Y | Y | `CreateSnapshotAsync()` exists; stored in-memory |
| Snapshot network transfer | Y | Y | |
| Membership changes | Y | Y | |
| Network RPC transport | Y | Y | `IRaftTransport` abstraction + in-memory transport baseline implemented |
### JetStream Clustering
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Meta-group governance | Y | Y | `JetStreamMetaGroup` tracks streams; no durable consensus |
| Per-stream replica group | Y | Y | `StreamReplicaGroup` + in-memory RAFT |
| Asset placement planner | Y | Y | `AssetPlacementPlanner` skeleton |
| Cross-cluster JetStream (gateways) | Y | Y | Requires functional gateways |
---
## 12. Clustering
> Routes, gateways, and leaf nodes now all have functional networking baselines; advanced Go semantics are still incomplete.
### Routes
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Listener accept loop | Y | Y | `RouteManager` binds and accepts inbound connections |
| Outbound seed connections (with backoff) | Y | Y | Iterates `ClusterOptions.Routes` with 250 ms retry |
| Route handshake (ROUTE `<serverId>`) | Y | Y | Bidirectional: sends own ID, reads peer ID |
| Remote subscription tracking | Y | Y | `ApplyRemoteSubscription` adds to SubList; `HasRemoteInterest` exposed |
| Subscription propagation (wire RS+/RS-) | Y | Y | Local SUB/UNSUB is propagated over route wire frames |
| Message routing (RMSG wire) | Y | Y | Routed publishes forward over RMSG to remote subscribers |
| RS+/RS- subscription protocol (wire) | Y | Y | Inbound RS+/RS- frames update remote-interest trie |
| Route pooling (3× per peer) | Y | Y | `ClusterOptions.PoolSize` defaults to 3 links per peer |
| Account-specific routes | Y | Y | |
| S2 compression on routes | Y | Y | |
| CONNECT info + topology gossip | Y | Y | Handshake is two-line text exchange only |
### Gateways
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Any networking (listener / outbound) | Y | Y | Listener + outbound remotes with retry are active |
| Gateway connection protocol | Y | Y | Baseline `GATEWAY` handshake implemented |
| Interest-only mode | Y | Y | Baseline A+/A- interest propagation implemented |
| Reply subject mapping (`_GR_.` prefix) | Y | Y | |
| Message forwarding to remote clusters | Y | Y | Baseline `GMSG` forwarding implemented |
### Leaf Nodes
| Feature | Go | .NET | Notes |
|---------|:--:|:----:|-------|
| Any networking (listener / spoke) | Y | Y | Listener + outbound remotes with retry are active |
| Leaf handshake / role negotiation | Y | Y | Baseline `LEAF` handshake implemented |
| Subscription sharing (LS+/LS-) | Y | Y | LS+/LS- propagation implemented |
| Loop detection (`$LDS.` prefix) | Y | Y | |
| Hub-and-spoke account mapping | Y | Y | Baseline LMSG forwarding works; advanced account remapping remains |
---
## Summary: Remaining Gaps
### Clustering (High Impact)
1. **Gateway advanced semantics** — reply remapping (`_GR_.`) and full interest-only behavior are not complete
2. **Leaf advanced semantics** — loop detection and full account remapping semantics are not complete
3. **Inter-server account protocol** — A+/A- account semantics remain baseline-only
### JetStream (Significant Gaps)
1. **Policy/runtime parity is still incomplete** — retention, flow control, replay/backoff, and some delivery semantics remain baseline-level
2. **FileStore scalability** — JSONL-based (not block/compressed/encrypted)
3. **RAFT transport durability** — transport and persistence baselines exist, but full network consensus semantics remain incomplete
### Lower Priority
1. **Dynamic buffer sizing** — delegated to Pipe, less optimized for long-lived connections
2. **`plist` optimization** — high-fanout nodes (>256 subs) not converted to array
3. **External auth callout / proxy auth** — custom auth interfaces not ported
4. **MQTT listener** — config parsed; no transport
5. **Inter-server account protocol (A+/A-)** — not implemented
---
## 13. JetStream Remaining Parity (2026-02-23)
### Newly Ported API Families
- `$JS.API.INFO`
- `$JS.API.SERVER.REMOVE`
- `$JS.API.ACCOUNT.PURGE.*`, `$JS.API.ACCOUNT.STREAM.MOVE.*`, `$JS.API.ACCOUNT.STREAM.MOVE.CANCEL.*`
- `$JS.API.STREAM.UPDATE.*`, `$JS.API.STREAM.DELETE.*`, `$JS.API.STREAM.NAMES`, `$JS.API.STREAM.LIST`
- `$JS.API.STREAM.PEER.REMOVE.*`
- `$JS.API.STREAM.MSG.GET.*`, `$JS.API.STREAM.MSG.DELETE.*`, `$JS.API.STREAM.PURGE.*`
- `$JS.API.DIRECT.GET.*`
- `$JS.API.STREAM.SNAPSHOT.*`, `$JS.API.STREAM.RESTORE.*`
- `$JS.API.CONSUMER.NAMES.*`, `$JS.API.CONSUMER.LIST.*`, `$JS.API.CONSUMER.DELETE.*.*`
- `$JS.API.CONSUMER.PAUSE.*.*`, `$JS.API.CONSUMER.RESET.*.*`, `$JS.API.CONSUMER.UNPIN.*.*`
- `$JS.API.CONSUMER.MSG.NEXT.*.*`
- `$JS.API.CONSUMER.LEADER.STEPDOWN.*.*`
- `$JS.API.STREAM.LEADER.STEPDOWN.*`, `$JS.API.META.LEADER.STEPDOWN`
### Runtime/Storage/RAFT Parity Additions
- JetStream publish precondition support for expected last sequence (`ErrorCode=10071` on mismatch).
- Pull consumer `no_wait` contract support (`TimedOut=false` on immediate empty fetch).
- Ack-all pending floor behavior via `AckProcessor.AckAll` and pending-count introspection.
- Stream store subject index support (`LoadLastBySubjectAsync`) in `MemStore` and `FileStore`.
- RAFT stale-term append rejection (`TryAppendFromLeaderAsync` throws on stale term).
- `/jsz` and `/varz` now expose JetStream API totals/errors from server stats.
- Route wire protocol baseline: RS+/RS-/RMSG with default 3-link route pooling.
- Gateway/Leaf wire protocol baselines: A+/A-/GMSG and LS+/LS-/LMSG.
- Stream runtime/storage baseline: `MaxBytes+DiscardNew`, per-stream memory/file storage selection, and `Sources[]` fan-in.
- Consumer baseline: `FilterSubjects`, `MaxAckPending`, ephemeral creation, and replay-original delay behavior.
- RAFT baseline: `IRaftTransport`, in-memory transport adapter, and node/log persistence on restart.
- Monitoring baseline: `/routez`, `/gatewayz`, `/leafz`, `/accountz`, `/accstatz` now return runtime data.
### Deep Operational Parity Closures (2026-02-23)
- Truth-matrix guardrails now enforce `differences.md`/parity-map alignment and contradiction detection.
- Internal JetStream client lifecycle is verified by runtime tests (`JetStreamInternalClientRuntimeTests`).
- Stream retention/runtime long-run guards now include retention-policy dispatch and dedupe-window expiry coverage.
- Consumer deliver-policy `LastPerSubject` now resolves the correct subject-scoped cursor.
- FileStore now persists a block-index manifest and reopens with manifest-backed index recovery.
- FileStore persisted payloads now use a versioned envelope with key-hash and payload-integrity validation.
- Deep runtime closure tests now cover flow/replay timing, RAFT append+convergence, governance, and cross-cluster forwarding paths.
### Remaining Explicit Deltas
- None after this deep operational parity cycle; stale contradictory notes were removed.