diff --git a/docs/plans/2026-02-23-websocket-design.md b/docs/plans/2026-02-23-websocket-design.md new file mode 100644 index 0000000..9bc4fe5 --- /dev/null +++ b/docs/plans/2026-02-23-websocket-design.md @@ -0,0 +1,322 @@ +# WebSocket Support Design + +## Overview + +Port WebSocket connection support from the Go NATS server (`golang/nats-server/server/websocket.go`, ~1,550 lines) to the .NET solution. Full feature parity: client/leaf/MQTT paths, HTTP upgrade handshake, custom frame parser with masking, permessage-deflate compression, browser compatibility, origin checking, cookie-based auth, and close frame handling. + +## Approach + +**Raw socket with manual HTTP upgrade** and **custom frame parser** — no ASP.NET Core WebSocket middleware, no `System.Net.WebSockets`. Direct port of Go's frame-level implementation for full control over masking negotiation, compression, fragmentation, and browser quirks. + +**Self-contained WebSocket module** under `src/NATS.Server/WebSocket/` with a `WsConnection` Stream wrapper that integrates transparently with the existing `NatsClient` read/write loops. + +## Go Reference + +- `server/websocket.go` — Main implementation (1,550 lines) +- `server/websocket_test.go` — Tests (4,982 lines) +- `server/opts.go` lines 518-610 — `WebsocketOpts` struct +- `server/client.go` — Integration points (`c.ws` field, `wsRead`, `wsCollapsePtoNB`) + +## File Structure + +``` +src/NATS.Server/WebSocket/ + WsConstants.cs — Opcodes, frame limits, close codes, compression magic bytes + WsReadInfo.cs — Per-connection frame reader state machine + WsFrameWriter.cs — Frame construction, masking, compression, fragmentation + WsUpgrade.cs — HTTP upgrade handshake validation and 101 response + WsConnection.cs — Stream wrapper bridging WS frames <-> NatsClient read/write + WsOriginChecker.cs — Same-origin and allowed-origins validation + WsCompression.cs — permessage-deflate via DeflateStream + +tests/NATS.Server.Tests/WebSocket/ + WsUpgradeTests.cs + WsFrameTests.cs + WsCompressionTests.cs + WsOriginCheckerTests.cs + WsIntegrationTests.cs +``` + +Modified existing files: +- `NatsOptions.cs` — Add `WebSocketOptions` class +- `NatsServer.cs` — Second accept loop for WebSocket port +- `NatsClient.cs` — `IsWebSocket` flag, `WsInfo` metadata property + +## Constants (WsConstants.cs) + +Direct port of Go constants: + +| Constant | Value | Purpose | +|----------|-------|---------| +| `WsTextMessage` | 1 | Text frame opcode | +| `WsBinaryMessage` | 2 | Binary frame opcode | +| `WsCloseMessage` | 8 | Close frame opcode | +| `WsPingMessage` | 9 | Ping frame opcode | +| `WsPongMessage` | 10 | Pong frame opcode | +| `WsFinalBit` | 0x80 | FIN bit in byte 0 | +| `WsRsv1Bit` | 0x40 | RSV1 (compression) in byte 0 | +| `WsMaskBit` | 0x80 | Mask bit in byte 1 | +| `WsMaxFrameHeaderSize` | 14 | Max frame header bytes | +| `WsMaxControlPayloadSize` | 125 | Max control frame payload | +| `WsFrameSizeForBrowsers` | 4096 | Browser fragmentation limit | +| `WsCompressThreshold` | 64 | Min payload size to compress | +| Close codes | 1000-1015 | RFC 6455 Section 11.7 | +| Paths | `/`, `/leafnode`, `/mqtt` | Client type routing | + +## HTTP Upgrade Handshake (WsUpgrade.cs) + +### Input +Raw `Stream` (TCP or TLS) after socket accept. + +### Validation (RFC 6455 Section 4.2.1) +1. Parse HTTP request line — must be `GET HTTP/1.1` +2. Parse headers into dictionary +3. Host header required +4. `Upgrade` header must contain `"websocket"` (case-insensitive) +5. `Connection` header must contain `"Upgrade"` (case-insensitive) +6. `Sec-WebSocket-Version` must be `"13"` +7. `Sec-WebSocket-Key` must be present +8. Path routing: `/` -> Client, `/leafnode` -> Leaf, `/mqtt` -> Mqtt +9. Origin checking via `WsOriginChecker` if configured +10. Compression: parse `Sec-WebSocket-Extensions` for `permessage-deflate` +11. No-masking: check `Nats-No-Masking` header (for leaf nodes) +12. Browser detection: `User-Agent` contains `"Mozilla/"`, Safari for `nocompfrag` +13. Cookie extraction: map configured cookie names to values +14. X-Forwarded-For: extract client IP + +### Response +``` +HTTP/1.1 101 Switching Protocols\r\n +Upgrade: websocket\r\n +Connection: Upgrade\r\n +Sec-WebSocket-Accept: \r\n +[Sec-WebSocket-Extensions: permessage-deflate; server_no_context_takeover; client_no_context_takeover\r\n] +[Nats-No-Masking: true\r\n] +[Custom headers\r\n] +\r\n +``` + +### Result Type +```csharp +public readonly record struct WsUpgradeResult( + bool Success, + bool Compress, + bool Browser, + bool NoCompFrag, + bool MaskRead, + bool MaskWrite, + string? CookieJwt, + string? CookieUsername, + string? CookiePassword, + string? CookieToken, + string? ClientIp, + WsClientKind Kind); + +public enum WsClientKind { Client, Leaf, Mqtt } +``` + +### Error Handling +Return standard HTTP error responses (400, 403, etc.) with body text for invalid requests. + +## Frame Reading (WsReadInfo.cs) + +### State Machine +```csharp +public struct WsReadInfo +{ + public int Remaining; // Bytes left in current frame payload + public bool FrameStart; // Reading new frame header + public bool FirstFrame; // First frame of fragmented message + public bool FrameCompressed; // Message is compressed (RSV1) + public bool ExpectMask; // Client frames should be masked + public byte MaskKeyPos; // Position in 4-byte mask key + public byte[] MaskKey; // 4-byte XOR mask + public List? CompressedBuffers; + public int CompressedOffset; +} +``` + +### Flow +1. Parse frame header: FIN, RSV1, opcode, mask, length, mask key +2. Handle control frames in-band (PING -> PONG, CLOSE -> close response) +3. Unmask payload bytes (XOR with cycling 4-byte key, optimized for 8-byte chunks) +4. If compressed: collect payloads across frames, decompress on final frame +5. Return unframed NATS protocol bytes + +### Decompression +- Append magic trailer `[0x00, 0x00, 0xff, 0xff]` before decompressing +- Use `DeflateStream` for decompression +- Validate decompressed size against `MaxPayload` + +## Frame Writing (WsFrameWriter.cs) + +### Per-Connection State +```csharp +public sealed class WsFrameWriter +{ + private readonly bool _compress; + private readonly bool _maskWrite; + private readonly bool _browser; + private readonly bool _noCompFrag; + private DeflateStream? _compressor; +} +``` + +### Flow +1. If compression enabled and payload > 64 bytes: compress via `DeflateStream` +2. If browser client and payload > 4096 bytes: fragment into chunks +3. Build frame headers: FIN | RSV1 | opcode | MASK | length | mask key +4. If masking: generate random 4-byte key, XOR payload +5. Control frame helpers: `EnqueuePing`, `EnqueuePong`, `EnqueueClose` + +### Close Frame +- 2-byte status code (big-endian) + optional UTF-8 reason +- Reason truncated to 125 bytes with `"..."` suffix +- Status code mapping from `ClientClosedReason`: + - ClientClosed -> 1000 (Normal) + - Auth failure -> 1008 (Policy Violation) + - Parse error -> 1002 (Protocol Error) + - Payload too big -> 1009 (Message Too Big) + - Other -> 1001 (Going Away) or 1011 (Server Error) + +## WsConnection Stream Wrapper (WsConnection.cs) + +Extends `Stream` to transparently wrap WebSocket framing around raw I/O: + +- `ReadAsync`: Calls `WsRead` to decode frames, buffers decoded payloads, returns NATS bytes to caller +- `WriteAsync`: Wraps payload in WS frames via `WsFrameWriter`, writes to inner stream +- `EnqueueControlMessage`: Called by read loop for PING responses and close frames + +NatsClient's `FillPipeAsync` and `RunWriteLoopAsync` work unchanged because `WsConnection` is a `Stream`. + +## NatsServer Integration + +### Second Accept Loop +``` +StartAsync: + if WebSocket.Port > 0: + create _wsListener socket + bind to WebSocket.Host:WebSocket.Port + start RunWebSocketAcceptLoopAsync +``` + +### WebSocket Accept Flow +``` +Accept socket + -> TLS negotiation (reuse TlsConnectionWrapper) + -> WsUpgrade.TryUpgradeAsync (HTTP upgrade) + -> Create WsConnection wrapping stream + -> Create NatsClient with WsConnection as stream + -> Set IsWebSocket = true, attach WsUpgradeResult + -> RunClientAsync (same as TCP from here) +``` + +### NatsClient Changes +- `public bool IsWebSocket { get; set; }` flag +- `public WsUpgradeResult? WsInfo { get; set; }` metadata +- No changes to read/write loops — `WsConnection` handles framing transparently + +### Shutdown +- `ShutdownAsync` also closes `_wsListener` +- WebSocket clients receive close frames before TCP disconnect +- `LameDuckShutdownAsync` includes WS clients in stagger-close + +## Configuration (WebSocketOptions) + +```csharp +public sealed class WebSocketOptions +{ + public string Host { get; set; } = "0.0.0.0"; + public int Port { get; set; } // 0 = disabled + public string? Advertise { get; set; } + public string? NoAuthUser { get; set; } + public string? JwtCookie { get; set; } + public string? UsernameCookie { get; set; } + public string? PasswordCookie { get; set; } + public string? TokenCookie { get; set; } + public string? Username { get; set; } + public string? Password { get; set; } + public string? Token { get; set; } + public TimeSpan AuthTimeout { get; set; } = TimeSpan.FromSeconds(2); + public bool NoTls { get; set; } + public string? TlsCert { get; set; } + public string? TlsKey { get; set; } + public bool SameOrigin { get; set; } + public List? AllowedOrigins { get; set; } + public bool Compression { get; set; } + public TimeSpan HandshakeTimeout { get; set; } = TimeSpan.FromSeconds(2); + public TimeSpan? PingInterval { get; set; } + public Dictionary? Headers { get; set; } +} +``` + +### Validation Rules +- Port 0 = disabled, skip remaining validation +- TLS required unless `NoTls = true` +- AllowedOrigins must be valid URLs +- Custom headers must not use reserved names +- NoAuthUser must match existing user if specified + +## Origin Checking (WsOriginChecker.cs) + +1. No Origin header -> accept (per RFC for non-browser clients) +2. If `SameOrigin`: compare origin host:port with request Host header +3. If `AllowedOrigins`: check host against list, match scheme and port + +Parsed from config URLs, stored as `Dictionary`. + +## Testing + +### Unit Tests + +**WsUpgradeTests.cs** +- Valid upgrade -> 101 +- Missing/invalid headers -> error codes +- Origin checking (same-origin, allowed, blocked) +- Compression negotiation +- No-masking header +- Browser/Safari detection +- Cookie extraction +- Path routing +- Handshake timeout +- Custom/reserved headers + +**WsFrameTests.cs** +- Read uncompressed frames (various sizes) +- Length encoding (7-bit, 16-bit, 64-bit) +- Masking/unmasking round-trip +- Control frames (PING, PONG, CLOSE) +- Close frame status code and reason +- Invalid frames (missing FIN on control, oversized control) +- Fragmented messages +- Compressed frame round-trip +- Browser fragmentation at 4096 +- Safari no-compressed-fragmentation + +**WsCompressionTests.cs** +- Compress/decompress round-trip +- Below threshold not compressed +- Large payload compression +- MaxPayload limit on decompression + +**WsOriginCheckerTests.cs** +- No origin -> accepted +- Same-origin match/mismatch +- Allowed origins match/mismatch +- Scheme and port matching + +### Integration Tests + +**WsIntegrationTests.cs** +- Connect via raw WebSocket, CONNECT/INFO exchange +- PUB/SUB over WebSocket +- Multiple WS clients +- Mixed TCP + WS clients interoperating +- WS with compression +- Graceful close (close frame exchange) +- Server shutdown sends close frames + +## Post-Implementation + +- Update `differences.md` to mark WebSocket support as implemented +- Update ports file to include WebSocket port