docs: add WebSocket support design document
Full port design for WebSocket connections from Go NATS server, including HTTP upgrade handshake, custom frame parser, compression, origin checking, and cookie-based auth.
This commit is contained in:
Joseph Doherty
322
docs/plans/2026-02-23-websocket-design.md
Normal file
322
docs/plans/2026-02-23-websocket-design.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# WebSocket Support Design
|
||||
|
||||
## Overview
|
||||
|
||||
Port WebSocket connection support from the Go NATS server (`golang/nats-server/server/websocket.go`, ~1,550 lines) to the .NET solution. Full feature parity: client/leaf/MQTT paths, HTTP upgrade handshake, custom frame parser with masking, permessage-deflate compression, browser compatibility, origin checking, cookie-based auth, and close frame handling.
|
||||
|
||||
## Approach
|
||||
|
||||
**Raw socket with manual HTTP upgrade** and **custom frame parser** — no ASP.NET Core WebSocket middleware, no `System.Net.WebSockets`. Direct port of Go's frame-level implementation for full control over masking negotiation, compression, fragmentation, and browser quirks.
|
||||
|
||||
**Self-contained WebSocket module** under `src/NATS.Server/WebSocket/` with a `WsConnection` Stream wrapper that integrates transparently with the existing `NatsClient` read/write loops.
|
||||
|
||||
## Go Reference
|
||||
|
||||
- `server/websocket.go` — Main implementation (1,550 lines)
|
||||
- `server/websocket_test.go` — Tests (4,982 lines)
|
||||
- `server/opts.go` lines 518-610 — `WebsocketOpts` struct
|
||||
- `server/client.go` — Integration points (`c.ws` field, `wsRead`, `wsCollapsePtoNB`)
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
src/NATS.Server/WebSocket/
|
||||
WsConstants.cs — Opcodes, frame limits, close codes, compression magic bytes
|
||||
WsReadInfo.cs — Per-connection frame reader state machine
|
||||
WsFrameWriter.cs — Frame construction, masking, compression, fragmentation
|
||||
WsUpgrade.cs — HTTP upgrade handshake validation and 101 response
|
||||
WsConnection.cs — Stream wrapper bridging WS frames <-> NatsClient read/write
|
||||
WsOriginChecker.cs — Same-origin and allowed-origins validation
|
||||
WsCompression.cs — permessage-deflate via DeflateStream
|
||||
|
||||
tests/NATS.Server.Tests/WebSocket/
|
||||
WsUpgradeTests.cs
|
||||
WsFrameTests.cs
|
||||
WsCompressionTests.cs
|
||||
WsOriginCheckerTests.cs
|
||||
WsIntegrationTests.cs
|
||||
```
|
||||
|
||||
Modified existing files:
|
||||
- `NatsOptions.cs` — Add `WebSocketOptions` class
|
||||
- `NatsServer.cs` — Second accept loop for WebSocket port
|
||||
- `NatsClient.cs` — `IsWebSocket` flag, `WsInfo` metadata property
|
||||
|
||||
## Constants (WsConstants.cs)
|
||||
|
||||
Direct port of Go constants:
|
||||
|
||||
| Constant | Value | Purpose |
|
||||
|----------|-------|---------|
|
||||
| `WsTextMessage` | 1 | Text frame opcode |
|
||||
| `WsBinaryMessage` | 2 | Binary frame opcode |
|
||||
| `WsCloseMessage` | 8 | Close frame opcode |
|
||||
| `WsPingMessage` | 9 | Ping frame opcode |
|
||||
| `WsPongMessage` | 10 | Pong frame opcode |
|
||||
| `WsFinalBit` | 0x80 | FIN bit in byte 0 |
|
||||
| `WsRsv1Bit` | 0x40 | RSV1 (compression) in byte 0 |
|
||||
| `WsMaskBit` | 0x80 | Mask bit in byte 1 |
|
||||
| `WsMaxFrameHeaderSize` | 14 | Max frame header bytes |
|
||||
| `WsMaxControlPayloadSize` | 125 | Max control frame payload |
|
||||
| `WsFrameSizeForBrowsers` | 4096 | Browser fragmentation limit |
|
||||
| `WsCompressThreshold` | 64 | Min payload size to compress |
|
||||
| Close codes | 1000-1015 | RFC 6455 Section 11.7 |
|
||||
| Paths | `/`, `/leafnode`, `/mqtt` | Client type routing |
|
||||
|
||||
## HTTP Upgrade Handshake (WsUpgrade.cs)
|
||||
|
||||
### Input
|
||||
Raw `Stream` (TCP or TLS) after socket accept.
|
||||
|
||||
### Validation (RFC 6455 Section 4.2.1)
|
||||
1. Parse HTTP request line — must be `GET <path> HTTP/1.1`
|
||||
2. Parse headers into dictionary
|
||||
3. Host header required
|
||||
4. `Upgrade` header must contain `"websocket"` (case-insensitive)
|
||||
5. `Connection` header must contain `"Upgrade"` (case-insensitive)
|
||||
6. `Sec-WebSocket-Version` must be `"13"`
|
||||
7. `Sec-WebSocket-Key` must be present
|
||||
8. Path routing: `/` -> Client, `/leafnode` -> Leaf, `/mqtt` -> Mqtt
|
||||
9. Origin checking via `WsOriginChecker` if configured
|
||||
10. Compression: parse `Sec-WebSocket-Extensions` for `permessage-deflate`
|
||||
11. No-masking: check `Nats-No-Masking` header (for leaf nodes)
|
||||
12. Browser detection: `User-Agent` contains `"Mozilla/"`, Safari for `nocompfrag`
|
||||
13. Cookie extraction: map configured cookie names to values
|
||||
14. X-Forwarded-For: extract client IP
|
||||
|
||||
### Response
|
||||
```
|
||||
HTTP/1.1 101 Switching Protocols\r\n
|
||||
Upgrade: websocket\r\n
|
||||
Connection: Upgrade\r\n
|
||||
Sec-WebSocket-Accept: <base64(SHA1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))>\r\n
|
||||
[Sec-WebSocket-Extensions: permessage-deflate; server_no_context_takeover; client_no_context_takeover\r\n]
|
||||
[Nats-No-Masking: true\r\n]
|
||||
[Custom headers\r\n]
|
||||
\r\n
|
||||
```
|
||||
|
||||
### Result Type
|
||||
```csharp
|
||||
public readonly record struct WsUpgradeResult(
|
||||
bool Success,
|
||||
bool Compress,
|
||||
bool Browser,
|
||||
bool NoCompFrag,
|
||||
bool MaskRead,
|
||||
bool MaskWrite,
|
||||
string? CookieJwt,
|
||||
string? CookieUsername,
|
||||
string? CookiePassword,
|
||||
string? CookieToken,
|
||||
string? ClientIp,
|
||||
WsClientKind Kind);
|
||||
|
||||
public enum WsClientKind { Client, Leaf, Mqtt }
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
Return standard HTTP error responses (400, 403, etc.) with body text for invalid requests.
|
||||
|
||||
## Frame Reading (WsReadInfo.cs)
|
||||
|
||||
### State Machine
|
||||
```csharp
|
||||
public struct WsReadInfo
|
||||
{
|
||||
public int Remaining; // Bytes left in current frame payload
|
||||
public bool FrameStart; // Reading new frame header
|
||||
public bool FirstFrame; // First frame of fragmented message
|
||||
public bool FrameCompressed; // Message is compressed (RSV1)
|
||||
public bool ExpectMask; // Client frames should be masked
|
||||
public byte MaskKeyPos; // Position in 4-byte mask key
|
||||
public byte[] MaskKey; // 4-byte XOR mask
|
||||
public List<byte[]>? CompressedBuffers;
|
||||
public int CompressedOffset;
|
||||
}
|
||||
```
|
||||
|
||||
### Flow
|
||||
1. Parse frame header: FIN, RSV1, opcode, mask, length, mask key
|
||||
2. Handle control frames in-band (PING -> PONG, CLOSE -> close response)
|
||||
3. Unmask payload bytes (XOR with cycling 4-byte key, optimized for 8-byte chunks)
|
||||
4. If compressed: collect payloads across frames, decompress on final frame
|
||||
5. Return unframed NATS protocol bytes
|
||||
|
||||
### Decompression
|
||||
- Append magic trailer `[0x00, 0x00, 0xff, 0xff]` before decompressing
|
||||
- Use `DeflateStream` for decompression
|
||||
- Validate decompressed size against `MaxPayload`
|
||||
|
||||
## Frame Writing (WsFrameWriter.cs)
|
||||
|
||||
### Per-Connection State
|
||||
```csharp
|
||||
public sealed class WsFrameWriter
|
||||
{
|
||||
private readonly bool _compress;
|
||||
private readonly bool _maskWrite;
|
||||
private readonly bool _browser;
|
||||
private readonly bool _noCompFrag;
|
||||
private DeflateStream? _compressor;
|
||||
}
|
||||
```
|
||||
|
||||
### Flow
|
||||
1. If compression enabled and payload > 64 bytes: compress via `DeflateStream`
|
||||
2. If browser client and payload > 4096 bytes: fragment into chunks
|
||||
3. Build frame headers: FIN | RSV1 | opcode | MASK | length | mask key
|
||||
4. If masking: generate random 4-byte key, XOR payload
|
||||
5. Control frame helpers: `EnqueuePing`, `EnqueuePong`, `EnqueueClose`
|
||||
|
||||
### Close Frame
|
||||
- 2-byte status code (big-endian) + optional UTF-8 reason
|
||||
- Reason truncated to 125 bytes with `"..."` suffix
|
||||
- Status code mapping from `ClientClosedReason`:
|
||||
- ClientClosed -> 1000 (Normal)
|
||||
- Auth failure -> 1008 (Policy Violation)
|
||||
- Parse error -> 1002 (Protocol Error)
|
||||
- Payload too big -> 1009 (Message Too Big)
|
||||
- Other -> 1001 (Going Away) or 1011 (Server Error)
|
||||
|
||||
## WsConnection Stream Wrapper (WsConnection.cs)
|
||||
|
||||
Extends `Stream` to transparently wrap WebSocket framing around raw I/O:
|
||||
|
||||
- `ReadAsync`: Calls `WsRead` to decode frames, buffers decoded payloads, returns NATS bytes to caller
|
||||
- `WriteAsync`: Wraps payload in WS frames via `WsFrameWriter`, writes to inner stream
|
||||
- `EnqueueControlMessage`: Called by read loop for PING responses and close frames
|
||||
|
||||
NatsClient's `FillPipeAsync` and `RunWriteLoopAsync` work unchanged because `WsConnection` is a `Stream`.
|
||||
|
||||
## NatsServer Integration
|
||||
|
||||
### Second Accept Loop
|
||||
```
|
||||
StartAsync:
|
||||
if WebSocket.Port > 0:
|
||||
create _wsListener socket
|
||||
bind to WebSocket.Host:WebSocket.Port
|
||||
start RunWebSocketAcceptLoopAsync
|
||||
```
|
||||
|
||||
### WebSocket Accept Flow
|
||||
```
|
||||
Accept socket
|
||||
-> TLS negotiation (reuse TlsConnectionWrapper)
|
||||
-> WsUpgrade.TryUpgradeAsync (HTTP upgrade)
|
||||
-> Create WsConnection wrapping stream
|
||||
-> Create NatsClient with WsConnection as stream
|
||||
-> Set IsWebSocket = true, attach WsUpgradeResult
|
||||
-> RunClientAsync (same as TCP from here)
|
||||
```
|
||||
|
||||
### NatsClient Changes
|
||||
- `public bool IsWebSocket { get; set; }` flag
|
||||
- `public WsUpgradeResult? WsInfo { get; set; }` metadata
|
||||
- No changes to read/write loops — `WsConnection` handles framing transparently
|
||||
|
||||
### Shutdown
|
||||
- `ShutdownAsync` also closes `_wsListener`
|
||||
- WebSocket clients receive close frames before TCP disconnect
|
||||
- `LameDuckShutdownAsync` includes WS clients in stagger-close
|
||||
|
||||
## Configuration (WebSocketOptions)
|
||||
|
||||
```csharp
|
||||
public sealed class WebSocketOptions
|
||||
{
|
||||
public string Host { get; set; } = "0.0.0.0";
|
||||
public int Port { get; set; } // 0 = disabled
|
||||
public string? Advertise { get; set; }
|
||||
public string? NoAuthUser { get; set; }
|
||||
public string? JwtCookie { get; set; }
|
||||
public string? UsernameCookie { get; set; }
|
||||
public string? PasswordCookie { get; set; }
|
||||
public string? TokenCookie { get; set; }
|
||||
public string? Username { get; set; }
|
||||
public string? Password { get; set; }
|
||||
public string? Token { get; set; }
|
||||
public TimeSpan AuthTimeout { get; set; } = TimeSpan.FromSeconds(2);
|
||||
public bool NoTls { get; set; }
|
||||
public string? TlsCert { get; set; }
|
||||
public string? TlsKey { get; set; }
|
||||
public bool SameOrigin { get; set; }
|
||||
public List<string>? AllowedOrigins { get; set; }
|
||||
public bool Compression { get; set; }
|
||||
public TimeSpan HandshakeTimeout { get; set; } = TimeSpan.FromSeconds(2);
|
||||
public TimeSpan? PingInterval { get; set; }
|
||||
public Dictionary<string, string>? Headers { get; set; }
|
||||
}
|
||||
```
|
||||
|
||||
### Validation Rules
|
||||
- Port 0 = disabled, skip remaining validation
|
||||
- TLS required unless `NoTls = true`
|
||||
- AllowedOrigins must be valid URLs
|
||||
- Custom headers must not use reserved names
|
||||
- NoAuthUser must match existing user if specified
|
||||
|
||||
## Origin Checking (WsOriginChecker.cs)
|
||||
|
||||
1. No Origin header -> accept (per RFC for non-browser clients)
|
||||
2. If `SameOrigin`: compare origin host:port with request Host header
|
||||
3. If `AllowedOrigins`: check host against list, match scheme and port
|
||||
|
||||
Parsed from config URLs, stored as `Dictionary<string, (string scheme, int port)>`.
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**WsUpgradeTests.cs**
|
||||
- Valid upgrade -> 101
|
||||
- Missing/invalid headers -> error codes
|
||||
- Origin checking (same-origin, allowed, blocked)
|
||||
- Compression negotiation
|
||||
- No-masking header
|
||||
- Browser/Safari detection
|
||||
- Cookie extraction
|
||||
- Path routing
|
||||
- Handshake timeout
|
||||
- Custom/reserved headers
|
||||
|
||||
**WsFrameTests.cs**
|
||||
- Read uncompressed frames (various sizes)
|
||||
- Length encoding (7-bit, 16-bit, 64-bit)
|
||||
- Masking/unmasking round-trip
|
||||
- Control frames (PING, PONG, CLOSE)
|
||||
- Close frame status code and reason
|
||||
- Invalid frames (missing FIN on control, oversized control)
|
||||
- Fragmented messages
|
||||
- Compressed frame round-trip
|
||||
- Browser fragmentation at 4096
|
||||
- Safari no-compressed-fragmentation
|
||||
|
||||
**WsCompressionTests.cs**
|
||||
- Compress/decompress round-trip
|
||||
- Below threshold not compressed
|
||||
- Large payload compression
|
||||
- MaxPayload limit on decompression
|
||||
|
||||
**WsOriginCheckerTests.cs**
|
||||
- No origin -> accepted
|
||||
- Same-origin match/mismatch
|
||||
- Allowed origins match/mismatch
|
||||
- Scheme and port matching
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**WsIntegrationTests.cs**
|
||||
- Connect via raw WebSocket, CONNECT/INFO exchange
|
||||
- PUB/SUB over WebSocket
|
||||
- Multiple WS clients
|
||||
- Mixed TCP + WS clients interoperating
|
||||
- WS with compression
|
||||
- Graceful close (close frame exchange)
|
||||
- Server shutdown sends close frames
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
- Update `differences.md` to mark WebSocket support as implemented
|
||||
- Update ports file to include WebSocket port
|
||||
Reference in New Issue
Block a user