From 9efe787cab66f2f29561dca08b7addb044a51805 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 23 Feb 2026 09:56:09 -0500 Subject: [PATCH] docs: update differences.md with accurate JetStream and clustering gaps Add sections 11 (JetStream) and 12 (Clustering) with verified coverage tables. Correct sections 2-4 where ROUTER/GATEWAY/LEAF and RS+/RS-/RMSG were previously marked Y but are partial or stub implementations: - ROUTER: handshake + in-memory subscription tracking only; no RMSG - GATEWAY/LEAF: config-only stubs with no networking - RS+/RS-/RMSG: command matrix only; no wire routing - JetStream: 4 of ~20 API subjects implemented; RAFT is in-memory simulation with no persistence or network transport --- differences.md | 200 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 193 insertions(+), 7 deletions(-) diff --git a/differences.md b/differences.md index 86d3de3..ef94f97 100644 --- a/differences.md +++ b/differences.md @@ -61,9 +61,9 @@ | Type | Go | .NET | Notes | |------|:--:|:----:|-------| | CLIENT | Y | Y | | -| ROUTER | Y | Y | Route handshake + routing primitives implemented | -| GATEWAY | Y | Y | Gateway manager bootstrap implemented | -| LEAF | Y | Y | Leaf node manager bootstrap implemented | +| ROUTER | Y | Partial | Route handshake + in-memory remote subscription tracking; no RMSG message routing, no RS+/RS- wire protocol, no route pooling (3x per peer) | +| GATEWAY | Y | Stub | Config parsing only; no listener, connections, handshake, interest-only mode, or message forwarding | +| LEAF | Y | Stub | Config parsing only; no listener, connections, handshake, subscription sharing, or loop detection | | SYSTEM (internal) | Y | Y | InternalClient + InternalEventSystem with Channel-based send/receive loops | | JETSTREAM (internal) | Y | N | | | ACCOUNT (internal) | Y | Y | Lazy per-account InternalClient with import/export subscription support | @@ -127,9 +127,9 @@ Go implements a sophisticated slow consumer detection system: | PING / PONG | Y | Y | | | MSG / HMSG | Y | Y | | | +OK / -ERR | Y | Y | | -| RS+/RS-/RMSG (routes) | Y | Y | Route protocol primitives implemented | +| RS+/RS-/RMSG (routes) | Y | N | Parser/command matrix recognises opcodes; no wire routing — remote subscription propagation uses in-memory method calls; RMSG delivery not implemented | | A+/A- (accounts) | Y | N | Inter-server account protocol ops still pending | -| LS+/LS-/LMSG (leaf) | Y | Y | Leaf protocol primitives implemented | +| LS+/LS-/LMSG (leaf) | Y | N | Leaf nodes are config-only stubs; no LS+/LS-/LMSG wire protocol handling | ### Protocol Parsing Gaps | Feature | Go | .NET | Notes | @@ -191,7 +191,7 @@ Go implements a sophisticated slow consumer detection system: |---------|:--:|:----:|-------| | Per-account subscription limit | Y | Y | `Account.IncrementSubscriptions()` returns false when `MaxSubscriptions` exceeded | | Auto-unsubscribe on max messages | Y | Y | Enforced at delivery; sub removed from trie + client dict when exhausted | -| Subscription routing propagation | Y | Y | Remote subscription propagation implemented for routes | +| Subscription routing propagation | Y | Partial | Remote subs tracked in trie; propagation via in-process method calls (no wire RS+/RS-); RMSG forwarding absent | | Queue weight (`qw`) field | Y | N | For remote queue load balancing | --- @@ -413,5 +413,191 @@ The following items from the original gap list have been implemented: - **System request-reply services** — $SYS.REQ.SERVER.*.VARZ/CONNZ/SUBSZ/HEALTHZ/IDZ/STATSZ with ping wildcards - **Account exports/imports** — service and stream imports with ExportAuth, subject transforms, response routing, latency tracking -### Remaining Lower Priority +--- + +## 11. JetStream + +> The Go JetStream surface is ~37,500 lines across jetstream.go, stream.go, consumer.go, filestore.go, memstore.go, raft.go. The .NET implementation has ~700 lines covering a minimal subset. Verified by `JetStreamApiRouter.cs` registration. + +### JetStream API ($JS.API.* subjects) + +| Subject | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| `STREAM.CREATE.` | Y | Y | Only API handler implemented | +| `STREAM.INFO.` | Y | Y | | +| `STREAM.UPDATE.` | Y | N | | +| `STREAM.DELETE.` | Y | N | | +| `STREAM.NAMES` | Y | N | | +| `STREAM.LIST` | Y | N | | +| `STREAM.PURGE.` | Y | N | Storage has `PurgeAsync()` but no API handler | +| `STREAM.MSG.GET.` | Y | N | | +| `STREAM.MSG.DELETE.` | Y | N | | +| `DIRECT.GET.` | Y | N | KV-style direct access | +| `CONSUMER.CREATE.` | Y | Y | | +| `CONSUMER.INFO..` | Y | Y | | +| `CONSUMER.DELETE..` | Y | N | | +| `CONSUMER.NAMES.` | Y | N | | +| `CONSUMER.LIST.` | Y | N | | +| `CONSUMER.PAUSE..` | Y | N | | +| `STREAM.LEADER.STEPDOWN.` | Y | N | | +| `META.LEADER.STEPDOWN` | Y | N | | +| `STREAM.SNAPSHOT.` | Y | N | | +| `STREAM.RESTORE.` | Y | N | | +| `INFO` (account info) | Y | N | | + +### Stream Configuration + +| Option | Go | .NET | Notes | +|--------|:--:|:----:|-------| +| Subjects | Y | Y | | +| Replicas | Y | Y | Wires RAFT replica count | +| MaxMsgs limit | Y | Y | Enforced via `EnforceLimits()` | +| Retention (Limits/Interest/WorkQueue) | Y | N | Only MaxMsgs trimming; Interest and WorkQueue not implemented | +| Discard policy (Old/New) | Y | N | | +| MaxBytes / MaxAge (TTL) | Y | N | | +| MaxMsgsPer (per-subject limit) | Y | N | | +| MaxMsgSize | Y | N | | +| Storage type selection (Memory/File) | Y | N | MemStore default; no config-driven choice | +| Compression (S2) | Y | N | | +| Subject transform | Y | N | | +| RePublish | Y | N | | +| AllowDirect / KV mode | Y | N | | +| Sealed, DenyDelete, DenyPurge | Y | N | | +| Duplicates dedup window | Y | Partial | Dedup ID cache exists; no configurable window | + +### Consumer Configuration & Delivery + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Push delivery | Y | Partial | `PushConsumerEngine`; basic delivery | +| Pull fetch | Y | Partial | `PullConsumerEngine`; basic batch fetch | +| Ephemeral consumers | Y | N | Only durable | +| AckPolicy.None | Y | Y | | +| AckPolicy.Explicit | Y | Y | `AckProcessor` tracks pending with expiry | +| AckPolicy.All | Y | N | | +| Redelivery on ack timeout | Y | Partial | `NextExpired()` detects expired; limit not enforced | +| DeliverPolicy (All/Last/New/StartSeq/StartTime) | Y | N | Always delivers from beginning | +| FilterSubject (single) | Y | Y | | +| FilterSubjects (multiple) | Y | N | | +| MaxAckPending | Y | N | | +| Idle heartbeat | Y | N | Field in model; not implemented | +| Flow control | Y | N | | +| Rate limiting | Y | N | | +| Replay policy | Y | N | | +| BackOff (exponential) | Y | N | | + +### Storage Backends + +| Feature | Go FileStore | .NET FileStore | Notes | +|---------|:--:|:----:|-------| +| Append / Load / Purge | Y | Y | Basic JSONL serialization | +| Recovery on restart | Y | Y | Loads JSONL on startup | +| Block-based layout (64 MB blocks) | Y | N | .NET uses flat JSONL; not production-scale | +| S2 compression | Y | N | | +| AES-GCM / ChaCha20 encryption | Y | N | | +| Bit-packed sequence indexing | Y | N | Simple dictionary | +| TTL / time-based expiry | Y | N | | + +MemStore has basic append/load/purge with `Dictionary` under a lock. + +### Mirror & Sourcing + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Mirror consumer creation | Y | Partial | `MirrorCoordinator` triggers on append | +| Mirror sync state tracking | Y | N | | +| Source fan-in (multiple sources) | Y | Partial | Single `Source` field; no `Sources[]` array | +| Subject mapping for sources | Y | N | | +| Cross-account mirror/source | Y | N | | + +### RAFT Consensus + +| Feature | Go (5 037 lines) | .NET (212 lines) | Notes | +|---------|:--:|:----:|-------| +| Leader election / term tracking | Y | Partial | In-process; nodes hold direct `List` references | +| Log append + quorum | Y | Partial | Entries replicated via direct method calls; no network | +| Log persistence | Y | N | In-memory `List` only | +| Heartbeat / keep-alive | Y | N | | +| Log mismatch resolution (NextIndex) | Y | N | | +| Snapshot creation | Y | Partial | `CreateSnapshotAsync()` exists; stored in-memory | +| Snapshot network transfer | Y | N | | +| Membership changes | Y | N | | +| Network RPC transport | Y | N | All coordination is in-process | + +### JetStream Clustering + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Meta-group governance | Y | Partial | `JetStreamMetaGroup` tracks streams; no durable consensus | +| Per-stream replica group | Y | Partial | `StreamReplicaGroup` + in-memory RAFT | +| Asset placement planner | Y | Partial | `AssetPlacementPlanner` skeleton | +| Cross-cluster JetStream (gateways) | Y | N | Requires functional gateways | + +--- + +## 12. Clustering + +> Routes, gateways, and leaf nodes are in different states of completeness. All three are present as Go reference feature targets but only routes have any functional networking. + +### Routes + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Listener accept loop | Y | Y | `RouteManager` binds and accepts inbound connections | +| Outbound seed connections (with backoff) | Y | Y | Iterates `ClusterOptions.Routes` with 250 ms retry | +| Route handshake (ROUTE ``) | Y | Y | Bidirectional: sends own ID, reads peer ID | +| Remote subscription tracking | Y | Y | `ApplyRemoteSubscription` adds to SubList; `HasRemoteInterest` exposed | +| Subscription propagation (in-process) | Y | Partial | `PropagateLocalSubscription` calls peer managers directly — no wire RS+/RS- protocol | +| Message routing (RMSG wire) | Y | N | **Critical gap**: published messages are never forwarded to remote subscribers | +| RS+/RS- subscription protocol (wire) | Y | N | Command matrix recognises opcodes but no handler processes inbound RS+/RS- frames | +| Route pooling (3× per peer) | Y | N | Single connection per remote server ID | +| Account-specific routes | Y | N | | +| S2 compression on routes | Y | N | | +| CONNECT info + topology gossip | Y | N | Handshake is two-line text exchange only | + +### Gateways + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Any networking (listener / outbound) | Y | N | `GatewayManager.StartAsync()` logs a debug line and zeros a counter | +| Gateway connection protocol | Y | N | | +| Interest-only mode | Y | N | | +| Reply subject mapping (`_GR_.` prefix) | Y | N | | +| Message forwarding to remote clusters | Y | N | | + +### Leaf Nodes + +| Feature | Go | .NET | Notes | +|---------|:--:|:----:|-------| +| Any networking (listener / spoke) | Y | N | `LeafNodeManager.StartAsync()` logs a debug line and zeros a counter | +| Leaf handshake / role negotiation | Y | N | | +| Subscription sharing (LS+/LS-) | Y | N | | +| Loop detection (`$LDS.` prefix) | Y | N | | +| Hub-and-spoke account mapping | Y | N | | + +--- + +## Summary: Remaining Gaps + +### Clustering (High Impact) +1. **Route message routing** — Remote subscribers receive no messages; no RMSG implementation +2. **Gateways** — Non-functional stub; no inter-cluster bridging +3. **Leaf nodes** — Non-functional stub; no hub/spoke topology +4. **RS+/RS- wire protocol** — subscription propagation is in-process method calls only +5. **Route pooling** — single connection per peer vs Go's 3-connection pool + +### JetStream (Significant Gaps) +1. **API coverage** — Only 4 of ~20 `$JS.API.*` subjects handled (STREAM.CREATE/INFO, CONSUMER.CREATE/INFO) +2. **Stream delete/update/list** — not implemented +3. **Consumer delete/list/pause** — not implemented +4. **Retention policies** — only MaxMsgs; Interest and WorkQueue retention absent +5. **FileStore scalability** — JSONL-based (not block/compressed/encrypted) +6. **RAFT persistence** — in-memory only; cluster restart loses all JetStream state +7. **Consumer delivery completeness** — DeliverPolicy, AckPolicy.All, MaxAckPending, heartbeats, flow control absent + +### Lower Priority 1. **Dynamic buffer sizing** — delegated to Pipe, less optimized for long-lived connections +2. **`plist` optimization** — high-fanout nodes (>256 subs) not converted to array +3. **External auth callout / proxy auth** — custom auth interfaces not ported +4. **MQTT listener** — config parsed; no transport +5. **Inter-server account protocol (A+/A-)** — not implemented