dohertj2/natsdotnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix E2E test gaps and add comprehensive E2E + parity test suites

Browse Source 
- Fix pull consumer fetch: send original stream subject in HMSG (not inbox)
  so NATS client distinguishes data messages from control messages
- Fix MaxAge expiry: add background timer in StreamManager for periodic pruning
- Fix JetStream wire format: Go-compatible anonymous objects with string enums,
  proper offset-based pagination for stream/consumer list APIs
- Add 42 E2E black-box tests (core messaging, auth, TLS, accounts, JetStream)
- Add ~1000 parity tests across all subsystems (gaps closure)
- Update gap inventory docs to reflect implementation status
This commit is contained in:
Joseph Doherty
2026-03-12 14:09:23 -04:00
parent 79c1ee8776
commit c30e67a69d
226 changed files with 17801 additions and 709 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
gaps/mqtt.md
View File

@@ -96,24 +96,24 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| mqttProtoLevel | mqtt.go:59 | PORTED | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:87-88 | Checked in ParseConnect |
| mqttConnFlag* constants | mqtt.go:62-68 | PORTED | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:96-104 | Decoded inline in ParseConnect |
| mqttPubFlag* constants | mqtt.go:71-75 | PORTED | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:161-163 | Decoded inline in ParsePublish |
| mqttSubscribeFlags | mqtt.go:78 | MISSING | | Subscribe flag validation not implemented in binary parser |
| mqttConnAckRC* constants | mqtt.go:85-91 | MISSING | | ConnAck return code constants not defined |
| mqttMaxPayloadSize | mqtt.go:94 | MISSING | | Max payload size constant not defined |
| mqttSubscribeFlags | mqtt.go:78 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:9 | Added SUBSCRIBE flags constant (`0x02`) and wired parser validation in `MqttBinaryDecoder.ParseSubscribe(..., flags)` |
| mqttConnAckRC* constants | mqtt.go:85-91 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:12 | Added MQTT 3.1.1 CONNACK return-code constants (`0x00`..`0x05`) for parity |
| mqttMaxPayloadSize | mqtt.go:94 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:20 | Added max payload constant and wired reader/writer remaining-length guardrails |
| mqttTopicLevelSep, mqttSingleLevelWC, mqttMultiLevelWC | mqtt.go:97-100 | PORTED | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:254-258 | Used in TranslateFilterToNatsSubject |
| mqttMultiLevelSidSuffix | mqtt.go:105 | MISSING | | Multi-level SID suffix for '#' wildcard not implemented |
| mqttPrefix, mqttSubPrefix | mqtt.go:108-113 | MISSING | | MQTT internal subject prefixes not defined |
| mqttStreamName, mqttStreamSubjectPrefix | mqtt.go:116-117 | MISSING | | JetStream stream naming not implemented |
| mqttRetainedMsgsStreamName | mqtt.go:120-121 | MISSING | | Retained messages stream not implemented |
| mqttSessStreamName | mqtt.go:124-125 | MISSING | | Session stream naming not implemented |
| mqttQoS2IncomingMsgsStreamName | mqtt.go:131-132 | MISSING | | QoS2 incoming stream not implemented |
| mqttOutStreamName, mqttPubRel* | mqtt.go:135-139 | MISSING | | PUBREL stream/subject not implemented |
| mqttDefaultAckWait | mqtt.go:145 | MISSING | | Default ack wait not defined (Go: 30s) |
| mqttMultiLevelSidSuffix | mqtt.go:105 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:29 | Added `MultiLevelSidSuffix = \" fwc\"` constant |
| mqttPrefix, mqttSubPrefix | mqtt.go:108-113 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:32 | Added MQTT internal subject prefixes (`Prefix`, `SubPrefix`) |
| mqttStreamName, mqttStreamSubjectPrefix | mqtt.go:116-117 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:36 | Added message stream name and subject-prefix constants |
| mqttRetainedMsgsStreamName | mqtt.go:120-121 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:38 | Added retained-message stream constants |
| mqttSessStreamName | mqtt.go:124-125 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:40 | Added session stream constants |
| mqttQoS2IncomingMsgsStreamName | mqtt.go:131-132 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:43 | Added QoS2 incoming stream constants |
| mqttOutStreamName, mqttPubRel* | mqtt.go:135-139 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:47 | Added outbound/PUBREL stream and subject-prefix constants |
| mqttDefaultAckWait | mqtt.go:145 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:23 | Added Go-parity default ack wait (`TimeSpan.FromSeconds(30)`) |
| mqttDefaultMaxAckPending | mqtt.go:149 | PARTIAL | src/NATS.Server/Mqtt/MqttFlowController.cs:15 | Default 1024 matches Go, but not wired to JetStream |
| mqttMaxAckTotalLimit | mqtt.go:153 | MISSING | | Max ack total limit (0xFFFF) not defined |
| mqttJSA* token constants | mqtt.go:156-177 | MISSING | | JetStream API reply subject tokens not implemented |
| mqttMaxAckTotalLimit | mqtt.go:153 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:26 | Added max ack total limit constant (`0xFFFF`) for flow/ack accounting parity |
| mqttJSA* token constants | mqtt.go:156-177 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:54 | Added JSA reply prefix, token positions, and stream/consumer/message token constants |
| mqttSessFlappingJailDur | mqtt.go:182 | PARTIAL | src/NATS.Server/Mqtt/MqttSessionStore.cs:96-106 | Flap detection exists but uses different default timing |
| sparkb* constants | mqtt.go:201-211 | MISSING | | Sparkplug B protocol constants not implemented |
| mqttNatsHeader* constants | mqtt.go:474-492 | MISSING | | NATS header names for MQTT message encoding not defined |
| sparkb* constants | mqtt.go:201-211 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:75 | Added Sparkplug birth/death constants and topic-prefix byte arrays |
| mqttNatsHeader* constants | mqtt.go:474-492 | PORTED | src/NATS.Server/Mqtt/MqttProtocolConstants.cs:83 | Added MQTT/NATS re-encoding header-name constants |
#### Core Types (lines 246-498)
@@ -122,14 +122,14 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| srvMQTT | mqtt.go:246 | PARTIAL | src/NATS.Server/Mqtt/MqttListener.cs:8-12 | Listener exists but no integration with Server struct or authOverride |
| mqttSessionManager | mqtt.go:253-256 | PARTIAL | src/NATS.Server/Mqtt/MqttSessionStore.cs:67 | Session store exists but not multi-account |
| mqttAccountSessionManager | mqtt.go:258-270 | PARTIAL | src/NATS.Server/Mqtt/MqttSessionStore.cs:67 | Single-account only, no JetStream backing, no retained msg sublist, no session hash |
| mqttJSA | mqtt.go:277-289 | MISSING | | JetStream API helper struct not implemented |
| mqttJSPubMsg | mqtt.go:291-296 | MISSING | | JS publish message type not implemented |
| mqttRetMsgDel | mqtt.go:298-301 | MISSING | | Retained message delete notification type not implemented |
| mqttJSA | mqtt.go:277-289 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttJsa`) | Added MQTT JetStream API helper model with account/reply-prefix/domain fields |
| mqttJSPubMsg | mqtt.go:291-296 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttJsPubMsg`) | Added JetStream publish message model (`Subject`, `Payload`, `ReplyTo`) |
| mqttRetMsgDel | mqtt.go:298-301 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttRetMsgDel`) | Added retained-message delete notification model (`Topic`, `Sequence`) |
| mqttSession | mqtt.go:303-344 | PARTIAL | src/NATS.Server/Mqtt/MqttSessionStore.cs:48-60 | MqttSessionData covers basic fields but missing pendingPublish/pendingPubRel maps, cpending, last_pi, maxp, tmaxack, JetStream consumer tracking |
| mqttPersistedSession | mqtt.go:346-353 | MISSING | | Persisted session JSON format not implemented |
| mqttRetainedMsg | mqtt.go:355-364 | PARTIAL | src/NATS.Server/Mqtt/MqttRetainedStore.cs:14 | MqttRetainedMessage exists but missing Origin, Flags, Source, Topic fields; no cache TTL |
| mqttRetainedMsgRef | mqtt.go:366-369 | MISSING | | Retained message reference (sseq + sub) not implemented |
| mqttSub | mqtt.go:375-391 | MISSING | | MQTT subscription metadata (qos, jsDur, prm, reserved) not implemented |
| mqttPersistedSession | mqtt.go:346-353 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttPersistedSession`) | Added persisted-session metadata model (`ClientId`, `LastPacketId`, `MaxAckPending`) |
| mqttRetainedMsg | mqtt.go:355-364 | PARTIAL | src/NATS.Server/Mqtt/MqttRetainedStore.cs:14 | Retained message model now includes `Origin`, `Flags`, `Source` in addition to topic/payload. Remaining: cache TTL + JetStream retention parity |
| mqttRetainedMsgRef | mqtt.go:366-369 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttRetainedMessageRef`) | Added retained-message reference model (`StreamSequence`, `Subject`) |
| mqttSub | mqtt.go:375-391 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttSub`) | Added MQTT subscription metadata model (`Filter`, `Qos`, `JsDur`, `Prm`, `Reserved`) |
| mqtt (client struct) | mqtt.go:393-408 | PARTIAL | src/NATS.Server/Mqtt/MqttConnection.cs:6-16 | MqttConnection exists but missing reader/writer, asm, sess, cid, rejectQoS2Pub, downgradeQoS2Sub |
| mqttPending | mqtt.go:410-414 | PARTIAL | src/NATS.Server/Mqtt/MqttQoS1Tracker.cs:89-96 | QoS1PendingMessage exists but missing sseq, jsAckSubject, jsDur fields |
| mqttConnectProto | mqtt.go:416-420 | PARTIAL | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:14-25 | MqttConnectInfo covers most fields but is a record struct, not mutable; missing rd (read deadline) |
@@ -137,9 +137,9 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| mqttReader | mqtt.go:427-433 | PARTIAL | src/NATS.Server/Mqtt/MqttPacketReader.cs:23-41 | MqttPacketReader handles fixed header; missing streaming buffer/partial packet support (pbuf, pstart) |
| mqttWriter | mqtt.go:435-437 | PORTED | src/NATS.Server/Mqtt/MqttPacketWriter.cs:3-38 | MqttPacketWriter covers write operations |
| mqttWill | mqtt.go:439-446 | PARTIAL | src/NATS.Server/Mqtt/MqttSessionStore.cs:35-42 | WillMessage exists but missing subject, mapped byte arrays; topic is string not bytes |
| mqttFilter | mqtt.go:448-453 | MISSING | | MQTT filter struct (filter, qos, ttopic) not implemented as a standalone type |
| mqttFilter | mqtt.go:448-453 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttFilter`) | Added standalone MQTT filter model (`Filter`, `Qos`, `TopicToken`) |
| mqttPublish | mqtt.go:455-463 | PARTIAL | src/NATS.Server/Mqtt/MqttBinaryDecoder.cs:31-37 | MqttPublishInfo covers most fields but missing subject, mapped byte arrays |
| mqttParsedPublishNATSHeader | mqtt.go:495-499 | MISSING | | Parsed NATS header struct for MQTT messages not implemented |
| mqttParsedPublishNATSHeader | mqtt.go:495-499 | PORTED | src/NATS.Server/Mqtt/MqttParityModels.cs (`MqttParsedPublishNatsHeader`) | Added parsed NATS-header model for MQTT publish flow (`Subject`, `Mapped`, publish/pubrel flags) |
#### Server Lifecycle Functions (lines 501-722)
@@ -363,8 +363,8 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| Go Symbol | Go File:Line | Status | .NET Equivalent | Notes |
|-----------|:-------------|--------|:----------------|-------|
| mqttWriter.WriteUint16 | mqtt.go:5850 | PARTIAL | src/NATS.Server/Mqtt/MqttPacketWriter.cs:12-14 | Inline in Write method, not a standalone helper |
| mqttWriter.WriteString | mqtt.go:5855 | MISSING | | Standalone string write (length-prefixed) not implemented |
| mqttWriter.WriteBytes | mqtt.go:5859 | MISSING | | Standalone bytes write (length-prefixed) not implemented |
| mqttWriter.WriteString | mqtt.go:5855 | PORTED | src/NATS.Server/Mqtt/MqttPacketWriter.cs:8 | Added standalone UTF-8 length-prefixed `WriteString` helper |
| mqttWriter.WriteBytes | mqtt.go:5859 | PORTED | src/NATS.Server/Mqtt/MqttPacketWriter.cs:11 | Added standalone length-prefixed `WriteBytes` helper with uint16 size guard |
| mqttWriter.WriteVarInt | mqtt.go:5864 | PORTED | src/NATS.Server/Mqtt/MqttPacketWriter.cs:19-37 | EncodeRemainingLength implemented |
| newMQTTWriter | mqtt.go:5878 | PARTIAL | src/NATS.Server/Mqtt/MqttPacketWriter.cs:3 | Static class, no constructor needed; Write method serves the purpose |
@@ -393,9 +393,9 @@ After porting work is completed:
| Status | Count |
|--------|-------|
| PORTED | 14 |
| PORTED | 39 |
| PARTIAL | 57 |
| MISSING | 119 |
| MISSING | 94 |
| NOT_APPLICABLE | 5 |
| DEFERRED | 0 |
| **Total** | **195** |
@@ -404,5 +404,7 @@ After porting work is completed:
| Date | Change | By |
|------|--------|----|
| 2026-02-25 | Ported MQTT constants/writer parity batch: internal subject/stream constants, JSA reply token constants, Sparkplug constants, MQTT→NATS header constants, plus standalone `MqttPacketWriter.WriteString/WriteBytes` helpers with targeted tests (`MqttProtocolConstantsParityBatch2Tests`). | codex |
| 2026-02-26 | Ported MQTT helper-model parity batch: added missing helper/data models (`MqttJsa`, `MqttJsPubMsg`, `MqttRetMsgDel`, `MqttPersistedSession`, `MqttRetainedMessageRef`, `MqttSub`, `MqttFilter`, `MqttParsedPublishNatsHeader`) and extended retained message model with `Origin`/`Flags`/`Source`; verified in `MqttModelParityBatch3Tests`. | codex |
| 2026-02-25 | Full gap analysis completed: 195 items analyzed. 14 PORTED, 57 PARTIAL, 119 MISSING, 5 NOT_APPLICABLE. Major gaps: JetStream integration (entire mqttJSA layer ~30 functions), binary protocol encoding (CONNACK/SUBACK/UNSUBACK/PUBLISH serialization), delivery callbacks (QoS0/QoS1/QoS2), account session management, retained message encoding/decoding, Sparkplug B support, NATS subject reverse mapping. .NET has solid foundation for packet reading/writing, connect parsing, basic pub/sub flow, QoS tracking, and retained store, but all are simplified/in-memory-only without JetStream backing. | claude |
| 2026-02-25 | File created with LLM analysis instructions | auto |