dohertj2/natsdotnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix E2E test gaps and add comprehensive E2E + parity test suites

Browse Source 
- Fix pull consumer fetch: send original stream subject in HMSG (not inbox)
  so NATS client distinguishes data messages from control messages
- Fix MaxAge expiry: add background timer in StreamManager for periodic pruning
- Fix JetStream wire format: Go-compatible anonymous objects with string enums,
  proper offset-based pagination for stream/consumer list APIs
- Add 42 E2E black-box tests (core messaging, auth, TLS, accounts, JetStream)
- Add ~1000 parity tests across all subsystems (gaps closure)
- Update gap inventory docs to reflect implementation status
This commit is contained in:
Joseph Doherty
2026-03-12 14:09:23 -04:00
parent 79c1ee8776
commit c30e67a69d
226 changed files with 17801 additions and 709 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
gaps/subscriptions.md
View File

@@ -100,14 +100,14 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| `level` (struct) | sublist.go:96 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:967` (private `TrieLevel`) | Inner class; `nodes`, `pwc`, `fwc` all present |
| `newNode()` | sublist.go:102 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:974` | Inline `new TrieNode()` |
| `newLevel()` | sublist.go:107 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:967` | Inline `new TrieLevel()` |
| `NewSublist(bool)` | sublist.go:117 | PARTIAL | `src/NATS.Server/Subscriptions/SubList.cs:11` | .NET `SubList` always starts with cache; no `enableCache` constructor param |
| `NewSublist(bool)` | sublist.go:117 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:40` | Added `SubList(bool enableCache)` constructor to explicitly control cache behavior |
| `NewSublistWithCache()` | sublist.go:125 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:11` | Default `new SubList()` has cache enabled |
| `NewSublistNoCache()` | sublist.go:130 | MISSING | — | No .NET equivalent; SubList always caches |
| `CacheEnabled()` | sublist.go:135 | MISSING | — | No public method to query whether cache is on; `CacheCount` exists but different semantics |
| `RegisterNotification()` | sublist.go:149 | MISSING | — | Go sends `true/false` on channel when first/last interest added/removed; .NET uses `InterestChanged` event which fires on every change but doesn't replicate channel-based deferred notify semantics |
| `RegisterQueueNotification()` | sublist.go:153 | MISSING | — | Queue-specific interest notification; no .NET equivalent |
| `ClearNotification()` | sublist.go:227 | MISSING | — | No .NET equivalent for removing a notification channel |
| `ClearQueueNotification()` | sublist.go:231 | MISSING | — | No .NET equivalent |
| `NewSublistNoCache()` | sublist.go:130 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:46` | Added `NewSublistNoCache()` factory returning `new SubList(false)` |
| `CacheEnabled()` | sublist.go:135 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:48` | Added `CacheEnabled()` to expose current cache mode |
| `RegisterNotification()` | sublist.go:149 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:50` | Added first/last-interest callback registration (`Action<bool>`) and transition notifications in `Insert`/`Remove` |
| `RegisterQueueNotification()` | sublist.go:153 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:57` | Added queue-scoped first/last-interest callback registration with immediate current-state callback and insert/remove transition tracking |
| `ClearNotification()` | sublist.go:227 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:52` | Added notification callback clearing method |
| `ClearQueueNotification()` | sublist.go:231 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:83` | Added queue-scoped notification de-registration across both insert/remove transition maps |
| `chkForInsertNotification()` | sublist.go:301 | NOT_APPLICABLE | — | Internal helper for channel notification; replaced by `InterestChanged` event emission in `Insert()` |
| `chkForRemoveNotification()` | sublist.go:317 | NOT_APPLICABLE | — | Internal helper; replaced by `InterestChanged` event emission in `Remove()` |
| `sendNotification()` | sublist.go:253 | NOT_APPLICABLE | — | Non-blocking channel send; Go-specific pattern, no .NET equivalent needed |
@@ -129,7 +129,7 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| `hasInterest()` | sublist.go:624 | NOT_APPLICABLE | — | Internal; maps to `HasInterestLevel()` private helper |
| `reduceCacheCount()` | sublist.go:675 | PORTED | `src/NATS.Server/Subscriptions/SubListCacheSweeper.cs:7` + `SubList.cs:458` | Background goroutine mapped to `SubListCacheSweeper` + `SweepCache()` |
| `isRemoteQSub()` | sublist.go:689 | NOT_APPLICABLE | — | Go has client.kind == ROUTER/LEAF; .NET uses `RemoteSubscription` model instead |
| `UpdateRemoteQSub()` | sublist.go:695 | MISSING | — | Go updates weight of remote qsub; .NET uses `ApplyRemoteSub()` for a different model (full add/remove) |
| `UpdateRemoteQSub()` | sublist.go:695 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:218` | Added remote queue-sub weight update path that mutates existing entry and bumps generation |
| `addNodeToResults()` | sublist.go:706 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:549` (private `AddNodeToResults()`) | Remote qsub weight expansion present in Go missing in .NET |
| `findQSlot()` | sublist.go:745 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:562` (inline in `AddNodeToResults`) | Inlined in .NET |
| `matchLevel()` | sublist.go:757 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:506` (private `MatchLevel()`) | Core trie descent algorithm |
@@ -144,33 +144,33 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| `Count()` | sublist.go:1023 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:41` (`Count` property) | |
| `CacheCount()` | sublist.go:1030 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:97` (`CacheCount` property) | |
| `SublistStats` (struct) | sublist.go:1038 | PORTED | `src/NATS.Server/Subscriptions/SubListStats.cs:3` | All public fields present; unexported fields `totFanout`, `cacheCnt`, `cacheHits` dropped (computed inline in `Stats()`) |
| `SublistStats.add()` | sublist.go:1052 | MISSING | — | Aggregates multiple SublistStats into one; used for cluster monitoring; no .NET equivalent |
| `SublistStats.add()` | sublist.go:1052 | PORTED | `src/NATS.Server/Subscriptions/SubListStats.cs:18` | Added `Add(SubListStats)` aggregation including cache-hit, fanout, and max-fanout rollups |
| `Stats()` | sublist.go:1076 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:580` | Full fanout stats computed |
| `numLevels()` | sublist.go:1120 | MISSING | — | Debug/test utility counting trie depth; not ported |
| `visitLevel()` | sublist.go:1126 | MISSING | — | Internal helper for `numLevels()`; not ported |
| `subjectHasWildcard()` | sublist.go:1159 | PARTIAL | `src/NATS.Server/Subscriptions/SubjectMatch.cs:50` (`IsLiteral()` — inverse) | .NET `!IsLiteral()` is equivalent but not a dedicated function |
| `numLevels()` | sublist.go:1120 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:1030` | Added internal trie-depth utility (`NumLevels`) for parity/debug verification |
| `visitLevel()` | sublist.go:1126 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:1223` | Added recursive depth traversal helper used by `NumLevels()` |
| `subjectHasWildcard()` | sublist.go:1159 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:72` | Added dedicated `SubjectHasWildcard()` helper |
| `subjectIsLiteral()` | sublist.go:1174 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:50` (`IsLiteral()`) | Exact equivalent |
| `IsValidPublishSubject()` | sublist.go:1187 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:67` | |
| `IsValidSubject()` | sublist.go:1192 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:9` | |
| `isValidSubject()` (internal, checkRunes) | sublist.go:1196 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:211` (`IsValidSubject(string, bool)`) | |
| `IsValidLiteralSubject()` | sublist.go:1236 | PARTIAL | `src/NATS.Server/Subscriptions/SubjectMatch.cs:50` (`IsLiteral()`) | `IsLiteral()` does not validate the subject first; `IsValidPublishSubject()` combines both |
| `IsValidLiteralSubject()` | sublist.go:1236 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:74` | Added dedicated `IsValidLiteralSubject()` helper mapped to publish-subject validation |
| `isValidLiteralSubject()` (tokens iter) | sublist.go:1241 | NOT_APPLICABLE | — | Takes `iter.Seq[string]` (Go 1.23 iterator); C# uses different iteration model |
| `ValidateMapping()` | sublist.go:1258 | MISSING | — | Validates a mapping destination subject string including `{{function()}}` syntax; no public .NET equivalent |
| `ValidateMapping()` | sublist.go:1258 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:138` | Added public destination-template validator with Go-compatible function parsing checks. |
| `analyzeTokens()` | sublist.go:1298 | NOT_APPLICABLE | — | Internal helper used only in `SubjectsCollide()`; logic inlined in .NET `SubjectsCollide()` |
| `tokensCanMatch()` | sublist.go:1314 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:199` (private `TokensCanMatch()`) | |
| `SubjectsCollide()` | sublist.go:1326 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:159` | |
| `numTokens()` | sublist.go:1374 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:118` (`NumTokens()`) | |
| `tokenAt()` | sublist.go:1389 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:132` (`TokenAt()`) | Go is 1-based index; .NET is 0-based |
| `tokenizeSubjectIntoSlice()` | sublist.go:1407 | NOT_APPLICABLE | — | Internal slice-reuse helper; .NET uses `Tokenize()` private method in SubList |
| `SubjectMatchesFilter()` | sublist.go:1421 | PARTIAL | `src/NATS.Server/JetStream/Storage/MemStore.cs:1175` (private), `FileStore.cs:773` (private) | Duplicated as private methods in MemStore and FileStore; not a public standalone function |
| `subjectIsSubsetMatch()` | sublist.go:1426 | MISSING | — | No public .NET equivalent; logic exists privately in MemStore/FileStore |
| `isSubsetMatch()` | sublist.go:1434 | MISSING | — | Internal; no public .NET equivalent |
| `isSubsetMatchTokenized()` | sublist.go:1444 | MISSING | — | Internal; no public .NET equivalent |
| `SubjectMatchesFilter()` | sublist.go:1421 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:205`; consumers: `src/NATS.Server/JetStream/Storage/MemStore.cs:1175`, `src/NATS.Server/JetStream/Storage/FileStore.cs:773` | Added standalone `SubjectMatch.SubjectMatchesFilter()` and switched JetStream stores to use it. |
| `subjectIsSubsetMatch()` | sublist.go:1426 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:207` (`SubjectIsSubsetMatch`) | Added direct port that tokenizes the subject and delegates to subset matching. |
| `isSubsetMatch()` | sublist.go:1434 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:213` (`IsSubsetMatch`) | Added token-array vs test-subject subset matcher. |
| `isSubsetMatchTokenized()` | sublist.go:1444 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:219` (`IsSubsetMatchTokenized`) | Added tokenized subset matcher with Go-compatible `*`/`>` handling. |
| `matchLiteral()` | sublist.go:1483 | PORTED | `src/NATS.Server/Subscriptions/SubjectMatch.cs:75` (`MatchLiteral()`) | |
| `addLocalSub()` | sublist.go:1552 | NOT_APPLICABLE | — | Filters by client kind (CLIENT/SYSTEM/JETSTREAM/ACCOUNT/LEAF); no .NET equivalent needed (client kind routing done elsewhere) |
| `Sublist.addNodeToSubs()` | sublist.go:1562 | NOT_APPLICABLE | — | Internal helper for `localSubs()`; not ported |
| `Sublist.collectLocalSubs()` | sublist.go:1581 | NOT_APPLICABLE | — | Internal helper for `localSubs()`; not ported |
| `Sublist.localSubs()` | sublist.go:1597 | MISSING | — | Returns only local-client subscriptions (excludes routes/gateways); no .NET equivalent |
| `Sublist.localSubs()` | sublist.go:1597 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:1015` | Added local-sub collector filtering to CLIENT/SYSTEM/JETSTREAM/ACCOUNT kinds with optional LEAF inclusion |
| `Sublist.All()` | sublist.go:1604 | PORTED | `src/NATS.Server/Subscriptions/SubList.cs:712` | |
| `Sublist.addAllNodeToSubs()` | sublist.go:1610 | NOT_APPLICABLE | — | Internal helper for `All()`; inlined in .NET |
| `Sublist.collectAllSubs()` | sublist.go:1627 | NOT_APPLICABLE | — | Internal; inlined in `CollectAllSubs()` private method in .NET |
@@ -182,26 +182,26 @@ Add rows to the Gap Inventory table below. Group by Go source file. Include the
| Go Symbol | Go File:Line | Status | .NET Equivalent | Notes |
|-----------|:-------------|--------|:----------------|-------|
| Transform type constants (`NoTransform`…`Random`) | subject_transform.go:43 | PARTIAL | `src/NATS.Server/Subscriptions/SubjectTransform.cs:682` (private `TransformType` enum) | `Random` (value 11 in Go) is absent from the .NET enum and switch statement; all others ported |
| Transform type constants (`NoTransform`…`Random`) | subject_transform.go:43 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:794` (private `TransformType` enum) | Added `Random` transform type and execution branch parity. |
| `subjectTransform` (struct) | subject_transform.go:61 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:11` | Internal fields mapped to `_source`, `_dest`, `_sourceTokens`, `_destTokens`, `_ops` |
| `SubjectTransformer` (interface) | subject_transform.go:73 | NOT_APPLICABLE | — | Go exports interface for polymorphism; C# uses concrete `SubjectTransform` class directly |
| `NewSubjectTransformWithStrict()` | subject_transform.go:81 | MISSING | — | Strict mode validates that all source wildcards are used in dest; no .NET equivalent |
| `NewSubjectTransformWithStrict()` | subject_transform.go:81 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:126` | Added strict factory variant that rejects transforms when any source wildcard is unused by destination mapping. |
| `NewSubjectTransform()` | subject_transform.go:198 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:31` (`Create()`) | Non-strict creation |
| `NewSubjectTransformStrict()` | subject_transform.go:202 | MISSING | — | Strict version for import mappings; no .NET equivalent |
| `NewSubjectTransformStrict()` | subject_transform.go:202 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:135` | Added strict convenience constructor delegating to strict factory mode. |
| `getMappingFunctionArgs()` | subject_transform.go:206 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:639` (private `GetFunctionArgs()`) | |
| `transformIndexIntArgsHelper()` | subject_transform.go:215 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:610` (private `ParseIndexIntArgs()`) | |
| `indexPlaceHolders()` | subject_transform.go:237 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:486` (`ParseDestToken()` + `ParseMustacheToken()`) | Split into two methods; `Random` branch missing |
| `transformTokenize()` | subject_transform.go:378 | MISSING | — | Converts `foo.*.*` to `foo.$1.$2`; used for import subject mapping reversal; no .NET equivalent |
| `transformUntokenize()` | subject_transform.go:399 | MISSING | — | Inverse of above; used in `reverse()`; no .NET equivalent |
| `indexPlaceHolders()` | subject_transform.go:237 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:586` (`ParseDestToken()` + `ParseMustacheToken()`) | Split into two methods, including `Random(...)` placeholder parsing branch. |
| `transformTokenize()` | subject_transform.go:378 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:156` | Added wildcard tokenization helper converting `*` capture positions into `$N` placeholders. |
| `transformUntokenize()` | subject_transform.go:399 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:173` | Added inverse helper converting `$N` placeholders back to `*` tokens. |
| `tokenizeSubject()` | subject_transform.go:414 | NOT_APPLICABLE | — | Internal tokenizer; .NET uses `string.Split('.')` or `Tokenize()` private method |
| `subjectTransform.Match()` | subject_transform.go:433 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:126` (`Apply()`) | Renamed; returns `null` instead of error on no-match |
| `subjectTransform.TransformSubject()` | subject_transform.go:456 | PARTIAL | `src/NATS.Server/Subscriptions/SubjectTransform.cs:126` (via `Apply()`) | `TransformSubject` (apply without match check) not separately exposed; `Apply()` always checks match |
| `subjectTransform.getRandomPartition()` | subject_transform.go:460 | MISSING | — | `Random` transform type not implemented in .NET |
| `subjectTransform.TransformSubject()` | subject_transform.go:456 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:251` | Added dedicated transform-only entrypoint (`TransformSubject`) that applies destination mapping without source match guard. |
| `subjectTransform.getRandomPartition()` | subject_transform.go:460 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:374` | Added random-partition helper and transform dispatch support (`random(n)` in range `[0,n)`, zero when `n<=0`). |
| `subjectTransform.getHashPartition()` | subject_transform.go:469 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:226` (`ComputePartition()` + `Fnv1A32()`) | FNV-1a 32-bit hash ported |
| `subjectTransform.TransformTokenizedSubject()` | subject_transform.go:482 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:144` (private `TransformTokenized()`) | All transform types except `Random` ported |
| `subjectTransform.reverse()` | subject_transform.go:638 | MISSING | — | Produces the inverse transform; used for import subject mapping; no .NET equivalent |
| `subjectTransform.TransformTokenizedSubject()` | subject_transform.go:482 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:261` (private `TransformTokenized()`) | Transform execution now includes `Random` and strict/helper parity branches. |
| `subjectTransform.reverse()` | subject_transform.go:638 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:188` | Added inverse-transform builder for wildcard-mapped transforms used in import mapping reversal scenarios. |
| `subjectInfo()` | subject_transform.go:666 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:451` (private `SubjectInfo()`) | |
| `ValidateMapping()` (in subject_transform.go context) | sublist.go:1258 | MISSING | — | Also defined via `NewSubjectTransform`; validates mapping destination with function syntax; no .NET public equivalent |
| `ValidateMapping()` (in subject_transform.go context) | sublist.go:1258 | PORTED | `src/NATS.Server/Subscriptions/SubjectTransform.cs:138` | Shared mapping validator exposed publicly and used for subject-transform destination validation parity. |
---
@@ -228,5 +228,9 @@ After porting work is completed:
| Date | Change | By |
|------|--------|----|
| 2026-02-26 | Executed subscriptions batch 4 subject-transform parity closures: added strict constructors (`NewSubjectTransformWithStrict`, `NewSubjectTransformStrict`), public mapping validator, random transform type/partition helper, tokenize/untokenize helpers, reverse-transform builder, and dedicated `TransformSubject` API with targeted tests (`SubjectTransformParityBatch3Tests`). | codex |
| 2026-02-25 | File created with LLM analysis instructions | auto |
| 2026-02-25 | Full gap inventory populated: analyzed sublist.go (~1,729 lines) and subject_transform.go (~689 lines) against all .NET Subscriptions/*.cs files. Counted 49 PORTED, 6 PARTIAL, 22 MISSING, 27 NOT_APPLICABLE, 0 DEFERRED. | claude-sonnet-4-6 |
| 2026-02-25 | Executed subscriptions batch 1: added cache-mode constructors/factories and first/last-interest notification APIs to `SubList`, added subject helper aliases to `SubjectMatch`, added targeted tests (`SubListCtorAndNotificationParityTests`), and reclassified 7 rows (4 MISSING + 3 PARTIAL) to PORTED | codex |
| 2026-02-25 | Executed subscriptions batch 2: added standalone subset/filter APIs to `SubjectMatch` (`SubjectMatchesFilter`, `SubjectIsSubsetMatch`, `IsSubsetMatch`, `IsSubsetMatchTokenized`), switched MemStore/FileStore subject filter helpers to use them, and added targeted tests (`SubjectSubsetMatchParityBatch1Tests`). Reclassified 4 open rows to PORTED. | codex |
| 2026-02-25 | Executed subscriptions batch 3: added queue-scoped notification APIs, remote queue-weight updater, `SubListStats.Add`, trie depth helpers (`NumLevels`/`VisitLevel`), and local-sub collection (`LocalSubs`) with targeted tests (`SubListParityBatch2Tests`). Reclassified 7 open rows to PORTED. | codex |