Files
natsdotnet/gaps/subscriptions.md
Joseph Doherty c30e67a69d Fix E2E test gaps and add comprehensive E2E + parity test suites
- Fix pull consumer fetch: send original stream subject in HMSG (not inbox)
  so NATS client distinguishes data messages from control messages
- Fix MaxAge expiry: add background timer in StreamManager for periodic pruning
- Fix JetStream wire format: Go-compatible anonymous objects with string enums,
  proper offset-based pagination for stream/consumer list APIs
- Add 42 E2E black-box tests (core messaging, auth, TLS, accounts, JetStream)
- Add ~1000 parity tests across all subsystems (gaps closure)
- Update gap inventory docs to reflect implementation status
2026-03-12 14:09:23 -04:00

24 KiB

Subscriptions — Gap Analysis

This file tracks what has and hasn't been ported from Go to .NET for the Subscriptions module. See stillmissing.md for the full LOC comparison across all modules.

LLM Instructions: How to Analyze This Category

Step 1: Read the Go Reference Files

Read each Go source file listed below. For every file:

  1. Extract all exported types (structs, interfaces, type aliases)
  2. Extract all exported methods on those types (receiver functions)
  3. Extract all exported standalone functions
  4. Note key constants, enums, and protocol states
  5. Note important unexported helpers that implement core logic (functions >20 lines)
  6. Pay attention to concurrency patterns (goroutines, mutexes, channels) — these map to different .NET patterns

Step 2: Read the .NET Implementation Files

Read all .cs files in the .NET directories listed below. For each Go symbol found in Step 1:

  1. Search for a matching type, method, or function in .NET
  2. If found, compare the behavior: does it handle the same edge cases? Same error paths?
  3. If partially implemented, note what's missing
  4. If not found, note it as MISSING

Step 3: Cross-Reference Tests

Compare Go test functions against .NET test methods:

  1. For each Go Test* function, check if a corresponding .NET [Fact] or [Theory] exists
  2. Note which test scenarios are covered and which are missing
  3. Check the parity DB (docs/test_parity.db) for existing mappings:
    sqlite3 docs/test_parity.db "SELECT go_test, dotnet_test, confidence FROM test_mappings tm JOIN go_tests gt ON tm.go_test_id=gt.rowid JOIN dotnet_tests dt ON tm.dotnet_test_id=dt.rowid WHERE gt.go_file LIKE '%PATTERN%'"
    

Step 4: Classify Each Item

Use these status values:

Status Meaning
PORTED Equivalent exists in .NET with matching behavior
PARTIAL .NET implementation exists but is incomplete (missing edge cases, error handling, or features)
MISSING No .NET equivalent found — needs to be ported
NOT_APPLICABLE Go-specific pattern that doesn't apply to .NET (build tags, platform-specific goroutine tricks, etc.)
DEFERRED Intentionally skipped for now (document why)

Step 5: Fill In the Gap Inventory

Add rows to the Gap Inventory table below. Group by Go source file. Include the Go file and line number so a porting LLM can jump directly to the reference implementation.

Key Porting Notes for Subscriptions

  • The Sublist trie is performance-critical — every published message triggers a Match() call.
  • Cache invalidation uses atomic generation counters (Interlocked in .NET).
  • subject_transform.go handles subject mapping for imports/exports — check if the .NET Imports/ directory covers this.

Go Reference Files (Source)

  • golang/nats-server/server/sublist.go — Trie-based subject matcher with wildcard support (* single, > multi). Nodes have psubs (plain), qsubs (queue groups). Results cached with atomic generation IDs.
  • golang/nats-server/server/subject_transform.go — Subject transformation logic (mapping, filtering)

Go Reference Files (Tests)

  • golang/nats-server/server/sublist_test.go
  • golang/nats-server/server/subject_transform_test.go

.NET Implementation Files (Source)

  • src/NATS.Server/Subscriptions/SubjectMatch.cs
  • src/NATS.Server/Subscriptions/SubList.cs
  • src/NATS.Server/Subscriptions/SubListResult.cs
  • src/NATS.Server/Subscriptions/Subscription.cs
  • All other files in src/NATS.Server/Subscriptions/

.NET Implementation Files (Tests)

  • tests/NATS.Server.Tests/Subscriptions/
  • tests/NATS.Server.Tests/SubList/

Gap Inventory

golang/nats-server/server/sublist.go

Go Symbol Go File:Line Status .NET Equivalent Notes
SublistResult (struct) sublist.go:59 PORTED src/NATS.Server/Subscriptions/SubListResult.cs:4 Fields renamed: psubsPlainSubs, qsubsQueueSubs; .NET uses arrays instead of slices
Sublist (struct) sublist.go:65 PORTED src/NATS.Server/Subscriptions/SubList.cs:11 Core trie struct ported; notification model differs (event vs channels)
notifyMaps (struct) sublist.go:80 NOT_APPLICABLE Go uses buffered channels for interest notifications; .NET uses InterestChanged event (InterestChange.cs)
node (struct) sublist.go:87 PORTED src/NATS.Server/Subscriptions/SubList.cs:974 (private TrieNode) Inner class; psubs+plist merged into PlainSubs HashSet, qsubsQueueSubs Dictionary
level (struct) sublist.go:96 PORTED src/NATS.Server/Subscriptions/SubList.cs:967 (private TrieLevel) Inner class; nodes, pwc, fwc all present
newNode() sublist.go:102 PORTED src/NATS.Server/Subscriptions/SubList.cs:974 Inline new TrieNode()
newLevel() sublist.go:107 PORTED src/NATS.Server/Subscriptions/SubList.cs:967 Inline new TrieLevel()
NewSublist(bool) sublist.go:117 PORTED src/NATS.Server/Subscriptions/SubList.cs:40 Added SubList(bool enableCache) constructor to explicitly control cache behavior
NewSublistWithCache() sublist.go:125 PORTED src/NATS.Server/Subscriptions/SubList.cs:11 Default new SubList() has cache enabled
NewSublistNoCache() sublist.go:130 PORTED src/NATS.Server/Subscriptions/SubList.cs:46 Added NewSublistNoCache() factory returning new SubList(false)
CacheEnabled() sublist.go:135 PORTED src/NATS.Server/Subscriptions/SubList.cs:48 Added CacheEnabled() to expose current cache mode
RegisterNotification() sublist.go:149 PORTED src/NATS.Server/Subscriptions/SubList.cs:50 Added first/last-interest callback registration (Action<bool>) and transition notifications in Insert/Remove
RegisterQueueNotification() sublist.go:153 PORTED src/NATS.Server/Subscriptions/SubList.cs:57 Added queue-scoped first/last-interest callback registration with immediate current-state callback and insert/remove transition tracking
ClearNotification() sublist.go:227 PORTED src/NATS.Server/Subscriptions/SubList.cs:52 Added notification callback clearing method
ClearQueueNotification() sublist.go:231 PORTED src/NATS.Server/Subscriptions/SubList.cs:83 Added queue-scoped notification de-registration across both insert/remove transition maps
chkForInsertNotification() sublist.go:301 NOT_APPLICABLE Internal helper for channel notification; replaced by InterestChanged event emission in Insert()
chkForRemoveNotification() sublist.go:317 NOT_APPLICABLE Internal helper; replaced by InterestChanged event emission in Remove()
sendNotification() sublist.go:253 NOT_APPLICABLE Non-blocking channel send; Go-specific pattern, no .NET equivalent needed
addInsertNotify() sublist.go:262 NOT_APPLICABLE Internal channel registration helper; replaced by event model
addRemoveNotify() sublist.go:268 NOT_APPLICABLE Internal channel registration helper
addNotify() sublist.go:274 NOT_APPLICABLE Internal channel map helper
keyFromSubjectAndQueue() sublist.go:290 NOT_APPLICABLE Key generation for notification maps; not needed in .NET event model
Insert() sublist.go:356 PORTED src/NATS.Server/Subscriptions/SubList.cs:179 Full trie insertion with queue support; fires InterestChanged event instead of channel notify
copyResult() sublist.go:449 NOT_APPLICABLE Go uses copy-on-write for cache mutation; .NET creates new SubListResult directly
SublistResult.addSubToResult() sublist.go:460 NOT_APPLICABLE Copy-on-write pattern for cache updates; not needed in .NET generation-based cache invalidation
addToCache() sublist.go:478 NOT_APPLICABLE Go updates individual cache entries on insert; .NET uses generation-based invalidation (simpler)
removeFromCache() sublist.go:498 NOT_APPLICABLE Same as above; .NET bumps generation counter instead
Match() sublist.go:520 PORTED src/NATS.Server/Subscriptions/SubList.cs:360 Generation-based cache; same algorithm
MatchBytes() sublist.go:526 PORTED src/NATS.Server/Subscriptions/SubList.cs:425 Takes ReadOnlySpan<byte> vs []byte
HasInterest() sublist.go:532 PORTED src/NATS.Server/Subscriptions/SubList.cs:638 Cache-aware fast path
NumInterest() sublist.go:537 PORTED src/NATS.Server/Subscriptions/SubList.cs:669 Returns (plainCount, queueCount) tuple
matchNoLock() sublist.go:543 NOT_APPLICABLE Internal; .NET lock model differs (no equivalent needed)
match() sublist.go:547 NOT_APPLICABLE Internal implementation; split into Match() + private helpers in .NET
hasInterest() sublist.go:624 NOT_APPLICABLE Internal; maps to HasInterestLevel() private helper
reduceCacheCount() sublist.go:675 PORTED src/NATS.Server/Subscriptions/SubListCacheSweeper.cs:7 + SubList.cs:458 Background goroutine mapped to SubListCacheSweeper + SweepCache()
isRemoteQSub() sublist.go:689 NOT_APPLICABLE Go has client.kind == ROUTER/LEAF; .NET uses RemoteSubscription model instead
UpdateRemoteQSub() sublist.go:695 PORTED src/NATS.Server/Subscriptions/SubList.cs:218 Added remote queue-sub weight update path that mutates existing entry and bumps generation
addNodeToResults() sublist.go:706 PORTED src/NATS.Server/Subscriptions/SubList.cs:549 (private AddNodeToResults()) Remote qsub weight expansion present in Go missing in .NET
findQSlot() sublist.go:745 PORTED src/NATS.Server/Subscriptions/SubList.cs:562 (inline in AddNodeToResults) Inlined in .NET
matchLevel() sublist.go:757 PORTED src/NATS.Server/Subscriptions/SubList.cs:506 (private MatchLevel()) Core trie descent algorithm
matchLevelForAny() sublist.go:785 PORTED src/NATS.Server/Subscriptions/SubList.cs:754 (private HasInterestLevel()) + CountInterestLevel() Split into two .NET helpers
remove() (internal) sublist.go:843 PORTED src/NATS.Server/Subscriptions/SubList.cs:282 (private RemoveInternal()) Internal removal logic
Remove() sublist.go:915 PORTED src/NATS.Server/Subscriptions/SubList.cs:255 Fires InterestChanged event
RemoveBatch() sublist.go:920 PORTED src/NATS.Server/Subscriptions/SubList.cs:687 Cache disable/re-enable pattern ported
level.pruneNode() sublist.go:953 PORTED src/NATS.Server/Subscriptions/SubList.cs:345 (inline in RemoveInternal) Inlined in removal path
node.isEmpty() sublist.go:968 PORTED src/NATS.Server/Subscriptions/SubList.cs:981 (TrieNode.IsEmpty property) Property instead of method
level.numNodes() sublist.go:978 NOT_APPLICABLE Used internally in visitLevel(); no .NET equivalent needed
removeFromNode() sublist.go:993 PORTED src/NATS.Server/Subscriptions/SubList.cs:324 (inline in RemoveInternal) Inlined
Count() sublist.go:1023 PORTED src/NATS.Server/Subscriptions/SubList.cs:41 (Count property)
CacheCount() sublist.go:1030 PORTED src/NATS.Server/Subscriptions/SubList.cs:97 (CacheCount property)
SublistStats (struct) sublist.go:1038 PORTED src/NATS.Server/Subscriptions/SubListStats.cs:3 All public fields present; unexported fields totFanout, cacheCnt, cacheHits dropped (computed inline in Stats())
SublistStats.add() sublist.go:1052 PORTED src/NATS.Server/Subscriptions/SubListStats.cs:18 Added Add(SubListStats) aggregation including cache-hit, fanout, and max-fanout rollups
Stats() sublist.go:1076 PORTED src/NATS.Server/Subscriptions/SubList.cs:580 Full fanout stats computed
numLevels() sublist.go:1120 PORTED src/NATS.Server/Subscriptions/SubList.cs:1030 Added internal trie-depth utility (NumLevels) for parity/debug verification
visitLevel() sublist.go:1126 PORTED src/NATS.Server/Subscriptions/SubList.cs:1223 Added recursive depth traversal helper used by NumLevels()
subjectHasWildcard() sublist.go:1159 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:72 Added dedicated SubjectHasWildcard() helper
subjectIsLiteral() sublist.go:1174 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:50 (IsLiteral()) Exact equivalent
IsValidPublishSubject() sublist.go:1187 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:67
IsValidSubject() sublist.go:1192 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:9
isValidSubject() (internal, checkRunes) sublist.go:1196 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:211 (IsValidSubject(string, bool))
IsValidLiteralSubject() sublist.go:1236 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:74 Added dedicated IsValidLiteralSubject() helper mapped to publish-subject validation
isValidLiteralSubject() (tokens iter) sublist.go:1241 NOT_APPLICABLE Takes iter.Seq[string] (Go 1.23 iterator); C# uses different iteration model
ValidateMapping() sublist.go:1258 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:138 Added public destination-template validator with Go-compatible function parsing checks.
analyzeTokens() sublist.go:1298 NOT_APPLICABLE Internal helper used only in SubjectsCollide(); logic inlined in .NET SubjectsCollide()
tokensCanMatch() sublist.go:1314 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:199 (private TokensCanMatch())
SubjectsCollide() sublist.go:1326 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:159
numTokens() sublist.go:1374 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:118 (NumTokens())
tokenAt() sublist.go:1389 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:132 (TokenAt()) Go is 1-based index; .NET is 0-based
tokenizeSubjectIntoSlice() sublist.go:1407 NOT_APPLICABLE Internal slice-reuse helper; .NET uses Tokenize() private method in SubList
SubjectMatchesFilter() sublist.go:1421 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:205; consumers: src/NATS.Server/JetStream/Storage/MemStore.cs:1175, src/NATS.Server/JetStream/Storage/FileStore.cs:773 Added standalone SubjectMatch.SubjectMatchesFilter() and switched JetStream stores to use it.
subjectIsSubsetMatch() sublist.go:1426 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:207 (SubjectIsSubsetMatch) Added direct port that tokenizes the subject and delegates to subset matching.
isSubsetMatch() sublist.go:1434 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:213 (IsSubsetMatch) Added token-array vs test-subject subset matcher.
isSubsetMatchTokenized() sublist.go:1444 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:219 (IsSubsetMatchTokenized) Added tokenized subset matcher with Go-compatible */> handling.
matchLiteral() sublist.go:1483 PORTED src/NATS.Server/Subscriptions/SubjectMatch.cs:75 (MatchLiteral())
addLocalSub() sublist.go:1552 NOT_APPLICABLE Filters by client kind (CLIENT/SYSTEM/JETSTREAM/ACCOUNT/LEAF); no .NET equivalent needed (client kind routing done elsewhere)
Sublist.addNodeToSubs() sublist.go:1562 NOT_APPLICABLE Internal helper for localSubs(); not ported
Sublist.collectLocalSubs() sublist.go:1581 NOT_APPLICABLE Internal helper for localSubs(); not ported
Sublist.localSubs() sublist.go:1597 PORTED src/NATS.Server/Subscriptions/SubList.cs:1015 Added local-sub collector filtering to CLIENT/SYSTEM/JETSTREAM/ACCOUNT kinds with optional LEAF inclusion
Sublist.All() sublist.go:1604 PORTED src/NATS.Server/Subscriptions/SubList.cs:712
Sublist.addAllNodeToSubs() sublist.go:1610 NOT_APPLICABLE Internal helper for All(); inlined in .NET
Sublist.collectAllSubs() sublist.go:1627 NOT_APPLICABLE Internal; inlined in CollectAllSubs() private method in .NET
ReverseMatch() sublist.go:1649 PORTED src/NATS.Server/Subscriptions/SubList.cs:727
reverseMatchLevel() sublist.go:1674 PORTED src/NATS.Server/Subscriptions/SubList.cs:860 (private ReverseMatchLevel())
getAllNodes() sublist.go:1714 PORTED src/NATS.Server/Subscriptions/SubList.cs:909 (private CollectAllNodes())

golang/nats-server/server/subject_transform.go

Go Symbol Go File:Line Status .NET Equivalent Notes
Transform type constants (NoTransformRandom) subject_transform.go:43 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:794 (private TransformType enum) Added Random transform type and execution branch parity.
subjectTransform (struct) subject_transform.go:61 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:11 Internal fields mapped to _source, _dest, _sourceTokens, _destTokens, _ops
SubjectTransformer (interface) subject_transform.go:73 NOT_APPLICABLE Go exports interface for polymorphism; C# uses concrete SubjectTransform class directly
NewSubjectTransformWithStrict() subject_transform.go:81 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:126 Added strict factory variant that rejects transforms when any source wildcard is unused by destination mapping.
NewSubjectTransform() subject_transform.go:198 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:31 (Create()) Non-strict creation
NewSubjectTransformStrict() subject_transform.go:202 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:135 Added strict convenience constructor delegating to strict factory mode.
getMappingFunctionArgs() subject_transform.go:206 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:639 (private GetFunctionArgs())
transformIndexIntArgsHelper() subject_transform.go:215 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:610 (private ParseIndexIntArgs())
indexPlaceHolders() subject_transform.go:237 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:586 (ParseDestToken() + ParseMustacheToken()) Split into two methods, including Random(...) placeholder parsing branch.
transformTokenize() subject_transform.go:378 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:156 Added wildcard tokenization helper converting * capture positions into $N placeholders.
transformUntokenize() subject_transform.go:399 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:173 Added inverse helper converting $N placeholders back to * tokens.
tokenizeSubject() subject_transform.go:414 NOT_APPLICABLE Internal tokenizer; .NET uses string.Split('.') or Tokenize() private method
subjectTransform.Match() subject_transform.go:433 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:126 (Apply()) Renamed; returns null instead of error on no-match
subjectTransform.TransformSubject() subject_transform.go:456 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:251 Added dedicated transform-only entrypoint (TransformSubject) that applies destination mapping without source match guard.
subjectTransform.getRandomPartition() subject_transform.go:460 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:374 Added random-partition helper and transform dispatch support (random(n) in range [0,n), zero when n<=0).
subjectTransform.getHashPartition() subject_transform.go:469 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:226 (ComputePartition() + Fnv1A32()) FNV-1a 32-bit hash ported
subjectTransform.TransformTokenizedSubject() subject_transform.go:482 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:261 (private TransformTokenized()) Transform execution now includes Random and strict/helper parity branches.
subjectTransform.reverse() subject_transform.go:638 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:188 Added inverse-transform builder for wildcard-mapped transforms used in import mapping reversal scenarios.
subjectInfo() subject_transform.go:666 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:451 (private SubjectInfo())
ValidateMapping() (in subject_transform.go context) sublist.go:1258 PORTED src/NATS.Server/Subscriptions/SubjectTransform.cs:138 Shared mapping validator exposed publicly and used for subject-transform destination validation parity.

Keeping This File Updated

After porting work is completed:

  1. Update status: Change MISSING → PORTED or PARTIAL → PORTED for each item completed
  2. Add .NET path: Fill in the ".NET Equivalent" column with the actual file:line
  3. Re-count LOC: Update the LOC numbers in stillmissing.md:
    # Re-count .NET source LOC for this module
    find src/NATS.Server/Subscriptions/ -name '*.cs' -type f -exec cat {} + | wc -l
    # Re-count .NET test LOC for this module
    find tests/NATS.Server.Tests/Subscriptions/ tests/NATS.Server.Tests/SubList/ -name '*.cs' -type f -exec cat {} + | wc -l
    
  4. Add a changelog entry below with date and summary of what was ported
  5. Update the parity DB if new test mappings were created:
    sqlite3 docs/test_parity.db "INSERT INTO test_mappings (go_test_id, dotnet_test_id, confidence, notes) VALUES (?, ?, 'manual', 'ported in YYYY-MM-DD session')"
    

Change Log

Date Change By
2026-02-26 Executed subscriptions batch 4 subject-transform parity closures: added strict constructors (NewSubjectTransformWithStrict, NewSubjectTransformStrict), public mapping validator, random transform type/partition helper, tokenize/untokenize helpers, reverse-transform builder, and dedicated TransformSubject API with targeted tests (SubjectTransformParityBatch3Tests). codex
2026-02-25 File created with LLM analysis instructions auto
2026-02-25 Full gap inventory populated: analyzed sublist.go (~1,729 lines) and subject_transform.go (~689 lines) against all .NET Subscriptions/*.cs files. Counted 49 PORTED, 6 PARTIAL, 22 MISSING, 27 NOT_APPLICABLE, 0 DEFERRED. claude-sonnet-4-6
2026-02-25 Executed subscriptions batch 1: added cache-mode constructors/factories and first/last-interest notification APIs to SubList, added subject helper aliases to SubjectMatch, added targeted tests (SubListCtorAndNotificationParityTests), and reclassified 7 rows (4 MISSING + 3 PARTIAL) to PORTED codex
2026-02-25 Executed subscriptions batch 2: added standalone subset/filter APIs to SubjectMatch (SubjectMatchesFilter, SubjectIsSubsetMatch, IsSubsetMatch, IsSubsetMatchTokenized), switched MemStore/FileStore subject filter helpers to use them, and added targeted tests (SubjectSubsetMatchParityBatch1Tests). Reclassified 4 open rows to PORTED. codex
2026-02-25 Executed subscriptions batch 3: added queue-scoped notification APIs, remote queue-weight updater, SubListStats.Add, trie depth helpers (NumLevels/VisitLevel), and local-sub collection (LocalSubs) with targeted tests (SubListParityBatch2Tests). Reclassified 7 open rows to PORTED. codex