Files
natsnet/docs/plans/2026-03-01-deferred-features-implementation-plan.md
Joseph Doherty 41ea272c8a chore(batch42-47): reconcile porting.db after all deferred feature batches
- Promoted 226 deferred features to verified (audit name mismatches)
- 1 stub remains (mqtt transferUniqueSessStreamsToMuxed — needs running server)
- Features: 3626 verified + 22 complete + 24 n/a + 1 stub = 3673 total (98.7%)
- Tests: 2066 verified + 307 n/a + 884 deferred = 3257 total
- Overall: 6057/6942 items complete (87.3%)
2026-03-01 10:18:54 -05:00

41 KiB

Deferred Features Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Implement 363 deferred features + 1 stub across 6 batches to bring feature coverage from 82% to ~92%.

Architecture: Each batch runs as a parallel Claude Code Sonnet agent in an isolated git worktree. Agents port full method bodies from Go to idiomatic C#. After each batch merges to main, run PortTracker audit to promote features to verified.

Tech Stack: .NET 10, C# latest, xUnit 3 + Shouldly + NSubstitute (testing standards but no new tests in this phase)


Execution Model

Each batch is dispatched as an Agent tool call with model: "sonnet" and isolation: "worktree". The agent receives a self-contained prompt with all context needed to implement the batch independently. After the agent completes, its worktree changes are merged to main.

Wave 1 (parallel): Batches 42 + 43 Wave 2 (parallel, after Wave 1): Batches 44 + 46 Wave 3 (after Batch 44): Batch 45 Wave 4 (after Batch 45): Batch 47

Merge Protocol (after each batch)

cd <worktree-path>
git add -A && git commit -m "feat(batch<N>): <description>"
# Back in main:
git merge <worktree-branch> --no-edit
# If porting.db conflicts: git checkout --ours porting.db
# Run: dotnet build dotnet/
# Run: dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
# Run audit: dotnet run --project tools/NatsNet.PortTracker -- audit --type all --db porting.db --execute

Shared Coding Standards (included in every batch prompt)

All agents must follow these rules:

  1. .NET 10, C# latest — file-scoped namespaces, nullable enabled, implicit usings
  2. Naming: PascalCase for public members, _camelCase for private fields
  3. Namespace: ZB.MOM.NatsNet.Server (or .Internal, .Mqtt for MQTT-specific code)
  4. Go→.NET patterns:
    • sync.RWMutexReaderWriterLockSlim
    • sync.MutexLock (C# 13)
    • atomic.Int64Interlocked.Read/Exchange/Add on long
    • chan struct{}Channel<bool> or CancellationTokenSource
    • time.DurationTimeSpan
    • goroutine + tickerSystem.Threading.Timer
  5. Do NOT use FluentAssertions or Moq
  6. Do NOT write new unit tests — implement features only
  7. Do NOT modify porting.db — it will be reconciled after merge
  8. Do NOT switch branches or merge — work only in the worktree branch
  9. Read the Go source for each method before porting — use: cat golang/nats-server/<file> | head -n <end> | tail -n <count>
  10. Build gate: dotnet build dotnet/ must pass after all changes
  11. Test gate: dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q must show 0 failures
  12. Commit all changes with a single commit at the end

Task 1: Batch 42 — Foundation Helpers (54 features + 1 stub)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/MessageTrace/MsgTraceTypes.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/MessageTrace/MsgTrace.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorHelpers.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs (if GatewayInterestMode.String not already there)
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Internal/MsgScheduling.cs

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 42: Foundation Helpers — 54 deferred features + 1 stub feature. These are standalone helper methods with no server runtime dependencies.

## Go Source Files to Read

- `golang/nats-server/server/msgtrace.go` (799 lines) — MsgTrace implementation
- `golang/nats-server/server/monitor.go` (4240 lines) — Monitor helper functions
- `golang/nats-server/server/gateway.go` — GatewayInterestMode.String (search for `func (mode InterestMode) String()`)
- `golang/nats-server/server/scheduler.go` lines 158-244 — getScheduledMessages

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/MessageTrace/MsgTraceTypes.cs` — existing type hierarchy (MsgTraceBase, MsgTraceIngress, etc.)
- `dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorTypes.cs` — existing Connz, ConnInfo, ConnzOptions types
- `dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorSortOptions.cs` — existing SortOpt types
- `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` — check if GatewayInterestMode.String exists
- `dotnet/src/ZB.MOM.NatsNet.Server/Internal/MsgScheduling.cs` — stub location for getScheduledMessages
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — NatsServer fields reference
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` — ClientConnection fields reference

## Features to Implement

### Group A: MsgTrace (28 features)

Create `MessageTrace/MsgTrace.cs` with a `MsgTrace` class that implements the trace event pipeline. Port these methods from `server/msgtrace.go`:

Feature IDs and methods:
- 2406: MsgTraceIngress.new (factory method)
- 2407: MsgTraceSubjectMapping.new (factory method)
- 2408: MsgTraceStreamExport.new (factory method)
- 2409: MsgTraceServiceImport.new (factory method)
- 2410: MsgTraceJetStream.new (factory method)
- 2411: MsgTraceEgress.new (factory method)
- 2412: MsgTraceEvents.UnmarshalJSON (deserialization)
- 2413: getTraceAs (helper to extract trace destination)
- 2420: client.isMsgTraceEnabled (check if tracing active)
- 2421: client.msgTraceSupport (check if client supports tracing)
- 2422: getConnName (get connection name for trace)
- 2423: getCompressionType (get compression type string)
- 2424: client.initMsgTrace (initialize tracing on a message)
- 2425: sample (probabilistic sampling check)
- 2426: genHeaderMapIfTraceHeadersPresent (parse trace headers)
- 2427: client.initAndSendIngressErrEvent (send error event)
- 2428: msgTrace.traceOnly (check if trace-only mode)
- 2429: msgTrace.setOriginAccountHeaderIfNeeded (set origin header)
- 2430: msgTrace.setHopHeader (set hop count header)
- 2431: msgTrace.setIngressError (mark ingress error)
- 2432: msgTrace.addSubjectMappingEvent (add mapping event)
- 2433: msgTrace.addEgressEvent (add egress event)
- 2434: msgTrace.addStreamExportEvent (add export event)
- 2435: msgTrace.addServiceImportEvent (add import event)
- 2436: msgTrace.addJetStreamEvent (add JS event)
- 2437: msgTrace.updateJetStreamEvent (update JS event)
- 2438: msgTrace.sendEventFromJetStream (send from JS context)
- 2439: msgTrace.sendEvent (send trace event)

### Group B: Gateway String (1 feature)

- 1265: GatewayInterestMode.String — check `Gateway/GatewayTypes.cs`. If `GatewayInterestModeExtensions.String()` already exists with full implementation, skip. Otherwise implement it by reading the Go `func (mode InterestMode) String()` in `server/gateway.go`.

### Group C: Monitor Helpers (25 features)

Create `Monitor/MonitorHelpers.cs` with standalone helper functions. Port from `server/monitor.go`:

- 2166: newSubsDetailList — create SubDetail list
- 2167: newSubsList — create simple subscription list
- 2170: createProxyInfo — build proxy metadata for ConnInfo
- 2171: makePeerCerts — extract peer certificate info
- 2173: decodeBool — decode bool from query param
- 2174: decodeUint64 — decode uint64 from query param
- 2175: decodeInt — decode int from query param
- 2176: decodeState — decode ConnState from query param
- 2177: decodeSubs — decode subscription filter from query param
- 2181: newSubDetail — create single SubDetail
- 2182: newClientSubDetail — create SubDetail from client subscription
- 2188: myUptime — format server uptime as human-readable string
- 2189: tlsCertNotAfter — get TLS certificate expiry
- 2194: urlsToStrings — convert URL list to string list
- 2196: getPinnedCertsAsSlice — get pinned cert hashes as slice
- 2200: getMonitorGWOptions — parse gateway monitor query options
- 2202: createOutboundRemoteGatewayz — build outbound remote gateway info
- 2203: createOutboundAccountsGatewayz — build outbound accounts gateway info
- 2204: createAccountOutboundGatewayz — build account outbound gateway info
- 2206: createInboundAccountsGatewayz — build inbound accounts gateway info
- 2207: createInboundAccountGatewayz — build inbound account gateway info
- 2213: ResponseHandler — HTTP response handler type/delegate
- 2214: handleResponse — generic HTTP response handler
- 2216: newExtServiceLatency — build external service latency info
- 2219: newExtImport — build external import info

### Group D: Scheduler Stub (1 feature)

- 2962: MsgScheduling.getScheduledMessages — replace the stub comment in `Internal/MsgScheduling.cs` (around line 201) with the full implementation. Read `golang/nats-server/server/scheduler.go` lines 158-244 for the Go source. This method takes two loader delegates (loadMsg and loadLast), iterates expired schedules, validates headers, strips schedule-specific headers, adds new headers, and returns sorted InMsg list.

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch42): implement foundation helpers — msgtrace, monitor helpers, scheduler"
```

Task 2: Batch 43 — Accounts Resolvers & Service Latency (37 features)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Accounts/AccountResolver.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ServiceExports.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Accounts/AccountTypes.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Accounts.cs

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 43: Accounts Resolvers & Service Latency — 37 deferred features related to account resolution, service export timers, latency tracking, JWT/claims validation, and distributed tracing headers.

## Go Source Files to Read

- `golang/nats-server/server/accounts.go` (4774 lines) — all account resolver and service export methods

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/AccountResolver.cs` — IAccountResolver interface, ResolverDefaultsOps base, MemoryAccountResolver
- `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/AccountTypes.cs` — AccountLimits, ServiceRespType, ServiceLatency, serviceExport, serviceImport types
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Accounts.cs` — existing account management methods
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — server fields (_accResolver, _accounts, etc.)
- `dotnet/src/ZB.MOM.NatsNet.Server/Auth/AuthTypes.cs` — User, NkeyUser types

## Features to Implement

### Group A: Type Methods (3 features)

- 150: ServiceRespType.String — string representation of ServiceRespType enum
- 183: NewMapDest — factory for mapping destination
- 208: ServiceLatency.merge — merge two latency measurements

### Group B: Tracing Headers (4 features)

Port from accounts.go — these create distributed tracing propagation headers:
- 250: newB3Header — create Zipkin B3 tracing header
- 251: newUberHeader — create Uber/Jaeger tracing header
- 252: newTraceCtxHeader — create W3C TraceContext header
- 253: shouldSample — probabilistic sampling decision

### Group C: Service Export Timers (4 features)

Port from accounts.go — service export response tracking:
- 258: serviceImport.isRespServiceImport — check if import is response-type
- 259: serviceExport.setResponseThresholdTimer — set response timeout timer
- 260: serviceExport.clearResponseThresholdTimer — clear response timeout timer
- 261: serviceExport.checkExpiredResponses — check and clean up expired responses

### Group D: Latency & Client Updates (2 features)

- 209: sanitizeLatencyMetric — clean up latency metric name
- 215: updateAllClientsServiceExportResponseTime — update response time for all clients

### Group E: Account Resolvers (18 features)

Extend `AccountResolver.cs` with concrete resolver implementations:

- 306: authAccounts — authenticate against configured accounts
- 307: Server.SetAccountResolver — set the account resolver on the server
- 310: Server.UpdateAccountClaims — update account from JWT claims
- 316: Server.updateAccountClaimsWithRefresh — update with refresh
- 318: buildPermissionsFromJwt — build permission set from JWT claims
- 319: buildInternalNkeyUser — build internal NKey user for system account
- 320: fetchAccount — fetch account by name or public key
- 326: resolverDefaultsOpsImpl.Store — default store implementation
- 327: MemAccResolver.Fetch — in-memory resolver fetch (may already exist — check first)
- 328: MemAccResolver.Store — in-memory resolver store (may already exist — check first)
- 330: NewURLAccResolver — factory for URL-based resolver
- 331: URLAccResolver.Fetch — HTTP-based account fetch
- 334: respondToUpdate — respond to account update request
- 335: handleListRequest — handle account list API request
- 336: handleDeleteRequest — handle account delete API request
- 337: getOperatorKeys — get operator signing keys
- 338: claimValidate — validate account JWT claims
- 339: removeCb — remove account update callback

### Group F: Directory Resolvers (5 features)

- 341: DirAccResolver.Fetch — directory-based resolver fetch
- 343: FetchTimeout — get fetch timeout constant
- 345: NewDirAccResolver — factory for directory resolver
- 346: Server.fetch — server-level fetch using configured resolver
- 347: NewCacheDirAccResolver — factory for cached directory resolver
- 349: CacheDirAccResolver.Reload — reload cached directory resolver

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch43): implement account resolvers and service latency tracking"
```

Task 3: Batch 44 — Events Core & Dispatch (46 features)

Files:

  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Events/EventHelpers.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Events/EventTypes.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Events.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs (or create partial)

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 44: Events Core & Dispatch — 46 deferred features that form the foundation of the internal event system: helpers, ServerInfo/ClientInfo methods, internal send/receive loops, and system subscription infrastructure.

## Go Source Files to Read

- `golang/nats-server/server/events.go` (3334 lines) — full event system

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/Events/EventTypes.cs` — SystemSubjects, EventMsgTypes, InternalState, PubMsg, ServerStatsMsg, ConnectEventMsg, etc.
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamEvents.cs` — existing PublishAdvisory, SendInternalAccountMsg methods
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — server fields (_sys InternalState, _sysAccAtomic, etc.)
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` — client connection fields
- `dotnet/src/ZB.MOM.NatsNet.Server/Internal/DataStructures/IpQueue.cs` — IpQueue used by event system

## Features to Implement

### Group A: Event Helpers (10 features)

Create `Events/EventHelpers.cs`:
- 863: newPubMsg — factory for internal publish message (uses pool)
- 882: routeStat — generate route stats for statz update
- 890: getHash — compute 8-char hash of a name
- 891: getHashSize — compute hash of specified size
- 912: getAcceptEncoding — parse accept-encoding for s2/snappy/gzip
- 935: remoteLatencySubjectForResponse — generate tracking subject for remote latency
- 939: totalSubs — sum subscription counts from results
- 945: accForClient — get account name for a client
- 946: issuerForClient — get issuer for a client
- 947: clearTimer — helper to clear a timer safely

### Group B: ServerInfo Capabilities (6 features)

Extend ServerInfo in `Events/EventTypes.cs`:
- 854: SetJetStreamEnabled — set JetStream capability flag
- 855: JetStreamEnabled — check JetStream capability
- 856: SetBinaryStreamSnapshot — set binary snapshot capability
- 857: BinaryStreamSnapshot — check binary snapshot capability
- 858: SetAccountNRG — set account NRG capability
- 859: AccountNRG — check account NRG capability

### Group C: ClientInfo Methods (3 features)

Extend ClientInfo in `Events/EventTypes.cs`:
- 860: forAssignmentSnap — minimal ClientInfo for assignment snapshots
- 861: forProposal — minimal ClientInfo for proposals
- 862: forAdvisory — minimal ClientInfo for JetStream advisories

### Group D: Message Pool (1 feature)

- 864: pubMsg.returnToPool — return PubMsg to pool

### Group E: Client Event Method (1 feature)

Add to ClientConnection (partial or directly):
- 875: client.sendInternalMsg — send internal message from client context

### Group F: Account Stats (1 feature)

- 920: Account.statz — compute account statistics (lock should be held on entry)

### Group G: Server Core Dispatch (24 features)

Create `NatsServer.Events.cs` (new partial class file):
- 865: internalReceiveLoop — background loop processing received internal messages
- 866: internalSendLoop — background loop sending internal messages
- 868: sendShutdownEvent — send server shutdown notification
- 871: sendInternalAccountSysMsg — send internal message to account's system subject
- 872: sendInternalMsgLocked — send internal message (lock held)
- 873: sendInternalMsg — send internal message
- 874: sendInternalResponse — send response to internal request
- 876: eventsRunning — check if events system is running (locked)
- 877: EventsEnabled — public check if events enabled (via system account)
- 878: eventsEnabled — internal check if events enabled (lock held)
- 892: Node — get node name (hash of server name)
- 893: initEventTracking — initialize event tracking infrastructure
- 909: filterRequest — check if request should be filtered for this server
- 927: noInlineCallback — wrap callback to prevent inline execution
- 928: noInlineCallbackStatsz — wrap statz callback
- 929: noInlineCallbackRecvQSelect — wrap receive queue callback
- 930: sysSubscribe — subscribe to system subject
- 931: sysSubscribeQ — subscribe to system subject with queue group
- 932: sysSubscribeInternal — subscribe with internal handler
- 933: systemSubscribe — core system subscription (used by all sysSubscribe variants)
- 934: sysUnsubscribe — unsubscribe from system subject
- 937: inboxReply — handle inbox reply (avoids supercluster-wide interest)
- 938: newRespInbox — generate new response inbox subject
- 948: wrapChk — wrap function with common lock checking

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch44): implement events core dispatch — send/receive loops, system subscriptions"
```

Task 4: Batch 45 — Events Server Methods (42 features)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Events.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.EventsRemote.cs

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 45: Events Server Methods — 42 deferred features that are all Server.* methods from events.go. These handle stats broadcasting, remote server tracking, account connection events, OCSP events, and config reload notifications.

IMPORTANT: This batch depends on Batch 44 (Events Core). The files `NatsServer.Events.cs` and `Events/EventHelpers.cs` should already exist with the core dispatch infrastructure (sendInternalMsg, sysSubscribe, internalSendLoop, etc.). Read those files first to understand the dispatch pattern before adding these methods.

## Go Source Files to Read

- `golang/nats-server/server/events.go` (3334 lines) — all event methods

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Events.cs` — core dispatch from Batch 44
- `dotnet/src/ZB.MOM.NatsNet.Server/Events/EventHelpers.cs` — helpers from Batch 44
- `dotnet/src/ZB.MOM.NatsNet.Server/Events/EventTypes.cs` — type definitions
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — server fields
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamEvents.cs` — existing advisory methods

## Features to Implement

### Group A: Stats & Heartbeat (10 features)

Add to `NatsServer.Events.cs`:
- 879: TrackedRemoteServers — count of tracked remote servers
- 881: updateServerUsage — update server resource usage tracking
- 883: sendStatsz — send server statistics
- 884: limitStatsz — rate-limit stats sending
- 885: heartbeatStatsz — periodic heartbeat stats sender
- 886: resetLastStatsz — reset last stats timestamp
- 887: sendStatszUpdate — send stats update message
- 888: startStatszTimer — start periodic stats timer
- 889: startRemoteServerSweepTimer — start orphan server sweep timer
- 880: checkRemoteServers — check for orphan remote servers

### Group B: Remote Server Tracking (8 features)

Create `NatsServer.EventsRemote.cs`:
- 897: accountClaimUpdate — receive and process account claim updates
- 898: processRemoteServerShutdown — process remote server shutdown notification
- 899: sameDomain — check if remote server is in same domain
- 900: remoteServerShutdown — handle remote server shutdown
- 901: remoteServerUpdate — handle remote server update
- 902: updateRemoteServer — update tracked remote server state
- 903: processNewServer — process new server joining cluster
- 904: updateNRGAccountStatus — update NRG account status

### Group C: Connection Events (6 features)

- 907: connsRequest — handle connection count request
- 908: leafNodeConnected — handle leaf node connect event
- 914: remoteConnsUpdate — update remote connection counts
- 921: accConnsUpdate — account connection count update
- 923: accountConnectEvent — send account connect/disconnect billing event
- 919: sendAccConnsUpdate — send account connections update

### Group D: Request Handlers (7 features)

- 894: userInfoReq — handle user info request
- 910: statszReq — handle stats request
- 911: idzReq — handle server identity request
- 913: zReq — generic monitoring request handler
- 940: debugSubscribers — debug active subscribers for a subject
- 941: nsubsRequest — handle subscription count request
- 942: reloadConfig — handle config reload notification

### Group E: Leaf Node & Gateway Events (4 features)

- 905: ensureGWsInterestOnlyForLeafNodes — ensure gateway interest-only for leaf accounts
- 917: sendLeafNodeConnect — send leaf node connect system event
- 918: sendLeafNodeConnectMsg — send leaf node connect message

### Group F: Auth & Error Events (3 features)

- 925: sendAuthErrorEvent — send authentication error event
- 926: sendAccountAuthErrorEvent — send account-specific auth error event
- 895: registerSystemImportsForExisting — register system imports for existing accounts

### Group G: OCSP Events (2 features)

- 949: sendOCSPPeerRejectEvent — send OCSP peer rejection event
- 950: sendOCSPPeerChainlinkInvalidEvent — send OCSP chain link invalid event

### Group H: Misc (2 features)

- 936: remoteLatencyUpdate — update remote latency tracking
- 943: kickClient — kick a client connection
- 944: ldmClient — handle LDM (Lame Duck Mode) for client

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch45): implement events server methods — stats, remote tracking, connection events"
```

Task 5: Batch 46 — Monitor Endpoints (45 features)

Files:

  • Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Monitor.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorTypes.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs (or partial)

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 46: Monitor Endpoints — 45 deferred features that implement the HTTP monitoring endpoints (Varz, Connz, Routez, Subsz, Gatewayz, Leafz, Accountz, Jsz, Healthz, Raftz) plus type methods and handler wrappers.

IMPORTANT: This batch depends on Batch 42 (Monitor Helpers). The file `Monitor/MonitorHelpers.cs` should already exist with standalone helper functions. Read it first.

## Go Source Files to Read

- `golang/nats-server/server/monitor.go` (4240 lines) — all monitoring endpoints

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorHelpers.cs` — helpers from Batch 42
- `dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorTypes.cs` — Connz, ConnInfo, ConnzOptions, etc.
- `dotnet/src/ZB.MOM.NatsNet.Server/Monitor/MonitorSortOptions.cs` — SortOpt types
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — server fields (_httpReqStats, _httpBasePath, etc.)
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Listeners.cs` — MonitorPaths constants
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` — client fields for connection info

## Features to Implement

### Group A: Type Methods (5 features)

Extend `Monitor/MonitorTypes.cs`:
- 2169: ConnInfo.fill — populate ConnInfo from a ClientConnection snapshot
- 2172: client.getRTT — get round-trip time for a client
- 2215: ClosedState.String — string representation of ClosedState
- 2226: HealthZErrorType.String — string representation of HealthZ error type
- 2227: HealthZErrorType.MarshalJSON — JSON serialization
- 2228: HealthZErrorType.UnmarshalJSON — JSON deserialization

### Group B: Connection Monitoring (2 features)

Create `NatsServer.Monitor.cs`:
- 2168: Server.Connz — build connection monitoring response
- 2178: Server.HandleConnz — HTTP handler wrapper for Connz

### Group C: Route Monitoring (2 features)

- 2179: Server.Routez — build route monitoring response
- 2180: Server.HandleRoutez — HTTP handler wrapper

### Group D: Subscription Monitoring (4 features)

- 2183: Server.Subsz — build subscription monitoring response
- 2184: Server.HandleSubsz — HTTP handler wrapper
- 2185: Server.HandleStacksz — stack trace handler
- 2186: Server.Ipqueuesz — internal queue monitoring
- 2187: Server.HandleIPQueuesz — HTTP handler wrapper

### Group E: Server Info (7 features)

- 2190: Server.HandleRoot — root HTTP handler (returns index of endpoints)
- 2191: Server.updateJszVarz — update JetStream stats in Varz
- 2192: Server.Varz — build Varz monitoring response
- 2193: Server.createVarz — create initial Varz structure
- 2195: Server.updateVarzConfigReloadableFields — update reloadable config fields
- 2197: Server.updateVarzRuntimeFields — update runtime fields (memory, goroutines, etc.)
- 2198: Server.HandleVarz — HTTP handler wrapper

### Group F: Gateway Monitoring (5 features)

- 2199: Server.Gatewayz — build gateway monitoring response
- 2201: Server.createOutboundsRemoteGatewayz — build outbound remote gateways info
- 2205: Server.createInboundsRemoteGatewayz — build inbound remote gateways info
- 2208: Server.HandleGatewayz — HTTP handler wrapper

### Group G: Leaf Node Monitoring (2 features)

- 2209: Server.Leafz — build leaf node monitoring response
- 2210: Server.HandleLeafz — HTTP handler wrapper

### Group H: Account Monitoring (6 features)

- 2211: Server.AccountStatz — build account stats response
- 2212: Server.HandleAccountStatz — HTTP handler wrapper
- 2217: Server.HandleAccountz — HTTP handler wrapper
- 2218: Server.Accountz — build account info response
- 2220: Server.accountInfo — get info for a single account
- 2221: Server.accountDetail — get detailed account info

### Group I: JetStream Monitoring (4 features)

- 2222: Server.JszAccount — get JetStream info for account
- 2223: Server.raftNodeToClusterInfo — convert raft node to ClusterInfo
- 2224: Server.Jsz — build JetStream monitoring response
- 2225: Server.HandleJsz — HTTP handler wrapper

### Group J: Health & Debug (6 features)

- 2229: Server.HandleHealthz — HTTP handler wrapper
- 2230: Server.healthz — internal health check logic
- 2231: Server.Healthz — build health check response
- 2232: Server.expvarz — Go-style expvar handler
- 2233: Server.profilez — profiling handler
- 2234: Server.HandleRaftz — Raft monitoring handler
- 2235: Server.Raftz — build Raft monitoring response

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch46): implement monitor endpoints — varz, connz, routez, healthz, etc."
```

Task 6: Batch 47 — MQTT Full Runtime (139 features)

Files:

  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttJsa.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttReader.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttWriter.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Mqtt.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Mqtt.cs

Agent Prompt

IMPORTANT: Before doing anything else, read the file AGENTS.md in the project root for full project context, build/test commands, and coding standards.

You are implementing Batch 47: MQTT Full Runtime — 139 deferred features covering the complete MQTT protocol implementation including JetStream API bridge, session management, account session management, server-side handlers, and client-side protocol parsing.

This is the largest batch. Organize your work into logical sub-groups and implement methodically.

IMPORTANT: This batch depends on Batches 44+45 (Events). The event dispatch infrastructure (sendInternalMsg, sysSubscribe, etc.) should already exist. Read `NatsServer.Events.cs` to understand the dispatch pattern.

## Go Source Files to Read

- `golang/nats-server/server/mqtt.go` (5882 lines) — complete MQTT implementation

## Existing .NET Files to Read First

- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs` — existing MQTT types (MqttWill, MqttConnectProto, MqttSession type def, etc.)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs` — packet types, flags, constants
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` — per-client MQTT state
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttReader.cs` — byte stream parser
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttWriter.cs` — byte stream builder
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSubjectConverter.cs` — topic/subject conversion
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` — server fields
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` — client connection

## Features to Implement

### Sub-batch A: mqttJSA — JetStream API Bridge (22 features)

Create `Mqtt/MqttJsa.cs`. The mqttJSA struct bridges MQTT to JetStream operations:

- 2269: newRequest — create new JS API request
- 2270: prefixDomain — add domain prefix to JS API subject
- 2271: newRequestEx — extended request with options
- 2272: newRequestExMulti — multi-request variant
- 2273: sendAck — send JS acknowledgment
- 2274: sendMsg — send message via JS
- 2275: createEphemeralConsumer — create ephemeral JS consumer
- 2276: createDurableConsumer — create durable JS consumer
- 2277: deleteConsumer — delete JS consumer
- 2278: createStream — create JS stream
- 2279: updateStream — update JS stream config
- 2280: lookupStream — look up JS stream by name
- 2281: deleteStream — delete JS stream
- 2282: loadLastMsgFor — load last message for subject
- 2283: loadLastMsgForMulti — load last message for multiple subjects
- 2284: loadNextMsgFor — load next message for subject
- 2285: loadMsg — load message by sequence
- 2286: storeMsgNoWait — store message without waiting for ack
- 2287: storeMsg — store message with ack
- 2288: storeSessionMsg — store session state message
- 2289: loadSessionMsg — load session state message
- 2290: deleteMsg — delete message by sequence

### Sub-batch B: mqttSession — Session State (15 features)

Create `Mqtt/MqttSession.cs`. Session state machine for MQTT client sessions:

- 2323: mqttSessionCreate — create new MQTT session
- 2324: save — persist session state
- 2325: clear — clear session state
- 2326: update — update session from loaded state
- 2327: bumpPI — bump packet identifier counter
- 2328: trackPublishRetained — track retained publish
- 2329: trackPublish — track QoS 1/2 publish
- 2330: untrackPublish — remove publish tracking
- 2331: trackAsPubRel — track PUBREL for QoS 2
- 2332: untrackPubRel — remove PUBREL tracking
- 2333: deleteConsumer — delete session consumer
- 2307: processQOS12Sub — process QoS 1/2 subscription
- 2308: processSub — process subscription
- 2375: cleanupFailedSub — clean up after failed subscription
- 2376: ensurePubRelConsumerSubscription — ensure PUBREL consumer exists
- 2377: processJSConsumer — process JetStream consumer for subscription

### Sub-batch C: mqttAccountSessionManager — Account-Level Manager (26 features)

Create `Mqtt/MqttAccountSessionManager.cs`. Manages sessions and retained messages per account:

- 2292: processJSAPIReplies — process JetStream API reply messages
- 2293: processRetainedMsg — process incoming retained message
- 2294: processRetainedMsgDel — process retained message deletion
- 2295: processSessionPersist — process session persistence message
- 2296: addSessToFlappers — add session to flapper tracking
- 2297: removeSessFromFlappers — remove session from flappers
- 2298: createSubscription — create internal subscription
- 2299: cleanupRetainedMessageCache — clean up stale retained messages
- 2300: sendJSAPIrequests — send pending JS API requests
- 2301: addRetainedMsg — add retained message to cache
- 2302: removeRetainedMsg — remove retained message from cache
- 2303: lockSession — acquire session lock
- 2304: unlockSession — release session lock
- 2305: addSession — add session to manager
- 2306: removeSession — remove session from manager
- 2309: processSubs — process subscription batch
- 2310: serializeRetainedMsgsForSub — serialize retained messages for subscription
- 2311: addRetainedSubjectsForSubject — add retained subjects for filter
- 2312: loadRetainedMessages — load retained messages from store
- 2316: createOrRestoreSession — create new or restore existing session
- 2317: deleteRetainedMsg — delete retained message
- 2318: notifyRetainedMsgDeleted — notify retained message deleted
- 2319: transferUniqueSessStreamsToMuxed — transfer unique session streams to muxed
- 2320: transferRetainedToPerKeySubjectStream — transfer retained to per-key subject stream
- 2321: getCachedRetainedMsg — get retained message from cache
- 2322: setCachedRetainedMsg — set retained message in cache

### Sub-batch D: Server MQTT Methods (17 features)

Create `NatsServer.Mqtt.cs`:

- 2252: startMQTT — start MQTT listener
- 2253: createMQTTClient — create MQTT client connection
- 2254: mqttConfigAuth — configure MQTT authentication
- 2255: validateMQTTOptions — validate MQTT server options
- 2260: mqttHandleClosedClient — handle MQTT client disconnect
- 2261: mqttUpdateMaxAckPending — update max ack pending setting
- 2262: mqttGetJSAForAccount — get JetStream API for account
- 2263: mqttStoreQoSMsgForAccountOnNewSubject — store QoS message on new subject
- 2266: getOrCreateMQTTAccountSessionManager — get/create session manager
- 2267: mqttCreateAccountSessionManager — create session manager
- 2268: mqttDetermineReplicas — determine replica count for MQTT streams
- 2336: mqttProcessConnect — process CONNECT packet
- 2338: mqttHandleWill — handle will message on disconnect
- 2344: mqttProcessPub — process PUBLISH packet
- 2345: mqttInitiateMsgDelivery — initiate message delivery
- 2346: mqttStoreQoS2MsgOnce — store QoS 2 message (exactly once)
- 2348: mqttProcessPubRel — process PUBREL packet
- 2350: mqttCheckPubRetainedPerms — check publish/retain permissions

### Sub-batch E: Client MQTT Methods (24 features)

Create `ClientConnection.Mqtt.cs`:

- 2257: getMQTTClientID — get MQTT client ID from connection
- 2258: mqttParse — main MQTT packet parser
- 2259: mqttTraceMsg — trace MQTT message
- 2334: mqttParseConnect — parse CONNECT packet
- 2335: mqttConnectTrace — trace CONNECT packet
- 2337: mqttEnqueueConnAck — enqueue CONNACK response
- 2339: mqttParsePub — parse PUBLISH packet
- 2347: mqttQoS2InternalSubject — build QoS 2 internal subject
- 2349: mqttHandlePubRetain — handle retained publish
- 2353: mqttEnqueuePubResponse — enqueue PUBACK/PUBREC/PUBREL/PUBCOMP
- 2355: mqttProcessPublishReceived — process received publish
- 2356: mqttProcessPubAck — process PUBACK
- 2357: mqttProcessPubRec — process PUBREC
- 2358: mqttProcessPubComp — process PUBCOMP
- 2362: mqttParseSubs — parse SUBSCRIBE packet
- 2363: mqttParseSubsOrUnsubs — parse SUBSCRIBE or UNSUBSCRIBE
- 2374: mqttProcessSubs — process subscriptions
- 2378: mqttSendRetainedMsgsToNewSubs — send retained messages to new subs
- 2379: mqttEnqueueSubAck — enqueue SUBACK
- 2380: mqttParseUnsubs — parse UNSUBSCRIBE packet
- 2381: mqttProcessUnsubs — process unsubscriptions
- 2382: mqttEnqueueUnsubAck — enqueue UNSUBACK
- 2384: mqttEnqueuePingResp — enqueue PINGRESP
- 2371: mqttEnqueuePublishMsgTo — enqueue PUBLISH to specific client

### Sub-batch F: Standalone Helpers (35 features)

Add to appropriate existing files or create helpers:

- 2264: mqttParsePublishNATSHeader — parse publish NATS header
- 2265: mqttParsePubRelNATSHeader — parse PUBREL NATS header
- 2291: isErrorOtherThan — check if error is not a specific type
- 2313: mqttEncodeRetainedMessage — encode retained message for storage
- 2314: mqttSliceHeaders — slice headers from message
- 2315: mqttDecodeRetainedMessage — decode retained message from storage
- 2340: mqttPubTrace — trace publish operation
- 2341: mqttComputeNatsMsgSize — compute NATS message size
- 2342: mqttNewDeliverableMessage — create deliverable message
- 2343: mqttNewDeliverablePubRel — create deliverable PUBREL
- 2351: generatePubPerms — generate publish permissions
- 2352: pubAllowed — check if publish is allowed
- 2354: mqttParsePIPacket — parse packet identifier packet
- 2359: mqttGetQoS — get QoS level from flags
- 2360: mqttIsRetained — check if message is retained
- 2361: sparkbParseBirthDeathTopic — parse SparkplugB birth/death topic
- 2364: mqttSubscribeTrace — trace subscribe operation
- 2365: mqttDeliverMsgCbQoS0 — QoS 0 delivery callback
- 2366: mqttDeliverMsgCbQoS12 — QoS 1/2 delivery callback
- 2367: mqttDeliverPubRelCb — PUBREL delivery callback
- 2368: mqttMustIgnoreForReservedSub — check if must ignore for reserved sub
- 2369: isMQTTReservedSubscription — check if subscription is reserved
- 2370: sparkbReplaceDeathTimestamp — replace SparkplugB death timestamp
- 2373: mqttMakePublishHeader — build PUBLISH packet header
- 2383: mqttUnsubscribeTrace — trace unsubscribe operation
- 2385: errOrTrace — log error or trace based on context
- 2386: mqttTopicToNATSPubSubject — convert MQTT topic to NATS pub subject
- 2387: mqttFilterToNATSSubject — convert MQTT filter to NATS subject
- 2388: mqttToNATSSubjectConversion — core topic/subject conversion
- 2389: natsSubjectStrToMQTTTopic — convert NATS subject string to MQTT topic
- 2390: natsSubjectToMQTTTopic — convert NATS subject bytes to MQTT topic
- 2391: mqttNeedSubForLevelUp — check if need sub for level-up wildcard
- 2404: newMQTTWriter — factory for MqttWriter

NOTE: For subject conversion methods (2386-2390), check `MqttSubjectConverter.cs` first — some may already be implemented. Only add what's missing.

## Build Verification

After all changes:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q
```

Both must pass with 0 failures.

## Commit

```bash
git add -A
git commit -m "feat(batch47): implement MQTT full runtime — JSA bridge, sessions, account manager, protocol handlers"
```

Task 7: Post-Merge Reconciliation

After all 6 batches merge to main:

Step 1: Build and test

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ -q

Step 2: Reset deferred features and re-audit

sqlite3 porting.db "UPDATE features SET status='unknown' WHERE status='deferred';"
sqlite3 porting.db "UPDATE features SET status='unknown' WHERE status='stub';"
dotnet run --project tools/NatsNet.PortTracker -- audit --type all --db porting.db --execute

Step 3: Generate report

./reports/generate-report.sh

Step 4: Commit reconciliation

git add porting.db
git add -f reports/current.md
git commit -m "chore: reconcile porting.db after deferred features batches 42-47"

Expected outcome: 363+ features promoted from deferred→verified, bringing total to ~98% feature coverage.