natsnet/docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md

# Batch 41 MQTT Client/IO Implementation Plan

> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.

**Goal:** Port and verify Batch 41 (`MQTT Client/IO`) so all 74 mapped features and 28 mapped tests are either truly implemented and verified or explicitly deferred with concrete blocker reasons.

**Architecture:** Implement in four feature groups (`19/19/17/19`) aligned to connect/publish/session handling, retained and QoS flow, subscription lifecycle, and reader/writer conversion I/O. Execute test waves after each group with strict evidence gates and explicit deferred handling for runtime-blocked cross-module tests.

**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)

**Design doc:** `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md`

---

## Batch 41 Scope

- Batch ID: `41`
- Name: `MQTT Client/IO`
- Dependency: Batch `40`
- Go source: `golang/nats-server/server/mqtt.go`
- Feature IDs: `2331-2404` (74)
- Test IDs: `115,1055,1056,1113,1258,1924,2170,2171,2182,2190,2191,2194,2195,2196,2199,2200,2204,2229,2234,2235,2236,2237,2238,2246,2251,2253,2285,3095`

Use this runtime path if `dotnet` is unavailable on PATH:

```bash
DOTNET=/usr/local/share/dotnet/dotnet
```

Expected source files (create/modify as needed):

- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs` (MQTT partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttReader*.cs` (new/partial)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttWriter*.cs` (new/partial)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` (new/partial)

Expected test files (create/modify as needed):

- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttExternalTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if missing)

---

## MANDATORY VERIFICATION PROTOCOL

> **NON-NEGOTIABLE:** Every Batch 41 feature/test item must pass this protocol before status promotion.

### 1. Dependency and Baseline Preflight (REQUIRED)

Before touching any status:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```

Start only when Batch 41 is ready:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 41 --db porting.db
```

Capture baseline:

```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

### 2. Per-Feature Verification Loop (REQUIRED per feature ID)

For each feature ID in active group:

1. Inspect mapping and Go behavior:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
   ```
2. Claim the feature:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
   ```
3. Implement minimal correct logic in mapped .NET class/method.
4. Add or update at least one behavioral test path covering this feature.
5. Run stub detection checks (source + test).
6. Run build gate.
7. Run targeted test gate for touched classes/methods.
8. Promote to `complete` only when gates pass.
9. Promote to `verified` only at task checkpoint.

### 3. Per-Test Verification Loop (REQUIRED per test ID)

For each mapped test ID in active wave:

1. Inspect mapped test details:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
   ```
2. Claim the test:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
   ```
3. Port full Arrange/Act/Assert behavior (no placeholders).
4. Run single-test filter and verify discovery:
   ```bash
   $DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
     --filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
   ```
5. Confirm output includes `Passed: 1, Failed: 0` (never accept `Passed: 0`).
6. Run class-level test filter.
7. Run stub detection + build gate.
8. Promote to `complete`; promote `verified` only at checkpoint.

### 4. Stub Detection Check (REQUIRED)

Run after each feature loop, each test loop, and before any status update.

Source stub scan:

```bash
src_changed=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server | rg "\\.cs$" || true)
if [ -n "$src_changed" ]; then
  echo "$src_changed" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|=>\\s*default;|return\\s+default;|return\\s+null;|throw\\s+new\\s+NotImplementedException)"
fi
```

Test stub scan:

```bash
test_changed=$(git diff --name-only -- dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$test_changed" ]; then
  echo "$test_changed" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|goFile\\s*=\\s*\"server/|ShouldContain\\(\"Should\"\\)|ShouldBe\\(string\\.Empty\\))"
fi
```

Assertion depth check for changed test files:

```bash
# per file: Shouldly assertion count / test count should be >= 1.5
```

### 5. Build Gate (REQUIRED)

```bash
$DOTNET build dotnet/
```

Run after every feature/test loop, before any batch-update, and at every checkpoint.

### 6. Test Gate (REQUIRED)

Run minimal relevant filters continuously, and full suite at checkpoints.

MQTT core filters:

```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.MqttExternalTests"
```

Cross-module mapped filters:

```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ImplBacklog.AuthCalloutTests|FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.LeafNodeHandlerTests|FullyQualifiedName~ImplBacklog.WebSocketHandlerTests|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3"
```

Checkpoint full unit gate:

```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

### 7. Status Update Protocol (HARD LIMIT: 15 IDs)

- Maximum `15` IDs per `feature batch-update` call.
- Maximum `15` IDs per `test batch-update` call.
- Never promote `verified` without checkpoint evidence.
- Deferred items must include explicit blocker reason.
- Status updates are scoped to the active task only.

Commands:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

$DOTNET run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```

### 8. Checkpoint Protocol Between Tasks (REQUIRED)

Before starting next task:

1. Run source + test stub detection scans.
2. Run build gate.
3. Run targeted test filters plus full unit test suite.
4. Apply status updates in <=15-ID chunks.
5. Run batch progress check:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
   ```
6. Commit checkpoint before moving on.

---

## ANTI-STUB GUARDRAILS

> **NON-NEGOTIABLE:** Placeholder behavior is not valid progress.

### Forbidden Patterns

Any of these in mapped feature/test bodies blocks status promotion:

- `throw new NotImplementedException(...)`
- Empty mapped method body
- `// TODO` or `// PLACEHOLDER` left in mapped logic
- `=> default;`, `return default;`, `return null;` in mapped feature methods
- `Assert.True(true)` or `Assert.Pass()`
- Placeholder template assertions not tied to behavior:
  - `var goFile = "server/..."` as test core
  - `"...".ShouldContain("Should")`
  - `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as primary assertion
- Test methods with only trivial constructor/null checks for complex protocol behavior

### Hard Limits

- Feature groups must stay <=20 IDs.
- Max 15 IDs per status batch-update command.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between tasks.
- No cross-task bulk status updates.
- No `verified` promotion without checkpoint evidence.

### If You Get Stuck (REQUIRED)

If an item needs infra or dependencies not available:

1. Stop work on that ID.
2. Do not add placeholders or force a pass.
3. Mark `deferred` with exact blocker reason.
4. Continue with the next unblocked ID.

Feature deferral:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```

Test deferral:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```

`deferred` with concrete reason is correct behavior. Stubs are not.

---

## Feature Groups (max ~20 each)

### Group A (19): session, CONNECT, publish ingress

IDs:
`2331,2332,2333,2334,2335,2336,2337,2338,2339,2340,2341,2342,2343,2344,2345,2346,2347,2348,2349`

### Group B (19): retained permissions, QoS ack flow, subscribe callbacks

IDs:
`2350,2351,2352,2353,2354,2355,2356,2357,2358,2359,2360,2361,2362,2363,2364,2365,2366,2367,2368`

### Group C (17): reserved-sub/sparkplug, subscribe/unsubscribe processing

IDs:
`2369,2370,2371,2372,2373,2374,2375,2376,2377,2378,2379,2380,2381,2382,2383,2384,2385`

### Group D (19): subject conversion and reader/writer I/O

IDs:
`2386,2387,2388,2389,2390,2391,2392,2393,2394,2395,2396,2397,2398,2399,2400,2401,2402,2403,2404`

## Test Waves (28 total)

- Wave T1 (10): `2170,2171,2190,2191,2194,2195,2196,2199,2200,2229`
- Wave T2 (11): `2182,2204,2234,2235,2236,2237,2238,2246,2251,2253,2285`
- Wave T3 (4): `115,1258,1924,3095`
- Wave T4 (3): `1055,1056,1113`

---

### Task 1: Preflight, Mapping Alignment, and Baseline

**Files:**
- Read: `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md`
- Read: `golang/nats-server/server/mqtt.go`
- Modify/Create: Batch 41 mapped class/method surfaces under `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/`, `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs`, and `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs`

**Steps:**
1. Verify Batch 40 completion/readiness and start Batch 41.
2. Ensure all mapped Batch 41 methods exist and compile.
3. Capture baseline build/test state.
4. Run checkpoint protocol.

### Task 2: Implement Group A + Wave T1 (Parser/Connect Core)

**Files:**
- Modify/Create: `ClientConnection` MQTT parse/connect files
- Modify/Create: `NatsServer` MQTT connect/publish ingress files
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/MqttExternalTests.Impltests.cs`

**Steps:**
1. Process all Group A feature IDs through per-feature loop.
2. Port/verify T1 parser/conversion tests through per-test loop.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol and commit.

### Task 3: Implement Group B + Wave T2 (Retain/QoS Core)

**Files:**
- Modify/Create: MQTT retained/perms and QoS ack handling files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`

**Steps:**
1. Process Group B features through per-feature loop.
2. Port/verify T2 tests tied to retain, QoS flags, pub ack flow.
3. Defer blocked IDs with explicit reasons (no stubs).
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.

### Task 4: Implement Group C + Remaining T2/T3 (Subscription Lifecycle)

**Files:**
- Modify/Create: MQTT subscription/publish delivery/session consumer files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/AuthCalloutTests.Impltests.cs`, `ImplBacklog/NatsConsumerTests.Impltests.cs`, `ImplBacklog/LeafNodeHandlerTests.Impltests.cs`, `ImplBacklog/WebSocketHandlerTests.Impltests.cs`

**Steps:**
1. Process Group C features through per-feature loop.
2. Complete remaining T2 tests and attempt T3 cross-module tests.
3. For cross-module blockers, defer with specific dependency reason.
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.

### Task 5: Implement Group D + T4 Cluster-Mapped Tests (Reader/Writer + Conversion)

**Files:**
- Modify/Create: `MqttReader*`, `MqttWriter*`, conversion helper files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if absent)

**Steps:**
1. Process Group D features through per-feature loop.
2. Port/verify reader/writer conversion tests first.
3. Attempt T4 cluster tests only where dependencies allow; otherwise defer explicitly.
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.

### Task 6: Final Batch 41 Verification and Closure

**Files:**
- Modify: `porting.db`
- Generate: `reports/current.md` (via report script)

**Steps:**
1. Run final source/test stub scans on all changed files.
2. Run build and full unit test suite.
3. Verify Batch 41 status:
   ```bash
   $DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
   $DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db
   ```
4. Generate report:
   ```bash
   ./reports/generate-report.sh
   ```
5. Commit final checkpoint.