From 8a126c4932e79d7af68ef409065b098a2541935b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 27 Feb 2026 17:27:51 -0500 Subject: [PATCH] Add batch plans for batches 37-41 (rounds 19-21) Generated design docs and implementation plans via Codex for: - Batch 37: Stream Messages - Batch 38: Consumer Lifecycle - Batch 39: Consumer Dispatch - Batch 40: MQTT Server/JSA - Batch 41: MQTT Client/IO All plans include mandatory verification protocol and anti-stub guardrails. Updated batches.md with file paths and planned status. All 42 batches (0-41) now have design docs and implementation plans. --- docs/batches.md | 10 +- ...6-02-27-batch-37-stream-messages-design.md | 157 ++++++ ...-37-stream-messages-implementation-plan.md | 457 +++++++++++++++++ ...2-27-batch-38-consumer-lifecycle-design.md | 165 ++++++ ...-consumer-lifecycle-implementation-plan.md | 481 ++++++++++++++++++ ...02-27-batch-39-consumer-dispatch-design.md | 159 ++++++ ...9-consumer-dispatch-implementation-plan.md | 418 +++++++++++++++ ...6-02-27-batch-40-mqtt-server-jsa-design.md | 136 +++++ ...-40-mqtt-server-jsa-implementation-plan.md | 427 ++++++++++++++++ ...26-02-27-batch-41-mqtt-client-io-design.md | 132 +++++ ...h-41-mqtt-client-io-implementation-plan.md | 388 ++++++++++++++ reports/current.md | 2 +- reports/report_f8dce79.md | 37 ++ 13 files changed, 2963 insertions(+), 6 deletions(-) create mode 100644 docs/plans/2026-02-27-batch-37-stream-messages-design.md create mode 100644 docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md create mode 100644 docs/plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md create mode 100644 docs/plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md create mode 100644 docs/plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md create mode 100644 docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md create mode 100644 reports/report_f8dce79.md diff --git a/docs/batches.md b/docs/batches.md index c77fca5..52f9058 100644 --- a/docs/batches.md +++ b/docs/batches.md @@ -51,8 +51,8 @@ | 34 | JS Cluster Consumers | [design](plans/2026-02-27-batch-34-js-cluster-consumers-design.md) | [plan](plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md) | planned | | 35 | JS Cluster Remaining | [design](plans/2026-02-27-batch-35-js-cluster-remaining-design.md) | [plan](plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md) | planned | | 36 | Stream Lifecycle | [design](plans/2026-02-27-batch-36-stream-lifecycle-design.md) | [plan](plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md) | planned | -| 37 | Stream Messages | | | not_planned | -| 38 | Consumer Lifecycle | | | not_planned | -| 39 | Consumer Dispatch | | | not_planned | -| 40 | MQTT Server/JSA | | | not_planned | -| 41 | MQTT Client/IO | | | not_planned | +| 37 | Stream Messages | [design](plans/2026-02-27-batch-37-stream-messages-design.md) | [plan](plans/2026-02-27-batch-37-stream-messages-implementation-plan.md) | planned | +| 38 | Consumer Lifecycle | [design](plans/2026-02-27-batch-38-consumer-lifecycle-design.md) | [plan](plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md) | planned | +| 39 | Consumer Dispatch | [design](plans/2026-02-27-batch-39-consumer-dispatch-design.md) | [plan](plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md) | planned | +| 40 | MQTT Server/JSA | [design](plans/2026-02-27-batch-40-mqtt-server-jsa-design.md) | [plan](plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md) | planned | +| 41 | MQTT Client/IO | [design](plans/2026-02-27-batch-41-mqtt-client-io-design.md) | [plan](plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md) | planned | diff --git a/docs/plans/2026-02-27-batch-37-stream-messages-design.md b/docs/plans/2026-02-27-batch-37-stream-messages-design.md new file mode 100644 index 0000000..0f64392 --- /dev/null +++ b/docs/plans/2026-02-27-batch-37-stream-messages-design.md @@ -0,0 +1,157 @@ +# Batch 37 Stream Messages Design + +**Date:** 2026-02-27 +**Batch:** 37 (`Stream Messages`) +**Scope:** 86 features + 13 unit tests +**Dependencies:** Batch `36` +**Go source:** `golang/nats-server/server/stream.go` (focus from ~line 4616 onward) + +## Problem + +Batch 37 is the stream message data plane: dedupe/message-id handling, direct-get APIs, inbound publish pipeline, consumer signaling, interest/pre-ack tracking, snapshot/restore, and monitor/replication accounting. It is a high-risk batch because most mapped methods are currently absent in .NET, while many mapped tests are still template placeholders. + +If this batch is implemented with stubs, Batch 38 (`Consumer Lifecycle`) and later stream/consumer correctness work will be blocked by false greens. + +## Context Findings + +### Required command outputs + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db` + - Status: `pending` + - Features: `86` (IDs `3294-3387` with mapped gaps) + - Tests: `13` + - Depends on: `36` + - Go file: `server/stream.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms dependency chain includes `36 -> 37` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`. + +### Feature ownership split (from `porting.db`) + +- `NatsStream`: 78 +- `JsOutQ`: 3 +- `JsPubMsg`: 2 +- `Account`: 1 +- `CMsg`: 1 +- `InMsg`: 1 + +### Test ownership split (from `porting.db`) + +- `RaftNodeTests`: 3 +- `JetStreamClusterTests2`: 2 +- `JetStreamEngineTests`: 2 +- `ConcurrencyTests1`: 1 +- `GatewayHandlerTests`: 1 +- `JetStreamClusterTests1`: 1 +- `JetStreamFileStoreTests`: 1 +- `JwtProcessorTests`: 1 +- `LeafNodeHandlerTests`: 1 + +Additional findings: + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` is currently minimal (357 lines) and does not contain mapped Batch 37 methods. +- `JsOutQ` does not currently exist in source and must be introduced. +- `JetStreamClusterTests1.Impltests.cs` and `RaftNodeTests.Impltests.cs` do not currently exist and must be created. +- Many `ImplBacklog` classes still use placeholder patterns (`var goFile = ...`, string-literal assertions), which must be replaced with behavioral tests. + +## Approaches + +### Approach A: Monolithic `NatsStream.cs` implementation + +Port all 86 features directly in `NatsStream.cs` and append helper logic in existing type files. + +- Pros: fewer files. +- Cons: poor reviewability, hard to isolate regressions, high merge conflict risk. + +### Approach B (Recommended): Message-domain partial decomposition with strict verification gates + +Convert `NatsStream` to partial and split message functionality by concern (headers/dedupe, direct-get, inbound pipeline, consumers/interest, snapshot/monitor), plus targeted type helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore path. + +- Pros: bounded review units, clearer ownership, easier per-group verification, aligns with anti-stub enforcement. +- Cons: requires class splitting and extra file setup. + +### Approach C: Test-wave-first before feature groups + +Port all 13 mapped tests first and drive production code entirely from failing tests. + +- Pros: stronger behavior pressure. +- Cons: high churn because many mapped APIs/types do not yet exist (`JsOutQ`, many stream methods), so red state will be noisy. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code organization strategy + +Keep mapped class ownership intact while splitting by concern: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` (convert to `partial`, keep core state/lifecycle) +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` (convert mapped message carrier types to partial if needed) +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs` (`InMsg.ReturnToPool`, `CMsg.ReturnToPool`, `JsPubMsg` helpers, `JsOutQ`) +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs` (`Account.RestoreStream`) + +Design intent: avoid duplicating existing `JetStreamHeaderHelpers` behavior; mapped `NatsStream` header methods should delegate where appropriate and preserve mapped method surface. + +### 2. Functional decomposition + +- **Headers + dedupe + scheduling metadata:** `unsubscribe`, `setupStore`, dedupe tables, msg-id checks, expected headers, TTL/schedule/batch header parsing, clustered checks. +- **Direct get + ingress path:** `queueInbound`, direct-get handlers, inbound publish entry points, `processJetStreamMsg`, `processJetStreamBatchMsg`. +- **Message carrier/outbound queue primitives:** `newCMsg`, pooled return methods, js pub message sizing/pool access, `JsOutQ.Send*`, unregister. +- **Consumers + interest + pre-ack:** signaling loop, consumer registry/listing, filtered-interest checks, pre-ack register/clear/ack. +- **Snapshot + restore + monitor traffic:** `snapshot`, `Account.RestoreStream`, orphan/replication checks, monitor running flags, replication traffic accounting. + +### 3. Test strategy + +Port mapped tests in three waves, replacing placeholders with behavior assertions: + +- **Wave T1 (5):** direct-get + rollup/mirror dedupe (`828,954,987,1642,1643`) +- **Wave T2 (4):** snapshot/restore/raft catchup (`383,2617,2672,2695`) +- **Wave T3 (4):** gateway/jwt/leaf/perf interaction (`635,1892,1961,2426`) + +Test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +### 4. Verification architecture + +- Mandatory per-feature and per-test loops with evidence before status promotion. +- Mandatory stub scan, build gate, and targeted/full test gates. +- Status changes chunked to max 15 IDs per `batch-update`. +- Mandatory checkpoints between every task. +- If blocked, keep item `deferred` with specific reason; never create fake-pass stubs. + +## Feature Grouping (for implementation plan) + +- Group A (18): `3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312` +- Group B (16): `3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330` +- Group C (16): `3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347` +- Group D (17): `3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368` +- Group E (19): `3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387` + +## Constraints + +- Planning only in this session; no implementation execution. +- Must follow `docs/standards/dotnet-standards.md` (`xUnit 3`, `Shouldly`, `NSubstitute`, nullable, naming conventions). +- Batch 37 work must not start unless Batch 36 is complete/ready. + +## Non-Goals + +- Executing Batch 37 code changes in this session. +- Expanding beyond mapped Batch 37 IDs. +- Introducing large integration harnesses unrelated to mapped tests. diff --git a/docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md b/docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md new file mode 100644 index 0000000..4d78bee --- /dev/null +++ b/docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md @@ -0,0 +1,457 @@ +# Batch 37 Stream Messages Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 37 (`Stream Messages`) from `server/stream.go` so all 86 mapped features and 13 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blockers. + +**Architecture:** Implement Batch 37 in five feature groups (`18/16/16/17/19`) using `NatsStream` partials plus message carrier helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore support. Execute three test waves with strict per-feature/per-test evidence loops and mandatory anti-stub gates before any status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-37-stream-messages-design.md` + +--- + +## Batch 37 Scope + +- Batch ID: `37` +- Name: `Stream Messages` +- Dependency: `36` +- Go source: `golang/nats-server/server/stream.go` +- Features: `86` (`3294-3387` mapped IDs) +- Tests: `13` + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Expected production files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs` + +Expected test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 37 feature and test must pass this loop. No shortcut status updates. + +### Dependency Preflight (before any status change) + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start only when Batch 37 is ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 37 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### What Counts as a Real Port + +A **feature** is real only if all are true: + +1. Mapped method/type exists with non-placeholder behavior. +2. Logic uses actual state/inputs (not hardcoded defaults). +3. At least one behavioral test exercises that code path. +4. No forbidden stub patterns are present. + +A **test** is real only if all are true: + +1. Arrange/Act/Assert is present. +2. It calls production code in `ZB.MOM.NatsNet.Server.*`. +3. Assertions validate behavior from Act output/effects. +4. It is not a template/literal stub. + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go context: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Claim work: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add or update focused behavioral test(s) for that feature. +4. Implement minimal real logic for the mapped method/type. +5. Run **Stub Detection Check** on changed files. +6. Run **Build Gate**. +7. Run targeted **Test Gate** (specific class/filter for touched area). +8. Promote only proven feature IDs to `complete` (or `verified` only after checkpoint evidence). + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go test location: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Claim work: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port full behavior (no template assertions). +4. Run single test method: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`). +6. Run touched class filter. +7. Run **Stub Detection Check** + **Build Gate**. +8. Promote only proven test IDs to `complete` (or `verified` only after checkpoint evidence). + +### Stub Detection Check (REQUIRED after every feature/test loop) + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$)" +fi +``` + +Any match blocks status promotion until fixed or deferred with reason. + +### Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run this: + +- after each feature loop +- after each test loop +- before every `batch-update` +- at every task checkpoint + +### Test Gate (REQUIRED) + +Minimum targeted gates by touched domain: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamClusterTests1|FullyQualifiedName~JetStreamClusterTests2" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~RaftNodeTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~JwtProcessorTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~ConcurrencyTests1" +``` + +Checkpoint/full gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Never pass more than `15` IDs per `feature batch-update` or `test batch-update`. +- Never update IDs outside current task scope. +- Never promote to `verified` without checkpoint evidence. +- For blocked items, keep `deferred` with explicit reason. + +Use: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting the next task: + +1. Run **Stub Detection Check** on current diff. +2. Run **Build Gate**. +3. Run targeted class test filters for touched classes. +4. Run full unit test gate. +5. Apply status updates in `<=15` ID chunks only. +6. Commit checkpoint before next task. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +Forbidden in mapped Batch 37 feature/test work: + +- `throw new NotImplementedException()` +- Empty mapped method bodies or no-op method shims +- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`) +- Template assertions that do not validate behavior: + - `var goFile = "server/..."` + - `"".ShouldContain("Should")` + - `GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as core assertion +- Constant-return shortcuts in mapped methods without behavioral justification: + - `=> default;` + - `return default;` + - `return null;` +- Catch-and-ignore patterns that hide failures + +### Hard Limits + +- Max feature group size: `~20` IDs. +- Max IDs per `feature batch-update`: `15`. +- Max IDs per `test batch-update`: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between every task. +- No `verified` promotion without full checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current ID immediately. +2. Do not write placeholder logic or fake assertions. +3. Mark item `deferred` with concrete blocker. +4. Continue with next unblocked item. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with explicit reason is correct. Stubbed progress is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (18): Dedupe and header extraction foundation + +IDs: +`3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312` + +### Group B (16): Scheduling/batch headers and direct-get ingress path + +IDs: +`3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330` + +### Group C (16): Message carriers, publish out queue, and internal loop hooks + +IDs: +`3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347` + +### Group D (17): Consumer registry and interest-state operations + +IDs: +`3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368` + +### Group E (19): Pre-ack, snapshot/restore, monitor/replication checks + +IDs: +`3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387` + +## Test Waves + +### Wave T1 (5): direct-get and rollup/mirror semantics + +IDs: +`828,954,987,1642,1643` + +### Wave T2 (4): snapshot/restore and raft recovery paths + +IDs: +`383,2617,2672,2695` + +### Wave T3 (4): gateway/jwt/leaf/perf regression coverage + +IDs: +`635,1892,1961,2426` + +--- + +### Task 1: Preflight, Baseline, and Batch Start + +**Files:** +- Read: `docs/plans/2026-02-27-batch-37-stream-messages-design.md` +- Read: `golang/nats-server/server/stream.go` + +**Steps:** +1. Run dependency preflight and verify Batch 36 completion. +2. Start Batch 37. +3. Capture baseline build/test evidence. +4. Do not change any status without baseline logs. + +### Task 2: Implement Feature Group A (18 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs` + +**Steps:** +1. Move Group A IDs to `stub` in `15 + 3` chunks. +2. Run Per-Feature Verification Loop for each ID. +3. Run required stub/build/test gates per ID. +4. Promote proven IDs in `<=15` chunks. +5. Run checkpoint protocol. + +### Task 3: Implement Feature Group B (16 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs` + +**Steps:** +1. Move Group B IDs to `stub` in `15 + 1` chunks. +2. Execute Per-Feature Verification Loop per ID. +3. Run mandatory gates. +4. Promote proven IDs in `<=15` chunks. +5. Run checkpoint protocol. + +### Task 4: Implement Feature Group C (16 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs` + +**Steps:** +1. Move Group C IDs to `stub` in `15 + 1` chunks. +2. Execute Per-Feature Verification Loop per ID. +3. Ensure `JsOutQ` exists with mapped methods. +4. Run mandatory gates. +5. Promote proven IDs in `<=15` chunks. +6. Run checkpoint protocol. + +### Task 5: Implement Feature Group D (17 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` + +**Steps:** +1. Move Group D IDs to `stub` in `15 + 2` chunks. +2. Execute Per-Feature Verification Loop per ID. +3. Run mandatory gates. +4. Promote proven IDs in `<=15` chunks. +5. Run checkpoint protocol. + +### Task 6: Implement Feature Group E (19 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs` + +**Steps:** +1. Move Group E IDs to `stub` in `15 + 4` chunks. +2. Execute Per-Feature Verification Loop per ID. +3. Run mandatory gates. +4. Promote proven IDs in `<=15` chunks. +5. Run checkpoint protocol. + +### Task 7: Port Test Wave T1 (5 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Move Wave T1 IDs to `stub`. +2. Execute Per-Test Verification Loop per ID. +3. Run mandatory stub/build/class/full test gates. +4. Promote proven IDs in one `<=15` batch. +5. Run checkpoint protocol. + +### Task 8: Port Test Wave T2 (4 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Move Wave T2 IDs to `stub`. +2. Execute Per-Test Verification Loop per ID. +3. Run mandatory gates. +4. Promote proven IDs in one `<=15` batch. +5. Run checkpoint protocol. + +### Task 9: Port Test Wave T3 (4 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` + +**Steps:** +1. Move Wave T3 IDs to `stub`. +2. Execute Per-Test Verification Loop per ID. +3. Run mandatory gates. +4. Promote proven IDs in one `<=15` batch. +5. Run checkpoint protocol. + +### Task 10: Batch Closeout + +**Files:** +- Modify: `porting.db` +- Modify: `reports/current.md` (via report script output) + +**Steps:** +1. Run final stub detection on current diff. +2. Run final build gate and full test gate. +3. Run dry-run audits for features and tests: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` +4. Complete batch: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 37 --db porting.db +``` +5. Regenerate summary report: +```bash +./reports/generate-report.sh +``` + +--- + +Plan complete and saved to `docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md`. Two execution options: + +1. Subagent-Driven (this session) - I dispatch fresh subagent per task, review between tasks, fast iteration. +2. Parallel Session (separate) - Open new session with `executeplan`, batch execution with checkpoints. + +Which approach? diff --git a/docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md b/docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md new file mode 100644 index 0000000..3ea96de --- /dev/null +++ b/docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md @@ -0,0 +1,165 @@ +# Batch 38 Consumer Lifecycle Design + +**Date:** 2026-02-27 +**Batch:** 38 (`Consumer Lifecycle`) +**Scope:** 96 features + 71 unit tests +**Dependencies:** Batches `34`, `36` +**Go source:** `golang/nats-server/server/consumer.go` (primary focus lines ~176-3761 for mapped features) + +## Problem + +Batch 38 carries the JetStream consumer control plane and lifecycle core: enum/config normalization, consumer creation/update paths, advisories, delivery-interest checks, ack processing, pending request tracking, store-state persistence, info snapshots, sampling, filtering, and wait-queue mechanics. + +The current .NET surface is far smaller than the mapped scope (`NatsConsumer.cs` is 246 lines vs. `consumer.go` at 6715 lines). If this batch is advanced with placeholders, Batch 39 (`Consumer Dispatch`) will inherit false-green behavior and brittle semantics. + +## Context Findings + +### Required command outputs + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db` + - Status: `pending` + - Features: `96` (IDs `584-684` with mapped gaps) + - Tests: `71` + - Depends on: `34,36` + - Go file: `server/consumer.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms dependency chain includes `34/36 -> 38 -> 39` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Environment note: `dotnet` is not on shell `PATH`; use `/usr/local/share/dotnet/dotnet`. + +### Dependency readiness + +- `batch ready` currently does **not** include Batch 38. +- Dependencies (`34`, `36`) are still `pending` and must be complete before Batch 38 starts. + +### Feature ownership split (from `porting.db`) + +- `NatsConsumer`: 77 +- `ConsumerAction`: 3 +- `PriorityPolicy`: 3 +- `NatsStream`: 3 +- `WaitingRequest`: 3 +- `DeliverPolicy`: 1 +- `AckPolicy`: 1 +- `ReplayPolicy`: 1 +- `SubjectTokens`: 1 +- `NatsServer`: 1 +- `Account`: 1 +- `JsPubMsg`: 1 + +### Test ownership split (from `porting.db`) + +- `NatsConsumerTests`: 38 +- `JetStreamEngineTests`: 20 +- `JetStreamClusterTests1`: 3 +- `JetStreamClusterTests3`: 3 +- `JetStreamClusterTests4`: 2 +- `ConcurrencyTests1`: 2 +- `AccountTests`: 1 +- `JetStreamBenchmarks`: 1 +- `MqttHandlerTests`: 1 + +Additional findings: + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` currently exposes only a small subset of mapped methods. +- `NatsStream` does not currently expose mapped `AddConsumer*` methods. +- `Account.CheckNewConsumerConfig` and `NatsServer.HasGatewayInterest` are not present. +- `JetStreamClusterTests1/3/4.Impltests.cs` and `JetStreamBenchmarks.Impltests.cs` do not currently exist. +- Existing `ImplBacklog` files for this scope heavily use placeholder patterns (`var goFile = ...`, literal-string assertions), requiring replacement with behavioral tests. + +## Approaches + +### Approach A: Expand a single `NatsConsumer.cs` monolith + +- Pros: fewer files. +- Cons: weak review boundaries, high merge conflict risk, difficult to validate incrementally across 96 features. + +### Approach B (Recommended): Partial-class decomposition by lifecycle domain + strict verification gates + +- Pros: bounded feature groups, cleaner method ownership, easier per-group verification and anti-stub enforcement. +- Cons: requires file reorganization and additional partial files. + +### Approach C: Port all mapped tests first, then backfill production code + +- Pros: forces behavior-driven progress. +- Cons: too noisy early because many mapped APIs/types do not yet exist; red phase would be broad and low-signal. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code organization strategy + +Use partial decomposition for consumer-heavy logic while keeping mapped ownership: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` (core state, constructor, shared locking) +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` (`AddConsumer*`) +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs` + +Design intent: keep method surfaces aligned to mapped `dotnet_class` + `dotnet_method` while avoiding one large file. + +### 2. Functional decomposition + +- **Enums and config validation:** consumer actions/policies string+JSON mapping, defaults, config checks. +- **Creation and lifecycle wiring:** `AddConsumer*`, assignment, monitor channels, leadership checks. +- **Advisories and delivery-interest:** internal subscribe/unsubscribe, advisory publication, gateway interest integration. +- **Ack and pending-request flow:** ack reply message parsing/building, `processAck/Nak/Term`, redelivery and pending-request transitions. +- **Store state and info path:** read/apply/write state, info snapshots, sampling, filter checks, pull request parsing, wait-queue recycling. + +### 3. Test strategy + +Port mapped tests by class waves and remove template assertions: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` (38) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` (20) +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` (3) +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` (3) +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` (2) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` (2) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` (1) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` (1) +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs` (1; may remain deferred if benchmark-only) + +### 4. Verification architecture + +Adopt Batch-0-style hard verification controls, adapted for feature + test work: + +- Mandatory per-feature loop (evidence required before status promotion). +- Mandatory per-test loop (single-test and class-level pass evidence). +- Stub detection required after each loop. +- Build + targeted test gates required before each status update. +- Status updates chunked to max 15 IDs per `batch-update`. +- Mandatory checkpoint between every task. +- Explicit deferred handling with reason when blocked; no placeholder stubs. + +## Feature Grouping (for implementation plan) + +- Group A (19): `584,585,586,587,588,589,590,591,592,594,595,596,597,598,599,600,601,602,603` +- Group B (19): `604,605,606,607,608,610,612,613,614,615,616,617,618,619,620,621,622,623,624` +- Group C (19): `625,626,627,629,630,631,632,633,635,636,637,638,639,640,641,642,643,644,645` +- Group D (19): `646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664` +- Group E (20): `665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684` + +## Constraints + +- Planning only in this session; no implementation execution. +- Must follow `docs/standards/dotnet-standards.md` (xUnit 3 + Shouldly + NSubstitute, nullable, naming conventions). +- Batch 38 implementation must start only after Batches 34 and 36 are complete/ready. + +## Non-Goals + +- Executing Batch 38 code changes in this session. +- Re-scoping features/tests outside mapped Batch 38 IDs. +- Fabricating benchmark/integration parity via fake-pass unit-test templates. diff --git a/docs/plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md b/docs/plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md new file mode 100644 index 0000000..8ac3435 --- /dev/null +++ b/docs/plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md @@ -0,0 +1,481 @@ +# Batch 38 Consumer Lifecycle Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 38 (`Consumer Lifecycle`) from `server/consumer.go` so all 96 mapped features and 71 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker notes. + +**Architecture:** Implement Batch 38 in five feature groups (`19/19/19/19/20`) using `NatsConsumer` partial decomposition plus supporting updates in `NatsStream`, `StoreTypes`, `StreamTypes`, `Account`, `NatsServer`, and `SubjectTokens`. Execute five test waves with strict per-feature/per-test evidence loops, mandatory stub detection, and checkpoint gates before status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md` + +--- + +## Batch 38 Scope + +- Batch ID: `38` +- Name: `Consumer Lifecycle` +- Dependencies: `34,36` +- Go source: `golang/nats-server/server/consumer.go` +- Features: `96` (`584-684` mapped IDs) +- Tests: `71` + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Primary production files (expected): + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs` + +Primary test files (expected): + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerPoliciesTests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerStateTests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 38 feature and test must pass this loop. No shortcut updates. + +### Dependency Preflight (before any status change) + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 34 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start only when Batch 38 is ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 38 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### What Counts as a Real Port + +A **feature** is real only if all are true: + +1. Mapped method/type exists with behavioral logic (no placeholder body). +2. Logic consumes real inputs/state (not constants that bypass behavior). +3. At least one behavioral test exercises the path. +4. Stub scan and gates are clean. + +A **test** is real only if all are true: + +1. Has Arrange/Act/Assert. +2. Calls production code in `ZB.MOM.NatsNet.Server.*`. +3. Assertions validate behavior derived from Act output/effects. +4. Contains no template-stub patterns. + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go context: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Claim feature: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add/adjust focused behavioral test(s) for that feature. +4. Implement minimal real logic for the mapped method/type. +5. Run **Stub Detection Check**. +6. Run **Build Gate**. +7. Run targeted **Test Gate** for touched classes. +8. Promote proven feature to `complete` (or `verified` only after checkpoint evidence). + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go test source: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Claim test: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port full behavior (no placeholders). +4. Run single test: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`). +6. Run class-level filter for touched class. +7. Run **Stub Detection Check** + **Build Gate**. +8. Promote proven test to `complete` (or `verified` only after checkpoint evidence). + +### Stub Detection Check (REQUIRED after every feature/test loop) + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$|throw new Exception\\(\"TODO\")" +fi +``` + +Any match blocks status promotion until fixed or explicitly deferred. + +### Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run after each feature loop, each test loop, before each `batch-update`, and at each checkpoint. + +### Test Gate (REQUIRED) + +Minimum targeted gates per touched domain: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream.NatsConsumerTests|FullyQualifiedName~JetStream.WaitQueueTests|FullyQualifiedName~JetStream.ConsumerPoliciesTests|FullyQualifiedName~JetStream.ConsumerStateTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3|FullyQualifiedName~ImplBacklog.JetStreamClusterTests4" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.AccountTests|FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.JetStreamBenchmarks" +``` + +Checkpoint/full gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Never send more than `15` IDs per `feature batch-update` or `test batch-update`. +- Never promote `verified` without checkpoint evidence. +- Never update IDs outside active task scope. +- Blocked items remain `deferred` with concrete reason. + +Use: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting the next task: + +1. Run **Stub Detection Check** on current diff. +2. Run **Build Gate**. +3. Run targeted **Test Gate** for touched classes. +4. Run full unit-test gate. +5. Apply status updates in `<=15` ID chunks only. +6. Commit checkpoint before moving on. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +Forbidden in mapped Batch 38 feature/test work: + +- `throw new NotImplementedException()` +- Empty/no-op mapped method bodies +- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`) +- Template assertions disconnected from behavior: + - `var goFile = "server/..."` + - `"".ShouldContain("Should")` + - `GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as a core assertion +- Constant-return shortcuts for mapped behavior without justification and coverage: + - `=> default;` + - `return default;` + - `return null;` +- Catch-and-ignore blocks used to force green tests + +### Hard Limits + +- Max feature group size: `~20`. +- Max IDs per `feature batch-update`: `15`. +- Max IDs per `test batch-update`: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between all tasks. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current feature/test ID immediately. +2. Do **not** write placeholder logic or fake assertions. +3. Mark the item `deferred` with a specific blocker reason. +4. Continue with the next unblocked ID. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with a concrete reason is correct. Stubbing is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (19): enums, subject filters, config defaults/validation, add-consumer and assignment entry + +IDs: +`584,585,586,587,588,589,590,591,592,594,595,596,597,598,599,600,601,602,603` + +### Group B (19): monitor channels, leadership/advisory path, delivery-interest bridge to server + +IDs: +`604,605,606,607,608,610,612,613,614,615,616,617,618,619,620,621,622,623,624` + +### Group C (19): delivery-subject/rate-limit/ack ingress and early delivery state transitions + +IDs: +`625,626,627,629,630,631,632,633,635,636,637,638,639,640,641,642,643,644,645` + +### Group D (19): proposals, ack floor updates, pending-request cluster flow, nak/term/ackWait foundation + +IDs: +`646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664` + +### Group E (20): store-state persistence, info snapshots, sampling/filtering, next-request parsing, wait-queue recycle + +IDs: +`665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684` + +## Test Waves + +### Wave T1 (19): consumer filters/actions/pinned behavior + +IDs: +`1220,1221,1222,1223,1224,1225,1226,1228,1239,1240,1241,1242,1243,1244,1248,1250,1259,1260,1263` + +### Wave T2 (19): consumer max-deliver/ack/rate/replay/multi-subject behavior + +IDs: +`1266,1268,1274,1278,1279,1280,1281,1282,1289,1311,1312,1313,1316,1318,1319,1320,1321,1325,1344` + +### Wave T3 (8): cluster consumer lifecycle consistency + +IDs: +`812,870,892,1123,1124,1126,1172,1173` + +### Wave T4 (12): core engine consumer lifecycle integration + +IDs: +`96,1472,1473,1491,1494,1497,1505,1513,1523,1525,1527,1546` + +### Wave T5 (13): perf/race/cross-protocol and benchmark-adjacent coverage + +IDs: +`749,1557,1558,1559,1561,1624,1647,1670,1671,1764,2212,2385,2388` + +--- + +### Task 1: Preflight, Baseline, and Batch Start + +**Files:** +- Read: `docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md` +- Read: `golang/nats-server/server/consumer.go` + +**Steps:** +1. Run dependency preflight for batches `34`, `36`, `38`. +2. Start Batch 38 only if ready. +3. Capture baseline build + full unit test. +4. Record the baseline counts before edits. + +### Task 2: Implement Feature Group A (19 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Modify/Create tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerPoliciesTests.cs` + +**Steps:** +1. Execute per-feature loop for each Group A ID. +2. Keep each status update batch at max 15 IDs. +3. Run checkpoint protocol. + +### Task 3: Implement Feature Group B (19 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs` + +**Steps:** +1. Execute per-feature loop for each Group B ID. +2. Validate advisory and delivery-interest paths with focused tests. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 4: Implement Feature Group C (19 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs` +- Modify tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs` + +**Steps:** +1. Execute per-feature loop for each Group C ID. +2. Cover ack message creation/parsing and deliver-subject update behavior. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 5: Implement Feature Group D (19 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs` +- Modify/Create tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerStateTests.cs` + +**Steps:** +1. Execute per-feature loop for each Group D ID. +2. Verify proposal forwarding, pending-request transitions, and `processNak/processTerm`. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 6: Implement Feature Group E (20 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs` +- Modify tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/WaitQueueTests.cs` + +**Steps:** +1. Execute per-feature loop for each Group E ID. +2. Verify state read/write/info sampling/filter parsing/wait-queue recycle behavior. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 7: Port Test Wave T1 (19 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` + +**Steps:** +1. Execute per-test loop for each T1 ID. +2. Replace template assertions with behavioral checks. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 8: Port Test Wave T2 (19 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` + +**Steps:** +1. Execute per-test loop for each T2 ID. +2. Validate ack/rate/replay/multi-subject semantics using produced feature behavior. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 9: Port Test Wave T3 (8 IDs) + +**Files:** +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` + +**Steps:** +1. Execute per-test loop for each T3 ID. +2. Keep cluster-only blockers as deferred with explicit reason if environment-dependent. +3. Run checkpoint protocol. + +### Task 10: Port Test Wave T4 (12 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Execute per-test loop for each T4 ID. +2. Verify integration behavior for `nextReqFromMsg`, filtering, ack reply and lifecycle interactions. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 11: Port Test Wave T5 (13 IDs) + +**Files:** +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` + +**Steps:** +1. Execute per-test loop for each T5 ID. +2. For benchmark-only cases not meaningful as unit tests, defer with explicit benchmark blocker reason. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol. + +### Task 12: Final Verification, Status Closure, and Batch Completion + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` + +**Steps:** +1. Run full stub detection across changed source and backlog tests. +2. Run full build + full unit-test suite. +3. Verify all Batch 38 IDs are `complete/verified/n_a/deferred(with reason)` with no silent stubs. +4. Run: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db +./reports/generate-report.sh +``` +5. Complete batch: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 38 --db porting.db +``` + diff --git a/docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md b/docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md new file mode 100644 index 0000000..93e3723 --- /dev/null +++ b/docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md @@ -0,0 +1,159 @@ +# Batch 39 Consumer Dispatch Design + +**Date:** 2026-02-27 +**Batch:** 39 (`Consumer Dispatch`) +**Scope:** 93 features + 53 unit tests +**Dependencies:** Batch `38` +**Go source:** `golang/nats-server/server/consumer.go` (dispatch-heavy region around lines 3925-6706) + +## Problem + +Batch 39 contains the JetStream consumer dispatch/data-plane logic: pull request queueing, next-message request parsing, pending accounting, heartbeats/flow-control, delivery tracking, redelivery queue semantics, ack reply parsing, stream purge/stop paths, and monitor/signal flow. + +If this batch is advanced with placeholders, the runtime behavior for pull consumers, redelivery correctness, and no-interest cleanup will look green in PortTracker while remaining behaviorally incomplete. + +## Context Findings + +### Required command outputs + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db` + - Status: `pending` + - Features: `93` (all currently `deferred`) + - Tests: `53` (all currently `deferred`) + - Depends on: `38` + - Go file: `server/consumer.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms dependency chain `38 -> 39` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall progress snapshot: `1924/6942 (27.7%)` + +Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`. + +### Batch ownership split + +Features by class: + +- `NatsConsumer`: 91 +- `NextMsgReq`: 1 +- `NatsStream`: 1 + +Tests by class: + +- `JetStreamEngineTests`: 34 +- `NatsConsumerTests`: 14 +- `ConcurrencyTests1`: 2 +- `JetStreamClusterTests1`: 1 +- `JwtProcessorTests`: 1 +- `RouteHandlerTests`: 1 + +### Current .NET baseline findings + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` is still a compact lifecycle shell and does not expose most Batch 39 mapped methods. +- `WaitQueue` and `WaitingRequest` exist in `StreamTypes.cs`, but only part of the Go dispatch behavior is represented. +- `ImplBacklog` files for Batch 39-related tests are mostly template placeholders and need full behavioral ports. + +## Approach Options + +### Approach A: Keep all dispatch work in `NatsConsumer.cs` + +- Pros: fewer files and simpler discovery. +- Cons: 90+ methods in one file, poor reviewability, high merge risk, and weaker checkpoint isolation. + +### Approach B (Recommended): Dispatch-focused partial decomposition for `NatsConsumer` + +- Pros: aligns with Go dispatch domains, improves review boundaries, supports group-by-group verification loops. +- Cons: more files and up-front organization overhead. + +### Approach C: Port all 53 tests first, then implement features + +- Pros: strict red/green pressure. +- Cons: too many missing APIs initially, broad low-signal red phase, high churn in test scaffolding. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code Organization + +Keep `NatsConsumer` as the central type but split dispatch concerns into partial files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` (core state, shared lock helpers, partial type anchor) +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` (`DeleteConsumer` mapping) +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` (`WaitQueue`, `WaitingRequest`, `PriorityGroup` helpers) + +### 2. Functional Decomposition + +- **Waiting queue and request ingress:** pending request map, pin assignment/ttl, request parsing (`ProcessNextMsgReq`, `ProcessResetReq`, `ProcessNextMsgRequest`). +- **Dispatch loop and pending accounting:** `GetNextMsg`, `ProcessWaiting`, `CheckAckFloor`, `LoopAndGatherMsgs`, pending counters. +- **Delivery and flow control:** heartbeat emission, flow-control reply generation, message delivery envelope conversion. +- **Redelivery and ack reply parsing:** pending tracking, redelivery queue operations, ack subject parsing (`ReplyInfo`, `AckReplyInfo`, seq extraction). +- **Lifecycle and stream-interest cleanup:** durable/push/pull helpers, purge/stop/cleanup, stream signals, monitor guard methods. + +### 3. Test Design + +Replace template tests with behavioral tests tied to mapped IDs: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` (14) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` (34) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` (2) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` (1) +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` (1) +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` (1) + +Also add/expand focused non-backlog unit tests where internal behaviors can be tested deterministically: + +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs` + +### 4. Verification Model (Design-level) + +Batch 39 execution must be evidence-driven: + +- Per-feature implementation loop with targeted tests. +- Per-test single-test and class-level verification. +- Stub scan before any status promotion. +- Build and test gates between groups. +- Status updates in max 15-ID chunks. +- Deferred-with-reason instead of stubbing when blocked. + +## Feature Groups (max ~20 each) + +- **Group A (18): waiting queue + pull request ingress** + `696,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715` +- **Group B (18): dispatch selection + ack floor + inbound loops** + `716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733` +- **Group C (19): delivery payload + flow control + redelivery foundation** + `734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752` +- **Group D (19): reply parsing + sequence selection + consumer identity helpers** + `753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,772` +- **Group E (19): stop/purge/no-interest/monitor signaling path** + `774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792` + +## Test Waves + +- **Wave T1 (14):** `1230,1232,1251,1261,1265,1267,1273,1277,1283,1284,1285,1286,1339,1370` +- **Wave T2 (17):** `1469,1484,1485,1486,1487,1488,1489,1490,1492,1493,1495,1496,1498,1499,1500,1501,1502` +- **Wave T3 (17):** `1508,1514,1515,1516,1517,1518,1519,1520,1521,1522,1526,1530,1531,1545,1547,1567,1665` +- **Wave T4 (5):** `814,1840,2389,2407,2858` + +## Risks and Mitigations + +- **Risk:** dispatch loops can mask bugs via timing-sensitive behavior. + **Mitigation:** isolate pure logic from timers/network side-effects; unit-test selectors and parsers first. +- **Risk:** placeholder tests could slip through due green-but-empty assertions. + **Mitigation:** mandatory stub scan and assertion-quality checks before status updates. +- **Risk:** over-bulk status updates can misclassify unverified IDs. + **Mitigation:** enforce max 15 IDs per update and per-task checkpoints. + +## Non-Goals + +- Executing Batch 39 implementation in this session. +- Re-scoping IDs outside Batch 39. +- Marking any feature/test as verified during planning. diff --git a/docs/plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md b/docs/plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md new file mode 100644 index 0000000..3ce4828 --- /dev/null +++ b/docs/plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md @@ -0,0 +1,418 @@ +# Batch 39 Consumer Dispatch Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 39 (`Consumer Dispatch`) from `server/consumer.go` so all 93 mapped features and 53 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker reasons. + +**Architecture:** Implement Batch 39 in five feature groups (`18/18/19/19/19`) with dispatch-focused `NatsConsumer` partial decomposition. After each feature group, execute mapped test waves and enforce strict evidence gates (stub scan, build gate, targeted tests, checkpoint) before any status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md` + +--- + +## Batch 39 Scope + +- Batch ID: `39` +- Name: `Consumer Dispatch` +- Dependency: `38` +- Go source: `golang/nats-server/server/consumer.go` +- Features: `93` +- Tests: `53` + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Primary production files (expected): + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` + +Primary test files (expected): + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 39 feature and test must pass this protocol. No shortcuts. + +### 1. Dependency Preflight and Baseline + +Run before touching statuses: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start Batch 39 only when ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 39 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 2. What Counts as a Real Port + +A **feature** is real only if all are true: + +1. Mapped method/type exists and contains behavioral logic (not placeholder control flow). +2. Code path is exercised by at least one behavioral test. +3. Stub scan, build gate, and targeted tests are green. +4. Status promotion has traceable evidence. + +A **test** is real only if all are true: + +1. Has Arrange/Act/Assert. +2. Calls production code from `ZB.MOM.NatsNet.Server.*`. +3. Uses meaningful Shouldly assertions on Act results/effects. +4. Contains no stub/template patterns. + +### 3. Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapped feature and Go source intent: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Claim the feature: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add/adjust at least one behavioral test for the feature. +4. Implement minimal correct logic. +5. Run **Stub Detection Check**. +6. Run **Build Gate**. +7. Run relevant **Test Gate** filters. +8. Promote proven feature to `complete` (or `verified` only after checkpoint evidence). + +### 4. Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapped test and Go source: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Claim the test: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port complete behavior. +4. Run single test and verify discovery: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`). +6. Run class-level filter. +7. Run **Stub Detection Check** + **Build Gate**. +8. Promote proven test to `complete` (or `verified` only after checkpoint evidence). + +### 5. Stub Detection Check (REQUIRED after each feature/test loop) + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$|throw new Exception\\(\"TODO\"\\))" +fi +``` + +Any match blocks status promotion until fixed or explicitly deferred. + +### 6. Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run after every feature loop, every test loop, before any `batch-update`, and at each task checkpoint. + +### 7. Test Gate (REQUIRED) + +Run the smallest relevant set after each change; run all of the following at checkpoints: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream.NatsConsumerTests|FullyQualifiedName~JetStream.NatsConsumerDispatchTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.JwtProcessorTests|FullyQualifiedName~ImplBacklog.RouteHandlerTests" +``` + +Checkpoint/full gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 8. Status Update Protocol (HARD LIMIT: 15 IDs per batch-update) + +- Never send more than `15` IDs per `feature batch-update` or `test batch-update` call. +- Never mark `verified` without checkpoint evidence. +- Never update IDs outside the active task scope. +- Keep blocked items `deferred` with specific reason text. + +Use: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### 9. Checkpoint Protocol Between Tasks (REQUIRED) + +Before moving to the next task: + +1. Run **Stub Detection Check**. +2. Run **Build Gate**. +3. Run **Test Gate** (targeted filters + full unit test suite). +4. Apply status updates in max-15-ID chunks only. +5. Commit the task checkpoint before continuing. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +The following are forbidden in Batch 39 feature/test work: + +- `throw new NotImplementedException()` +- Empty mapped method bodies / no-op ports +- `TODO` / `PLACEHOLDER` comments as unresolved implementation markers +- `Assert.True(true)` / `Assert.Pass()` +- Template assertions disconnected from behavior: + - `var goFile = "server/..."` + - `"".ShouldContain("Should")` + - `GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as primary assertion +- Placeholder return shortcuts in mapped code: + - `=> default;` + - `return default;` + - `return null;` +- Catch-and-ignore blocks used only to force green tests + +### Hard Limits + +- Max feature group size: `~20`. +- Max IDs per `feature batch-update`: `15`. +- Max IDs per `test batch-update`: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between tasks. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current feature/test immediately. +2. Do **not** stub or fake-pass. +3. Mark item `deferred` with a specific blocker reason. +4. Continue with the next unblocked item. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with concrete reason is the correct fallback; stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (18): waiting queue and pull ingress + +IDs: +`696,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715` + +### Group B (18): dispatch core, ack floor, inbound loops + +IDs: +`716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733` + +### Group C (19): delivery, flow-control, redelivery basics + +IDs: +`734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752` + +### Group D (19): reply parsing, seq selection, identity helpers + +IDs: +`753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,772` + +### Group E (19): purge/stop/no-interest/monitor signaling + +IDs: +`774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792` + +## Test Waves + +### Wave T1 (14): `NatsConsumerTests` + +IDs: +`1230,1232,1251,1261,1265,1267,1273,1277,1283,1284,1285,1286,1339,1370` + +### Wave T2 (17): `JetStreamEngineTests` (pull/ack-first half) + +IDs: +`1469,1484,1485,1486,1487,1488,1489,1490,1492,1493,1495,1496,1498,1499,1500,1501,1502` + +### Wave T3 (17): `JetStreamEngineTests` (pending/purge-second half) + +IDs: +`1508,1514,1515,1516,1517,1518,1519,1520,1521,1522,1526,1530,1531,1545,1547,1567,1665` + +### Wave T4 (5): cross-suite integration/backlog edges + +IDs: +`814,1840,2389,2407,2858` + +--- + +### Task 1: Preflight and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md` +- Read: `golang/nats-server/server/consumer.go` + +**Steps:** +1. Run dependency preflight and confirm Batch 39 readiness. +2. Start Batch 39. +3. Capture baseline build + full unit test results. +4. Record baseline pass/fail/skip counts. + +### Task 2: Implement Feature Group A + Test Wave T1 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` + +**Steps:** +1. Process each feature ID in Group A through the per-feature loop. +2. Port all tests in Wave T1 with per-test loop. +3. Run checkpoint protocol. +4. Update Group A/T1 statuses in max-15-ID chunks. +5. Commit. + +### Task 3: Implement Feature Group B + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs` + +**Steps:** +1. Process Group B feature IDs through per-feature loop. +2. Add deterministic unit tests for dispatch loop and ack-floor transitions. +3. Run checkpoint protocol. +4. Update Group B feature statuses in max-15-ID chunks. +5. Commit. + +### Task 4: Implement Feature Group C + Test Wave T2 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Process Group C feature IDs through per-feature loop. +2. Port Wave T2 tests through per-test loop. +3. Run checkpoint protocol. +4. Update Group C/T2 statuses in max-15-ID chunks. +5. Commit. + +### Task 5: Implement Feature Group D + Test Wave T3 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Process Group D feature IDs through per-feature loop. +2. Port Wave T3 tests through per-test loop. +3. Run checkpoint protocol. +4. Update Group D/T3 statuses in max-15-ID chunks. +5. Commit. + +### Task 6: Implement Feature Group E + Test Wave T4 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +**Steps:** +1. Process Group E feature IDs through per-feature loop. +2. Port Wave T4 tests through per-test loop. +3. Run checkpoint protocol. +4. Update Group E/T4 statuses in max-15-ID chunks. +5. Commit. + +### Task 7: Final Batch 39 Verification and Closure + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` + +**Steps:** +1. Run full stub scan over changed source + test files. +2. Run full build gate. +3. Run full unit test suite; require `Failed: 0`. +4. Run: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` +5. If all IDs are complete/verified/deferred-with-reason, execute batch completion: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 39 --db porting.db +``` +6. Generate updated report and commit: +```bash +./reports/generate-report.sh +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests porting.db reports/ +git commit -m "feat(batch39): implement consumer dispatch features and tests" +``` diff --git a/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md b/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md new file mode 100644 index 0000000..81a798d --- /dev/null +++ b/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md @@ -0,0 +1,136 @@ +# Batch 40 MQTT Server/JSA Design + +**Date:** 2026-02-27 +**Batch:** 40 (`MQTT Server/JSA`) +**Scope:** 78 features + 323 unit tests +**Dependencies:** Batches `19` (Accounts Core), `27` (JetStream Core) +**Go source:** `golang/nats-server/server/mqtt.go` (Batch scope through line ~3500) + +## Problem + +Batch 40 is the core MQTT server-side control plane and JetStream adapter layer: + +- MQTT listener/bootstrap and option validation +- MQTT parser/trace/header helpers +- Per-account session manager and retained-message handling +- MQTT JetStream request/reply adapter methods +- MQTT session persistence and packet identifier tracking + +If this batch is advanced with placeholder implementations or placeholder tests, PortTracker can look healthy while MQTT behavior remains non-functional. + +## Context Findings + +### Required command outputs + +Executed with explicit dotnet path: + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db` + - Status: `pending` + - Features: `78` (all `deferred`) + - Tests: `323` (all `deferred`) + - Depends on: `19,27` + - Go file: `server/mqtt.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms dependency chain includes `19 -> 40` and `27 -> 40` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall snapshot: `1924/6942 (27.7%)` + +Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`. + +### Current .NET baseline + +- MQTT types/constants exist in: + - `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs` +- Core behavior is mostly stubs: + - `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- Batch 40 mapped features target classes not fully represented yet (notably `MqttJetStreamAdapter`). +- Batch 40 mapped tests are mostly placeholder-template tests in `ImplBacklog` files with non-behavioral assertions. + +## Approach Options + +### Approach A: Keep all MQTT behavior in one file (`MqttHandler.cs`) + +- Pros: minimal file churn. +- Cons: difficult review boundaries, high merge risk, and weak checkpoint isolation for 78 features. + +### Approach B: Port tests first, then features + +- Pros: strict red/green pressure. +- Cons: too many missing APIs and placeholder methods produce noisy red phase with low signal. + +### Approach C (Recommended): Domain-sliced feature groups + staged test waves + +- Pros: aligns with Go method clusters, enables per-group verification and status updates, and supports strict anti-stub gates. +- Cons: requires planned decomposition across several files and explicit mapping alignment. + +**Decision:** Approach C. + +## Proposed Design + +### 1. Code organization and ownership + +Implement Batch 40 in MQTT-focused partial/domain files so each group has clear ownership: + +- `NatsServer` MQTT entrypoints (start/config/session manager bridge) +- `ClientConnection` MQTT parser/client-id/trace helpers +- `MqttJetStreamAdapter` request/reply and stream/consumer/message operations +- `MqttAccountSessionManager` retained/session orchestration +- `MqttSession` lifecycle/persistence/pending publish tracking +- Stateless helpers for retained-message encoding/decoding and header parsing + +This keeps behavioral slices reviewable while preserving .NET naming and namespace rules. + +### 2. Feature grouping strategy (<= 20 per group) + +- Group A (20): MQTT bootstrap/parser/auth and request-builder foundations +- Group B (20): JSA operations + JS API reply processing bridge +- Group C (20): session-manager retained/subscription core +- Group D (18): retained encode/decode + session lifecycle tracking + +Each group is independently verifiable and status-updated in controlled ID chunks. + +### 3. Test strategy for 323 mapped tests + +Use wave-based execution by test class risk profile: + +- MQTT-focused and deterministic first (`MqttHandlerTests`, helper-centric classes) +- then consumer/engine deterministic slices +- cluster/supercluster/benchmark/norace-heavy tests last, with explicit deferral when runtime infra is required + +Rule: deferred with concrete blocker reason is valid; fake pass/stub is not. + +### 4. Mapping alignment + +Because mapped class names include `MqttJetStreamAdapter` and method names using Go-style labels, include an early mapping-alignment checkpoint: + +- either implement mapped classes/methods directly, +- or add well-documented wrappers/adapters that preserve mapped method discoverability. + +No status promotion happens before this alignment compiles and has test coverage. + +### 5. Verification model (design-level) + +Batch 40 execution must be evidence-driven and include: + +- per-feature verification loop +- per-test verification loop +- stub detection scan before any promotion +- build gate and targeted/full test gates +- checkpoint protocol between tasks +- status updates with hard limit of 15 IDs per command + +## Risks and Mitigations + +- Risk: large count of integration/cluster tests may encourage placeholder conversions. + Mitigation: mandatory anti-stub patterns and forced deferred-with-reason fallback. +- Risk: mapping/class drift between tracker metadata and code structure. + Mitigation: explicit mapping-alignment task before major implementation. +- Risk: broad behavioral surface across parser/session/JSA code paths. + Mitigation: 4 feature groups with mandatory checkpoint gates after each task. + +## Non-Goals + +- Executing Batch 40 in this planning session. +- Re-scoping IDs outside Batch 40. +- Mass-promoting statuses without per-item evidence. diff --git a/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md b/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md new file mode 100644 index 0000000..9746435 --- /dev/null +++ b/docs/plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md @@ -0,0 +1,427 @@ +# Batch 40 MQTT Server/JSA Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 40 (`MQTT Server/JSA`) from `server/mqtt.go` so all 78 mapped features and 323 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker reasons. + +**Architecture:** Implement Batch 40 in four feature groups (`20/20/20/18`) aligned to MQTT bootstrap/parser, JSA adapter, account session manager, and session persistence/retained-message codec behavior. After each feature group, execute mapped test waves with strict evidence gates before any status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md` + +--- + +## Batch 40 Scope + +- Batch ID: `40` +- Name: `MQTT Server/JSA` +- Dependencies: `19`, `27` +- Go source: `golang/nats-server/server/mqtt.go` +- Features: `78` +- Tests: `323` + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Expected production files (create/modify as needed): + +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT partials) +- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` (or MQTT partial) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttJetStreamAdapter*.cs` (new if absent) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs` (new/partials) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` (new/partials) + +Expected test files (create/modify as needed): + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` +- Create missing mapped backlog files as needed: + - `JetStreamClusterTests1.Impltests.cs` + - `JetStreamClusterTests3.Impltests.cs` + - `JetStreamClusterTests4.Impltests.cs` + - `JetStreamSuperClusterTests.Impltests.cs` + - `JetStreamClusterLongTests.Impltests.cs` + - `JetStreamSourcingScalingTests.Impltests.cs` + - `JetStreamBenchmarks.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 40 feature and test must pass this protocol. No shortcuts. + +### 1. Dependency Preflight and Baseline + +Run before any status changes: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 19 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start batch only when ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 40 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 2. Per-Feature Verification Loop (REQUIRED for each feature ID) + +1. Inspect mapping and Go source intent: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` + +2. Claim the feature: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` + +3. Add/adjust at least one behavioral test path for this feature. +4. Implement minimal correct logic (no placeholders). +5. Run **Stub Detection Check**. +6. Run **Build Gate**. +7. Run relevant **Test Gate** filters. +8. Promote feature to `complete` only after evidence is captured. +9. Promote to `verified` only during task checkpoint after full gates pass. + +### 3. Per-Test Verification Loop (REQUIRED for each test ID) + +1. Inspect mapped test: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` + +2. Claim the test: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` + +3. Port full Arrange/Act/Assert behavior. +4. Run single-test filter and verify discovery: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` + +5. Confirm `Passed: 1, Failed: 0` (never accept `Passed: 0`). +6. Run class-level filter. +7. Run **Stub Detection Check** and **Build Gate**. +8. Promote test to `complete`; defer `verified` to checkpoint. + +### 4. Stub Detection Check (REQUIRED) + +Run after each feature loop, each test loop, and before any status promotion: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass\(\)|var goFile = \"server/|\.ShouldContain\(\"Should\"\)|GetRequiredApiLevel\(new Dictionary\)\.ShouldBe\(string\.Empty\)|=>\s*default;|return\s+default;|return\s+null;\s*$)" +fi +``` + +Any match blocks status promotion until fixed or explicitly deferred. + +### 5. Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run after each feature loop, each test loop, before any batch-update, and at every task checkpoint. + +### 6. Test Gate (REQUIRED) + +Run smallest relevant filters continuously; run all sections at checkpoints. + +MQTT-first filter: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.MessageTracerTests|FullyQualifiedName~ImplBacklog.ConfigReloaderTests" +``` + +Deterministic JS/consumer filter: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamBatchingTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests|FullyQualifiedName~ImplBacklog.JetStreamJwtTests|FullyQualifiedName~ImplBacklog.JetStreamFileStoreTests|FullyQualifiedName~ImplBacklog.JetStreamLeafNodeTests" +``` + +Cluster/concurrency filter: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.JetStreamClusterTests2|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3|FullyQualifiedName~ImplBacklog.JetStreamClusterTests4|FullyQualifiedName~ImplBacklog.JetStreamSuperClusterTests|FullyQualifiedName~ImplBacklog.JetStreamClusterLongTests|FullyQualifiedName~ImplBacklog.JetStreamSourcingScalingTests|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.ConcurrencyTests2|FullyQualifiedName~ImplBacklog.JetStreamBenchmarks" +``` + +Checkpoint/full gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 7. Status Update Protocol (HARD LIMIT: 15 IDs per batch-update) + +- Never send more than `15` IDs per `feature batch-update` command. +- Never send more than `15` IDs per `test batch-update` command. +- Never promote `verified` without checkpoint evidence. +- Never update IDs outside active task scope. +- Deferred items must include explicit blocker reason. + +Use: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### 8. Checkpoint Protocol Between Tasks (REQUIRED) + +Before moving to the next task: + +1. Run **Stub Detection Check**. +2. Run **Build Gate**. +3. Run **Test Gate** (targeted filters + full unit suite). +4. Apply status updates in <=15-ID chunks only. +5. Run: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db +``` + +6. Commit checkpoint before continuing. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +The following patterns are forbidden for Batch 40 feature/test work: + +- `throw new NotImplementedException(...)` in mapped feature methods +- Empty method bodies for mapped features +- `TODO` / `PLACEHOLDER` markers left in mapped logic +- `Assert.True(true)` / `Assert.Pass()` +- Placeholder template assertions disconnected from behavior: + - `var goFile = "server/..."` + - `"".ShouldContain("Should")` + - `GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as primary assertion +- Placeholder return shortcuts in mapped code: + - `=> default;` + - `return default;` + - `return null;` + +### Hard Limits + +- Max feature group size: `~20` features. +- Max IDs per `feature batch-update`: `15`. +- Max IDs per `test batch-update`: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between tasks. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop current item immediately. +2. Do not stub or fake-pass. +3. Mark item `deferred` with explicit blocker reason. +4. Continue with next unblocked feature/test. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with concrete reason is correct. Stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20): bootstrap/parser/auth/JSA request foundations + +IDs: +`2252,2253,2254,2255,2257,2258,2259,2260,2261,2262,2263,2264,2265,2266,2267,2268,2269,2270,2271,2272` + +### Group B (20): JSA operations + reply processing bridge + +IDs: +`2273,2274,2275,2276,2277,2278,2279,2280,2281,2282,2283,2284,2285,2286,2287,2288,2289,2290,2291,2292` + +### Group C (20): account session manager retained/subscription core + +IDs: +`2293,2294,2295,2296,2297,2298,2299,2300,2301,2302,2303,2304,2305,2306,2307,2308,2309,2310,2311,2312` + +### Group D (18): retained encode/decode + session lifecycle/publish tracking + +IDs: +`2313,2314,2315,2316,2317,2318,2319,2320,2321,2322,2323,2324,2325,2326,2327,2328,2329,2330` + +## Test Waves (323 total) + +- Wave T1 (51): `MqttHandlerTests`, `MessageTracerTests`, `ConfigReloaderTests` +- Wave T2 (89): `NatsConsumerTests`, `JetStreamBatchingTests`, `JetStreamEngineTests`, `JetStreamJwtTests`, `JetStreamFileStoreTests`, `JetStreamLeafNodeTests` +- Wave T3 (70): `JetStreamClusterTests1`, `JetStreamClusterTests2` +- Wave T4 (77): `JetStreamClusterTests3`, `JetStreamClusterTests4`, `JetStreamSuperClusterTests`, `JetStreamClusterLongTests`, `JetStreamSourcingScalingTests` +- Wave T5 (36): `ConcurrencyTests1`, `ConcurrencyTests2`, `JetStreamBenchmarks` + +ID extraction helper: + +```bash +sqlite3 -header -column porting.db " +SELECT bt.test_id AS id, t.dotnet_class, t.name +FROM batch_tests bt +JOIN unit_tests t ON t.id=bt.test_id +WHERE bt.batch_id=40 + AND t.dotnet_class IN () +ORDER BY bt.test_id;" +``` + +--- + +### Task 1: Preflight, Mapping Alignment, and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md` +- Read: `golang/nats-server/server/mqtt.go` +- Modify (if needed): `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/*` for class/method mapping alignment + +**Steps:** +1. Run dependency preflight and start Batch 40. +2. Confirm mapped class/method targets compile (notably `MqttJetStreamAdapter`). +3. Capture baseline build + full unit test counts. +4. Run checkpoint protocol. + +### Task 2: Implement Feature Group A + Test Wave T1 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT methods) +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs` (MQTT parse/trace helpers) +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/MessageTracerTests.Impltests.cs`, `ImplBacklog/ConfigReloaderTests.Impltests.cs` + +**Steps:** +1. Process each Group A feature through per-feature loop. +2. Port/verify Wave T1 tests through per-test loop. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol and commit. + +### Task 3: Implement Feature Group B + Test Wave T2 (Part 1) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttJetStreamAdapter*.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs` +- Modify tests: `ImplBacklog/JetStreamBatchingTests.Impltests.cs`, `ImplBacklog/JetStreamEngineTests.Impltests.cs`, `ImplBacklog/NatsConsumerTests.Impltests.cs`, `ImplBacklog/JetStreamJwtTests.Impltests.cs`, `ImplBacklog/JetStreamFileStoreTests.Impltests.cs`, `ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` + +**Steps:** +1. Process each Group B feature through per-feature loop. +2. Port deterministic tests from Wave T2 first; defer runtime-blocked tests with reason. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol and commit. + +### Task 4: Implement Feature Group C + Test Wave T3 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` +- Modify/Create tests: `ImplBacklog/JetStreamClusterTests1.Impltests.cs`, `ImplBacklog/JetStreamClusterTests2.Impltests.cs` + +**Steps:** +1. Process each Group C feature through per-feature loop. +2. Execute Wave T3 with strict infra eligibility checks. +3. Defer cluster-runtime blocked tests explicitly; do not stub. +4. Apply status updates in <=15-ID chunks. +5. Run checkpoint protocol and commit. + +### Task 5: Implement Feature Group D + Test Waves T4/T5 + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` +- Modify/Create tests: + - `ImplBacklog/JetStreamClusterTests3.Impltests.cs` + - `ImplBacklog/JetStreamClusterTests4.Impltests.cs` + - `ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` + - `ImplBacklog/JetStreamClusterLongTests.Impltests.cs` + - `ImplBacklog/JetStreamSourcingScalingTests.Impltests.cs` + - `ImplBacklog/ConcurrencyTests1.Impltests.cs` + - `ImplBacklog/ConcurrencyTests2.Impltests.cs` + - `ImplBacklog/JetStreamBenchmarks.Impltests.cs` + +**Steps:** +1. Process each Group D feature through per-feature loop. +2. Execute Waves T4/T5 with strict defer-with-reason handling. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol and commit. + +### Task 6: Batch 40 Closure Verification + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Steps:** +1. Run final stub detection across changed MQTT source/test files. +2. Run `build` and full unit test suite. +3. Verify Batch 40 state: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +4. Generate updated report: + +```bash +./reports/generate-report.sh +``` + +5. Final checkpoint commit. + diff --git a/docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md b/docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md new file mode 100644 index 0000000..07aa707 --- /dev/null +++ b/docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md @@ -0,0 +1,132 @@ +# Batch 41 MQTT Client/IO Design + +**Date:** 2026-02-27 +**Batch:** 41 (`MQTT Client/IO`) +**Scope:** 74 features + 28 unit tests +**Dependency:** Batch `40` (`MQTT Server/JSA`) +**Go source:** `golang/nats-server/server/mqtt.go` (line ~3503 through ~5882) + +## Problem + +Batch 41 is the MQTT client protocol and I/O execution surface: + +- CONNECT/PUBLISH/SUBSCRIBE/UNSUBSCRIBE parse and response paths +- QoS1/QoS2 flow control and PUBREL lifecycle +- retain handling and retained-message permission checks +- topic/filter <-> subject conversion logic +- byte-level reader/writer utilities used across MQTT packet handling + +This batch can easily appear "complete" via placeholders because many mapped tests are currently template-style and several mapped test IDs require non-MQTT infrastructure. The design must force evidence-based completion and explicit deferrals instead of stubs. + +## Context Findings + +### Required command outputs + +Executed with explicit runtime path because `dotnet` is not on PATH in this shell: + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db` + - Status: `pending` + - Features: `74` (all currently `deferred`) + - Tests: `28` (all currently `deferred`) + - Depends on: `40` + - Go file: `server/mqtt.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms Batch 41 is the final batch and depends only on Batch 40. +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Current snapshot: `1924/6942 (27.7%)`. + +### Current .NET baseline + +- MQTT types/constants exist in: + - `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs` +- Batch 41 mapped methods are not implemented yet and many target methods/classes are currently absent. +- `MqttHandler.cs` still contains broad NotImplemented stubs in server extension methods. +- `MqttHandlerTests.Impltests.cs` and related backlog files still include placeholder assertion patterns. +- Mapped test set includes cross-module classes (`AuthCalloutTests`, `NatsConsumerTests`, `LeafNodeHandlerTests`, `WebSocketHandlerTests`, `JetStreamClusterTests3`) in addition to MQTT-specific classes. + +## Approach Options + +### Approach A: Single-file incremental port in `MqttHandler.cs` + +- Pros: minimal file churn. +- Cons: high merge conflict risk, poor reviewability, weak ownership boundaries for 74 methods. + +### Approach B: Tests-first for all 28 IDs before feature work + +- Pros: strict red/green discipline. +- Cons: too many missing method surfaces creates high-noise red phase; many test IDs depend on deferred non-B41 features. + +### Approach C (Recommended): Feature-sliced implementation groups + test waves with explicit deferred protocol + +- Pros: aligns with Go function clusters, keeps each group <=20 features, enables mandatory checkpoint evidence, and handles cross-module tests without fake pass pressure. +- Cons: requires initial mapping-alignment step and disciplined status batching. + +**Decision:** Approach C. + +## Proposed Design + +### 1. File and component layout + +Implement Batch 41 in focused MQTT slices (new files/partials as needed), not a single monolith: + +- `ClientConnection` MQTT packet handlers/parsers + - `MqttParseConnect`, `MqttParsePub`, `MqttParseSubsOrUnsubs`, `MqttProcessPublishReceived`, enqueue acks +- `NatsServer` MQTT publish/session bridge handlers + - `MqttProcessConnect`, `MqttProcessPub`, `MqttProcessPubRel`, retained permissions audit +- `MqttSession` QoS2 consumer/pubrel helpers + - `TrackAsPubRel`, `UntrackPubRel`, `DeleteConsumer`, `ProcessJSConsumer` +- `MqttReader` / `MqttWriter` byte I/O utilities + - varint, length-prefixed strings/bytes, publish header encoding +- Stateless conversion/trace/helper methods + - topic/filter conversion, reserve-sub logic, trace formatters, Sparkplug-B helpers + +### 2. Feature grouping model (max ~20 each) + +- Group A (19): session + connect + initial publish path (`2331-2349`) +- Group B (19): retained/perms + QoS ack processing + subscribe callbacks (`2350-2368`) +- Group C (17): reserved-sub/sparkplug + subscribe processing + unsubscribe/ping (`2369-2385`) +- Group D (19): conversion + reader/writer I/O (`2386-2404`) + +### 3. Test-wave model (28 tests) + +- Wave T1 (10): deterministic MQTT parser/conversion tests (`2170,2171,2190,2191,2194,2195,2196,2199,2200,2229`) +- Wave T2 (11): publish/retain/session behavior tests (`2182,2204,2234,2235,2236,2237,2238,2246,2251,2253,2285`) +- Wave T3 (4): cross-module integration-touching tests (`115,1258,1924,3095`) +- Wave T4 (3): cluster-dependent tests (`1055,1056,1113`) + +### 4. Deferred strategy by design + +Because some mapped tests depend on deferred non-B41 features, completion criteria is: + +- implement and verify what is truly executable in local unit-test context, +- mark blockers `deferred` with specific reason and evidence, +- never substitute with placeholders. + +### 5. Verification architecture + +The implementation plan will enforce: + +- per-feature verification loop before promotion, +- per-test verification loop with discovered/pass evidence, +- stub detection checks on source and tests, +- build/test gates before every status update, +- status updates limited to <=15 IDs per batch-update, +- mandatory checkpoints between task groups. + +## Risks and Mitigations + +- Risk: class/method mapping drift (mapped methods not yet present). + - Mitigation: dedicated mapping-alignment preflight task before feature status changes. +- Risk: placeholder tests pass without exercising MQTT logic. + - Mitigation: anti-stub scans + assertion minimum checks + single-test evidence requirement. +- Risk: cluster/integration tests block throughput. + - Mitigation: explicit deferred-with-reason path and continue with unblocked items. +- Risk: large parser/I/O surface causes hidden regressions. + - Mitigation: incremental group checkpoints with full build + targeted/full test gates. + +## Non-Goals + +- Executing Batch 41 during this planning session. +- Marking statuses without verification evidence. +- Re-scoping features outside Batch 41 except documented test dependency blockers. diff --git a/docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md b/docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md new file mode 100644 index 0000000..b9817dc --- /dev/null +++ b/docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md @@ -0,0 +1,388 @@ +# Batch 41 MQTT Client/IO Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 41 (`MQTT Client/IO`) so all 74 mapped features and 28 mapped tests are either truly implemented and verified or explicitly deferred with concrete blocker reasons. + +**Architecture:** Implement in four feature groups (`19/19/17/19`) aligned to connect/publish/session handling, retained and QoS flow, subscription lifecycle, and reader/writer conversion I/O. Execute test waves after each group with strict evidence gates and explicit deferred handling for runtime-blocked cross-module tests. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md` + +--- + +## Batch 41 Scope + +- Batch ID: `41` +- Name: `MQTT Client/IO` +- Dependency: Batch `40` +- Go source: `golang/nats-server/server/mqtt.go` +- Feature IDs: `2331-2404` (74) +- Test IDs: `115,1055,1056,1113,1258,1924,2170,2171,2182,2190,2191,2194,2195,2196,2199,2200,2204,2229,2234,2235,2236,2237,2238,2246,2251,2253,2285,3095` + +Use this runtime path if `dotnet` is unavailable on PATH: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Expected source files (create/modify as needed): + +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs` (MQTT partials) +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT partials) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttReader*.cs` (new/partial) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttWriter*.cs` (new/partial) +- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` (new/partial) + +Expected test files (create/modify as needed): + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttExternalTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if missing) + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 41 feature/test item must pass this protocol before status promotion. + +### 1. Dependency and Baseline Preflight (REQUIRED) + +Before touching any status: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start only when Batch 41 is ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 41 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 2. Per-Feature Verification Loop (REQUIRED per feature ID) + +For each feature ID in active group: + +1. Inspect mapping and Go behavior: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db + ``` +2. Claim the feature: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db + ``` +3. Implement minimal correct logic in mapped .NET class/method. +4. Add or update at least one behavioral test path covering this feature. +5. Run stub detection checks (source + test). +6. Run build gate. +7. Run targeted test gate for touched classes/methods. +8. Promote to `complete` only when gates pass. +9. Promote to `verified` only at task checkpoint. + +### 3. Per-Test Verification Loop (REQUIRED per test ID) + +For each mapped test ID in active wave: + +1. Inspect mapped test details: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db + ``` +2. Claim the test: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db + ``` +3. Port full Arrange/Act/Assert behavior (no placeholders). +4. Run single-test filter and verify discovery: + ```bash + $DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal + ``` +5. Confirm output includes `Passed: 1, Failed: 0` (never accept `Passed: 0`). +6. Run class-level test filter. +7. Run stub detection + build gate. +8. Promote to `complete`; promote `verified` only at checkpoint. + +### 4. Stub Detection Check (REQUIRED) + +Run after each feature loop, each test loop, and before any status update. + +Source stub scan: + +```bash +src_changed=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server | rg "\\.cs$" || true) +if [ -n "$src_changed" ]; then + echo "$src_changed" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|=>\\s*default;|return\\s+default;|return\\s+null;|throw\\s+new\\s+NotImplementedException)" +fi +``` + +Test stub scan: + +```bash +test_changed=$(git diff --name-only -- dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$test_changed" ]; then + echo "$test_changed" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|goFile\\s*=\\s*\"server/|ShouldContain\\(\"Should\"\\)|ShouldBe\\(string\\.Empty\\))" +fi +``` + +Assertion depth check for changed test files: + +```bash +# per file: Shouldly assertion count / test count should be >= 1.5 +``` + +### 5. Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run after every feature/test loop, before any batch-update, and at every checkpoint. + +### 6. Test Gate (REQUIRED) + +Run minimal relevant filters continuously, and full suite at checkpoints. + +MQTT core filters: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.MqttExternalTests" +``` + +Cross-module mapped filters: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ImplBacklog.AuthCalloutTests|FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.LeafNodeHandlerTests|FullyQualifiedName~ImplBacklog.WebSocketHandlerTests|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3" +``` + +Checkpoint full unit gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### 7. Status Update Protocol (HARD LIMIT: 15 IDs) + +- Maximum `15` IDs per `feature batch-update` call. +- Maximum `15` IDs per `test batch-update` call. +- Never promote `verified` without checkpoint evidence. +- Deferred items must include explicit blocker reason. +- Status updates are scoped to the active task only. + +Commands: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### 8. Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting next task: + +1. Run source + test stub detection scans. +2. Run build gate. +3. Run targeted test filters plus full unit test suite. +4. Apply status updates in <=15-ID chunks. +5. Run batch progress check: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db + ``` +6. Commit checkpoint before moving on. + +--- + +## ANTI-STUB GUARDRAILS + +> **NON-NEGOTIABLE:** Placeholder behavior is not valid progress. + +### Forbidden Patterns + +Any of these in mapped feature/test bodies blocks status promotion: + +- `throw new NotImplementedException(...)` +- Empty mapped method body +- `// TODO` or `// PLACEHOLDER` left in mapped logic +- `=> default;`, `return default;`, `return null;` in mapped feature methods +- `Assert.True(true)` or `Assert.Pass()` +- Placeholder template assertions not tied to behavior: + - `var goFile = "server/..."` as test core + - `"...".ShouldContain("Should")` + - `GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as primary assertion +- Test methods with only trivial constructor/null checks for complex protocol behavior + +### Hard Limits + +- Feature groups must stay <=20 IDs. +- Max 15 IDs per status batch-update command. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between tasks. +- No cross-task bulk status updates. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +If an item needs infra or dependencies not available: + +1. Stop work on that ID. +2. Do not add placeholders or force a pass. +3. Mark `deferred` with exact blocker reason. +4. Continue with the next unblocked ID. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with concrete reason is correct behavior. Stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (19): session, CONNECT, publish ingress + +IDs: +`2331,2332,2333,2334,2335,2336,2337,2338,2339,2340,2341,2342,2343,2344,2345,2346,2347,2348,2349` + +### Group B (19): retained permissions, QoS ack flow, subscribe callbacks + +IDs: +`2350,2351,2352,2353,2354,2355,2356,2357,2358,2359,2360,2361,2362,2363,2364,2365,2366,2367,2368` + +### Group C (17): reserved-sub/sparkplug, subscribe/unsubscribe processing + +IDs: +`2369,2370,2371,2372,2373,2374,2375,2376,2377,2378,2379,2380,2381,2382,2383,2384,2385` + +### Group D (19): subject conversion and reader/writer I/O + +IDs: +`2386,2387,2388,2389,2390,2391,2392,2393,2394,2395,2396,2397,2398,2399,2400,2401,2402,2403,2404` + +## Test Waves (28 total) + +- Wave T1 (10): `2170,2171,2190,2191,2194,2195,2196,2199,2200,2229` +- Wave T2 (11): `2182,2204,2234,2235,2236,2237,2238,2246,2251,2253,2285` +- Wave T3 (4): `115,1258,1924,3095` +- Wave T4 (3): `1055,1056,1113` + +--- + +### Task 1: Preflight, Mapping Alignment, and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md` +- Read: `golang/nats-server/server/mqtt.go` +- Modify/Create: Batch 41 mapped class/method surfaces under `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/`, `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs`, and `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` + +**Steps:** +1. Verify Batch 40 completion/readiness and start Batch 41. +2. Ensure all mapped Batch 41 methods exist and compile. +3. Capture baseline build/test state. +4. Run checkpoint protocol. + +### Task 2: Implement Group A + Wave T1 (Parser/Connect Core) + +**Files:** +- Modify/Create: `ClientConnection` MQTT parse/connect files +- Modify/Create: `NatsServer` MQTT connect/publish ingress files +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs` +- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/MqttExternalTests.Impltests.cs` + +**Steps:** +1. Process all Group A feature IDs through per-feature loop. +2. Port/verify T1 parser/conversion tests through per-test loop. +3. Apply status updates in <=15-ID chunks. +4. Run checkpoint protocol and commit. + +### Task 3: Implement Group B + Wave T2 (Retain/QoS Core) + +**Files:** +- Modify/Create: MQTT retained/perms and QoS ack handling files +- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs` + +**Steps:** +1. Process Group B features through per-feature loop. +2. Port/verify T2 tests tied to retain, QoS flags, pub ack flow. +3. Defer blocked IDs with explicit reasons (no stubs). +4. Apply <=15-ID status updates. +5. Run checkpoint protocol and commit. + +### Task 4: Implement Group C + Remaining T2/T3 (Subscription Lifecycle) + +**Files:** +- Modify/Create: MQTT subscription/publish delivery/session consumer files +- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/AuthCalloutTests.Impltests.cs`, `ImplBacklog/NatsConsumerTests.Impltests.cs`, `ImplBacklog/LeafNodeHandlerTests.Impltests.cs`, `ImplBacklog/WebSocketHandlerTests.Impltests.cs` + +**Steps:** +1. Process Group C features through per-feature loop. +2. Complete remaining T2 tests and attempt T3 cross-module tests. +3. For cross-module blockers, defer with specific dependency reason. +4. Apply <=15-ID status updates. +5. Run checkpoint protocol and commit. + +### Task 5: Implement Group D + T4 Cluster-Mapped Tests (Reader/Writer + Conversion) + +**Files:** +- Modify/Create: `MqttReader*`, `MqttWriter*`, conversion helper files +- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if absent) + +**Steps:** +1. Process Group D features through per-feature loop. +2. Port/verify reader/writer conversion tests first. +3. Attempt T4 cluster tests only where dependencies allow; otherwise defer explicitly. +4. Apply <=15-ID status updates. +5. Run checkpoint protocol and commit. + +### Task 6: Final Batch 41 Verification and Closure + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Steps:** +1. Run final source/test stub scans on all changed files. +2. Run build and full unit test suite. +3. Verify Batch 41 status: + ```bash + $DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db + $DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db + ``` +4. Generate report: + ```bash + ./reports/generate-report.sh + ``` +5. Commit final checkpoint. diff --git a/reports/current.md b/reports/current.md index ca13c83..b95e36d 100644 --- a/reports/current.md +++ b/reports/current.md @@ -1,6 +1,6 @@ # NATS .NET Porting Status Report -Generated: 2026-02-27 22:01:32 UTC +Generated: 2026-02-27 22:27:52 UTC ## Modules (12 total) diff --git a/reports/report_f8dce79.md b/reports/report_f8dce79.md new file mode 100644 index 0000000..b95e36d --- /dev/null +++ b/reports/report_f8dce79.md @@ -0,0 +1,37 @@ +# NATS .NET Porting Status Report + +Generated: 2026-02-27 22:27:52 UTC + +## Modules (12 total) + +| Status | Count | +|--------|-------| +| verified | 12 | + +## Features (3673 total) + +| Status | Count | +|--------|-------| +| deferred | 2377 | +| n_a | 24 | +| stub | 1 | +| verified | 1271 | + +## Unit Tests (3257 total) + +| Status | Count | +|--------|-------| +| deferred | 2640 | +| n_a | 187 | +| verified | 430 | + +## Library Mappings (36 total) + +| Status | Count | +|--------|-------| +| mapped | 36 | + + +## Overall Progress + +**1924/6942 items complete (27.7%)**