diff --git a/docs/batches.md b/docs/batches.md index d643f76..5ef37a5 100644 --- a/docs/batches.md +++ b/docs/batches.md @@ -37,14 +37,14 @@ | 20 | Accounts Resolvers | [design](plans/2026-02-27-batch-20-accounts-resolvers-design.md) | [plan](plans/2026-02-27-batch-20-accounts-resolvers-implementation-plan.md) | planned | | 21 | Events + MsgTrace | [design](plans/2026-02-27-batch-21-events-msgtrace-design.md) | [plan](plans/2026-02-27-batch-21-events-msgtrace-implementation-plan.md) | planned | | 22 | Monitoring | [design](plans/2026-02-27-batch-22-monitoring-design.md) | [plan](plans/2026-02-27-batch-22-monitoring-implementation-plan.md) | planned | -| 23 | Routes | | | not_planned | -| 24 | Leaf Nodes | | | not_planned | -| 25 | Gateways | | | not_planned | -| 26 | WebSocket | | | not_planned | -| 27 | JetStream Core | | | not_planned | -| 28 | JetStream API | | | not_planned | -| 29 | JetStream Batching | | | not_planned | -| 30 | Raft Part 1 | | | not_planned | +| 23 | Routes | [design](plans/2026-02-27-batch-23-routes-design.md) | [plan](plans/2026-02-27-batch-23-routes-implementation-plan.md) | planned | +| 24 | Leaf Nodes | [design](plans/2026-02-27-batch-24-leaf-nodes-design.md) | [plan](plans/2026-02-27-batch-24-leaf-nodes-implementation-plan.md) | planned | +| 25 | Gateways | [design](plans/2026-02-27-batch-25-gateways-design.md) | [plan](plans/2026-02-27-batch-25-gateways-implementation-plan.md) | planned | +| 26 | WebSocket | [design](plans/2026-02-27-batch-26-websocket-design.md) | [plan](plans/2026-02-27-batch-26-websocket-implementation-plan.md) | planned | +| 27 | JetStream Core | [design](plans/2026-02-27-batch-27-jetstream-core-design.md) | [plan](plans/2026-02-27-batch-27-jetstream-core-implementation-plan.md) | planned | +| 28 | JetStream API | [design](plans/2026-02-27-batch-28-jetstream-api-design.md) | [plan](plans/2026-02-27-batch-28-jetstream-api-implementation-plan.md) | planned | +| 29 | JetStream Batching | [design](plans/2026-02-27-batch-29-jetstream-batching-design.md) | [plan](plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md) | planned | +| 30 | Raft Part 1 | [design](plans/2026-02-27-batch-30-raft-part-1-design.md) | [plan](plans/2026-02-27-batch-30-raft-part-1-implementation-plan.md) | planned | | 31 | Raft Part 2 | | | not_planned | | 32 | JS Cluster Meta | | | not_planned | | 33 | JS Cluster Streams | | | not_planned | diff --git a/docs/plans/2026-02-27-batch-23-routes-design.md b/docs/plans/2026-02-27-batch-23-routes-design.md new file mode 100644 index 0000000..9cb88bd --- /dev/null +++ b/docs/plans/2026-02-27-batch-23-routes-design.md @@ -0,0 +1,146 @@ +# Batch 23 Routes Design + +**Date:** 2026-02-27 +**Batch:** 23 (`Routes`) +**Scope:** Design only. No implementation in this document. + +## Problem + +Batch 23 ports NATS route/cluster behavior from `golang/nats-server/server/route.go` into .NET. + +- Features: `52` (currently `deferred`) +- Tests: `5` (currently `deferred`) +- Dependencies: batches `16`, `18` +- Go source: `server/route.go` +- Batch status: `pending` + +This batch is the route control plane: route connect negotiation, route INFO gossip, per-account route subscriptions, route pooling, route lifecycle, and compression negotiation. + +## Context Findings + +Collected with: + +- `dotnet run --project tools/NatsNet.PortTracker -- batch show 23 --db porting.db` +- `dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` +- `dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + +Key findings: + +- Most Batch 23 methods do not yet exist in .NET runtime code. +- Existing route surface in .NET is mostly type definitions (`Routes/RouteTypes.cs`) and some route-adjacent stubs in `NatsServer.Lifecycle.cs`. +- The 5 mapped tests are all in `RouteHandlerTests` and currently deferred. +- Existing `ProtocolParserTests` and `ServerTests` already cover route-adjacent parser/compression helpers and should be part of the feature verification gate. + +## Constraints and Success Criteria + +Constraints: + +- Follow `.NET 10` and project standards (`xUnit 3`, `Shouldly`, `NSubstitute`, no FluentAssertions/Moq). +- Keep implementation idiomatic C# while preserving Go behavior semantics. +- No placeholder implementations or fake-pass tests. +- Feature groups must be <= ~20 features. +- Batch status changes require evidence and guarded update cadence. + +Success criteria: + +- All 52 Batch 23 features are implemented and promoted with evidence (`deferred -> stub -> complete -> verified`) or explicitly deferred with blocker reason. +- All 5 mapped Batch 23 tests are real behavioral tests and passing. +- Route-related regression gates pass (`ProtocolParserTests`, `ServerTests`, lifecycle route tests, and mapped route ImplBacklog tests). + +## Approaches + +### Approach A: Single large route partial + +Implement all route behavior in one large `NatsServer.Routes.cs` + one `ClientConnection.Routes.cs` file. + +Trade-offs: + +- Pros: fewer files. +- Cons: very large diff, difficult review, high regression risk, weak checkpointing. + +### Approach B (Recommended): Domain-segmented route partials + +Split route implementation into bounded domains aligned with `route.go` sections and Batch 23 grouping. + +Trade-offs: + +- Pros: clear ownership boundaries, strong per-group build/test gates, easier status evidence, lower merge risk. +- Cons: more files and more coordination. + +### Approach C: Tests-first for all 5 mapped tests before feature work + +Rewrite all five mapped tests first, then implement features until tests pass. + +Trade-offs: + +- Pros: immediate test signal. +- Cons: only 5 mapped tests under-cover 52 features, so this risks false confidence without strict per-feature verification. + +## Recommended Design + +Use **Approach B**. + +### Architecture + +Organize Batch 23 into three feature domains: + +1. Route protocol parsing and permission handling (inbound route msgs, account sub/unsub, permission checks, route info processing) +2. Route subscription propagation and route creation pipeline (remote sub/unsub map operations, protocol emission, route object creation) +3. Route lifecycle and cluster connectivity (start routing, reconnect/validate routes, duplicate handling, gossip/implicit route handling, route iteration helpers) + +Mapped tests are executed after feature groups, with additional regression gates from parser/server test classes before `complete -> verified` promotion. + +### Proposed File Map + +Primary production files: + +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/IProtocolHandler.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteTypes.cs` + +Primary mapped test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +Related regression test files: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Protocol/ProtocolParserTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ServerTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Server/ServerLifecycleStubFeaturesTests.cs` + +### Data and Control Flow + +- Route connection establishment: connect -> route CONNECT/INFO exchange -> route registration -> sub interest sync. +- Route message path: parse routed protocol args -> permission checks -> route/local interest updates -> inbound delivery handling. +- Route topology updates: implicit route processing + forwarding new route info to known remotes + duplicate-route handling. + +### Error Handling Strategy + +- Mirror Go error semantics for parse and negotiation failures (return error / close connection path). +- Use explicit guard clauses for malformed protocols and invalid route state transitions. +- Keep blocked infrastructure behavior deferred with explicit reason instead of stubs. + +### Verification Strategy + +- Per-feature loop: read Go implementation, implement minimum C# behavior, build, run smallest related tests, record evidence. +- Per-group gates: mandatory stub scan, build gate, targeted test gate. +- Pre-verify gate: all related route test classes and mapped tests must pass before promoting any Batch 23 feature to `verified`. + +### Risks and Mitigations + +- Risk: Route lifecycle races/regressions due lock-sensitive logic. + - Mitigation: keep route iteration/removal code in focused partials; include lifecycle regression tests in every gate. +- Risk: Placeholder test ports in `ImplBacklog` produce false green. + - Mitigation: explicit anti-stub scans and assertion quality checks before status updates. +- Risk: Large route.go surface causes drift in status updates. + - Mitigation: strict ID chunking (<=15 per update) and evidence ledger per update. + +## Design Approval Basis + +This design is derived from the explicit request for Batch 23 planning, mandatory verification protocol parity with Batch 0, anti-stub guardrails, and grouped execution without implementation. diff --git a/docs/plans/2026-02-27-batch-23-routes-implementation-plan.md b/docs/plans/2026-02-27-batch-23-routes-implementation-plan.md new file mode 100644 index 0000000..20b7e35 --- /dev/null +++ b/docs/plans/2026-02-27-batch-23-routes-implementation-plan.md @@ -0,0 +1,551 @@ +# Batch 23 Routes Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 23 (`Routes`) by implementing 52 deferred route features from `server/route.go` and replacing/verifying the 5 mapped route tests with real behavioral coverage. + +**Architecture:** Implement route behavior in focused partials for `ClientConnection` and `NatsServer`, plus a small `RouteHandler` helper surface for batch-mapped helper methods. Execute in three feature groups (18/14/20 IDs), each with strict per-feature verification loops and group gates, then run the mapped test wave and only promote features `complete -> verified` after all related route regression tests are green. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-23-routes-design.md` + +--- + +## Batch 23 Scope + +- Batch ID: `23` +- Name: `Routes` +- Dependencies: `16`, `18` +- Go source: `golang/nats-server/server/route.go` +- Features: `52` +- Tests: `5` + +Reference commands: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 23 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +Primary production files: + +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/IProtocolHandler.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteTypes.cs` + +Mapped test file: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +Related regression test files: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Protocol/ProtocolParserTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ServerTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Server/ServerLifecycleStubFeaturesTests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 23 feature/test update must follow this protocol. + +### Per-Feature Verification Loop (REQUIRED for every feature ID) + +1. Read the tracked feature metadata and Go source before coding: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +# then open golang/nats-server/server/route.go at go_line_number..(go_line_number + go_line_count) +``` + +2. Implement the mapped C# behavior in the planned file(s) without placeholders. +3. Build immediately after each feature (or tightly coupled pair): + +```bash +dotnet build dotnet/ +``` + +4. Run the smallest related tests immediately: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ServerTests|FullyQualifiedName~ServerLifecycleStubFeaturesTests|FullyQualifiedName~RouteHandlerTests" +``` + +5. Record evidence (feature ID, build result, test result) before moving on. +6. If red, fix first. Do not stack unresolved failures. + +### Stub Detection Check (REQUIRED after every feature group and test wave) + +Run these checks before any status promotion: + +```bash +# 1) Forbidden stub markers +grep -R -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs + +# 2) Empty method bodies in mapped feature files +grep -R -n -E "^[[:space:]]*(public|internal|private|protected)[^{;=]*\)[[:space:]]*\{[[:space:]]*\}$" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs + +# 3) Fake ImplBacklog placeholders +grep -n -E "(var goFile = \"server/|ShouldContain\(\"Should\"\)|ShouldNotBeNullOrWhiteSpace\(\))" \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs +``` + +Any hit must be removed or explicitly deferred. No exceptions. + +### Build Gate (REQUIRED after each feature group) + +```bash +dotnet build dotnet/ +``` + +`Build succeeded` is required before moving any group features to `complete`. + +### Test Gate (REQUIRED before any `complete -> verified` promotion) + +All related route tests must pass: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Protocol.ProtocolParserTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ServerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Server.ServerLifecycleStubFeaturesTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.RouteHandlerTests" +``` + +All 5 Batch 23 mapped tests must pass before any Batch 23 feature is marked `verified`. + +### Status Update Protocol (REQUIRED) + +- Max `15` IDs per `feature batch-update` or `test batch-update` command. +- Required progression: + - `deferred -> stub` when active work begins. + - `stub -> complete` only after clean stub checks + build gate. + - `complete -> verified` only after global route test gate passes. +- Evidence required for each update chunk: + - successful build output, + - successful related test output, + - clean stub scan output, + - explicit ID list and reason. + +Status command templates: + +```bash +# features (<=15 IDs) +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +# tests (<=15 IDs) +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol (REQUIRED between tasks) + +After each task (every feature group and the test wave): + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "" +``` + +Do not start the next task until this checkpoint is complete. + +### If You Get Stuck (REQUIRED) + +1. Stop on the blocked ID. +2. Remove partial placeholder code (do not leave stubs). +3. Mark blocked item deferred with explicit reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db +# or +dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db +``` + +4. Continue to the next unblocked ID. +5. Never write fake-pass code or tests to bypass blockers. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +These patterns are forbidden in Batch 23 feature and test work: + +- `throw new NotImplementedException()` in mapped methods +- Empty method bodies for mapped route features +- `// TODO` or `// PLACEHOLDER` in mapped route code paths +- Trivial default-return placeholders for non-trivial route logic (`return null`, `return false`, `return 0`, `return string.Empty`) +- ImplBacklog placeholder tests that only assert string literals (for example `"...".ShouldContain("Should")`) +- Test bodies that do not exercise route production code + +### Hard Limits + +- Maximum `20` features per feature group +- Maximum `15` IDs per status update command +- One feature group active at a time +- Zero unresolved stub-scan hits in touched Batch 23 files +- Mandatory checkpoint (full build + full test + commit) between tasks + +--- + +## Feature Groups (<=20 IDs each) + +### Group 1 (18 IDs): Route protocol and info/perms foundation + +IDs: + +`2895,2896,2897,2898,2899,2900,2901,2902,2903,2904,2905,2906,2908,2909,2910,2911,2912,2913` + +Focus: + +- Account SUB/UNSUB handling and reply-sub cleanup +- Routed args parsing and inbound routed message setup +- Route CONNECT and route INFO processing +- Route compression negotiation and remote permission update +- Implicit route filtering/forwarding and import/export permission checks + +Target files: + +- `ClientConnection.Routes.cs` +- `NatsServer.Routes.InfoAndPerms.cs` +- `Routes/RouteHandler.cs` + +### Group 2 (14 IDs): Remote route subs and sub-proto fanout + +IDs: + +`2914,2915,2916,2917,2918,2919,2920,2921,2922,2923,2924,2925,2926,2927` + +Focus: + +- Routed-sub key/account extraction +- Remote sub/unsub parsing and state mutation +- Route SUB/UNSUB protocol buffer generation +- Sending full subscription interest to a route +- Route object creation and delayed-info decision helper + +Target files: + +- `ClientConnection.Routes.cs` +- `NatsServer.Routes.Subscriptions.cs` +- `Routes/RouteHandler.cs` + +### Group 3 (20 IDs): Route lifecycle, solicitation, dedupe, iteration + +IDs: + +`2928,2929,2930,2931,2932,2933,2935,2936,2937,2938,2939,2940,2941,2942,2943,2944,2945,2947,2948,2950` + +Focus: + +- Initial route INFO JSON and `addRoute` +- Solicited/duplicate route detection and upgrade path +- Route import filters +- Route accept loop, start routing, connect/reconnect/validate +- Route connect processing, remove-all-except behavior +- Duplicate server-name check and route iterators by pool index + +Target files: + +- `ClientConnection.Routes.cs` +- `NatsServer.Routes.Connections.cs` +- `NatsServer.Lifecycle.cs` (for harmonization of remove/iterate paths) +- `Routes/RouteHandler.cs` + +--- + +## Mapped Test Wave (5 IDs) + +IDs: + +`2798,2822,2823,2844,2850` + +Target file: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +Required behavior focus: + +- Cluster advertise startup error behavior +- TLS route certificate implicit allow pass/fail behavior +- Origin cluster route msg arg parsing behavior +- Route compression behavior + +--- + +## Task 1: Preflight and Batch Start + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-23-routes-design.md` +- Read: `golang/nats-server/server/route.go` + +**Step 1: Confirm batch context** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 23 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 2: Start batch** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 23 --db porting.db +``` + +**Step 3: Baseline gate** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 4: Checkpoint commit** + +```bash +git add porting.db +git commit -m "chore(batch23): start routes batch" +``` + +--- + +## Task 2: Implement Feature Group 1 (18 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs` + +**Step 1: Move Group 1 to `stub` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2895,2896,2897,2898,2899,2900,2901,2902,2903,2904,2905,2906,2908,2909,2910" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2911,2912,2913" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 18 IDs** + +**Step 3: Run Stub Detection Check + Build Gate + Group-related tests** + +```bash +# mandatory stub checks (see protocol) +# then: +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~RouteHandlerTests" +``` + +**Step 4: Move Group 1 IDs to `complete` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2895,2896,2897,2898,2899,2900,2901,2902,2903,2904,2905,2906,2908,2909,2910" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2911,2912,2913" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch23): implement route protocol and info/perms foundation" +``` + +--- + +## Task 3: Implement Feature Group 2 (14 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs` + +**Step 1: Move Group 2 to `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2914,2915,2916,2917,2918,2919,2920,2921,2922,2923,2924,2925,2926,2927" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 14 IDs** + +**Step 3: Run Stub Detection Check + Build Gate + Group-related tests** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ServerTests|FullyQualifiedName~RouteHandlerTests" +``` + +**Step 4: Move Group 2 IDs to `complete`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2914,2915,2916,2917,2918,2919,2920,2921,2922,2923,2924,2925,2926,2927" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch23): implement route subscription propagation and creation helpers" +``` + +--- + +## Task 4: Implement Feature Group 3 (20 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs` + +**Step 1: Move Group 3 to `stub` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2928,2929,2930,2931,2932,2933,2935,2936,2937,2938,2939,2940,2941,2942,2943" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2944,2945,2947,2948,2950" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 20 IDs** + +**Step 3: Run Stub Detection Check + Build Gate + Group-related tests** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ServerTests|FullyQualifiedName~ServerLifecycleStubFeaturesTests|FullyQualifiedName~RouteHandlerTests" +``` + +**Step 4: Move Group 3 IDs to `complete` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2928,2929,2930,2931,2932,2933,2935,2936,2937,2938,2939,2940,2941,2942,2943" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2944,2945,2947,2948,2950" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch23): implement route lifecycle, solicitation, dedupe, and iteration" +``` + +--- + +## Task 5: Implement and Verify Mapped Tests (5 IDs) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +**Step 1: Move mapped tests to `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2798,2822,2823,2844,2850" --set-status stub --db porting.db --execute +``` + +**Step 2: For each test ID (2798, 2822, 2823, 2844, 2850), run per-test loop** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db +# port behavior from routes_test.go line range +# run single test: +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~" +``` + +**Step 3: Run Stub Detection Check on test file + class-level gate** + +```bash +grep -n -E "(NotImplementedException|Assert\.True\(true\)|TODO|PLACEHOLDER|ShouldContain\(\"Should\"\))" \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs + +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.RouteHandlerTests" +``` + +**Step 4: Move mapped tests to `complete`, then `verified` only after full Test Gate** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2798,2822,2823,2844,2850" --set-status complete --db porting.db --execute +``` + +**Step 5: Run full Route Test Gate from protocol section and promote statuses** + +```bash +# tests verified +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2798,2822,2823,2844,2850" --set-status verified --db porting.db --execute + +# features verified in chunks <=15 +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2895,2896,2897,2898,2899,2900,2901,2902,2903,2904,2905,2906,2908,2909,2910" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2911,2912,2913,2914,2915,2916,2917,2918,2919,2920,2921,2922,2923,2924,2925" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2926,2927,2928,2929,2930,2931,2932,2933,2935,2936,2937,2938,2939,2940,2941" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "2942,2943,2944,2945,2947,2948,2950" --set-status verified --db porting.db --execute +``` + +**Step 6: Run Checkpoint Protocol and commit** + +```bash +git commit -m "test(batch23): port and verify mapped route tests" +``` + +--- + +## Task 6: Batch Closure and Reporting + +**Files:** + +- Modify: `porting.db` +- Generate: `reports/current.md` + +**Step 1: Final build + full unit test sweep** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 2: Final Batch 23 stub audit** + +```bash +grep -R -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Routes.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.InfoAndPerms.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Subscriptions.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Routes.Connections.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Routes/RouteHandler.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs +``` + +**Step 3: Verify batch status and complete batch** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 23 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch complete 23 --db porting.db +``` + +**Step 4: Generate report and final commit** + +```bash +./reports/generate-report.sh +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/ +git commit -m "feat(batch23): complete routes implementation and verification" +``` diff --git a/docs/plans/2026-02-27-batch-24-leaf-nodes-design.md b/docs/plans/2026-02-27-batch-24-leaf-nodes-design.md new file mode 100644 index 0000000..cc3dcc2 --- /dev/null +++ b/docs/plans/2026-02-27-batch-24-leaf-nodes-design.md @@ -0,0 +1,152 @@ +# Batch 24 Leaf Nodes Design + +**Date:** 2026-02-27 +**Batch:** 24 (`Leaf Nodes`) +**Scope:** Design only. No implementation in this document. + +## Problem + +Batch 24 ports leaf-node connection handling from `golang/nats-server/server/leafnode.go` into the .NET server. + +- Features: `67` (all currently `deferred`) +- Tests: `2` (both currently `deferred`) +- Dependencies: batches `19` and `23` +- Go source: `server/leafnode.go` +- Batch status: `pending` + +Leaf-node behavior touches client protocol parsing, solicited/accepted leaf lifecycle, account interest propagation, and cross-cluster subscription semantics. + +## Context Findings + +Collected with: + +- `dotnet run --project tools/NatsNet.PortTracker -- batch show 24 --db porting.db` +- `dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` +- `dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + +Additional repository findings: + +- .NET already has leaf data models in `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeTypes.cs` (`Leaf`, `LeafNodeCfg`, `LeafConnectInfo`) but the Batch 24 behavior surface is largely unimplemented. +- `ClientConnection` currently has explicit leaf placeholders (`IsHubLeafNode`, `RemoteCluster`) and parser-level leaf argument helpers still marked as stubs/delegates. +- `LeafNodeHandlerTests.Impltests.cs` and `RouteHandlerTests.Impltests.cs` still contain placeholder-style tests and do not currently include the two Batch 24 mapped methods. +- Batch 24 mapped tests are: + - `1966` (`LeafNodeHandlerTests.LeafNodeRoutedSubKeyDifferentBetweenLeafSubAndRoutedSub_ShouldSucceed`) + - `2825` (`RouteHandlerTests.ClusterQueueGroupWeightTrackingLeak_ShouldSucceed`) + +## Constraints and Success Criteria + +Constraints: + +- Follow project standards in `docs/standards/dotnet-standards.md`. +- Keep behavior equivalent to Go intent, but idiomatic C# (.NET 10, nullable enabled). +- No placeholder implementations, no fake-pass tests, no status promotion without evidence. +- Feature work must be grouped into chunks of <= ~20 features. +- Batch 24 execution must wait for dependency readiness (batches 19 and 23). + +Success criteria: + +- All 67 features are either implemented and verified, or explicitly deferred with a concrete blocker reason. +- Both mapped tests are implemented as real behavioral tests and pass. +- Leaf-related parser/account regression tests pass before `complete -> verified`. +- Batch 24 can be completed through PortTracker without overrides for unverifiable work. + +## Approaches + +### Approach A: Single monolithic leaf file + +Implement all 67 methods in one large `NatsServer.LeafNodes.cs` and keep most `ClientConnection` work in `ClientConnection.cs`. + +Trade-offs: + +- Pros: fewer files to navigate. +- Cons: very high review risk, difficult evidence tracking, poor checkpoint isolation. + +### Approach B (Recommended): Domain-segmented partials by leaf lifecycle stage + +Split implementation by responsibility: + +1. Leaf validation/config/bootstrap +2. Connect/handshake/lifecycle +3. Interest propagation/account/subscription maps +4. Message/sub/unsub processing and websocket solicit path + +Trade-offs: + +- Pros: aligns to `leafnode.go` sections, cleaner verification gates, easier status chunking and rollback. +- Cons: more file coordination. + +### Approach C: Test-first only on mapped tests, then fill remaining features + +Start with the two mapped tests and then backfill features. + +Trade-offs: + +- Pros: immediate test signal. +- Cons: two tests under-cover 67 features, making false confidence likely unless strict feature-level verification is enforced. + +## Recommended Design + +Use **Approach B** with four feature groups (18/18/14/17 IDs) mapped to Go line ranges and .NET class ownership. + +### Architecture + +- `ClientConnection` partials handle leaf protocol parsing, leaf connect payload processing, sub/unsub deltas, permission violations, and websocket solicitation. +- `NatsServer` partials handle remote solicitation, outbound reconnect/connect loops, listener accept loop, leaf info dissemination, and connection registration lifecycle. +- `LeafNode` helper surface (`LeafNodeHandler` + `LeafNodeCfg` methods) centralizes validation and key generation utilities used by both server and client paths. +- `Account` updates cover `UpdateLeafNodesEx` semantics and gateway/leaf interest propagation coupling. + +### Proposed File Map + +Primary production files: + +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.LeafNodes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.ConfigAndConnect.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.Subscriptions.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeHandler.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeTypes.cs` (for `LeafNodeCfg` method surface) +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` + +Mapped test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +Related regression tests: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Protocol/ProtocolParserTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Accounts/ResolverDefaultsOpsTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ClientTests.cs` + +### Data and Control Flow + +- Outbound flow: configured remote leaf nodes -> validate -> pick URL -> connect/proxy/ws path -> leaf CONNECT/INFO handshake -> register connection -> initialize subscription map. +- Inbound flow: accept loop -> create leaf client -> process INFO/CONNECT -> negotiate compression/perms -> account registration and interest sync. +- Interest propagation: account sub changes -> leaf smap deltas -> leaf SUB/UNSUB protocol emission -> inbound leaf message/sub updates -> permission enforcement. + +### Error Handling Strategy + +- Preserve Go failure modes for malformed protocol args, invalid remote config, and loop/duplicate detection paths. +- Prefer explicit guard clauses with clear exceptions over silent default returns. +- On infrastructure blockers, defer with reason rather than introducing placeholders. + +### Verification Strategy + +- Enforce per-feature verification loop (Go read, C# implementation, build, related test run). +- Enforce group-level stub scans and build gate before any `stub -> complete`. +- Enforce full leaf-related test gate before any `complete -> verified`. +- Enforce checkpoint protocol (full build + full tests + commit) between every task. + +### Risks and Mitigations + +- Risk: race-prone account/smap updates regress existing account behavior. + - Mitigation: isolate account update logic and require `ResolverDefaultsOpsTests` in every verification wave. +- Risk: parser and runtime leaf arg handling diverge. + - Mitigation: keep parser and client leaf arg updates in the same feature group and verify via `ProtocolParserTests`. +- Risk: backlog tests remain placeholders and create false green. + - Mitigation: mandatory anti-stub scans and method-level filters for mapped tests. + +## Design Approval Basis + +This design is based on Batch 24 metadata from PortTracker, existing Batch 0 verification standards, and explicit requirements for feature+test anti-stub guardrails and grouped execution. diff --git a/docs/plans/2026-02-27-batch-24-leaf-nodes-implementation-plan.md b/docs/plans/2026-02-27-batch-24-leaf-nodes-implementation-plan.md new file mode 100644 index 0000000..47ccec5 --- /dev/null +++ b/docs/plans/2026-02-27-batch-24-leaf-nodes-implementation-plan.md @@ -0,0 +1,647 @@ +# Batch 24 Leaf Nodes Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 24 (`Leaf Nodes`) by implementing 67 deferred leaf-node features from `server/leafnode.go` and implementing/verifying the 2 mapped tests with real behavioral coverage. + +**Architecture:** Implement leaf-node behavior in domain-focused partials for `ClientConnection` and `NatsServer`, plus helper logic in `LeafNodeHandler`/`LeafNodeCfg` and account propagation updates in `Account`. Execute in four feature groups (18/18/14/17 IDs), each with a strict per-feature verification loop and hard gates. Only promote `complete -> verified` after both mapped tests and all related regression tests pass. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-24-leaf-nodes-design.md` + +--- + +## Batch 24 Scope + +- Batch ID: `24` +- Name: `Leaf Nodes` +- Dependencies: `19`, `23` +- Go source: `golang/nats-server/server/leafnode.go` +- Features: `67` +- Tests: `2` + +Reference commands: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 24 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +If `dotnet` is not on `PATH` in your shell, use `/usr/local/share/dotnet/dotnet` in the same commands. + +Primary production files: + +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.LeafNodes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.ConfigAndConnect.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.Subscriptions.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeHandler.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` + +Mapped test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +Related regression test files: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Protocol/ProtocolParserTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Accounts/ResolverDefaultsOpsTests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ClientTests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 24 feature/test update must follow this protocol. + +### Per-Feature Verification Loop (REQUIRED for every feature ID) + +1. Read tracked metadata and Go source before coding: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +# then open golang/nats-server/server/leafnode.go at go_line_number..(go_line_number + go_line_count) +``` + +2. Write/adjust C# implementation for that specific feature. +3. Build immediately after each feature (or tightly coupled pair): + +```bash +dotnet build dotnet/ +``` + +4. Run the smallest related tests immediately (feature-local gate): + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ResolverDefaultsOpsTests|FullyQualifiedName~ClientTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~RouteHandlerTests" +``` + +5. Record evidence per feature ID: Go line(s) read, files touched, build status, test command, test summary. +6. If any command is red, fix before moving to the next ID. + +### Stub Detection Check (REQUIRED after every feature group and test wave) + +Run before any status promotion: + +```bash +# 1) Forbidden markers in mapped production and test files +grep -R -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs + +# 2) Empty method bodies in mapped production files +grep -R -n -E "^[[:space:]]*(public|internal|private|protected)[^{;=]*\\)[[:space:]]*\\{[[:space:]]*\\}$" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs +``` + +Any hit must be fixed or deferred with a reason. Do not promote status with unresolved hits. + +### Build Gate (REQUIRED after each feature group) + +```bash +dotnet build dotnet/ +``` + +`Build succeeded` is mandatory before any `stub -> complete` update. + +### Test Gate (REQUIRED before any `complete -> verified` promotion) + +All related tests must pass: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Protocol.ProtocolParserTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Accounts.ResolverDefaultsOpsTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ClientTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~LeafNodeHandlerTests.LeafNodeRoutedSubKeyDifferentBetweenLeafSubAndRoutedSub_ShouldSucceed" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RouteHandlerTests.ClusterQueueGroupWeightTrackingLeak_ShouldSucceed" +``` + +No Batch 24 feature may be marked `verified` until both mapped tests and this regression gate are green. + +### Status Update Protocol (REQUIRED) + +- Never include more than `15` IDs in a single `feature batch-update` or `test batch-update`. +- Required progression: + - `deferred -> stub` when active work begins. + - `stub -> complete` only after clean stub checks + build gate. + - `complete -> verified` only after full Test Gate passes. +- Evidence is required for each update chunk: + - build output summary, + - related test output summary, + - clean stub scan output, + - explicit IDs being promoted and why. + +Command templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +After each task (feature group or test wave), before starting the next task: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "" +``` + +Do not continue if checkpoint gates fail. + +### If You Get Stuck (REQUIRED) + +1. Stop on the blocked ID. +2. Remove partial placeholders; do not leave stubs in-tree. +3. Mark the blocked item `deferred` with a specific reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db + +dotnet run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +4. Move to next unblocked ID. +5. Do not create fake-pass code/tests to force progress. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +The following are forbidden in Batch 24 feature and test work: + +- `throw new NotImplementedException()` in mapped feature methods +- Empty method bodies for mapped features +- `// TODO` or `// PLACEHOLDER` in mapped code paths +- Non-semantic default placeholders (`return null;`, `return false;`, `return 0;`, `return string.Empty;`) for non-trivial methods +- ImplBacklog fake assertions such as `"...".ShouldContain("Should")` +- Tests that do not call production code +- Tests with no meaningful assertions on behavior under test + +### Hard Limits + +- Maximum `20` features per feature task +- Maximum `15` IDs per status-update command +- One feature group active at a time +- Zero unresolved stub-scan hits before any promotion +- Mandatory checkpoint (full build + full tests + commit) between tasks + +--- + +## Feature Groups (<=20 IDs each) + +### Group 1 (18 IDs): Identity, validation, and remote-config bootstrap + +IDs: + +`1979,1980,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997` + +Focus: + +- Leaf role identity checks (`solicited`, `spoke`, `isolated`) +- Remote solicitation validity checks and TLS config updates +- Leaf config construction and URL/delay helpers +- Validation helpers (`validateLeafNode*`) and proxy tunnel setup + +### Group 2 (18 IDs): Connect loop, listener, CONNECT/INFO foundation + +IDs: + +`1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015` + +Focus: + +- Outbound connect/reconnect and observer/migration timer handling +- Listener accept loop and leaf client creation path +- Leaf CONNECT payload generation and INFO handling +- Compression negotiation and server leaf-info broadcast helpers + +### Group 3 (14 IDs): Connection registration and account/smap propagation + +IDs: + +`2016,2017,2018,2019,2021,2022,2024,2025,2026,2027,2029,2030,2031,2032` + +Focus: + +- Remote URL update flow and host/port/IP advertisement +- Leaf connection registration and inbound CONNECT processing +- Account permission and subscription map initialization +- Gateway interest update and account leaf propagation (`UpdateLeafNodesEx`) + +### Group 4 (17 IDs): Sub keys, sub/unsub and inbound message pipeline + +IDs: + +`2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049` + +Focus: + +- Routed-sub key generation (`keyFromSub`, `keyFromSubWithOrigin`) +- Leaf sub protocol serialization and sub/unsub handlers +- Leaf MSG/HMSG arg parsing and inbound message processing +- Permission violation/error paths +- TLS/ws solicit helpers and resume/finish connect process + +--- + +## Mapped Test Wave (2 IDs) + +IDs: + +`1966,2825` + +Target methods: + +- `LeafNodeHandlerTests.LeafNodeRoutedSubKeyDifferentBetweenLeafSubAndRoutedSub_ShouldSucceed` +- `RouteHandlerTests.ClusterQueueGroupWeightTrackingLeak_ShouldSucceed` + +Target files: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +--- + +## Task 1: Preflight and Dependency Gate + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-24-leaf-nodes-design.md` +- Read: `golang/nats-server/server/leafnode.go` + +**Step 1: Confirm batch context** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 24 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 2: Verify dependency readiness** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Expected: Batch `24` is listed as ready. If not, stop implementation and keep planning-only state. + +**Step 3: Start batch and run baseline gate** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 24 --db porting.db +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 4: Checkpoint commit** + +```bash +git add porting.db +git commit -m "chore(batch24): start leaf nodes batch" +``` + +--- + +## Task 2: Implement Feature Group 1 (18 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeHandler.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.ConfigAndConnect.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeTypes.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` + +**Step 1: Move Group 1 to `stub` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1979,1980,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1995,1996,1997" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 18 IDs** + +**Step 3: Run mandatory stub checks + build gate + related tests** + +```bash +# run stub detection check from protocol section +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ClientTests" +``` + +**Step 4: Move Group 1 to `complete` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1979,1980,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1995,1996,1997" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch24): implement leaf validation and remote config bootstrap" +``` + +--- + +## Task 3: Implement Feature Group 2 (18 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.LeafNodes.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.ConfigAndConnect.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeTypes.cs` + +**Step 1: Move Group 2 to `stub` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2013,2014,2015" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 18 IDs** + +**Step 3: Run mandatory stub checks + build gate + related tests** + +```bash +# run stub detection check from protocol section +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ClientTests" +``` + +**Step 4: Move Group 2 to `complete` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2013,2014,2015" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch24): implement leaf connect lifecycle and handshake foundation" +``` + +--- + +## Task 4: Implement Feature Group 3 (14 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.Subscriptions.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.LeafNodes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` + +**Step 1: Move Group 3 to `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2016,2017,2018,2019,2021,2022,2024,2025,2026,2027,2029,2030,2031,2032" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 14 IDs** + +**Step 3: Run mandatory stub checks + build gate + related tests** + +```bash +# run stub detection check from protocol section +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ResolverDefaultsOpsTests|FullyQualifiedName~ClientTests|FullyQualifiedName~ProtocolParserTests" +``` + +**Step 4: Move Group 3 to `complete`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2016,2017,2018,2019,2021,2022,2024,2025,2026,2027,2029,2030,2031,2032" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch24): implement leaf registration and account interest propagation" +``` + +--- + +## Task 5: Implement Feature Group 4 (17 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.LeafNodes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/LeafNodeHandler.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.LeafNodes.ConfigAndConnect.cs` + +**Step 1: Move Group 4 to `stub` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2048,2049" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 17 IDs** + +**Step 3: Run mandatory stub checks + build gate + related tests** + +```bash +# run stub detection check from protocol section +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ResolverDefaultsOpsTests|FullyQualifiedName~ClientTests" +``` + +**Step 4: Move Group 4 to `complete` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2048,2049" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch24): implement leaf sub and inbound message pipeline" +``` + +--- + +## Task 6: Implement and Verify the 2 Mapped Tests + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` + +**Step 1: Move tests to `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "1966,2825" --set-status stub --db porting.db --execute +``` + +**Step 2: Per-test loop for each ID** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test show 1966 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- test show 2825 --db porting.db + +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~LeafNodeHandlerTests.LeafNodeRoutedSubKeyDifferentBetweenLeafSubAndRoutedSub_ShouldSucceed" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~RouteHandlerTests.ClusterQueueGroupWeightTrackingLeak_ShouldSucceed" +``` + +**Step 3: Mandatory stub scan on test files + class-level run** + +```bash +grep -n -E "(NotImplementedException|Assert\\.True\\(true\\)|TODO|PLACEHOLDER|ShouldContain\\(\"Should\"\\))" \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs + +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~LeafNodeHandlerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RouteHandlerTests" +``` + +**Step 4: Move tests to `complete`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "1966,2825" --set-status complete --db porting.db --execute +``` + +**Step 5: Run Checkpoint Protocol and commit** + +```bash +git commit -m "test(batch24): port mapped leaf-node tests" +``` + +--- + +## Task 7: Verification Gate and Promote to `verified` + +**Files:** + +- Modify: `porting.db` + +**Step 1: Run full Batch 24 test gate** + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Protocol.ProtocolParserTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.Accounts.ResolverDefaultsOpsTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ClientTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~LeafNodeHandlerTests.LeafNodeRoutedSubKeyDifferentBetweenLeafSubAndRoutedSub_ShouldSucceed" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RouteHandlerTests.ClusterQueueGroupWeightTrackingLeak_ShouldSucceed" +``` + +**Step 2: Promote tests to `verified`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "1966,2825" --set-status verified --db porting.db --execute +``` + +**Step 3: Promote all features to `verified` in chunks <=15** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1979,1980,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2021,2022,2024,2025,2026" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2027,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2043,2044,2045,2046,2047,2048,2049" --set-status verified --db porting.db --execute +``` + +**Step 4: Run Checkpoint Protocol and commit** + +```bash +git commit -m "feat(batch24): verify all leaf-node features and mapped tests" +``` + +--- + +## Task 8: Batch Closure and Reporting + +**Files:** + +- Modify: `porting.db` +- Generate: `reports/current.md` + +**Step 1: Final build/test sweep** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 2: Final Batch 24 stub audit** + +```bash +grep -R -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/LeafNode/*.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs +``` + +**Step 3: Verify batch closure** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 24 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch complete 24 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 4: Generate report and final commit** + +```bash +./reports/generate-report.sh +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/ +git commit -m "feat(batch24): complete leaf nodes implementation and verification" +``` + diff --git a/docs/plans/2026-02-27-batch-25-gateways-design.md b/docs/plans/2026-02-27-batch-25-gateways-design.md new file mode 100644 index 0000000..5ec1c33 --- /dev/null +++ b/docs/plans/2026-02-27-batch-25-gateways-design.md @@ -0,0 +1,161 @@ +# Batch 25 Gateways Design + +**Date:** 2026-02-27 +**Batch:** 25 (`Gateways`) +**Scope:** Design only. No implementation in this document. + +## Problem + +Batch 25 ports gateway connection handling from `golang/nats-server/server/gateway.go` into the .NET server. + +- Features: `86` (all currently `deferred`) +- Tests: `59` (all currently `deferred`) +- Dependencies: batches `19` and `23` +- Go source: `server/gateway.go` +- Batch status: `pending` + +This batch is one of the highest fan-in networking areas: it touches server startup, inbound/outbound connection lifecycle, route gossip, account interest propagation, and reply-subject mapping. + +## Context Findings + +Collected with: + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 25 --db porting.db` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db` + +Repository observations: + +- `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` already contains gateway data types, but method behavior is mostly unimplemented and includes at least one explicit TODO (`GwReplyMapping.Get`). +- `NatsServer.Init.cs` still comments gateway initialization as deferred (`s.NewGateway(opts)`). +- `NatsServer.Lifecycle.cs` includes gateway placeholders (`GetAllGatewayConnections`, `RemoveRemoteGatewayConnection`, `GatewayUpdateSubInterest`) that need full parity behavior. +- `ClientConnection.cs` does not yet contain gateway protocol/message handlers (`ProcessGatewayConnect`, `ProcessGatewayInfo`, `ProcessGatewayRSub`, `SendMsgToGateways`, etc.). +- `GatewayHandlerTests.Impltests.cs` currently has many placeholder-style tests (for example assertions against literal `"Should"` strings), so Batch 25 test work must include anti-stub cleanup. +- `batch ready` currently does not list Batch 25 as ready, so implementation must not start until dependencies are complete. + +## Constraints and Success Criteria + +Constraints: + +- Follow `docs/standards/dotnet-standards.md` (.NET 10, nullable enabled, xUnit 3 + Shouldly + NSubstitute). +- Keep behavior equivalent to Go intent in `gateway.go`; do not line-by-line transliterate goroutine patterns. +- No stubs/fake-pass tests/status promotion without evidence. +- Execute in feature groups of at most ~20 features. +- Dependency order is strict: Batch 25 execution only starts when batches 19 and 23 are complete. + +Success criteria: + +- All 86 features are either implemented and verified, or explicitly deferred with concrete blocker reasons. +- All 59 mapped tests are real behavioral tests and pass. +- Gateway regression paths remain green across server/account/route/leaf/monitoring touchpoints. +- Batch 25 can be completed in PortTracker without audit override due to placeholder code. + +## Approaches + +### Approach A: Single-file gateway implementation + +Place nearly all gateway methods into one large server file and keep client changes in `ClientConnection.cs`. + +Trade-offs: + +- Pros: fewer files. +- Cons: very high review risk, poor parallelism, hard evidence tracking, difficult rollback. + +### Approach B (Recommended): Domain-segmented partials by gateway lifecycle + +Split implementation into focused partials: + +1. Config/bootstrap/listener and outbound solicitation +2. Handshake/INFO/gossip and connection registry +3. Interest/subscription propagation +4. Reply-map and inbound/outbound message processing + +Trade-offs: + +- Pros: maps directly to `gateway.go` sections, clean feature grouping, tighter verification loops, lower merge risk. +- Cons: more files and coordination. + +### Approach C: Test-first backlog replacement then backfill feature methods + +Replace all gateway-related placeholder tests first, then implement production behavior until tests pass. + +Trade-offs: + +- Pros: immediate feedback. +- Cons: current mapped tests underrepresent some feature paths unless supplemented with targeted regression gates. + +## Recommended Design + +Use **Approach B** with five feature groups (`19/18/17/16/16`) and four test waves (`15/16/18/10`) tied to `gateway.go` line ranges and mapped test IDs. + +### Architecture + +- `NatsServer` partials own gateway server state transitions: initialization, accept loop, solicitation/reconnect, remote registration, route gossip, account-level interest operations, and reply-map lifecycle. +- `ClientConnection` partials own gateway protocol operations: CONNECT/INFO handling, RSub/RUnsub/account commands, inbound gateway message pipeline, and outbound gateway send decisions. +- `Gateway` helper surface owns deterministic utilities: hash/prefix helpers, routed-reply parsing, string/proto builders, and gateway option validation. +- Existing `GatewayTypes` remains the source of gateway state objects but receives method implementations and lock-safe helpers. + +### Proposed File Map + +Primary production files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayHandler.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConfigAndStartup.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConnectionsAndGossip.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.Interest.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ReplyMap.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Protocol.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Messages.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Init.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` + +Mapped test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` + +### Data and Control Flow + +- Startup flow: validate options -> initialize gateway state (`NewGateway`) -> start accept loop -> solicit remotes after configured delay. +- Handshake flow: outbound connection receives INFO -> emits CONNECT -> registers outbound; inbound accepts CONNECT -> validates/authorizes -> registers inbound. +- Gossip flow: new gateway URLs and cluster metadata propagate to routes and inbound gateways. +- Interest flow: account/subscription changes update per-account gateway state and trigger RS+/RS-/A+/A- commands. +- Message flow: publish routing checks local interest, gateway interest mode, queue weights, and reply mapping before fanout. +- Reply flow: `_GR_`/legacy reply prefixes are tracked, mapped, and expired through periodic cleanup. + +### Error Handling Strategy + +- Preserve Go behavior for protocol errors (`wrong gateway`, malformed gateway commands, invalid account commands). +- Keep explicit guard clauses and structured log messages around malformed INFO/CONNECT and URL validation failures. +- On transient dial failures use reconnect/backoff paths; on unrecoverable config violations fail fast. +- For blocked items, defer with concrete reason instead of placeholder logic. + +### Verification Strategy + +- Use per-feature and per-test loops (read Go source, implement, build, run targeted tests). +- Enforce mandatory stub scans for both production and test files after each group. +- Require build gate after each feature group and full gateway-related test gate before `verified`. +- Enforce checkpoint protocol between tasks: full build + full unit test sweep + commit. + +### Risks and Mitigations + +- Risk: race regressions in gateway interest maps and reply map. + - Mitigation: include no-race mapped tests (`2376`, `2490`) in mandatory verification wave. +- Risk: mismatch between route gossip and gateway URL updates. + - Mitigation: include monitoring and reload mapped tests (`2127`, `2131`, `2747`) in verification gate. +- Risk: placeholder test drift in `GatewayHandlerTests`. + - Mitigation: anti-stub guardrails with explicit forbidden patterns and evidence-based status updates. + +## Design Approval Basis + +This design is based on Batch 25 tracker metadata, current repository state, and the mandatory verification/anti-stub guardrail model from `docs/plans/2026-02-27-batch-0-implementable-tests-plan.md`, adapted for both features and tests. diff --git a/docs/plans/2026-02-27-batch-25-gateways-implementation-plan.md b/docs/plans/2026-02-27-batch-25-gateways-implementation-plan.md new file mode 100644 index 0000000..87ca58d --- /dev/null +++ b/docs/plans/2026-02-27-batch-25-gateways-implementation-plan.md @@ -0,0 +1,862 @@ +# Batch 25 Gateways Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 25 (`Gateways`) by implementing 86 deferred gateway features from `server/gateway.go` and implementing/verifying all 59 mapped gateway-related tests as real behavioral tests. + +**Architecture:** Implement gateway behavior in domain-focused partials for `NatsServer` and `ClientConnection`, with helper logic in `Gateway/GatewayHandler.cs` and state helpers in `GatewayTypes.cs`. Execute in five feature groups (19/18/17/16/16) and four test waves (15/16/18/10), each guarded by strict per-feature/per-test verification loops, anti-stub scans, and build/test gates before status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-25-gateways-design.md` + +--- + +## Batch 25 Scope + +- Batch ID: `25` +- Name: `Gateways` +- Dependencies: `19`, `23` +- Go source: `golang/nats-server/server/gateway.go` +- Features: `86` +- Tests: `59` + +Reference commands: + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 25 --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +If `dotnet` is on `PATH`, equivalent commands may use `dotnet` directly. + +Primary production files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayHandler.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConfigAndStartup.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConnectionsAndGossip.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.Interest.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ReplyMap.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Protocol.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Messages.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Init.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` + +Mapped test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 25 feature and test update must follow this protocol. + +### Per-Feature Verification Loop (REQUIRED for every feature ID) + +1. Read tracker metadata and Go source before coding: + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +# then open golang/nats-server/server/gateway.go at go_line_number..(go_line_number + go_line_count) +``` + +2. Write/adjust C# implementation for that specific feature. +3. Build immediately after each feature (or tightly coupled feature pair): + +```bash +/usr/local/share/dotnet/dotnet build dotnet/ +``` + +4. Run the smallest related test filter immediately after the build: + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~ProtocolParserTests|FullyQualifiedName~ClientTests|FullyQualifiedName~NatsServerTests" +``` + +5. Record evidence for the feature ID: Go lines read, files touched, build result, test command, pass/fail summary. +6. If any command is red, fix before starting the next feature ID. + +### Per-Test Verification Loop (REQUIRED for every test ID) + +1. Read Go test source from tracker metadata: + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` + +2. Port behavior (not syntax) into mapped `.Impltests.cs` method. +3. Run the single method filter and confirm discovery: + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` + +4. Confirm the run reports `Passed: 1` and `Failed: 0`. +5. Add the test ID to verification candidates only after a green individual run. + +### Stub Detection Check (REQUIRED after every feature group and test wave) + +Run before any status promotion: + +```bash +# Forbidden markers in mapped production + test files +grep -R -n -E "(NotImplementedException|TODO|PLACEHOLDER|Assert\\.True\\(true\\)|ShouldContain\\(\"Should\"\\))" \ + --include="Gateway*.cs" \ + --include="NatsServer.Gateways*.cs" \ + --include="ClientConnection.Gateways*.cs" \ + --include="NatsServer.Init.cs" \ + --include="NatsServer.Lifecycle.cs" \ + --include="GatewayHandlerTests.Impltests.cs" \ + --include="LeafNodeHandlerTests.Impltests.cs" \ + --include="MonitoringHandlerTests.Impltests.cs" \ + --include="JetStreamEngineTests.Impltests.cs" \ + --include="JetStreamSuperClusterTests.Impltests.cs" \ + --include="ConcurrencyTests1.Impltests.cs" \ + --include="ConcurrencyTests2.Impltests.cs" \ + --include="ConfigReloaderTests.Impltests.cs" \ + --include="NatsServerTests.Impltests.cs" \ + dotnet/src/ZB.MOM.NatsNet.Server \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog + +# Empty method bodies in mapped production files +grep -R -n -E "^[[:space:]]*(public|internal|private|protected)[^{;=]*\\)[[:space:]]*\\{[[:space:]]*\\}$" \ + --include="Gateway*.cs" \ + --include="NatsServer.Gateways*.cs" \ + --include="ClientConnection.Gateways*.cs" \ + dotnet/src/ZB.MOM.NatsNet.Server +``` + +Any hit must be fixed or deferred with a reason before status promotion. + +### Build Gate (REQUIRED after each feature group) + +```bash +/usr/local/share/dotnet/dotnet build dotnet/ +``` + +`Build succeeded` is mandatory before any `stub -> complete` feature update. + +### Test Gate (REQUIRED before any `complete -> verified` feature promotion) + +All Batch 25 mapped/related tests must pass: + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests" +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~LeafNodeHandlerTests.LeafNodeAndGatewaysSingleMsgPerQueueGroup_ShouldSucceed|FullyQualifiedName~LeafNodeHandlerTests.LeafNodeQueueGroupWeightCorrectOnConnectionCloseInSuperCluster_ShouldSucceed" +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~MonitoringHandlerTests.MonitorGatewayURLsUpdated_ShouldSucceed|FullyQualifiedName~MonitoringHandlerTests.MonitorGatewayzAccounts_ShouldSucceed" +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ConcurrencyTests1.NoRaceGatewayNoMissingReplies_ShouldSucceed|FullyQualifiedName~ConcurrencyTests2.NoRaceConnectionObjectReleased_ShouldSucceed" +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ConfigReloaderTests.ConfigReloadClusterAdvertise_ShouldSucceed|FullyQualifiedName~NatsServerTests.ReconnectErrorReports_ShouldSucceed" +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests.JetStreamStreamConfigClone_ShouldSucceed|FullyQualifiedName~JetStreamSuperClusterTests.JetStreamSuperClusterInterestOnlyMode_ShouldSucceed" +``` + +No Batch 25 feature is eligible for `verified` until this gate is green. + +### Status Update Protocol (REQUIRED) + +- Never include more than `15` IDs in any `feature batch-update` or `test batch-update`. +- Required progression: + - `deferred -> stub` at work start + - `stub -> complete` only after clean stub scan + build gate + - `complete -> verified` only after full Test Gate +- Evidence is required for every update chunk: + - build summary, + - test summary, + - stub-scan summary, + - exact IDs being promoted. + +Command templates: + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +After each task, before starting the next task: + +```bash +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "" +``` + +Do not continue if checkpoint gates fail. + +### If You Get Stuck (REQUIRED) + +1. Stop on the blocked feature/test ID. +2. Remove partial placeholder code (no TODO stubs left behind). +3. Mark blocked item as `deferred` with specific reason: + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db + +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +4. Move to the next unblocked ID. +5. Never ship or verify a stub to keep momentum. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +The following patterns are forbidden in Batch 25 production and test code: + +- `throw new NotImplementedException()` +- Empty method body `{ }` for mapped methods +- `// TODO` and `// PLACEHOLDER` in mapped code paths +- Placeholder return defaults for non-trivial methods (`return false;`, `return 0;`, `return string.Empty;`, `return null;`) without behavioral logic +- `Assert.True(true)` or similar tautological assertions +- `"...".ShouldContain("Should")` placeholder assertion pattern in `ImplBacklog` tests +- Tests that do not execute production code under test + +### Hard Limits + +- Maximum `20` features per feature task +- Maximum `15` IDs per status update command +- Maximum `1` feature group active at a time +- Zero unresolved stub-scan matches before status promotion +- Mandatory checkpoint gate (full build + full tests + commit) between tasks + +--- + +## Feature Groups (<=20 IDs each) + +### Group 1 (19 IDs): Gateway bootstrap, option validation, startup, solicitation + +IDs: + +`1263,1264,1265,1266,1268,1269,1270,1271,1272,1273,1275,1276,1277,1278,1279,1280,1281,1282,1283` + +Focus: + +- Solicit delay controls and gateway mode helpers +- Gateway option validation and hash helpers +- `NewGateway`, startup accept loop, solicitation/reconnect bootstrap +- Early inbound existence and gateway creation path + +### Group 2 (18 IDs): CONNECT/INFO handshake, gossip, and remote config propagation + +IDs: + +`1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298,1299,1300,1301` + +Focus: + +- `SendGatewayConnect`, `ProcessGatewayConnect`, `ProcessGatewayInfo` +- Gateway-to-route gossip/config propagation +- Outbound/inbound count and remote lookup helpers +- Initial gateway connection-attempt tracking + +### Group 3 (17 IDs): Gateway URL/state bookkeeping and connection registry ordering + +IDs: + +`1302,1303,1304,1305,1306,1307,1308,1309,1310,1311,1312,1313,1315,1316,1317,1318,1319` + +Focus: + +- GatewayCfg URL mutation/accessors and TLS hostname cache +- Cluster URL add/remove and async INFO broadcast +- Gateway URL/name lookup +- Inbound/outbound registration and ordered outbound list maintenance + +### Group 4 (16 IDs): Interest mode transitions and outbound send decisions + +IDs: + +`1320,1321,1323,1324,1325,1326,1327,1328,1329,1330,1331,1333,1334,1335,1336,1337` + +Focus: + +- Inbound connection enumeration and address helpers +- Account/RSub/RUnsub protocol handlers +- Account switch-to-interest-only logic +- Routed-reply detection/prefix logic and send decision path + +### Group 5 (16 IDs): No-interest handling, reply-map routing, inbound message pipeline + +IDs: + +`1338,1339,1340,1341,1342,1343,1344,1345,1346,1347,1348,1349,1350,1351,1352,1353` + +Focus: + +- Account/subject no-interest handling and A- emission +- Route hash table helpers and routed reply subject extraction +- `HandleGatewayReply`, `ProcessInboundGatewayMsg`, all-subs receive start/complete +- Reply-map tracking and expiration worker + +--- + +## Test Waves + +### Wave 1 (15 IDs): Basic connect/listen/auth/TLS/reconnect scenarios + +IDs: + +`600,601,605,608,609,610,611,612,613,615,616,617,618,620,621` + +### Wave 2 (16 IDs): Interest propagation, queue subs, and service import baseline + +IDs: + +`624,627,630,631,632,634,638,639,640,644,645,647,650,651,652,653` + +### Wave 3 (18 IDs): Interest-mode edge cases, TLS reload/discovery, slow consumer/no-race gateway paths + +IDs: + +`660,661,662,663,664,667,669,670,672,673,674,675,676,677,679,680,681,687` + +### Wave 4 (10 IDs): Cross-module integration tests (JetStream/Leaf/Monitor/NoRace/Reload/Server) + +IDs: + +`1426,1715,1962,1963,2127,2131,2376,2490,2747,2899` + +--- + +## Task 1: Preflight and Dependency Gate + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-25-gateways-design.md` +- Read: `golang/nats-server/server/gateway.go` + +**Step 1: Confirm batch metadata** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 25 --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 2: Verify dependency readiness** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Expected: Batch `25` appears only after dependencies `19` and `23` are complete. + +**Step 3: Start batch only if ready** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch start 25 --db porting.db +``` + +**Step 4: Baseline build/test gate** + +```bash +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 5: Checkpoint commit** + +```bash +git add porting.db +git commit -m "chore(batch25): start gateways batch" +``` + +--- + +## Task 2: Implement Feature Group 1 (19 IDs) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` +- Create: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayHandler.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConfigAndStartup.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Init.cs` + +**Step 1: Move Group 1 to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1263,1264,1265,1266,1268,1269,1270,1271,1272,1273,1275,1276,1277,1278,1279" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1280,1281,1282,1283" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-feature verification loop for all Group 1 IDs** + +**Step 3: Run stub detection check + build gate + targeted tests** + +```bash +# run stub detection commands from protocol section +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~NatsServerTests" +``` + +**Step 4: Move Group 1 to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1263,1264,1265,1266,1268,1269,1270,1271,1272,1273,1275,1276,1277,1278,1279" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1280,1281,1282,1283" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): implement gateway bootstrap and solicitation" +``` + +--- + +## Task 3: Implement Feature Group 2 (18 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConnectionsAndGossip.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Protocol.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Protocol/ProtocolParser.cs` + +**Step 1: Move Group 2 to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1299,1300,1301" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-feature verification loop for all Group 2 IDs** + +**Step 3: Run stub detection check + build gate + targeted tests** + +```bash +# run stub detection commands from protocol section +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~ClientTests" +``` + +**Step 4: Move Group 2 to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1299,1300,1301" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): implement gateway handshake and gossip" +``` + +--- + +## Task 4: Implement Feature Group 3 (17 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ConnectionsAndGossip.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayTypes.cs` + +**Step 1: Move Group 3 to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1302,1303,1304,1305,1306,1307,1308,1309,1310,1311,1312,1313,1315,1316,1317" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1318,1319" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-feature verification loop for all Group 3 IDs** + +**Step 3: Run stub detection check + build gate + targeted tests** + +```bash +# run stub detection commands from protocol section +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~MonitoringHandlerTests" +``` + +**Step 4: Move Group 3 to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1302,1303,1304,1305,1306,1307,1308,1309,1310,1311,1312,1313,1315,1316,1317" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1318,1319" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): implement gateway URL and registry bookkeeping" +``` + +--- + +## Task 5: Implement Feature Group 4 (16 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.Interest.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Protocol.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Messages.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayHandler.cs` + +**Step 1: Move Group 4 to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1320,1321,1323,1324,1325,1326,1327,1328,1329,1330,1331,1333,1334,1335,1336" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1337" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-feature verification loop for all Group 4 IDs** + +**Step 3: Run stub detection check + build gate + targeted tests** + +```bash +# run stub detection commands from protocol section +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~ConcurrencyTests1" +``` + +**Step 4: Move Group 4 to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1320,1321,1323,1324,1325,1326,1327,1328,1329,1330,1331,1333,1334,1335,1336" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1337" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): implement gateway interest and outbound send logic" +``` + +--- + +## Task 6: Implement Feature Group 5 (16 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Gateways.ReplyMap.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.Gateways.Messages.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Gateway/GatewayHandler.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` + +**Step 1: Move Group 5 to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1338,1339,1340,1341,1342,1343,1344,1345,1346,1347,1348,1349,1350,1351,1352" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1353" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-feature verification loop for all Group 5 IDs** + +**Step 3: Run stub detection check + build gate + targeted tests** + +```bash +# run stub detection commands from protocol section +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~ConcurrencyTests1|FullyQualifiedName~ConcurrencyTests2|FullyQualifiedName~NatsServerTests" +``` + +**Step 4: Move Group 5 to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1338,1339,1340,1341,1342,1343,1344,1345,1346,1347,1348,1349,1350,1351,1352" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1353" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): implement gateway reply map and inbound message pipeline" +``` + +--- + +## Task 7: Port and Verify Test Wave 1 (15 IDs) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` + +**Step 1: Move Wave 1 tests to `stub`** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "600,601,605,608,609,610,611,612,613,615,616,617,618,620,621" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-test verification loop for each ID** + +**Step 3: Run class-level gate and stub scan** + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests" +# run stub detection commands from protocol section +``` + +**Step 4: Move Wave 1 tests to `complete`** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "600,601,605,608,609,610,611,612,613,615,616,617,618,620,621" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "test(batch25): port gateway connect and tls baseline tests" +``` + +--- + +## Task 8: Port and Verify Test Wave 2 (16 IDs) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` + +**Step 1: Move Wave 2 tests to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "624,627,630,631,632,634,638,639,640,644,645,647,650,651,652" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "653" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-test verification loop for each ID** + +**Step 3: Run class-level gate and stub scan** + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests" +# run stub detection commands from protocol section +``` + +**Step 4: Move Wave 2 tests to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "624,627,630,631,632,634,638,639,640,644,645,647,650,651,652" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "653" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "test(batch25): port gateway interest and service-import tests" +``` + +--- + +## Task 9: Port and Verify Test Wave 3 (18 IDs) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` + +**Step 1: Move Wave 3 tests to `stub` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "660,661,662,663,664,667,669,670,672,673,674,675,676,677,679" --set-status stub --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "680,681,687" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-test verification loop for each ID** + +**Step 3: Run class-level gate and stub scan** + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests" +# run stub detection commands from protocol section +``` + +**Step 4: Move Wave 3 tests to `complete` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "660,661,662,663,664,667,669,670,672,673,674,675,676,677,679" --set-status complete --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "680,681,687" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "test(batch25): port gateway edge, reload, and slow-consumer tests" +``` + +--- + +## Task 10: Port and Verify Test Wave 4 (10 IDs, cross-module) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` + +**Step 1: Move Wave 4 tests to `stub`** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "1426,1715,1962,1963,2127,2131,2376,2490,2747,2899" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute per-test verification loop for each ID** + +**Step 3: Run targeted class gates and stub scan** + +```bash +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~MonitoringHandlerTests|FullyQualifiedName~JetStreamEngineTests|FullyQualifiedName~JetStreamSuperClusterTests|FullyQualifiedName~ConcurrencyTests1|FullyQualifiedName~ConcurrencyTests2|FullyQualifiedName~ConfigReloaderTests|FullyQualifiedName~NatsServerTests" +# run stub detection commands from protocol section +``` + +**Step 4: Move Wave 4 tests to `complete`** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "1426,1715,1962,1963,2127,2131,2376,2490,2747,2899" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git commit -m "test(batch25): port cross-module gateway integration tests" +``` + +--- + +## Task 11: Verification Gate and Promote to `verified` + +**Files:** + +- Modify: `porting.db` + +**Step 1: Run full Batch 25 test gate** + +```bash +# run all commands from the Test Gate subsection in MANDATORY VERIFICATION PROTOCOL +``` + +**Step 2: Promote all 59 tests to `verified` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "600,601,605,608,609,610,611,612,613,615,616,617,618,620,621" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "624,627,630,631,632,634,638,639,640,644,645,647,650,651,652" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "653,660,661,662,663,664,667,669,670,672,673,674,675,676,677" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "679,680,681,687,1426,1715,1962,1963,2127,2131,2376,2490,2747,2899" --set-status verified --db porting.db --execute +``` + +**Step 3: Promote all 86 features to `verified` in <=15 chunks** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1263,1264,1265,1266,1268,1269,1270,1271,1272,1273,1275,1276,1277,1278,1279" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1280,1281,1282,1283,1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1295,1296,1297,1298,1299,1300,1301,1302,1303,1304,1305,1306,1307,1308,1309" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1310,1311,1312,1313,1315,1316,1317,1318,1319,1320,1321,1323,1324,1325,1326" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1327,1328,1329,1330,1331,1333,1334,1335,1336,1337,1338,1339,1340,1341,1342" --set-status verified --db porting.db --execute +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1343,1344,1345,1346,1347,1348,1349,1350,1351,1352,1353" --set-status verified --db porting.db --execute +``` + +**Step 4: Run checkpoint protocol and commit** + +```bash +git commit -m "feat(batch25): verify gateway features and tests" +``` + +--- + +## Task 12: Batch Closure and Reporting + +**Files:** + +- Modify: `porting.db` +- Generate: `reports/current.md` + +**Step 1: Final build/test sweep** + +```bash +/usr/local/share/dotnet/dotnet build dotnet/ +/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 2: Final Batch 25 stub audit** + +```bash +# rerun full stub detection commands from MANDATORY VERIFICATION PROTOCOL +``` + +**Step 3: Verify batch closure** + +```bash +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 25 --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch complete 25 --db porting.db +/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 4: Generate report and final commit** + +```bash +./reports/generate-report.sh +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/ +git commit -m "feat(batch25): complete gateways implementation and verification" +``` diff --git a/docs/plans/2026-02-27-batch-26-websocket-design.md b/docs/plans/2026-02-27-batch-26-websocket-design.md new file mode 100644 index 0000000..09ccb9f --- /dev/null +++ b/docs/plans/2026-02-27-batch-26-websocket-design.md @@ -0,0 +1,128 @@ +# Batch 26 (WebSocket) Design + +## Context Snapshot + +- Batch: `26 - WebSocket` +- Scope: `37 features`, `86 tests` +- Dependencies: `16 (Client Core first half)`, `18 (Server Core)` +- Go source: `golang/nats-server/server/websocket.go` (+ `websocket_test.go` for tests) +- Current state: + - Batch status is `pending`. + - Batch is **not ready** (`batch ready` currently lists only batches 0, 1, 5, 8). + - .NET websocket code is largely stubbed in `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/WebSocketTypes.cs`. + - Multiple websocket-related stubs exist outside the websocket folder (`ClientConnection`, `NatsServer.Lifecycle`, `NatsServer.Listeners`). + +## Goals + +1. Port all 37 Batch 26 websocket features from `server/websocket.go` with behaviorally equivalent logic. +2. Port and verify all 86 Batch 26 mapped tests with no fake/stub assertions. +3. Remove websocket-specific stubs (`NotImplementedException`, placeholder comments, always-false flags) in runtime paths touched by this batch. +4. Keep implementation idiomatic C# while preserving protocol correctness (frame parsing, masking, compression, upgrade validation, close/status handling). + +## Non-Goals + +1. Rewriting unrelated subsystems (MQTT, full leaf routing internals, non-websocket JetStream internals). +2. Performance tuning beyond parity with Go logic and existing .NET architecture. +3. Advancing blocked dependencies (Batches 16/18) inside this batch plan. + +## Approaches Considered + +### Approach A: Monolithic Port in `WebSocketTypes.cs` + +- Description: Implement all websocket behavior inside the existing single file. +- Pros: Minimal file churn. +- Cons: Hard to review/test, encourages oversized methods, mixes protocol codec, HTTP upgrade, and server lifecycle logic in one place. + +### Approach B (Recommended): Split by Concern with Partial Integration + +- Description: Keep websocket-focused primitives in `WebSocket/` and wire server/client integration through partial class methods in `NatsServer` and `ClientConnection`. +- Pros: Clear boundaries, easier per-feature verification, easier targeted tests, aligns with existing partial-class architecture in `NatsServer`. +- Cons: More file edits and some type movement from current stubs. + +### Approach C: Rely on `System.Net.WebSockets` High-Level Abstractions + +- Description: Use framework websocket APIs for framing/handshake and adapt behavior. +- Pros: Less low-level frame code. +- Cons: Poor fit for NATS semantics (manual handshake headers, no-masking extension, explicit frame control, compression toggles, client/leaf modes). Higher behavior drift risk vs Go. + +## Recommended Design (Approach B) + +### 1) Component Boundaries + +1. Frame/read-path primitives: + - `WsReadInfo` state machine (`init`, `Read`, `ReadByte`, `NextCBuf`, `Decompress`, `Unmask`). + - Static helpers for `WsGet`, `WsIsControlFrame`, frame-header creation/fill, masking utilities, close payload creation. +2. Client websocket behavior: + - `ClientConnection` websocket methods (`WsRead`, control frame handling/enqueue, close mapping, outbound collapse). + - `IsWebSocket()` no longer hardcoded false once websocket state exists on connection. +3. HTTP upgrade + option validation: + - Handshake validation and response (`WsUpgrade`, header checks, accept key, compression negotiation, origin checks). + - Options parsing/validation (`ValidateWebsocketOptions`, origin/header settings, auth override). +4. Server lifecycle/listener integration: + - Start/close websocket listener and include it in readiness checks. + - Keep server INFO websocket connect URLs updated and propagated. + +### 2) Data Flow + +1. Upgrade flow: + - HTTP request validation -> origin check -> extension negotiation -> connection hijack -> 101 response -> websocket context creation. +2. Inbound frame flow: + - Header decode -> mask enforcement -> payload size handling -> optional control-frame handling -> optional decompression -> payload slices returned for parser. +3. Outbound frame flow: + - Pending buffers -> optional compression -> fragmentation rules (browser/safari constraints) -> optional masking -> queued frames + close frame semantics. + +### 3) Error Handling Model + +1. Protocol violations map to websocket close frames with RFC status codes, then close/read termination behavior. +2. Handshake failures return HTTP status + server-side error logging. +3. Decompression limit violations map to payload errors (max payload). +4. Any runtime branch that cannot be implemented due dependency/runtime gap is marked `deferred` with explicit reason, never stubbed. + +### 4) File-Level Plan + +Likely touched files: + +- Source: + - `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/WebSocketTypes.cs` (replace stubs/split responsibilities) + - `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/WebSocketConstants.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Listeners.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Init.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` +- Tests: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` + - New backlog classes likely required by mapping: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CoreBenchmarks.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/PublishBenchmarks.Impltests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeProxyTests.Impltests.cs` + +### 5) Test Strategy + +1. Use strict ID-driven mapping from PortTracker for all 86 tests. +2. For websocket functional tests, port assertions against frame semantics, handshake negotiation, header behavior, compression behavior, and outbound framing. +3. For benchmark-mapped test IDs: + - If deterministic behavioral assertions can be extracted, port as regular unit tests. + - If benchmark-only and non-deterministic/perf-only, mark `n_a` or `deferred` with explicit reason and evidence. +4. Add anti-stub checks to prevent placeholder tests passing as real. + +### 6) Risks and Mitigations + +1. Risk: Dependency batches 16/18 incomplete. + - Mitigation: strict preflight gate, do not start batch until dependencies are complete. +2. Risk: Silent stub regressions in broad ImplBacklog files. + - Mitigation: mandatory grep-based stub scans and per-ID evidence before status updates. +3. Risk: Frame/compression edge-case mismatch. + - Mitigation: prioritize low-level frame tests first, then server/upgrade paths. +4. Risk: Benchmarks misclassified as verified tests. + - Mitigation: explicit benchmark decision gate (real deterministic test vs `n_a`/`deferred` with reason). + +## Design Decision + +Proceed with **Approach B** (split-by-concern websocket port with partial integration into `NatsServer`/`ClientConnection`), executed via a strict verification-first implementation plan. + diff --git a/docs/plans/2026-02-27-batch-26-websocket-implementation-plan.md b/docs/plans/2026-02-27-batch-26-websocket-implementation-plan.md new file mode 100644 index 0000000..242e1e2 --- /dev/null +++ b/docs/plans/2026-02-27-batch-26-websocket-implementation-plan.md @@ -0,0 +1,504 @@ +# Batch 26 (WebSocket) Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 26 WebSocket behavior end-to-end (37 features, 86 tests) from `server/websocket.go` and mapped Go tests without introducing stubs or fake test passes. + +**Architecture:** Implement websocket protocol logic in focused websocket components, wire runtime behavior through `ClientConnection` and `NatsServer` partials, and execute ID-driven feature/test verification loops with hard build/test/status gates. Do low-level frame primitives first, then HTTP upgrade/options/listener integration, then mapped tests (including explicit benchmark classification handling). + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-26-websocket-design.md` + +--- + +## Batch Scope + +- Batch: `26 - WebSocket` +- Dependencies: Batches `16`, `18` (must be complete before execution) +- Features: `37` +- Tests: `86` + +### Feature Groups (max ~20 each) + +1. **Group A (20 features):** `3506,3507,3509,3510,3511,3512,3513,3514,3515,3516,3517,3518,3519,3520,3521,3522,3523,3524,3525,3541` +2. **Group B (17 features):** `3526,3527,3528,3529,3530,3531,3532,3533,3534,3535,3536,3537,3538,3539,3540,3542,3543` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every feature and test in this batch must follow this protocol. Skipping any step is a plan violation. + +### What Counts as a Real Feature/Test + +1. **Real feature**: + - Behavior implemented in runtime code (not placeholder return/no-op/throw stub). + - Mapped to the Go method intent for that feature ID. + - Covered by at least one related passing test. +2. **Real test**: + - Uses Arrange/Act/Assert. + - Calls relevant production code. + - Has meaningful Shouldly assertions. + - Is not a placeholder pattern. + +### Per-Feature Verification Loop (REQUIRED for each feature ID) + +1. Read mapped Go method location and surrounding logic: + - `dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db` + - `sed -n ',p' golang/nats-server/server/websocket.go` +2. Write/adjust C# implementation for only that behavior slice. +3. Build immediately: + - `dotnet build dotnet/` +4. Run related targeted tests (single method/class filter first). +5. Record evidence (feature ID, Go line(s), test(s) run, pass summary) before status update. + +### Stub Detection Check (REQUIRED after each feature/test mini-batch) + +Run both scans and resolve all hits before continuing: + +```bash +# Production code stub scan +rg -n --pcre2 "(NotImplementedException|TODO: session 23|TODO: websocket|// TODO|// PLACEHOLDER|=>\\s*throw\\s+new\\s+NotImplementedException)" \ + dotnet/src/ZB.MOM.NatsNet.Server/WebSocket \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs + +# Empty/trivial method body scan in touched websocket-related files +rg -n --pcre2 "^\\s*(public|private|internal|protected).*\\)\\s*\\{\\s*\\}\\s*$" \ + dotnet/src/ZB.MOM.NatsNet.Server/WebSocket \ + dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs \ + dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs + +# Test stub scan +rg -n --pcre2 "(Assert\\.True\\(true\\)|Assert\\.Pass\\(|NotImplementedException|// TODO|// PLACEHOLDER|ShouldBe\\(true\\);\\s*$)" \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog +``` + +### Build Gate (REQUIRED after each feature group) + +After Group A and Group B, full build must pass: + +```bash +dotnet build dotnet/ --verbosity minimal +``` + +If build fails, no status updates are allowed. + +### Test Gate (REQUIRED before marking features verified) + +1. Run all related tests for the feature group. +2. Confirm `Failed: 0`. +3. Only after test gate passes can feature IDs move to `verified`. + +Example: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~WebSocketHandlerTests" --verbosity normal +``` + +### Status Update Protocol (REQUIRED) + +1. Use max **15 IDs per `batch-update`** command. +2. Never update status without captured evidence for those exact IDs. +3. Keep feature and test updates separate. +4. If audit rejects update and behavior is truly implemented, use `--override ""` with explicit evidence note. + +Examples: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "3506-3517,3518" --set-status complete --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "3075,3076,3077,3078,3079,3080,3082,3083,3084,3085,3086,3087,3088,3089,3093" \ + --set-status verified --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Between every task below: + +1. Run full build: + - `dotnet build dotnet/ --verbosity minimal` +2. Run full unit tests: + - `dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal` +3. If regressions exist, fix before next task. +4. Commit checkpoint: + - `git add ` + - `git commit -m ""` + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +Any appearance below in touched Batch 26 feature/test methods is disallowed: + +1. `throw new NotImplementedException(...)` +2. Empty method body `{ }` for mapped feature methods +3. Trivial constant-return placeholder (`return true;`, `return null;`, `return default;`) where real behavior is required +4. `Assert.True(true)` or equivalent always-pass assertions +5. `// TODO`, `// PLACEHOLDER`, `TODO: session 23` in mapped code paths +6. Test methods that only assert non-null on unrelated constants without exercising websocket behavior + +### Hard Limits + +1. Max `~20` feature IDs per implementation group (already enforced by Group A/B). +2. Max `15` IDs per status batch update. +3. Max `1` feature group status finalization cycle at a time (finish Group A fully before Group B verification updates). +4. No marking `verified` for features until related tests pass. +5. No marking tests `verified` without direct pass evidence. + +### If You Get Stuck (REQUIRED) + +1. Do **not** leave stubs. +2. Mark blocked feature/test IDs as `deferred` with explicit reason. +3. Add a code comment near deferred code/test: + - `// DEFERRED(batch26): ` +4. Continue with next unblocked ID. +5. Revisit deferred IDs only after blocker is resolved. + +--- + +### Task 1: Dependency and Batch Readiness Gate + +**Files:** +- Modify: none +- Validate context: `porting.db`, batch metadata + +**Step 1: Verify dependencies are complete** + +Run: +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 16 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 18 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` +Expected: batch 26 is ready only after 16/18 are complete. + +**Step 2: Start batch 26** + +Run: +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 26 --db porting.db +``` +Expected: start succeeds; no dependency error. + +**Step 3: Materialize execution manifest** + +Run: +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 26 --db porting.db +``` +Expected: explicit list of 37 features + 86 tests used as task checklist. + +**Step 4: Commit checkpoint** + +```bash +git add porting.db +git commit -m "chore(batch26): start websocket batch after dependency gate" +``` + +--- + +### Task 2: Implement Feature Group A (20 features, frame/core path) + +**Feature IDs:** +- `3506,3507,3509,3510,3511,3512,3513,3514,3515,3516,3517,3518,3519,3520,3521,3522,3523,3524,3525,3541` + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/WebSocketTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/WebSocketConstants.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` +- Add or modify focused websocket helper files under: `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` + +**Step 1: Write/port failing tests for Group A behaviors** + +Start with tests covering: control-frame detection, masking/unmasking, close message creation, frame header generation, read/decompression paths, close enqueue, outbound collapse. + +Run: +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~WebSocketHandlerTests.WSIsControlFrame_ShouldSucceed|FullyQualifiedName~WebSocketHandlerTests.WSUnmask_ShouldSucceed" +``` +Expected: initially fail until implementation lands. + +**Step 2: Implement minimal code per feature using per-feature loop** + +For each feature ID in Group A, execute the mandatory loop (Go source -> C# -> build -> related tests). + +**Step 3: Run Group A build and focused tests** + +Run: +```bash +dotnet build dotnet/ --verbosity minimal +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~WebSocketHandlerTests" --verbosity normal +``` +Expected: no failures in mapped Group A coverage. + +**Step 4: Run mandatory stub detection checks** + +Run all scans from the MANDATORY VERIFICATION PROTOCOL section. +Expected: zero matches in touched methods/tests. + +**Step 5: Update feature statuses (chunked <=15 IDs)** + +Example chunks: +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "3506,3507,3509,3510,3511,3512,3513,3514,3515,3516,3517,3518,3519,3520,3521" \ + --set-status complete --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "3522,3523,3524,3525,3541" --set-status complete --db porting.db --execute +``` + +**Step 6: Commit checkpoint** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server/WebSocket dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs porting.db +git commit -m "feat(batch26): implement websocket frame/core feature group A" +``` + +--- + +### Task 3: Implement Feature Group B (17 features, upgrade/options/server integration) + +**Feature IDs:** +- `3526,3527,3528,3529,3530,3531,3532,3533,3534,3535,3536,3537,3538,3539,3540,3542,3543` + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Init.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Listeners.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- Modify: websocket source files in `dotnet/src/ZB.MOM.NatsNet.Server/WebSocket/` +- Test: websocket/backlog test files listed in later tasks + +**Step 1: Write failing tests for upgrade/origin/options behavior** + +Focus on IDs mapped to: +- `TestWSUpgradeConnDeadline`, `TestWSCompressNegotiation`, `TestWSSetHeader`, `TestWSSetOriginOptions`. + +**Step 2: Implement feature-by-feature with mandatory loop** + +For each Group B feature: +- Read Go source section from `websocket.go`. +- Implement in .NET with lock/lifecycle parity. +- Build and run targeted tests before next feature. + +**Step 3: Remove related integration stubs** + +Eliminate websocket placeholder behavior in: +- readiness checks +- listener lists +- websocket server shutdown path +- websocket connect URL propagation +- websocket client creation path + +**Step 4: Build + focused tests + stub scans** + +Run: +```bash +dotnet build dotnet/ --verbosity minimal +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~WebSocketHandlerTests" --verbosity normal +``` + +**Step 5: Update feature statuses (chunked <=15 IDs)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "3526,3527,3528,3529,3530,3531,3532,3533,3534,3535,3536,3537,3538,3539,3540" \ + --set-status complete --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "3542,3543" --set-status complete --db porting.db --execute +``` + +**Step 6: Commit checkpoint** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs dotnet/src/ZB.MOM.NatsNet.Server/WebSocket porting.db +git commit -m "feat(batch26): implement websocket upgrade/options/server feature group B" +``` + +--- + +### Task 4: Port WebSocket Functional Tests (22 tests in `websocket_test.go`) + +**Test IDs:** +- `3075,3076,3077,3078,3079,3080,3082,3083,3084,3085,3086,3087,3088,3089,3093,3097,3098,3099,3102,3113,3117,3132` + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` + +**Step 1: Port test-by-test from Go source** + +For each test ID: +1. `test show ` +2. Read Go body in `websocket_test.go` +3. Port behavior to Shouldly assertions +4. Run single test + +Single-test command template: +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.WebSocketHandlerTests." \ + --verbosity normal +``` + +**Step 2: Run class-level gate** + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~WebSocketHandlerTests" --verbosity normal +``` + +**Step 3: Stub scan + assertion quality check** + +```bash +rg -n "(Assert\\.True\\(true\\)|NotImplementedException|// TODO|// PLACEHOLDER)" \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs +``` + +**Step 4: Update test statuses in chunks <=15** + +Use chunks with pass evidence recorded for each chunk. + +**Step 5: Commit checkpoint** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs porting.db +git commit -m "test(batch26): port websocket functional tests" +``` + +--- + +### Task 5: Port Cross-Module Functional Tests Dependent on WebSocket (11 tests) + +**Test IDs:** +- `409,465,466,1353,1902,1903,1975,1986,2371,2384,2488` + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Create if missing: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeProxyTests.Impltests.cs` + +**Step 1: Ensure mapped class files exist** + +Create missing files/classes before test-porting if absent (especially `LeafNodeProxyTests`). + +**Step 2: Port tests ID-by-ID with single-test loop** + +Use the same per-test loop and evidence policy as Task 4. + +**Step 3: Run focused class gates** + +Run each touched class with `dotnet test --filter "FullyQualifiedName~"`. + +**Step 4: Update statuses in <=15 chunks** + +Mark `verified` only with evidence. +If blocked by missing infra, mark `deferred` and add `// DEFERRED(batch26): `. + +**Step 5: Commit checkpoint** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "test(batch26): port cross-module websocket-dependent tests" +``` + +--- + +### Task 6: Resolve Benchmark-Mapped Test IDs (53 tests) + +**Benchmark test IDs:** +- `154,274,275,276,277,3134,3135,3136,3137,3138,3139,3140,3141,3142,3143,3144,3145,3146,3147,3148,3149,3150,3151,3152,3153,3154,3155,3156,3157,3158,3159,3160,3161,3162,3163,3164,3165,3166,3167,3168,3169,3170,3171,3172,3173,3174,3175,3176,3177,3179,3180,3181,3182` + +**Files:** +- Modify/create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` +- Create if missing: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CoreBenchmarks.Impltests.cs` +- Create if missing: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/PublishBenchmarks.Impltests.cs` + +**Step 1: Benchmark decision gate per ID** + +For each benchmark ID choose exactly one: +1. Convert to deterministic functional test with real assertions and mark `verified`. +2. Mark `n_a` (benchmark-only/perf harness requirement) with explicit reason. +3. Mark `deferred` if runtime infra blocker exists. + +**Step 2: Run tests for converted benchmark IDs** + +Use class-level and method-level gates as in previous tasks. + +**Step 3: Apply status updates in <=15 chunks** + +Separate commands by status (`verified`, `n_a`, `deferred`). + +**Step 4: Commit checkpoint** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "test(batch26): resolve benchmark-mapped websocket test IDs" +``` + +--- + +### Task 7: Final Verification, Status Closure, and Batch Completion + +**Files:** +- Modify: `porting.db` +- Optional report artifacts: `reports/current.md` (if generator updates it) + +**Step 1: Full verification run** + +```bash +dotnet build dotnet/ --verbosity minimal +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/ --verbosity normal +``` + +**Step 2: Promote complete feature IDs to verified** + +Use chunked updates with evidence references. + +**Step 3: Audit and reconcile** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` + +**Step 4: Complete batch** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 26 --db porting.db +``` +Expected: completion succeeds only when all batch items are in valid terminal statuses. + +**Step 5: Generate report and commit** + +```bash +./reports/generate-report.sh +git add porting.db reports/ +git commit -m "feat(batch26): complete websocket batch with verified implementation and tests" +``` + +--- + +## Execution Notes + +1. If `dotnet` is not on `PATH`, use absolute binary (for this machine): `/usr/local/share/dotnet/dotnet`. +2. Always keep evidence for each status update chunk (test output snippet + feature/test IDs). +3. Never move a feature to `verified` before its related tests are passing. +4. Never convert a blocked item into a stub just to make progress. + diff --git a/docs/plans/2026-02-27-batch-27-jetstream-core-design.md b/docs/plans/2026-02-27-batch-27-jetstream-core-design.md new file mode 100644 index 0000000..11cfe37 --- /dev/null +++ b/docs/plans/2026-02-27-batch-27-jetstream-core-design.md @@ -0,0 +1,126 @@ +# Batch 27 JetStream Core Design + +## Context + +- Batch: `27` (`JetStream Core`) +- Scope: `82` features, `77` tests +- Dependencies: batches `5`, `8`, `19` +- Primary Go source: `golang/nats-server/server/jetstream.go` +- PortTracker status snapshot (2026-02-27): batch `27` is `pending` + +Batch 27 is the first full behavior layer for JetStream server lifecycle and per-account resource management. It spans server bootstrap/shutdown wiring, account-level JetStream enable/disable and usage tracking, limits enforcement, and config validation helpers consumed by later JetStream API and clustering batches. + +## Constraints and Success Criteria + +### Constraints + +- No placeholder implementations or fake-pass tests. +- Dependency order is mandatory: batch 27 cannot start until batches `5`, `8`, and `19` are complete. +- Keep feature execution chunked to `~20` features per task group. +- Keep status updates evidence-based and chunked to max `15` IDs per `batch-update` call. +- Follow .NET standards: .NET 10, nullable, xUnit 3 + Shouldly + NSubstitute. + +### Success Criteria + +- All 82 feature IDs are moved to `verified` or kept `deferred` with explicit blocker notes. +- All 77 test IDs are moved to `verified` or kept `deferred` with explicit blocker notes. +- No forbidden stub patterns in touched production or test files. +- Build and test gates pass at every checkpoint. + +## Approaches Considered + +### Approach A (Recommended): Incremental in-place behavior port with grouped feature waves + +- Implement missing methods in existing core files (`NatsServer` partials, `Account`, `JetStream`/`JsAccount` core files). +- Execute five feature groups (max ~20 each), then port mapped tests in staged class waves. +- Promote statuses only after per-feature/per-test verification evidence. + +Pros: +- Lowest structural risk. +- Best audit trace from Go line ranges to C# methods. +- Works well with strict status-evidence protocol. + +Cons: +- Large diffs in a few high-churn files. + +### Approach B: Structural refactor first (convert types to partial files), then port behavior + +- First split large classes into new partial files, then implement features. + +Pros: +- Cleaner long-term organization. + +Cons: +- Extra upfront churn before behavior parity. +- Harder to isolate regressions during a large dependency-heavy batch. + +### Approach C: Signature-first skeleton pass, fill behavior later + +- Add methods quickly to satisfy compile, then return later to complete behavior. + +Pros: +- Fast apparent progress. + +Cons: +- Violates anti-stub requirements and creates false status inflation. + +Decision: **Approach A**. + +## Proposed Design + +### 1. Source Partitioning by Runtime Responsibility + +Implement Batch 27 behavior in four logical zones: + +- `NatsServer` JetStream lifecycle/config orchestration methods. +- `Account` JetStream account enrollment, limits assignment, account usage projections. +- `JetStream` engine-level limits/resource checks and global usage stats. +- `JsAccount` per-account synchronized usage, cluster usage sync, tier selection. + +This mirrors the Go file layout while staying idiomatic in C#. + +### 2. Feature Grouping Strategy (Max ~20/group) + +Use five feature groups with contiguous IDs and behavior cohesion: + +1. Group 1 (`20`): `1368-1387` — server enable/disable bootstrap, encryption setup, exports, account wiring +2. Group 2 (`19`): `1388,1389,1390,1391,1392,1394,1396,1397,1398,1399,1400,1401,1402,1403,1404,1405,1406,1407,1408` — runtime state flags/getters and account enable/update core +3. Group 3 (`18`): `1409-1426` — account usage snapshots and foundational limits helpers +4. Group 4 (`17`): `1427-1443` — jsAccount limits enforcement and stream resource reservation/release +5. Group 5 (`8`): `1444-1451` — dynamic config/validation/error helper tail + +### 3. Test Porting Strategy + +Port tests by class waves after core feature groups are green: + +- Wave A (smaller cross-module classes, 19 tests): `AccountTests`, `EventsHandlerTests`, `MonitoringHandlerTests`, `JetStreamVersioningTests`, `MqttHandlerTests`, `MessageTracerTests`, `ConcurrencyTests2`, `NatsServerTests` +- Wave B (mid-size JetStream/storage classes, 17 tests): `JetStreamFileStoreTests`, `NatsConsumerTests`, `ConcurrencyTests1` +- Wave C (heavy class, 41 tests): `JetStreamEngineTests` in sub-batches; keep perf/benchmark-style tests deferred if they require unsupported harness/runtime shape. + +### 4. Verification Model + +The implementation plan defines a mandatory protocol with: + +- Per-feature verification loop (read Go lines -> implement -> build/test evidence) +- Per-test verification loop (single-test pass evidence) +- Stub scan required after each group/class +- Build gate and test gate before status promotion +- Status updates in max-15 ID chunks +- Required checkpoints between tasks + +### 5. Defer-Not-Stub Policy + +If blocked by infrastructure/runtime gaps (cluster setup, benchmark harness, missing dependent behavior), keep item `deferred` with explicit reason in PortTracker; do not write placeholder code/tests. + +## Risks and Mitigations + +- Risk: dependency batches still pending. + - Mitigation: hard preflight gate before `batch start 27`. +- Risk: large JetStreamEngine test class can hide shallow ports. + - Mitigation: single-test evidence + assertion density checks + periodic class gates. +- Risk: status inflation without proof. + - Mitigation: max-15 update chunks and evidence checklist per update. + +## Design Approval Note + +This design is based on the explicit constraints in your request and current repository state. The implementation plan materializes this design into executable, checkpointed tasks without running any batch work. diff --git a/docs/plans/2026-02-27-batch-27-jetstream-core-implementation-plan.md b/docs/plans/2026-02-27-batch-27-jetstream-core-implementation-plan.md new file mode 100644 index 0000000..a01df40 --- /dev/null +++ b/docs/plans/2026-02-27-batch-27-jetstream-core-implementation-plan.md @@ -0,0 +1,591 @@ +# Batch 27 JetStream Core Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 27 JetStream core behavior (82 features, 77 tests) from `server/jetstream.go` into .NET without stubs, with evidence-based status updates. + +**Architecture:** Implement JetStream core behavior in grouped waves across `NatsServer`, `Account`, `JetStream`, and `JsAccount` code paths, then port mapped tests by class waves with strict per-test verification evidence. Promote status only through `stub -> complete -> verified` after mandatory build/test/stub gates. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-27-jetstream-core-design.md` + +--- + +## Batch 27 Scope + +- Batch ID: `27` +- Dependencies: `5`, `8`, `19` +- Go source: `golang/nats-server/server/jetstream.go` +- Feature count: `82` +- Test count: `77` + +Primary source targets: + +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Lifecycle.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServerTypes.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` (new or existing core behavior files) + +Primary test targets: + +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamVersioningTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** every feature and test in Batch 27 must follow this protocol. No exceptions. + +### Per-Feature Verification Loop (REQUIRED for each feature ID) + +1. Read mapping and Go span: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db + sed -n ',p' golang/nats-server/server/jetstream.go + ``` +2. Implement exactly one feature behavior in mapped C# target. +3. Build immediately: + ```bash + dotnet build dotnet/ + ``` +4. Run smallest related test filter (feature-local) and confirm discovery: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~" + ``` +5. If green, add feature ID to the current `complete-candidate` list. +6. If red, fix before touching the next feature ID. + +### Per-Test Verification Loop (REQUIRED for each test ID) + +1. Read mapping and Go span: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db + ``` +2. Port one test method (Arrange/Act/Assert, calls production code). +3. Run single-test filter and capture output: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.." \ + --verbosity normal + ``` +4. Verify output includes `Passed: 1, Failed: 0` (not `Passed: 0`). +5. Only then add test ID to `verified-candidate` list. + +### Stub Detection Check (REQUIRED after each feature group and test class) + +```bash +rg -n "(NotImplementedException|Assert\.True\(true\)|Assert\.Pass|// TODO|// PLACEHOLDER|throw new Exception\(\"TODO\"\))" \ + dotnet/src/ZB.MOM.NatsNet.Server \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog + +rg -n "^\s*(public|internal|private)\s+.*\)\s*\{\s*\}$" \ + dotnet/src/ZB.MOM.NatsNet.Server +``` + +Any new match in touched files blocks status promotion. + +### Build Gate (REQUIRED) + +- Required after every feature group and before every status promotion: + ```bash + dotnet build dotnet/ + ``` +- `Build succeeded` is mandatory. + +### Test Gate (REQUIRED) + +Before setting any feature to `verified`, all relevant Batch 27 backlog classes touched in the current wave must pass. + +Minimum gate commands: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.AccountTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.EventsHandlerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamFileStoreTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamVersioningTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.MonitoringHandlerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.MessageTracerTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.ConcurrencyTests1" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.ConcurrencyTests2" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsServerTests" +``` + +### Status Update Protocol (REQUIRED, hard limit enforced) + +- **Max 15 IDs per `feature batch-update` or `test batch-update` command.** +- Status progression: + - Start work: `deferred/not_started -> stub` + - After feature/test evidence: `stub -> complete` + - After full gate evidence: `complete -> verified` +- Evidence required per update chunk: + - latest build success output + - latest relevant test success output + - latest stub-scan output (no unresolved new matches) + +Command template: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At end of every task (before next task): + +1. Run build gate: + ```bash + dotnet build dotnet/ + ``` +2. Run test checkpoint: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal + ``` +3. Record checkpoint summary (pass/fail/skip, stub scan result, updated IDs). +4. Commit checkpoint before next task. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +The following indicate fake progress and are forbidden in Batch 27 feature/test code: + +- `throw new NotImplementedException()` +- Empty feature method body (`{ }`) for mapped methods +- `Assert.True(true)` / equivalent trivial pass assertion +- `Assert.Pass()` +- `// TODO` or `// PLACEHOLDER` in newly ported feature/test method bodies +- Returning trivial constants (`null`, `0`, `false`, `string.Empty`) where Go behavior is non-trivial +- Test methods that do not invoke production code + +### Hard Limits + +- Max `~20` features per feature task group. +- Max `15` IDs per status update command. +- One test class at a time for test verification/status promotion. +- No `verified` updates when any touched file still contains unresolved stub-pattern matches. +- No cross-group status promotion; only update IDs in the active task group. + +### If You Get Stuck (MANDATORY) + +1. Stop the current ID immediately. +2. Do **not** leave placeholder code/tests behind. +3. Mark blocked item as `deferred` with explicit reason: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db + ``` +4. Move to next unblocked ID. +5. Deferred with reason is correct; stubbing is not. + +--- + +## Feature Group Map (Max ~20 Features Each) + +### Group 1 (20 IDs): Server Enable/Disable Bootstrap and Account Wiring + +`1368,1369,1370,1371,1372,1373,1374,1375,1376,1377,1378,1379,1380,1381,1382,1383,1384,1385,1386,1387` + +### Group 2 (19 IDs): Engine State/Getters and Account Enable Core + +`1388,1389,1390,1391,1392,1394,1396,1397,1398,1399,1400,1401,1402,1403,1404,1405,1406,1407,1408` + +### Group 3 (18 IDs): Usage Foundations and Limits Helpers + +`1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423,1424,1425,1426` + +### Group 4 (17 IDs): jsAccount Limits Enforcement and Resource Accounting + +`1427,1428,1429,1430,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441,1442,1443` + +### Group 5 (8 IDs): Dynamic Config/Validation/Error Tail + +`1444,1445,1446,1447,1448,1449,1450,1451` + +--- + +## Test Wave Map + +### Wave A (19 IDs): Cross-Module Small/Medium Classes + +`100,101,102,314,345,1804,1806,2160,2161,2243,2244,2245,2248,2259,2343,2345,2346,2478,2890` + +### Wave B (17 IDs): Storage/Consumer/Concurrency Support + +`356,379,389,390,391,392,463,1295,1302,1308,1314,1336,2397,2403,2409,2424,2430` + +### Wave C (41 IDs): JetStreamEngine Heavy Class + +`1466,1467,1529,1548,1549,1550,1581,1583,1584,1589,1595,1600,1604,1608,1609,1617,1620,1621,1629,1631,1632,1648,1649,1650,1651,1654,1666,1672,1673,1675,1683,1684,1689,1721,1723,1724,1732,1739,1740,1744,1780` + +--- + +## Task 1: Preflight, Dependency Gate, and Batch Start + +**Files:** +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-27-jetstream-core-design.md` +- Read: `golang/nats-server/server/jetstream.go` + +**Step 1: Verify dependency batches are complete** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 5 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 8 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 19 --db porting.db +``` + +Expected: all required predecessor work for Batch 27 is complete/verified enough to start. + +**Step 2: Start Batch 27** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 27 --db porting.db +``` + +**Step 3: Capture baseline health** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 4: Checkpoint commit** + +```bash +git add porting.db +git commit -m "chore(batch27): start jetstream core batch" +``` + +--- + +## Task 2: Implement Feature Group 1 (20 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (JetStream lifecycle/config methods) +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify/create as needed: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` + +**Step 1: Mark Group 1 IDs as `stub` (max 15 per call)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1368,1369,1370,1371,1372,1373,1374,1375,1376,1377,1378,1379,1380,1381,1382" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1383,1384,1385,1386,1387" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 20 IDs** + +**Step 3: Run mandatory stub scan + build gate + targeted test gate** + +**Step 4: Mark Group 1 as `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1368,1369,1370,1371,1372,1373,1374,1375,1376,1377,1378,1379,1380,1381,1382" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1383,1384,1385,1386,1387" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch27): implement jetstream bootstrap and account wiring" +``` + +--- + +## Task 3: Implement Feature Group 2 (19 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` + +**Step 1: Mark Group 2 IDs as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1388,1389,1390,1391,1392,1394,1396,1397,1398,1399,1400,1401,1402,1403,1404" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1405,1406,1407,1408" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 19 IDs** + +**Step 3: Run mandatory stub scan + build gate + targeted test gate** + +**Step 4: Mark Group 2 as `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1388,1389,1390,1391,1392,1394,1396,1397,1398,1399,1400,1401,1402,1403,1404" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1405,1406,1407,1408" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch27): implement jetstream engine state and account enable core" +``` + +--- + +## Task 4: Implement Feature Group 3 (18 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` + +**Step 1: Mark Group 3 IDs as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1424,1425,1426" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 18 IDs** + +**Step 3: Run mandatory stub scan + build gate + targeted test gate** + +**Step 4: Mark Group 3 as `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1424,1425,1426" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch27): implement jetstream usage foundations and limit helpers" +``` + +--- + +## Task 5: Implement Feature Group 4 (17 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` + +**Step 1: Mark Group 4 IDs as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1427,1428,1429,1430,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1442,1443" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 17 IDs** + +**Step 3: Run mandatory stub scan + build gate + targeted test gate** + +**Step 4: Mark Group 4 as `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1427,1428,1429,1430,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1442,1443" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch27): implement jsaccount limit checks and resource accounting" +``` + +--- + +## Task 6: Implement Feature Group 5 (8 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/*.cs` + +**Step 1: Mark Group 5 IDs as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1444,1445,1446,1447,1448,1449,1450,1451" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all 8 IDs** + +**Step 3: Run mandatory stub scan + build gate + targeted test gate** + +**Step 4: Mark Group 5 as `complete`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1444,1445,1446,1447,1448,1449,1450,1451" --set-status complete --db porting.db --execute +``` + +**Step 5: Run checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch27): implement jetstream config validation and error tail" +``` + +--- + +## Task 7: Port Test Wave A (19 IDs) + +**Files:** +- Modify backlog class files for: `AccountTests`, `EventsHandlerTests`, `MonitoringHandlerTests`, `JetStreamVersioningTests`, `MqttHandlerTests`, `MessageTracerTests`, `ConcurrencyTests2`, `NatsServerTests` + +**Step 1: Mark Wave A tests as `stub` (max 15 IDs)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "100,101,102,314,345,1804,1806,2160,2161,2243,2244,2245,2248,2259,2343" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2345,2346,2478,2890" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each ID** + +**Step 3: Class-level gates + stub scans for each touched class** + +**Step 4: Mark Wave A tests `verified` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "100,101,102,314,345,1804,1806,2160,2161,2243,2244,2245,2248,2259,2343" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2345,2346,2478,2890" --set-status verified --db porting.db --execute +``` + +**Step 5: Checkpoint protocol and commit** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "test(batch27): port wave-a cross-module jetstream tests" +``` + +--- + +## Task 8: Port Test Wave B (17 IDs) + +**Files:** +- Modify backlog class files: `JetStreamFileStoreTests`, `NatsConsumerTests`, `ConcurrencyTests1` + +**Step 1: Mark Wave B tests as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "356,379,389,390,391,392,463,1295,1302,1308,1314,1336,2397,2403,2409" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2424,2430" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each ID** + +**Step 3: Class-level gates + stub scans for each touched class** + +**Step 4: Mark Wave B tests `verified` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "356,379,389,390,391,392,463,1295,1302,1308,1314,1336,2397,2403,2409" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "2424,2430" --set-status verified --db porting.db --execute +``` + +**Step 5: Checkpoint protocol and commit** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "test(batch27): port wave-b filestore-consumer-concurrency tests" +``` + +--- + +## Task 9: Port Test Wave C (JetStreamEngineTests, 41 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Step 1: Mark Wave C tests as `stub` (3 chunks)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1466,1467,1529,1548,1549,1550,1581,1583,1584,1589,1595,1600,1604,1608,1609" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1617,1620,1621,1629,1631,1632,1648,1649,1650,1651,1654,1666,1672,1673,1675" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1683,1684,1689,1721,1723,1724,1732,1739,1740,1744,1780" --set-status stub --db porting.db --execute +``` + +**Step 2: Port in sub-batches of 8-10 tests** + +For each sub-batch: + +1. Run per-test verification loop for each method. +2. Run class gate for `JetStreamEngineTests`. +3. Run stub scan and assertion-density check. +4. If unsupported benchmark/perf/runtime behavior is encountered, defer with reason immediately. + +**Step 3: Mark verified/deferred with evidence (max 15 IDs per call)** + +**Step 4: Checkpoint protocol and commit** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db +git commit -m "test(batch27): port jetstream engine heavy test wave" +``` + +--- + +## Task 10: Final Verification, Status Closure, and Batch Completion + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Step 1: Final feature verification promotion (chunked max 15 IDs)** + +Promote each feature group from `complete -> verified` only after all relevant tests are green. + +**Step 2: Final test status reconciliation** + +- Ensure every Batch 27 test ID is `verified` or `deferred` with explicit reason. + +**Step 3: Full repo verification gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/ +``` + +**Step 4: Complete batch + summary checks** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 27 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +./reports/generate-report.sh +``` + +**Step 5: Final commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/ +git commit -m "feat(batch27): complete jetstream core features and mapped tests" +``` + +--- + +## Execution Notes + +- Do not execute this plan while dependencies (`5`,`8`,`19`) are still blocked. +- Do not bypass verification protocol for speed. +- Deferred-with-reason is acceptable; stubbed code/tests are not. diff --git a/docs/plans/2026-02-27-batch-28-jetstream-api-design.md b/docs/plans/2026-02-27-batch-28-jetstream-api-design.md new file mode 100644 index 0000000..474fc06 --- /dev/null +++ b/docs/plans/2026-02-27-batch-28-jetstream-api-design.md @@ -0,0 +1,141 @@ +# Batch 28 JetStream API Design + +**Date:** 2026-02-27 +**Batch:** 28 (`JetStream API`) +**Scope:** 55 features + 2 unit tests from `server/jetstream_api.go` / `server/jetstream_test.go` +**Dependencies:** batches `5` (`JetStream Errors`), `27` (`JetStream Core`) + +## Problem + +Batch 28 ports the JetStream API request/response surface, including subject dispatch, request decoding, account usage tracking, stream/consumer management handlers, delayed API responses, and API audit publication. This is the behavioral bridge between JetStream core state (Batch 27) and external API subjects (`$JS.API.*`). + +## Context Findings + +### Required Command Findings + +- `batch show 28 --db porting.db` + - Batch 28 is `pending` + - 55 features and 2 tests are all `deferred` + - Go source scope is `server/jetstream_api.go` +- `batch list --db porting.db` + - Batch 28 depends on batches 5 and 27 + - Batch 36 (`Stream Lifecycle`) depends on batch 28, so this is an unblocker batch +- `report summary --db porting.db` + - Features verified: 1271 / 3673 + - Unit tests verified: 430 / 3257 + - Large deferred backlog confirms need for strict anti-stub controls +- `batch ready --db porting.db` + - Ready batches currently: 0, 1, 5, 8 + - Batch 28 is not startable until batch 27 is complete (and batch 5 remains satisfied) + +### Batch 28 Mapping Shape + +Feature distribution by mapped .NET class: + +- `NatsServer`: 43 features (~4006 Go LOC) +- `JetStreamApi`: 6 features (~91 Go LOC) +- `Account`: 4 features (~56 Go LOC) +- `JsAccount`: 1 feature (~23 Go LOC) +- `JetStreamEngine`: 1 feature (~86 Go LOC) + +Complexity hot spots (Go LOC): + +- `#1501 Server.jsConsumerCreateRequest` (224) +- `#1477 Server.jsStreamInfoRequest` (223) +- `#1498 Server.processStreamRestore` (207) +- `#1504 Server.jsConsumerInfoRequest` (204) +- `#1484 Server.jsLeaderServerStreamMoveRequest` (162) + +Test mapping: + +- `#1716 TestIsJSONObjectOrArray` -> `JetStreamEngineTests.IsJSONObjectOrArray_ShouldSucceed` (depends on feature `#1488`) +- `#1719 TestJetStreamDelayedAPIResponses` -> `JetStreamEngineTests.JetStreamDelayedAPIResponses_ShouldSucceed` (includes dependency on feature `#1400` from batch 27) + +## Approaches + +### Approach A: Single-file monolithic port in one `NatsServer` partial + +- Put almost all Batch 28 methods into one large `NatsServer` file. +- Pros: fast to scaffold. +- Cons: high review/debug risk, weak separation for helper classes (`JetStreamApi`, `JetStreamEngine`), poor maintainability. + +### Approach B (Recommended): Layered API pipeline with grouped handler waves + +- Keep class boundaries aligned with mapping (`JetStreamApi`, `JetStreamEngine`, `NatsServer`, `Account`, `JsAccount`). +- Implement by feature groups (~20 max each) using API lifecycle phases: dispatch/helpers -> stream leader/metadata handlers -> message/consumer/snapshot handlers. +- Pros: strongest audit alignment, lower regression risk, clean status-evidence workflow. +- Cons: more upfront planning discipline. + +### Approach C: Signature-first sweep then fill behavior later + +- Add method signatures quickly, defer real logic. +- Pros: apparent short-term progress. +- Cons: violates anti-stub policy and creates status inflation risk. + +Decision: **Approach B**. + +## Proposed Design + +### 1. API Pipeline Architecture + +Implement a deterministic request pipeline: + +1. Subject mapping + dispatch (`GenerateJSMappingTable`, `ApiDispatch`, routed request processing) +2. Request metadata extraction (`GetRequestInfo`, `UnmarshalRequest`, subject parsing helpers) +3. Account checks/tracking (`CheckJetStream`, `TrackAPI`, `TrackAPIErr`, reservation/limits checks) +4. Handler execution (stream, consumer, leader/control operations) +5. Response publishing (immediate or delayed) and advisory audit emission + +### 2. Component Boundaries + +- `JetStreamApi` class: pure/static helpers (`GenerateJSMappingTable`, subject parsers, JSON-shape helpers, delayed queue helper) +- `JetStreamEngine` class: API entry on engine (`ApiDispatch`) +- `NatsServer` partials: handler implementations and response transport +- `Account` / `JsAccount`: API counters, enablement checks, tiered reservation and non-clustered limits checks + +### 3. Handler Grouping Strategy + +Use 3 feature groups (max ~20): + +1. **Group A (20):** IDs `1452,1454-1472` + Dispatch, delayed responder core, request parsing, account tracking, account info, naming helpers. +2. **Group B (18):** IDs `1473-1490` + Stream CRUD/list/info plus leader step-down, peer/remove/move operations and request-shape validators. +3. **Group C (17):** IDs `1491-1507` + Message delete/get, consumer unpin/pause/create/list/info/delete, purge/restore/snapshot, API advisory. + +### 4. Testing Strategy + +- Tracked tests first-class: + - `#1716` (JSON object/array detector) + - `#1719` (delayed API response sequencing/cancellation/reload behavior) +- Add focused helper tests where needed for deterministic per-feature verification, but only promote tracked IDs through PortTracker workflow. +- Keep infra-heavy scenarios deferred with explicit reasons when prerequisites from batch 27/runtime harness are missing. + +### 5. Error Handling and Deferred Policy + +- Use explicit API error responses (`JsApiError`-based) for request validation and unavailable-state paths. +- Any blocked item is explicitly marked `deferred` with reason; no placeholder production logic, no fake-pass tests. + +## Risks and Mitigations + +- **Risk:** Batch 28 execution before dependencies complete. + - **Mitigation:** hard preflight gate (`batch show 5`, `batch show 27`, then `batch start 28`). +- **Risk:** High-LOC handlers become partially ported. + - **Mitigation:** per-feature loop, stub scan, build/test gates before status promotion. +- **Risk:** Tracker status drift from actual code quality. + - **Mitigation:** max-15 ID updates, evidence checklist per update chunk, checkpoint between tasks. + +## Success Criteria + +- Batch 28 features are `verified` or explicitly `deferred` with concrete blocker reasons. +- Batch 28 tests are `verified` or explicitly `deferred` with concrete blocker reasons. +- No new stub patterns in touched source/test files. +- Build and required test gates pass at each checkpoint. +- Status updates are chunked and evidence-backed. + +## Non-Goals + +- Executing Batch 28 implementation in this document. +- Reworking unrelated JetStream/Raft architecture outside mapped Batch 28 behavior. +- Changing porting tracker mappings unless audit requires explicit remap. diff --git a/docs/plans/2026-02-27-batch-28-jetstream-api-implementation-plan.md b/docs/plans/2026-02-27-batch-28-jetstream-api-implementation-plan.md new file mode 100644 index 0000000..6331e1f --- /dev/null +++ b/docs/plans/2026-02-27-batch-28-jetstream-api-implementation-plan.md @@ -0,0 +1,550 @@ +# Batch 28 JetStream API Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 28 JetStream API behavior (55 features, 2 tests) from `server/jetstream_api.go` with strict no-stub execution and evidence-based PortTracker status updates. + +**Architecture:** Implement Batch 28 in three feature waves aligned to API lifecycle (dispatch/helpers, stream-control handlers, message/consumer/snapshot handlers), then port and verify the two mapped tests. Keep class/method mapping audit-safe (`NatsServer`, `JetStreamApi`, `JetStreamEngine`, `Account`, `JsAccount`) and enforce mandatory verification checkpoints before every status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-28-jetstream-api-design.md` + +--- + +## Batch 28 Scope + +- Batch ID: `28` +- Name: `JetStream API` +- Dependencies: `5`, `27` +- Go source: `golang/nats-server/server/jetstream_api.go` +- Feature IDs: `1452,1454-1507` (55 total; `1453` not present) +- Test IDs: `1716`, `1719` + +Primary implementation files: + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamApi.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamEngine.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamApi.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` (state fields only if required) +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` (`JsAccount` methods) + +Primary test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Create/Modify (supporting focused tests if needed): `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamApiTests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every feature and test in this batch must satisfy this protocol. Skipping steps is a plan violation. + +### Preflight Dependency Gate (REQUIRED before any feature work) + +1. Verify dependency batches are complete enough to start Batch 28: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- batch show 5 --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db + ``` +2. Start batch only when ready: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- batch start 28 --db porting.db + ``` +3. Baseline gates: + ```bash + dotnet build dotnet/ + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + ``` + +### Per-Feature Verification Loop (REQUIRED for each feature ID) + +1. Read mapped feature details and Go span: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db + sed -n ',p' golang/nats-server/server/jetstream_api.go + ``` +2. Mark the single active feature `stub` before coding: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db + ``` +3. Implement only that feature behavior in mapped class/method. +4. **Build gate (per feature):** + ```bash + dotnet build dotnet/ + ``` +5. **Test gate (per feature):** run the smallest focused tests that execute the changed path: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamApi" + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" + ``` +6. If both gates are green and no stub markers are introduced, promote feature to `complete`: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status complete --db porting.db + ``` +7. Promote to `verified` only after group-level gates pass. + +### Per-Test Verification Loop (REQUIRED for each test ID) + +1. Read mapping and Go test body: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db + sed -n ',p' golang/nats-server/server/jetstream_test.go + ``` +2. Port/replace one C# test method with real Arrange/Act/Assert behavior that calls production code. +3. Run the single test and confirm it is discovered: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.JetStreamEngineTests." \ + --verbosity normal + ``` +4. Validate summary contains `Passed: 1, Failed: 0` (not `Passed: 0`). +5. Only then set test status to `complete`; move to `verified` after class-level and batch-level gates. + +### Stub Detection Check (REQUIRED after every feature group and after test edits) + +Run against touched files only: + +```bash +git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \ + | rg "\.cs$" \ + | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass|throw new Exception\(\"TODO\"\))" +``` + +If any match appears in newly touched code, stop and fix or defer; do not promote status. + +### Build Gate (REQUIRED) + +Must pass: + +- after every feature ID loop +- after every feature group +- before any `batch-update` status command +- after test task completion + +Command: + +```bash +dotnet build dotnet/ +``` + +### Test Gate (REQUIRED) + +Minimum required gates for Batch 28: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests.IsJSONObjectOrArray_ShouldSucceed" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests.JetStreamDelayedAPIResponses_ShouldSucceed" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +``` + +At task checkpoints and final closure, run full unit test suite: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Allowed progress path: `deferred/not_started -> stub -> complete -> verified` +- Never update more than 15 feature IDs in one `feature batch-update` call +- Never update more than 15 test IDs in one `test batch-update` call +- Never set `verified` without captured build + test + stub-scan evidence +- If audit mismatch occurs, use explicit override reason only when evidence is documented + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At the end of each task before moving on: + +1. Run stub detection check. +2. Run build gate. +3. Run required test gate for touched classes. +4. Record updated IDs and evidence output. +5. Commit checkpoint. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +Any of the following in newly touched methods means the item is a stub and must not be promoted: + +- `throw new NotImplementedException()` +- Empty mapped method body (`{ }`) +- Placeholder comments (`// TODO`, `// PLACEHOLDER`, `// later`) +- Fake success tests (`Assert.True(true)`, `Assert.Pass()`) +- Tests that do not call production code +- Trivial constant-return placeholders for non-trivial handlers (`return null;`, `return false;`, `return 0;`, `return string.Empty;`) without behavior-equivalent Go logic + +### Hard Limits + +- Max feature group size: `~20` +- Max IDs per `batch-update`: `15` +- One active feature implementation loop at a time +- No status promotion if build gate is red +- No status promotion if stub scan has unresolved matches +- No cross-group status updates (only IDs in current task) + +### If You Get Stuck (MANDATORY) + +1. Stop implementing the blocked ID immediately. +2. Do **not** leave placeholder code or fake tests. +3. Mark blocked item `deferred` with an explicit reason: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db + ``` +4. Move to the next unblocked ID in the same task. +5. Deferred with reason is correct behavior; stubbing is not. + +--- + +## Feature Group Map (Max ~20 per task) + +### Group A (20 features): Dispatch, request parsing, account tracking, delayed responder core + +`1452,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467,1468,1469,1470,1471,1472` + +### Group B (18 features): Stream CRUD/list/info and control-plane handlers + +`1473,1474,1475,1476,1477,1478,1479,1480,1481,1482,1483,1484,1485,1486,1487,1488,1489,1490` + +### Group C (17 features): Message/consumer APIs, purge/restore/snapshot, API advisory + +`1491,1492,1493,1494,1495,1496,1497,1498,1499,1500,1501,1502,1503,1504,1505,1506,1507` + +### Batch 28 Tests + +`1716,1719` + +--- + +### Task 1: Dependency Gate and Batch Initialization + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-28-jetstream-api-design.md` +- Read: `golang/nats-server/server/jetstream_api.go` + +**Step 1: Verify Batch 28 dependencies are complete** + +Run: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 5 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Expected: + +- Batch 28 appears ready (or dependencies explicitly complete enough to start). + +**Step 2: Start Batch 28** + +Run: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 28 --db porting.db +``` + +Expected: + +- Batch status transitions to in-progress without dependency validation failure. + +**Step 3: Baseline gates** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +Expected: + +- Build succeeds; baseline failures (if any) are known before feature edits. + +**Step 4: Commit checkpoint** + +```bash +git add porting.db +git commit -m "chore(batch28): start jetstream api batch" +``` + +--- + +### Task 2: Implement Feature Group A (20 IDs) + +**Files:** + +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamApi.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamEngine.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamApi.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Optional state wiring: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` + +**Step 1: Mark Group A as `stub` in chunks (max 15 IDs)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1452,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1468,1469,1470,1471,1472" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute the Per-Feature Verification Loop for each Group A ID** + +Expected: + +- Each ID reaches `complete` individually only after build+test+stub-scan evidence. + +**Step 3: Run Group A gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +``` + +Expected: + +- Build succeeds and no regression in JetStream engine backlog class. + +**Step 4: Promote Group A IDs to `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1452,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1468,1469,1470,1471,1472" --set-status complete --db porting.db --execute +``` + +**Step 5: Checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch28): implement jetstream api dispatch and account request core" +``` + +--- + +### Task 3: Implement Feature Group B (18 IDs) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamApi.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamApi.cs` +- Modify (if needed): `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + +**Step 1: Mark Group B as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1473,1474,1475,1476,1477,1478,1479,1480,1481,1482,1483,1484,1485,1486,1487" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1488,1489,1490" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute the Per-Feature Verification Loop for each Group B ID** + +Expected: + +- Stream and control handlers are fully behavior-ported, not placeholders. + +**Step 3: Run Group B gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +``` + +**Step 4: Promote Group B IDs to `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1473,1474,1475,1476,1477,1478,1479,1480,1481,1482,1483,1484,1485,1486,1487" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1488,1489,1490" --set-status complete --db porting.db --execute +``` + +**Step 5: Checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch28): implement jetstream api stream and control handlers" +``` + +--- + +### Task 4: Implement Feature Group C (17 IDs) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamApi.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + +**Step 1: Mark Group C as `stub` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1491,1492,1493,1494,1495,1496,1497,1498,1499,1500,1501,1502,1503,1504,1505" --set-status stub --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1506,1507" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute the Per-Feature Verification Loop for each Group C ID** + +Expected: + +- Consumer/message/snapshot and advisory handlers behave per Go source intent. + +**Step 3: Run Group C gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" +``` + +**Step 4: Promote Group C IDs to `complete` (chunked)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1491,1492,1493,1494,1495,1496,1497,1498,1499,1500,1501,1502,1503,1504,1505" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1506,1507" --set-status complete --db porting.db --execute +``` + +**Step 5: Checkpoint protocol and commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server porting.db +git commit -m "feat(batch28): implement jetstream api consumer/message/snapshot handlers" +``` + +--- + +### Task 5: Port and Verify Batch 28 Tests (2 IDs) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Optional support tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamApiTests.cs` + +**Step 1: Mark both tests `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1716,1719" --set-status stub --db porting.db --execute +``` + +**Step 2: Implement `IsJSONObjectOrArray_ShouldSucceed` (ID 1716) with table-style cases from Go** + +Run: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~IsJSONObjectOrArray_ShouldSucceed" --verbosity normal +``` + +Expected: + +- `Passed: 1, Failed: 0` + +**Step 3: Implement `JetStreamDelayedAPIResponses_ShouldSucceed` (ID 1719)** + +Run: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamDelayedAPIResponses_ShouldSucceed" --verbosity normal +``` + +Expected: + +- `Passed: 1, Failed: 0` + +**Step 4: Class-level gate for `JetStreamEngineTests`** + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamEngineTests" --verbosity normal +``` + +**Step 5: Promote tests to `complete`, then `verified`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1716,1719" --set-status complete --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- test batch-update --ids "1716,1719" --set-status verified --db porting.db --execute +``` + +**Step 6: Checkpoint protocol and commit** + +```bash +git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests porting.db +git commit -m "test(batch28): port jetstream api object-shape and delayed-response tests" +``` + +--- + +### Task 6: Final Verification, Status Closure, and Batch Completion + +**Files:** + +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Step 1: Full gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 2: Final stub scan on touched files** + +```bash +git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \ + | rg "\.cs$" \ + | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass)" +``` + +Expected: + +- No unresolved matches in touched files. + +**Step 3: Promote feature IDs to `verified` in max-15 chunks** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1452,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1468,1469,1470,1471,1472,1473,1474,1475,1476,1477,1478,1479,1480,1481,1482" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1483,1484,1485,1486,1487,1488,1489,1490,1491,1492,1493,1494,1495,1496,1497" --set-status verified --db porting.db --execute +dotnet run --project tools/NatsNet.PortTracker -- feature batch-update --ids "1498,1499,1500,1501,1502,1503,1504,1505,1506,1507" --set-status verified --db porting.db --execute +``` + +**Step 4: Validate batch/report visibility** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 28 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 5: Complete batch and generate report** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 28 --db porting.db +./reports/generate-report.sh +``` + +**Step 6: Final commit** + +```bash +git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests porting.db reports/current.md +git commit -m "feat(batch28): complete jetstream api feature and test port" +``` + +--- + +## Execution Notes + +- This plan is intentionally strict on status evidence because Batch 28 is a high-fanout dependency for later stream/consumer lifecycle batches. +- If dependency batch 27 remains incomplete when execution starts, Task 1 must stop early and keep Batch 28 unstarted; do not bypass dependency enforcement. diff --git a/docs/plans/2026-02-27-batch-29-jetstream-batching-design.md b/docs/plans/2026-02-27-batch-29-jetstream-batching-design.md new file mode 100644 index 0000000..6824d3e --- /dev/null +++ b/docs/plans/2026-02-27-batch-29-jetstream-batching-design.md @@ -0,0 +1,135 @@ +# Batch 29 JetStream Batching Design + +**Date:** 2026-02-27 +**Batch:** 29 (`JetStream Batching`) +**Scope:** 12 features + 3 unit tests +**Dependencies:** batch `27` (`JetStream Core`) +**Go source:** `golang/nats-server/server/jetstream_batching.go` (+ mapped tests in `server/raft_test.go`) + +## Problem + +Batch 29 ports JetStream atomic batch internals: batch lifecycle/store setup, staged consistency bookkeeping, apply-path rejection/cleanup, and pre-proposal header validation (`checkMsgHeadersPreClusteredProposal`). This batch also includes 3 Raft-node behavior tests that depend on batch cleanup correctness. + +## Context Findings + +### Required command outputs (captured) + +- `batch show 29 --db porting.db` + - Batch 29 is `pending` + - 12 features + 3 tests are all `deferred` + - Dependency: batch 27 + - Go file: `server/jetstream_batching.go` +- `batch list --db porting.db` + - Batch 29 sits after batch 28 and depends on 27 + - Batch 40 (`MQTT Server/JSA`) depends on 27 as well; keeping 29 high quality prevents later churn in JetStream/Raft behavior +- `report summary --db porting.db` + - Features verified: 1271 / 3673 + - Tests verified: 430 / 3257 + - Deferred backlog remains dominant, so no-stub discipline is mandatory + +### Batch 29 mapped IDs + +Features: + +- `1508` `batching.newBatchGroup` +- `1509` `getBatchStoreDir` +- `1510` `newBatchStore` +- `1511` `batchGroup.readyForCommit` +- `1512` `batchGroup.cleanup` +- `1513` `batchGroup.cleanupLocked` +- `1514` `batchGroup.stopLocked` +- `1515` `batchStagedDiff.commit` +- `1516` `batchApply.clearBatchStateLocked` +- `1517` `batchApply.rejectBatchStateLocked` +- `1518` `batchApply.rejectBatchState` +- `1519` `checkMsgHeadersPreClusteredProposal` (largest surface, ~423 Go LOC) + +Tests: + +- `2654` `TestNRGMultipleStopsDontPanic` -> `RaftNodeTests.NRGMultipleStopsDontPanic_ShouldSucceed` +- `2674` `TestNRGKeepRunningOnServerShutdown` -> `RaftNodeTests.NRGKeepRunningOnServerShutdown_ShouldSucceed` +- `2718` `TestNRGInitSingleMemRaftNodeDefaults` -> `RaftNodeTests.NRGInitSingleMemRaftNodeDefaults_ShouldSucceed` + +### Existing .NET baseline + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs` exists but is partial and contains stub-like behavior (`ReadyForCommit` comment indicates stub). +- No current `RaftNodeTests` file exists under `dotnet/tests/...`; mapped test targets are not implemented yet. +- JetStream batching test file currently contains deferred placeholders and does not cover Batch 29 mapped Raft tests. + +## Approaches + +### Approach A: Single massive `JetStreamBatching.cs` pass in one shot + +- Pros: fewer commits, direct throughput. +- Cons: high defect risk around cross-cutting map/counter/header logic, hard to validate incrementally. + +### Approach B (Recommended): Two feature waves + one test wave with strict evidence gates + +- Wave 1: batch/store lifecycle primitives (`1508-1514`) +- Wave 2: staged/apply/header semantics (`1515-1519`) +- Wave 3: mapped Raft tests (`2654,2674,2718`) +- Pros: manageable review units, easier causality between feature changes and tests, strongest anti-stub control. +- Cons: more status updates/checkpoints. + +### Approach C: Signature-first fill (compile now, behavior later) + +- Pros: quick apparent progress. +- Cons: violates anti-stub goals and creates false tracker progress. + +Decision: **Approach B**. + +## Proposed Design + +### 1. Component boundaries + +- Keep batching logic centered in `JetStreamBatching.cs` for mapped methods. +- Add narrow helper methods/types only when required to preserve method-level mapping and testability. +- Keep heavy validation (`checkMsgHeadersPreClusteredProposal`) behaviorally aligned with Go checks: pre-check ordering, duplicate/msg-id checks, counter increment path, expected sequence checks, scheduling/rollup constraints, and discard policy checks. + +### 2. Data and concurrency model + +- Preserve lock expectations from Go comments by using existing `Lock`/`ReaderWriterLockSlim` conventions. +- Preserve inflight/global counters and cleanup semantics as deterministic state transitions. +- Ensure timer cleanup and commit readiness are race-safe and idempotent. + +### 3. Feature grouping strategy (max ~20) + +- **Group A (7 features):** `1508-1514` + - Batch group creation, store dir/store construction, commit readiness, cleanup and stop paths. +- **Group B (5 features):** `1515-1519` + - Staged diff commit state, batch apply clear/reject, and full header pre-check function. + +### 4. Test strategy + +- Create/port mapped tests in `RaftNodeTests` (or equivalent mapped class file) with real Arrange/Act/Assert behavior. +- Keep tests deterministic and non-networked where possible; if runtime infrastructure is missing, explicitly defer with reason. +- Add focused unit tests for `JetStreamBatching` helpers as needed to verify feature behavior before promoting feature status. + +### 5. Status and evidence design + +- Status transitions must be evidence-backed: `stub -> complete -> verified`. +- Chunked updates (max 15 IDs) to prevent bulk unverifiable promotion. +- Checkpoint between tasks: stub scan + build + targeted tests + tracker updates. + +## Risks and Mitigations + +- **Risk:** `1519` complexity causes partial/placeholder implementation. + - **Mitigation:** isolate into dedicated task, require feature-level gates and explicit defer-if-blocked path. +- **Risk:** Mapped Raft tests need runtime hooks not yet available. + - **Mitigation:** mark `deferred` with exact blocker reason; no fake-pass tests. +- **Risk:** Tracker drift from actual behavior. + - **Mitigation:** per-ID evidence, max-15 update chunk, and post-task checkpoints. + +## Success Criteria + +- All Batch 29 features and tests are either: + - `verified` with captured verification evidence, or + - `deferred` with explicit blocker reason. +- No new stub patterns introduced in touched production/test files. +- Build and relevant test gates are green at each task checkpoint. + +## Non-Goals + +- Executing implementation in this design doc. +- Refactoring unrelated JetStream or Raft subsystems beyond mapped Batch 29 behavior. +- Broad integration-test harness expansion beyond what Batch 29 requires. diff --git a/docs/plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md b/docs/plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md new file mode 100644 index 0000000..abefa83 --- /dev/null +++ b/docs/plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md @@ -0,0 +1,479 @@ +# Batch 29 JetStream Batching Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 29 JetStream batching behavior (12 features, 3 tests) from `server/jetstream_batching.go` and mapped `raft_test.go` cases without introducing stubs or unverifiable tracker updates. + +**Architecture:** Execute in two feature groups and one test group. Group A ports lifecycle/store primitives (`1508-1514`), Group B ports staged/apply/header checks (`1515-1519`, including the large `checkMsgHeadersPreClusteredProposal` surface), then mapped Raft tests are ported/verified. Every ID follows strict per-item evidence gates before status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-29-jetstream-batching-design.md` + +--- + +## Batch 29 Scope + +- Batch ID: `29` +- Name: `JetStream Batching` +- Dependency: `27` +- Go source: `golang/nats-server/server/jetstream_batching.go` +- Features: `1508-1519` (12 total) +- Tests: `2654,2674,2718` + +Primary implementation files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs` +- Optional split (recommended for readability of feature `1519`): + - Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.HeaderChecks.cs` +- Optional supporting updates only if required by compile/runtime behavior: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs` + +Primary test files: + +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeTests.cs` +- Modify or Create focused batching tests as needed: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs` +- Existing deferred batching integration placeholders (do not convert to fake unit tests): + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingTests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Applies to every feature ID and test ID in Batch 29. + +### Preflight Dependency Gate (REQUIRED before Task 1 coding) + +1. Verify dependency and readiness: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db + ``` +2. Start the batch only when dependency checks pass: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- batch start 29 --db porting.db + ``` +3. Baseline gates: + ```bash + dotnet build dotnet/ + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + ``` + +### Per-Feature Verification Loop (REQUIRED for each feature ID) + +For each feature in current task group: + +1. Inspect mapping and Go span: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db + sed -n ',p' golang/nats-server/server/jetstream_batching.go + ``` +2. Mark that feature as `stub` before editing: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db + ``` +3. Implement mapped behavior in .NET (no placeholders). +4. Run **Build Gate**. +5. Run **Test Gate** (focused tests for changed behavior). +6. Run **Stub Detection Check** on touched files. +7. If gates are green and no stubs are detected, promote feature to `complete`: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status complete --db porting.db + ``` +8. Promote feature IDs to `verified` only after task-level checkpoint passes. + +### Per-Test Verification Loop (REQUIRED for each test ID) + +1. Inspect mapped test details and Go source: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db + sed -n ',p' golang/nats-server/server/raft_test.go + ``` +2. Mark test `stub` before editing: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db + ``` +3. Implement real Arrange/Act/Assert test calling production code. +4. Run single-test command: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.JetStream.RaftNodeTests." \ + --verbosity normal + ``` +5. Confirm summary includes `Passed: 1, Failed: 0` (not `Passed: 0`). +6. Run class-level test gate: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.JetStream.RaftNodeTests" + ``` +7. Run Stub Detection Check. +8. Promote test to `complete`, then `verified` at checkpoint when evidence is complete. + +### Stub Detection Check (REQUIRED after every feature/test loop and each task) + +Run against touched code: + +```bash +git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \ + | rg "\.cs$" \ + | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|throw new Exception\\(\"TODO\"\\)|=>\\s*default;|=>\\s*null;)" +``` + +Any match in touched methods means the item is not eligible for `complete` or `verified`. + +### Build Gate (REQUIRED) + +`dotnet build dotnet/` is mandatory: + +- after each feature implementation loop +- after each test implementation loop +- before any `batch-update` +- at every task checkpoint + +### Test Gate (REQUIRED) + +Minimum per-task gates: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests" +``` + +Checkpoint and final gates: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Allowed path: `deferred/not_started -> stub -> complete -> verified` +- Use `batch-update` with **at most 15 IDs** per call. +- Never update IDs outside the current task group. +- Never mark `verified` without corresponding build/test/stub-scan evidence. +- If audit rejects status change, use `--override ""` only with explicit evidence. + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At the end of each task before starting the next task: + +1. Run Stub Detection Check. +2. Run Build Gate. +3. Run task-relevant Test Gate. +4. Run full unit suite: + ```bash + dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ + ``` +5. Update status IDs for current task only (max 15 IDs per command). +6. Commit checkpoint changes. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +Any of these in touched methods means the work is a stub and must not be promoted: + +- `throw new NotImplementedException()` +- Empty mapped method bodies +- Placeholder comments: `// TODO`, `// PLACEHOLDER`, `// later` +- Fake-pass assertions: `Assert.True(true)`, `Assert.Pass()` +- Tests that never call production code in `ZB.MOM.NatsNet.Server` +- Constant-return placeholders for non-trivial logic (`return null;`, `return 0;`, `return false;`, `return string.Empty;`) without Go-equivalent behavior +- Catch-all exception swallowing used to force pass behavior + +### Hard Limits + +- Max feature IDs per group: `~20` +- Max IDs per `feature/test batch-update`: `15` +- One active feature loop at a time +- No `verified` promotion unless build + targeted tests + stub scan are green +- One checkpoint commit per task minimum +- Feature `1519` must not be merged as partial/no-op behavior + +### If You Get Stuck (MANDATORY) + +1. Stop work on the blocked ID immediately. +2. Do **not** leave placeholder code or fake tests. +3. Mark the blocked item `deferred` with specific reason: + ```bash + dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db + dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db + ``` +4. Move to the next unblocked ID in the same task. +5. Deferred-with-reason is correct behavior; stubs are not. + +--- + +## Feature/Test Grouping (max ~20 per group) + +### Group A (7 features): Batch lifecycle + store creation + +`1508,1509,1510,1511,1512,1513,1514` + +### Group B (5 features): Staged/apply state + pre-proposal header checks + +`1515,1516,1517,1518,1519` + +### Group C (3 tests): Raft-node behavioral tests mapped to batch 29 + +`2654,2674,2718` + +--- + +### Task 1: Preflight and Batch Start + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-29-jetstream-batching-design.md` +- Read: `golang/nats-server/server/jetstream_batching.go` +- Read: `golang/nats-server/server/raft_test.go` (mapped lines) + +**Step 1: Dependency/readiness checks** + +Run: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Expected: + +- Batch 29 is startable, dependency 27 satisfied. + +**Step 2: Start batch** + +Run: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 29 --db porting.db +``` + +Expected: + +- Batch transitions to in-progress. + +**Step 3: Baseline build/test** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +Expected: + +- Known baseline captured before edits. + +**Step 4: Checkpoint protocol and commit** + +--- + +### Task 2: Implement Group A Features (`1508-1514`) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs` +- Optional modify if needed by compile/runtime wiring: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs` + +**Step 1: Mark Group A as `stub` (single chunk, 7 IDs <= 15)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1508,1509,1510,1511,1512,1513,1514" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each ID in Group A** + +Expected: + +- Lifecycle/store methods are behaviorally implemented and individually gated. + +**Step 3: Task-level gates + checkpoint protocol** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching" +``` + +**Step 4: Promote eligible Group A IDs to `complete` (blocked IDs stay `deferred`)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status complete --db porting.db --execute +``` + +**Step 5: After checkpoint evidence, promote eligible Group A IDs to `verified`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status verified --db porting.db --execute +``` + +--- + +### Task 3: Implement Group B Features (`1515-1519`) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs` +- Optional create/modify to isolate large method: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.HeaderChecks.cs` +- Optional supporting updates if required by dependencies: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamErrors.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs` + +**Step 1: Mark Group B as `stub` (single chunk, 5 IDs <= 15)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "1515,1516,1517,1518,1519" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each ID in Group B** + +Notes: + +- Treat `1519` as its own micro-loop: implement incrementally but only promote when full gate passes. +- Validate error-path parity (duplicate msg ID, expected sequence checks, schedule/rollup validation, discard-new limits). + +**Step 3: Task-level gates + checkpoint protocol** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftTypesTests" +``` + +**Step 4: Promote eligible Group B IDs to `complete` (blocked IDs stay `deferred`)** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status complete --db porting.db --execute +``` + +**Step 5: After checkpoint evidence, promote eligible Group B IDs to `verified`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status verified --db porting.db --execute +``` + +--- + +### Task 4: Implement Group C Tests (`2654,2674,2718`) + +**Files:** + +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeTests.cs` +- Optional modify if helper setup reuse is needed: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` +- Source references: + - `golang/nats-server/server/raft_test.go` + +**Step 1: Mark tests as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "2654,2674,2718" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each test ID** + +Expected: + +- Each mapped test method is real, discovered, and passing individually. + +**Step 3: Task-level gates + checkpoint protocol** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests" +``` + +**Step 4: Promote eligible tests to `complete`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status complete --db porting.db --execute +``` + +**Step 5: After checkpoint evidence, promote eligible tests to `verified`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status verified --db porting.db --execute +``` + +--- + +### Task 5: Batch 29 Final Verification and Closeout + +**Files:** + +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Step 1: Final mandatory gates** + +Run: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +Expected: + +- All Batch 29 IDs are `verified` or `deferred` with explicit reasons. + +**Step 2: Complete batch** + +Run: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 29 --db porting.db +``` + +**Step 3: Generate report + commit** + +```bash +./reports/generate-report.sh +git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \ + dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream \ + porting.db reports/ +git commit -m "feat(batch29): port jetstream batching and mapped raft tests" +``` diff --git a/docs/plans/2026-02-27-batch-30-raft-part-1-design.md b/docs/plans/2026-02-27-batch-30-raft-part-1-design.md new file mode 100644 index 0000000..c742b5b --- /dev/null +++ b/docs/plans/2026-02-27-batch-30-raft-part-1-design.md @@ -0,0 +1,127 @@ +# Batch 30 Raft Part 1 Design + +**Date:** 2026-02-27 +**Batch:** 30 (`Raft Part 1`) +**Scope:** 85 features + 414 unit tests +**Dependencies:** batches `4` (Logging), `18` (Server Core) +**Go source:** `golang/nats-server/server/raft.go` + +## Problem + +Batch 30 is the first major Raft execution tranche and includes foundational node bootstrap, election/follower/leader loops (through `runCatchup`), append-entry/vote encoding helpers, and server-level Raft-node registration and lookup methods. The mapped test surface is very large (414 tests) and includes both direct Raft tests and broad JetStream cluster regressions. + +The design goal is to make Batch 30 executable in a deterministic, evidence-driven way without repeating the placeholder-test/stub drift seen in earlier backlog files. + +## Context Findings + +### Required command results + +- `batch show 30 --db porting.db` + - Status: `pending` + - Features: `85` (all currently `deferred`) + - Tests: `414` (all currently `deferred`) + - Depends on: `4,18` +- `batch list --db porting.db` + - Batch 30 is ordered before Batch 31 (`Raft Part 2`) and is a dependency anchor for JetStream cluster batches. +- `report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + - Deferred backlog remains dominant, so verification rigor is mandatory. + +### Feature-map and codebase findings + +- Batch 30 feature IDs are concentrated in `raft.go` line ranges `137-3237` plus package/helper functions in `4354-4753`. +- Existing .NET Raft baseline exists in `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs`, but many mapped Batch 30 method targets are still missing or only partially approximated. +- Server-level mapped methods (`BootstrapRaftNode`, `InitRaftNode`, `StartRaftNode`, `RegisterRaftNode`, etc.) do not currently exist as first-class `NatsServer` methods. + +### Test-map findings (critical) + +- Batch 30 tests are highly skewed: + - `366` tests map to feature `2683` (`raft.shutdown`) + - `3` tests map to feature `2715` (`appendEntry.encode`) + - remaining tests map to external feature IDs (45 tests) +- Only `19` tests come from `server/raft_test.go`; the rest are mostly JetStream cluster/supercluster/concurrency/MQTT regression tests that rely on raft behavior transitively. +- Existing `ImplBacklog/*.Impltests.cs` files contain many superficial placeholder-style tests and cannot be treated as verification evidence. + +## Approaches + +### Approach A: Full-surface implementation (85 features + all 414 tests in one pass) + +- Pros: maximal immediate tracker movement. +- Cons: extremely high risk, weak causality, and likely stub/fake-pass relapse. + +### Approach B (Recommended): Feature-first Raft core in five groups, then two-tier test strategy + +- Tier 1: implement and verify all 85 Batch 30 features in grouped, dependency-ordered slices. +- Tier 2: port direct Raft unit tests first, then process broader mapped regression tests class-by-class with explicit defer rules when runtime infra is missing. +- Pros: coherent sequencing, auditable status transitions, supports strict anti-stub controls. +- Cons: more checkpoints and status operations. + +### Approach C: Infra-first (build cluster harness before feature completion) + +- Pros: can unlock larger integration tests earlier. +- Cons: violates YAGNI for this batch and delays core feature parity. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Architecture and file strategy + +- Keep Raft model/codec/state-machine logic centered in `JetStream/RaftTypes.cs`, but split into partial files when method count becomes unreviewable (for example: lifecycle, codecs, follower/leader loops). +- Add explicit `NatsServer` Raft integration surface in a dedicated partial (`NatsServer.Raft.cs`) instead of reflection-only lifecycle hooks. +- Preserve existing locking/concurrency primitives already used in repo (`ReaderWriterLockSlim`, `Interlocked`, `Channel`, `IpQueue`), mapping Go intent rather than line-by-line syntax. + +### 2. Feature grouping (max ~20 per group) + +- **Group A (15):** bootstrap/init/server registry and early apply/snapshot prep + `2599,2600,2601,2602,2603,2607,2608,2609,2610,2611,2612,2613,2614,2615,2629` +- **Group B (14):** snapshot lifecycle + leader-state helpers/campaign hooks + `2634,2637,2639,2645,2646,2647,2651,2652,2653,2659,2663,2664,2665,2674` +- **Group C (20):** node runtime loop and follower pipeline scaffolding + `2683,2684,2685,2686,2687,2688,2689,2690,2691,2692,2693,2694,2695,2696,2697,2698,2701,2702,2703,2704` +- **Group D (19):** entry construction/encoding + membership-change handlers + `2705,2706,2707,2708,2709,2710,2711,2712,2714,2715,2716,2717,2718,2719,2720,2721,2722,2723,2724` +- **Group E (17):** leader loop/catchup + peer-state and vote persistence helpers + `2725,2727,2728,2729,2731,2732,2757,2762,2763,2764,2770,2771,2773,2774,2775,2781,2782` + +### 3. Test design + +- **Tier T1 (Raft-direct tests, highest value first):** `server/raft_test.go` mapped IDs (`2618,2619,2621,2623,2625,2632,2633,2639,2642,2675,2700,2701,2706,2707,2708,2709,2710,2711,2713`). +- **Tier T2 (transitive regression tests):** remaining mapped classes (`JetStreamClusterTests*`, `JetStreamSuperClusterTests`, `ConcurrencyTests*`, `MqttHandlerTests`, etc.) processed only with real behavioral assertions; otherwise explicitly deferred with reason. +- Existing placeholder tests are not accepted as evidence; they must be replaced or remain deferred. + +### 4. Verification model + +- Enforce per-feature and per-test loops with mandatory build/test gates before any status promotions. +- Enforce max `15` IDs per `feature/test batch-update` command. +- Require checkpoint protocol between tasks (stub scan, build, targeted tests, full unit test, then status updates). + +### 5. Error handling and stuck policy + +- If a feature/test is blocked by missing infrastructure, missing prerequisite behavior, or non-deterministic harness gaps: + - do not leave placeholders, + - do not force fake-pass tests, + - mark `deferred` with explicit blocker reason, + - continue with next unblocked ID in current group. + +## Risks and Mitigations + +- **Risk:** `raft.shutdown` mapping concentration (366 tests) causes noisy/non-actionable test wave. + **Mitigation:** process tests by class and behavioral relevance; require per-test evidence and deferred reasons for infra-blocked cases. +- **Risk:** large `RaftTypes.cs` file becomes unreviewable. + **Mitigation:** allow controlled partial-file split while preserving namespaces and API shape. +- **Risk:** false progress via stubbed placeholders in `ImplBacklog`. + **Mitigation:** mandatory anti-stub scans and hard promotion gates. + +## Success Criteria + +- All 85 features are either `verified` with evidence or `deferred` with specific blocker reason (no placeholders). +- Batch 30 mapped tests are processed under the same rule: verified only with real execution evidence, otherwise deferred with explicit reason. +- No new stub/fake-pass patterns in touched production or test files. +- The implementation plan includes strict verification and anti-stub guardrails adapted to both features and tests. + +## Non-Goals + +- Executing implementation in this design document. +- Completing downstream Batch 31+ raft/clustering behavior that is outside Batch 30 feature IDs. +- Building new distributed integration infrastructure beyond what is needed for deterministic unit-level verification. diff --git a/docs/plans/2026-02-27-batch-30-raft-part-1-implementation-plan.md b/docs/plans/2026-02-27-batch-30-raft-part-1-implementation-plan.md new file mode 100644 index 0000000..e22413a --- /dev/null +++ b/docs/plans/2026-02-27-batch-30-raft-part-1-implementation-plan.md @@ -0,0 +1,550 @@ +# Batch 30 Raft Part 1 Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 30 (`Raft Part 1`) features from `server/raft.go` with evidence-backed status updates, while processing mapped tests with strict no-stub discipline. + +**Architecture:** Implement the 85 features in five Raft-focused groups (max 20 each), progressing from server bootstrap/state setup to follower/leader loops and codec/persistence helpers. Use a dual-track test strategy: direct raft tests first, then mapped transitive regressions by class with explicit defer reasons when infrastructure is unavailable. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-30-raft-part-1-design.md` + +--- + +## Batch 30 Scope + +- Batch ID: `30` +- Name: `Raft Part 1` +- Dependencies: `4,18` +- Go source: `golang/nats-server/server/raft.go` +- Features: `85` +- Tests: `414` + +Primary implementation files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Create/Modify (recommended split for reviewability): + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.ServerIntegration.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Snapshots.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.RunLoop.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Codecs.cs` +- Create/Modify for server-level mapped methods: + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Raft.cs` +- Modify (remove dangling TODO when method exists): + - `dotnet/src/ZB.MOM.NatsNet.Server/Config/ReloadOptions.cs` + +Primary test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeCoreTests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Server/NatsServerRaftTests.cs` +- Create/Modify mapped backlog classes as needed for Batch 30 test IDs: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + - existing `ImplBacklog/*.Impltests.cs` files for mapped IDs in this batch + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Applies to every feature ID and test ID touched in Batch 30. + +### Preflight Dependency Gate (REQUIRED before Task 1 coding) + +1. Verify dependency and readiness: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 4 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 18 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 30 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +2. Start batch only when ready: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 30 --db porting.db +``` + +3. Capture baseline: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED for each feature ID) + +For each feature in active task group: + +1. Inspect mapping and Go span: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +sed -n ',p' golang/nats-server/server/raft.go +``` + +2. Mark feature `stub` before editing: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` + +3. Add or update a focused test that fails first for that feature behavior. +4. Run that focused test to confirm red. +5. Implement minimal production code for green. +6. Run Build Gate. +7. Run Test Gate (focused). +8. Run Stub Detection Check. +9. If all gates are green, mark feature `complete`: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status complete --db porting.db +``` + +10. Promote to `verified` only at task checkpoint after cumulative evidence is clean. + +### Per-Test Verification Loop (REQUIRED for each test ID) + +1. Inspect test mapping and Go source: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db +sed -n ',p' +``` + +2. Mark test `stub` before editing: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` + +3. Write real Arrange/Act/Assert test that invokes production code. +4. Run single test: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." \ + --verbosity normal +``` + +5. Confirm `Passed: 1, Failed: 0` (if `Passed: 0`, test discovery failed). +6. Run class-level filter for cumulative validation. +7. Run Stub Detection Check. +8. Mark `complete` only after test evidence is real; mark `verified` only at checkpoint. + +### Stub Detection Check (REQUIRED after each feature/test loop and each task) + +Run against touched C# files: + +```bash +git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \ + | rg "\.cs$" \ + | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass|=>\s*default;|=>\s*null;|return\s+null;\s*$|return\s+0;\s*$|return\s+false;\s*$)" +``` + +Any hit in touched mapped methods/tests blocks promotion to `complete` or `verified`. + +### Build Gate (REQUIRED) + +Run `dotnet build dotnet/`: + +- after each feature loop, +- before every status batch update, +- at each task checkpoint. + +### Test Gate (REQUIRED) + +Minimum focused gates per task: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~Raft" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~NatsServerRaft" +``` + +For touched ImplBacklog classes: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog." +``` + +Checkpoint and final gates: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Allowed progression: `deferred/not_started -> stub -> complete -> verified` +- Use `feature/test batch-update` with at most `15` IDs per command. +- Never update IDs outside current task/group. +- Never mark `verified` without evidence from build + tests + stub scan. + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At the end of every task: + +1. Run Stub Detection Check. +2. Run Build Gate. +3. Run focused Test Gate for touched areas. +4. Run full unit test suite. +5. Apply status updates for current task only (max 15 IDs per command). +6. Commit checkpoint. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +Any of these in touched mapped code is a stub and blocks promotion: + +- `throw new NotImplementedException()` +- Empty mapped method bodies for non-trivial behavior +- Placeholder comments (`// TODO`, `// PLACEHOLDER`, `// later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`) +- Tests that do not call production code +- Constant-return placeholders for complex logic (`return null;`, `return 0;`, `return false;`, `return string.Empty;`) +- Catch-all exception swallowing that forces tests to pass +- Reflection-only assertions that never validate observable raft behavior + +### Hard Limits + +- Max feature IDs per implementation group: `~20` +- Max IDs per `feature/test batch-update`: `15` +- One active feature loop at a time +- One active test loop at a time +- No `verified` status without green build + tests + stub scan evidence +- Mandatory checkpoint after each task before moving forward + +### If You Get Stuck + +1. Stop the blocked item immediately. +2. Do not leave placeholder implementation or fake-pass tests. +3. Mark blocked item `deferred` with explicit reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db +``` + +4. Add a short code comment only where needed to explain defer reason in test source. +5. Continue with next unblocked ID in the same group. + +Deferred-with-reason is correct. Stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (15) + +`2599,2600,2601,2602,2603,2607,2608,2609,2610,2611,2612,2613,2614,2615,2629` + +### Group B (14) + +`2634,2637,2639,2645,2646,2647,2651,2652,2653,2659,2663,2664,2665,2674` + +### Group C (20) + +`2683,2684,2685,2686,2687,2688,2689,2690,2691,2692,2693,2694,2695,2696,2697,2698,2701,2702,2703,2704` + +### Group D (19) + +`2705,2706,2707,2708,2709,2710,2711,2712,2714,2715,2716,2717,2718,2719,2720,2721,2722,2723,2724` + +### Group E (17) + +`2725,2727,2728,2729,2731,2732,2757,2762,2763,2764,2770,2771,2773,2774,2775,2781,2782` + +--- + +## Test Groups + +### T1: Direct Raft tests (19 IDs, highest priority) + +`2618,2619,2621,2623,2625,2632,2633,2639,2642,2675,2700,2701,2706,2707,2708,2709,2710,2711,2713` + +### T2: Transitive mapped regressions (395 IDs) + +- Process class-by-class from mapped ImplBacklog classes (`JetStreamClusterTests*`, `JetStreamSuperClusterTests`, `ConcurrencyTests*`, `MqttHandlerTests`, etc.). +- Replace placeholder tests only when real deterministic verification is possible. +- Otherwise keep `deferred` with explicit reason. + +--- + +### Task 1: Preflight, Batch Start, and Baseline + +**Files:** + +- Read: `docs/standards/dotnet-standards.md` +- Read: `docs/plans/2026-02-27-batch-30-raft-part-1-design.md` +- Read: `golang/nats-server/server/raft.go` + +**Step 1: Run preflight dependency gate** + +Run the Preflight Dependency Gate commands. + +**Step 2: Start Batch 30** + +Run `batch start 30` command. + +**Step 3: Capture baseline build and tests** + +Run baseline build/test commands. + +**Step 4: Checkpoint protocol and commit** + +--- + +### Task 2: Implement Feature Group A (15 features) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.Raft.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Config/ReloadOptions.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/Server/NatsServerRaftTests.cs` + +**Step 1: Mark Group A features as `stub` (split into max-15 chunks)** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2599,2600,2601,2602,2603,2607,2608,2609,2610,2611,2612,2613,2614,2615,2629" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each Group A ID** + +**Step 3: Run task-level gates and checkpoint protocol** + +**Step 4: Promote eligible Group A IDs to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 3: Implement Feature Group B (14 features) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Snapshots.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeCoreTests.cs` + +**Step 1: Mark Group B as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2634,2637,2639,2645,2646,2647,2651,2652,2653,2659,2663,2664,2665,2674" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each Group B ID** + +**Step 3: Run task-level gates and checkpoint protocol** + +**Step 4: Promote eligible Group B IDs to `complete` then `verified`** + +--- + +### Task 4: Implement Feature Group C (20 features) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.RunLoop.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeCoreTests.cs` + +**Step 1: Mark first 15 Group C IDs as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2683,2684,2685,2686,2687,2688,2689,2690,2691,2692,2693,2694,2695,2696,2697" --set-status stub --db porting.db --execute +``` + +**Step 2: Mark remaining 5 Group C IDs as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2698,2701,2702,2703,2704" --set-status stub --db porting.db --execute +``` + +**Step 3: Execute Per-Feature Verification Loop for all Group C IDs** + +**Step 4: Run task-level gates and checkpoint protocol** + +**Step 5: Promote eligible Group C IDs to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 5: Implement Feature Group D (19 features) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Codecs.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + +**Step 1: Mark Group D features `stub` in two chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2705,2706,2707,2708,2709,2710,2711,2712,2714,2715,2716,2717,2718,2719,2720" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2721,2722,2723,2724" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all Group D IDs** + +**Step 3: Run task-level gates and checkpoint protocol** + +**Step 4: Promote eligible Group D IDs to `complete` then `verified`** + +--- + +### Task 6: Implement Feature Group E (17 features) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.RunLoop.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Codecs.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeCoreTests.cs` + +**Step 1: Mark Group E features `stub` in two chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2725,2727,2728,2729,2731,2732,2757,2762,2763,2764,2770,2771,2773,2774,2775" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2781,2782" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for all Group E IDs** + +**Step 3: Run task-level gates and checkpoint protocol** + +**Step 4: Promote eligible Group E IDs to `complete` then `verified`** + +--- + +### Task 7: Port and Verify T1 Direct Raft Tests (19 tests) + +**Files:** + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + +**Step 1: Mark T1 tests `stub` in <=15-ID chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "2618,2619,2621,2623,2625,2632,2633,2639,2642,2675,2700,2701,2706,2707,2708" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "2709,2710,2711,2713" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each T1 test ID** + +**Step 3: Run class-level and task-level test gates** + +**Step 4: Promote eligible T1 tests to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 8: Process T2 Transitive Mapped Tests (395 tests) + +**Files:** + +- Modify only touched mapped files under `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/` + +**Step 1: Generate T2 working manifest (exclude T1 IDs)** + +```bash +sqlite3 -header -column porting.db " +SELECT ut.id, ut.dotnet_class, ut.dotnet_method, ut.go_file, ut.go_line_number +FROM batch_tests bt +JOIN unit_tests ut ON ut.id=bt.test_id +WHERE bt.batch_id=30 + AND ut.id NOT IN (2618,2619,2621,2623,2625,2632,2633,2639,2642,2675,2700,2701,2706,2707,2708,2709,2710,2711,2713) +ORDER BY ut.dotnet_class, ut.id;" +``` + +**Step 2: Work class-by-class using Per-Test Verification Loop** + +**Step 3: For blocked tests, apply If-You-Get-Stuck protocol (keep deferred with reason, do not stub)** + +**Step 4: Apply status updates in <=15-ID chunks for truly verified tests only** + +**Step 5: Run checkpoint protocol after each class** + +--- + +### Task 9: Final Verification, Audit, and Batch Readiness Review + +**Files:** + +- Modify: `porting.db` +- Optional update: `reports/current.md` (via report script) + +**Step 1: Run full gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +**Step 2: Run audit checks** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 30 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` + +**Step 3: Complete batch only if all items satisfy completion criteria** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 30 --db porting.db +``` + +If completion criteria fail, keep blocked IDs deferred with explicit reasons and stop; do not force status changes. + +--- + +## Execution Notes + +- If `dotnet` is not on PATH in your shell, use `/usr/local/share/dotnet/dotnet` as a drop-in replacement for commands above. +- Do not run implementation from this planning session. diff --git a/reports/current.md b/reports/current.md index f83cd4d..a6eb769 100644 --- a/reports/current.md +++ b/reports/current.md @@ -1,6 +1,6 @@ # NATS .NET Porting Status Report -Generated: 2026-02-27 20:43:15 UTC +Generated: 2026-02-27 21:33:11 UTC ## Modules (12 total) diff --git a/reports/report_dc3e162.md b/reports/report_dc3e162.md new file mode 100644 index 0000000..a6eb769 --- /dev/null +++ b/reports/report_dc3e162.md @@ -0,0 +1,37 @@ +# NATS .NET Porting Status Report + +Generated: 2026-02-27 21:33:11 UTC + +## Modules (12 total) + +| Status | Count | +|--------|-------| +| verified | 12 | + +## Features (3673 total) + +| Status | Count | +|--------|-------| +| deferred | 2377 | +| n_a | 24 | +| stub | 1 | +| verified | 1271 | + +## Unit Tests (3257 total) + +| Status | Count | +|--------|-------| +| deferred | 2640 | +| n_a | 187 | +| verified | 430 | + +## Library Mappings (36 total) + +| Status | Count | +|--------|-------| +| mapped | 36 | + + +## Overall Progress + +**1924/6942 items complete (27.7%)**