dohertj2/natsnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reconcile batch 0 with 553 implementable tests and add design/plan docs

Browse Source 
Map 553 deferred tests whose feature dependencies are all verified into
batch 0 (batch_tests table). Add Codex-generated design and implementation
plan for batch 0 execution.
This commit is contained in:
Joseph Doherty
2026-02-27 13:20:29 -05:00
parent c5e0416793
commit ee0827d7d4
5 changed files with 638 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

156
docs/plans/2026-02-27-batch-0-implementable-tests-design.md Normal file
View File

@@ -0,0 +1,156 @@
# Batch 0 Implementable Tests Design
**Date:** 2026-02-27
**Scope:** Plan how to port/implement Batch 0 unit tests only (no execution in this document).
## Problem
Batch 0 is defined as: deferred tests whose feature dependencies are already verified/complete/n_a.
However, current tracker output is inconsistent:
- `batch show 0` reports `Tests: 0`
- `report summary` reports `2640 deferred` unit tests
- direct DB query shows **553 deferred tests** already satisfy Batch 0 dependency rules
- all 553 have `dotnet_class` + `dotnet_method` mappings and method names already present in test source
This means Batch 0 implementation planning must treat the DB `batch_tests` mapping as stale and use a dependency query as the source of truth.
## Context Findings
### Command Findings
- `batch show 0 --db porting.db`: Batch metadata exists but has no mapped tests.
- `batch list --db porting.db`: total mapped tests across batches = 2087.
- `report summary --db porting.db`: deferred tests = 2640.
- Gap: `2640 - 2087 = 553` deferred tests are currently unassigned to any batch.
### Source-of-Truth Query (Batch 0 Candidates)
```sql
WITH implementable AS (
SELECT t.id
FROM unit_tests t
WHERE t.status = 'deferred'
AND EXISTS (
SELECT 1
FROM dependencies d
JOIN features f ON f.id = d.target_id AND d.target_type = 'feature'
WHERE d.source_type = 'unit_test' AND d.source_id = t.id
)
AND NOT EXISTS (
SELECT 1
FROM dependencies d
JOIN features f ON f.id = d.target_id AND d.target_type = 'feature'
WHERE d.source_type = 'unit_test'
AND d.source_id = t.id
AND f.status NOT IN ('verified', 'complete', 'n_a')
)
)
SELECT COUNT(*) FROM implementable; -- 553
```
### Candidate Distribution (553 tests)
All 553 are in module `server` and currently land in `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/*.Impltests.cs`.
Largest classes:
- `JetStreamEngineTests` (89)
- `MonitoringHandlerTests` (76)
- `MqttHandlerTests` (56)
- `LeafNodeHandlerTests` (47)
- `NatsConsumerTests` (35)
- `JwtProcessorTests` (28)
- `RouteHandlerTests` (25)
## Approaches
### Approach A: Use Existing `batch_tests` Mapping Only
- Work only from `batch show 0`.
- Pros: Uses existing CLI flow exactly.
- Cons: Batch 0 is empty today, so this blocks all useful work and misses 553 valid candidates.
### Approach B: Query-Driven Batch 0 (No DB Mapping Changes)
- Treat the SQL dependency query as authoritative for execution order/status updates.
- Pros: No batch table mutation; can start immediately.
- Cons: `batch show 0` remains misleading; no built-in batch completion visibility.
### Approach C (Recommended): Query-Driven Execution + Batch 0 Mapping Reconciliation
- First, reconcile `batch_tests` for Batch 0 from dependency query.
- Then implement tests by class waves using a manifest generated from the same query.
- Pros: Correct batch visibility, supports `batch show 0`, preserves workflow consistency.
- Cons: One upfront tracker-maintenance step.
## Recommended Design
### 1. Batch 0 Candidate Manifest
Create a reproducible manifest (CSV/SQL output) containing:
- `test_id`
- `dotnet_class`
- `dotnet_method`
- `go_file`
- `go_method`
The manifest is regenerated from DB each session to avoid stale lists.
### 2. Batch Mapping Reconciliation
Insert missing candidate tests into `batch_tests(batch_id=0)` only when absent.
Do not reassign already-mapped tests in other batches.
### 3. Implementation Model
Port tests class-by-class in `ImplBacklog` files first (lowest-risk), then optionally move mature tests to domain folders later.
Per test workflow:
1. Open Go test body at mapped `go_file + go_method`.
2. Classify test as:
- `implementable-unit` (no running server/cluster required)
- `runtime-blocked` (needs real server/cluster infra) -> keep `deferred`, add note
3. Replace placeholder/assertion-only body with behavior-faithful xUnit 3 + Shouldly test.
4. Run method and class filters.
5. Update tracker status only after green test evidence.
### 4. Wave Sequencing
Order by lowest infra risk first:
1. Accounts/Auth/JWT/Options/reload
2. Routes/Gateways/Leaf/WebSocket
3. Events/Monitoring/MsgTrace/Server core
4. JetStream lightweight deterministic tests
5. JetStream heavy + MQTT + concurrency edge cases
### 5. Verification Gates
- Gate 1: Method-level run passes after each test port.
- Gate 2: Class-level run passes before status updates.
- Gate 3: Wave-level run passes before commit.
- Gate 4: End-of-batch query returns zero remaining implementable deferred tests.
## Error Handling and Risk Controls
- If Go test requires unavailable runtime infra, do not force a brittle port; keep `deferred` with explicit note.
- If ported test reveals feature regression, stop and create a feature-fix branch of work before marking test `verified`.
- Keep DB updates idempotent: only update statuses for IDs proven by passing runs.
- Avoid mass `test batch-update ... verified` without per-class pass evidence.
## Success Criteria
- Batch 0 mapping reflects true implementable set (or documented query-based equivalent).
- All implementable unit tests in the set are ported and passing.
- Runtime-blocked tests remain deferred with explicit reason notes.
- `report summary` and Batch 0 progress trend down deterministically after each wave.
## Non-Goals
- No server/cluster integration infrastructure build-out in this batch.
- No refactoring of production server code except where discovered test failures require follow-up work.
- No attempt to complete non-Batch-0 deferred tests.

444
docs/plans/2026-02-27-batch-0-implementable-tests-plan.md Normal file
View File

@@ -0,0 +1,444 @@
# Batch 0 Implementable Tests Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify all currently implementable Batch 0 deferred tests (553 candidates) whose feature dependencies are already verified, while keeping runtime-blocked tests deferred with explicit notes.
**Architecture:** Use a query-driven manifest as the Batch 0 source of truth, then execute class-by-class test porting in `ImplBacklog` files with tight red/green verification loops. After each class wave, update only the proven test IDs in PortTracker (`verified` for passing ports, `deferred` with notes for runtime-blocked tests). Keep production code unchanged unless a test reveals a real feature regression.
**Tech Stack:** .NET 10, xUnit 3, Shouldly, NSubstitute, SQLite (`porting.db`), PortTracker CLI
**Design doc:** `docs/plans/2026-02-27-batch-0-implementable-tests-design.md`
---
### Task 1: Generate Batch 0 Manifest and Reconcile Mapping
**Files:**
- Modify: `porting.db` (`batch_tests`, `implementation_batches`)
- Create: `/tmp/batch0-implementable-tests.csv`
**Step 1: Generate manifest**
Run:
```bash
sqlite3 -header -csv porting.db "
WITH implementable AS (
SELECT t.id, t.dotnet_class, t.dotnet_method, t.go_file, t.go_method
FROM unit_tests t
WHERE t.status='deferred'
AND EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
)
AND NOT EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
AND f.status NOT IN ('verified','complete','n_a')
)
)
SELECT * FROM implementable ORDER BY dotnet_class, id;
" > /tmp/batch0-implementable-tests.csv
```
Expected: CSV created with ~553 rows.
**Step 2: Reconcile Batch 0 mapping**
Run:
```bash
sqlite3 porting.db "
INSERT INTO batch_tests (batch_id, test_id)
SELECT 0, t.id
FROM unit_tests t
WHERE t.status='deferred'
AND EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
)
AND NOT EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
AND f.status NOT IN ('verified','complete','n_a')
)
AND NOT EXISTS (SELECT 1 FROM batch_tests bt WHERE bt.test_id=t.id);
UPDATE implementation_batches
SET test_count=(SELECT COUNT(*) FROM batch_tests WHERE batch_id=0)
WHERE id=0;
"
```
Expected: `batch show 0` now reports Batch 0 tests > 0.
**Step 3: Verify mapping**
Run:
```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 0 --db porting.db
```
Expected: Batch 0 includes mapped tests.
**Step 4: Commit**
```bash
git add porting.db
git commit -m "chore(porttracker): reconcile batch 0 implementable test mappings"
```
---
### Task 2: Apply Porting Template to One Seed Test Per Class
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/*.Impltests.cs` (one method per class)
- Read: `golang/nats-server/server/*_test.go`
**Step 1: Replace one placeholder method body with behavior-faithful assertions**
Template:
```csharp
[Fact] // T:<id>
public void <DotnetMethod>()
{
// Arrange (from Go test setup)
var subject = /* translated setup */;
// Act
var result = /* invocation under test */;
// Assert (exact behavioral intent from Go)
result.ShouldBe(/* expected */);
}
```
**Step 2: Run method-level filter**
Run:
```bash
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.<ClassName>.<DotnetMethod>"
```
Expected: fail first if placeholder previously masked behavior; pass after fixing setup/assertions.
**Step 3: Repeat for all 28 classes as seed tests**
Expected: each class has at least one real, non-placeholder port.
**Step 4: Commit**
```bash
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog
git commit -m "test(batch0): replace seed placeholders with behavior ports across classes"
```
---
### Task 3: Port Account/Auth/Options Cluster (Core Low-Infra)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ServerOptionsTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CertificateStoreWindowsTests.Impltests.cs`
**Step 1: Port methods for these classes from Go tests**
Reference Go files:
- `golang/nats-server/server/accounts_test.go`
- `golang/nats-server/server/auth_test.go`
- `golang/nats-server/server/auth_callout_test.go`
- `golang/nats-server/server/jwt_test.go`
- `golang/nats-server/server/opts_test.go`
- `golang/nats-server/server/reload_test.go`
**Step 2: Tag runtime-blocked cases**
If Go method requires live server/cluster/process orchestration, keep deferred and add explicit reason note later.
**Step 3: Run per-class tests**
Run one class at a time:
```bash
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.AccountTests"
```
Expected: class green before status updates.
**Step 4: Update statuses for proven IDs**
Run:
```bash
IDS=$(sqlite3 porting.db "
WITH implementable AS (
SELECT t.id,t.dotnet_class
FROM unit_tests t
WHERE t.status='deferred'
AND EXISTS (SELECT 1 FROM dependencies d JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id)
AND NOT EXISTS (SELECT 1 FROM dependencies d JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
AND f.status NOT IN ('verified','complete','n_a'))
)
SELECT group_concat(id, ',') FROM implementable
WHERE dotnet_class IN ('AccountTests','AuthHandlerTests','AuthCalloutTests','JwtProcessorTests','ServerOptionsTests','ConfigReloaderTests','CertificateStoreWindowsTests');
")
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "$IDS" --set-status verified --db porting.db --execute
```
Expected: IDs verified only after passing evidence.
**Step 5: Commit**
```bash
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
git commit -m "test(batch0): port account/auth/options implementable tests"
```
---
### Task 4: Port Routing and Edge Transport Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs`
**Step 1: Port deterministic tests from Go route/gateway/leaf/websocket files**
**Step 2: Keep runtime-only topology tests deferred with notes**
**Step 3: Run class filters**
```bash
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.RouteHandlerTests"
```
Expected: pass per class.
**Step 4: Update statuses for passing IDs and commit**
Use `test batch-update --ids "$IDS" --set-status verified --execute` for this class set, then commit.
---
### Task 5: Port Server Introspection and Events Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs`
**Step 1: Replace placeholder assertions with behavior-based assertions from Go tests**
**Step 2: Validate log/JSON/monitor payload contracts with strict Shouldly checks**
**Step 3: Run class filters and update statuses**
Expected: each class green before DB status updates.
**Step 4: Commit**
```bash
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
git commit -m "test(batch0): port monitoring/events/server implementable tests"
```
---
### Task 6: Port Concurrency and Process-Control Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`
**Step 1: Port race-safe deterministic cases only**
Use timeout-safe async assertions and avoid nondeterministic sleep-based checks.
**Step 2: Run each class repeatedly**
Run:
```bash
for i in 1 2 3; do
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConcurrencyTests1";
done
```
Expected: stable pass across repeated runs.
**Step 3: Update statuses + commit**
Mark passing IDs verified; keep flaky/runtime-bound cases deferred with notes.
---
### Task 7: Port JetStream Small/Deterministic Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamVersioningTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamTpmTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/StorageEngineTests.Impltests.cs`
**Step 1: Port only no-cluster/no-live-server methods**
**Step 2: For cluster-required tests, keep deferred and annotate notes**
Example note: `deferred: requires running JetStream cluster/runtime`.
**Step 3: Run class filters; update statuses; commit**
Expected: deterministic classes pass; blocked cases stay deferred with reasons.
---
### Task 8: Port JetStream Consumer/Engine Heavy Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
**Step 1: Work test-by-test with strict triage**
For each method:
1. Port and run method-level filter.
2. If runtime-bound, revert method to deferred state marker and keep status deferred.
3. If unit-portable, finish port and keep in verified candidate list.
**Step 2: Run full class filters**
Expected: passing set is stable.
**Step 3: Update statuses and commit**
Use class-specific ID lists from manifest to avoid accidental status drift.
---
### Task 9: Port MQTT Cluster
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttExternalTests.Impltests.cs`
**Step 1: Port unit-level parser/options/utility tests**
**Step 2: Defer broker-runtime tests with explicit notes**
**Step 3: Run class filters; update statuses; commit**
Expected: no silent skips; each deferred item has rationale.
---
### Task 10: Final Batch 0 Verification and Status Closure
**Files:**
- Modify: `porting.db`
- Optional update: `reports/current.md` (via report generation script)
**Step 1: Run targeted regression sweep for all touched Batch 0 classes**
Run:
```bash
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
Expected: all touched classes pass.
**Step 2: Validate remaining implementable deferred count**
Run:
```bash
sqlite3 porting.db "
WITH implementable AS (
SELECT t.id
FROM unit_tests t
WHERE t.status='deferred'
AND EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
)
AND NOT EXISTS (
SELECT 1 FROM dependencies d
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
WHERE d.source_type='unit_test' AND d.source_id=t.id
AND f.status NOT IN ('verified','complete','n_a')
)
)
SELECT COUNT(*) FROM implementable;
"
```
Expected: `0` (or only explicitly documented runtime-blocked exceptions).
**Step 3: Verify Batch 0 visibility**
Run:
```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 0 --db porting.db
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
```
Expected: Batch 0 test count reduced/closed and summary reflects updates.
**Step 4: Final commit**
```bash
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/current.md
git commit -m "test(batch0): port implementable deferred tests and close batch 0"
```
---
## Batch 0 Working Set (Current)
- Total implementable candidates: **553**
- Primary files: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/*.Impltests.cs`
- Highest-volume classes:
- `JetStreamEngineTests` (89)
- `MonitoringHandlerTests` (76)
- `MqttHandlerTests` (56)
- `LeafNodeHandlerTests` (47)
- `NatsConsumerTests` (35)
## TDD Guardrails
- Never bulk-mark IDs `verified` before class-level green runs.
- Preserve `T:<id>` comments to keep DB traceability.
- If production behavior mismatches Go spec, stop and open feature-fix work rather than weakening assertions.
- Keep tests deterministic: avoid timing races and process-global side effects where possible.