398 lines
17 KiB
Markdown
398 lines
17 KiB
Markdown
# Batch 0 Implementable Tests Implementation Plan
|
|
|
|
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
|
|
|
|
**Goal:** Port and verify all currently implementable Batch 0 deferred tests (553 candidates) whose feature dependencies are already verified, while keeping runtime-blocked tests deferred with explicit notes.
|
|
|
|
**Architecture:** Use a query-driven manifest as the Batch 0 source of truth, then execute class-by-class test porting in `ImplBacklog` files with tight red/green verification loops. After each class wave, update only the proven test IDs in PortTracker (`verified` for passing ports, `deferred` with notes for runtime-blocked tests). Keep production code unchanged unless a test reveals a real feature regression.
|
|
|
|
**Tech Stack:** .NET 10, xUnit 3, Shouldly, NSubstitute, SQLite (`porting.db`), PortTracker CLI
|
|
|
|
**Design doc:** `docs/plans/2026-02-27-batch-0-implementable-tests-design.md`
|
|
|
|
---
|
|
|
|
## MANDATORY VERIFICATION PROTOCOL
|
|
|
|
> **This section is NON-NEGOTIABLE. Every task MUST follow this protocol. Skipping any step is a plan violation.**
|
|
|
|
### What Counts as a Real Test
|
|
|
|
A test is **real** (eligible for `verified` status) ONLY if ALL of these are true:
|
|
|
|
1. **Has Arrange/Act/Assert** — the method body contains setup, an invocation of production code, and at least one Shouldly assertion on the result
|
|
2. **Calls production code** — the test invokes at least one method from `ZB.MOM.NatsNet.Server.*` (not just test helpers or constants)
|
|
3. **Has meaningful assertions** — uses `ShouldBe`, `ShouldContain`, `ShouldThrow`, `ShouldNotBeNull`, etc. on values derived from the Act step
|
|
4. **Is NOT a stub** — does not contain `throw new NotImplementedException`, `Assert.True(true)`, `Assert.Pass()`, `// TODO`, or an empty body
|
|
|
|
### Stub Detection Check (REQUIRED after every class)
|
|
|
|
After porting all tests in a class, run this stub scan **before** running the tests:
|
|
|
|
```bash
|
|
# Scan for stub patterns — ANY match means the test is NOT real
|
|
grep -n -E "(NotImplementedException|Assert\.True\(true\)|Assert\.Pass|\.ShouldBe\(true\);$|// TODO|// PLACEHOLDER)" \
|
|
dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/<ClassName>.Impltests.cs
|
|
```
|
|
|
|
**If any matches are found**: fix them or reclassify the test as deferred. Do NOT proceed to run tests until all stubs are eliminated.
|
|
|
|
### Assertion Count Check (REQUIRED after every class)
|
|
|
|
```bash
|
|
# Count Shouldly assertions per test file — must average >= 1.5 per test method
|
|
grep -c "\.Should" dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/<ClassName>.Impltests.cs
|
|
grep -c "\[Fact\]\|\[Theory\]" dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/<ClassName>.Impltests.cs
|
|
```
|
|
|
|
**If assertion count / test count < 1.5**: review tests for shallow coverage. Tests with only one trivial assertion (e.g., `result.ShouldNotBeNull()`) on a complex method need strengthening.
|
|
|
|
### Per-Test Verification Loop
|
|
|
|
For EVERY test method ported (not per-class, not per-wave — PER TEST):
|
|
|
|
1. **Read the Go test** — open the Go source at the line from `test show <id>` and understand what behavior it verifies
|
|
2. **Write the C# port** — translate the behavioral intent (not line-for-line)
|
|
3. **Run the single test** and capture output:
|
|
```bash
|
|
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
|
|
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.<ClassName>.<MethodName>" \
|
|
--verbosity normal 2>&1 | tail -20
|
|
```
|
|
4. **Verify the output** shows `Passed: 1, Failed: 0` — if it shows `Passed: 0` the test was not discovered (name mismatch)
|
|
5. **Only after green**: add the test ID to the verified-candidates list for this class
|
|
|
|
### Class-Level Gate (REQUIRED before any status updates)
|
|
|
|
After all tests in a class are individually verified:
|
|
|
|
1. **Run the full class**:
|
|
```bash
|
|
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
|
|
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.<ClassName>" \
|
|
--verbosity normal 2>&1 | tail -5
|
|
```
|
|
2. **Parse the summary line** — extract `Passed: N, Failed: M, Skipped: S`
|
|
3. **Verify**: `Failed` MUST be `0`, and `Passed` MUST equal the number of tests you ported
|
|
4. **If mismatch**: investigate before proceeding — do not mark anything verified
|
|
|
|
### Status Update Protocol
|
|
|
|
- **NEVER use `test batch-update` with more than 15 IDs at once** — process in chunks of max 15
|
|
- **NEVER mark a test verified unless you have captured and reviewed its individual pass output**
|
|
- **For each batch-update command**, list the specific test IDs and the evidence (class run output showing pass count)
|
|
- **Deferred tests MUST have a reason** — when keeping a test deferred, update its notes with why (e.g., "requires running NATS server", "needs cluster topology")
|
|
|
|
### Checkpoint Protocol (REQUIRED between tasks)
|
|
|
|
After completing each task (Tasks 2-9), before starting the next:
|
|
|
|
1. Run the full test suite to check for regressions:
|
|
```bash
|
|
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal 2>&1 | tail -10
|
|
```
|
|
2. Report the total pass/fail/skip counts
|
|
3. If any pre-existing tests broke, fix them before continuing
|
|
4. Commit the work for this task before starting the next
|
|
|
|
---
|
|
|
|
### Task 1: Reconcile Batch 0 Mapping (**DONE** — commit ee0827d)
|
|
|
|
Batch 0 now has 553 tests mapped. This task is complete.
|
|
|
|
---
|
|
|
|
### Task 2: Port Account/Auth/Options Cluster (89 tests)
|
|
|
|
**Max scope: 7 classes, ~89 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` (18 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthHandlerTests.Impltests.cs` (3 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs` (17 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` (28 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ServerOptionsTests.Impltests.cs` (3 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` (19 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CertificateStoreWindowsTests.Impltests.cs` (1 test)
|
|
|
|
**For EACH class, follow this sequence:**
|
|
|
|
1. Read all Go test sources for the class (use `test show <id>` to get file+line for each test)
|
|
2. Port each test method following the **Per-Test Verification Loop** above
|
|
3. Run **Stub Detection Check** on the class file
|
|
4. Run **Assertion Count Check** on the class file
|
|
5. Run **Class-Level Gate** and capture output
|
|
6. Record verified IDs (only tests with individual + class-level green evidence)
|
|
7. Update status in chunks of max 15:
|
|
```bash
|
|
dotnet run --project tools/NatsNet.PortTracker -- \
|
|
test batch-update --ids "<max 15 comma-separated IDs>" --set-status verified --db porting.db --execute
|
|
```
|
|
8. For any test that requires live server/cluster infra, keep deferred and note the reason
|
|
|
|
**After ALL 7 classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port account/auth/options implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Port Routing and Edge Transport Cluster (111 tests)
|
|
|
|
**Max scope: 4 classes, ~111 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` (25 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` (21 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` (47 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` (18 tests)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2 (per-test loop, stub check, assertion check, class gate, chunked status updates).
|
|
|
|
**After ALL 4 classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port routing/gateway/leaf/websocket implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Port Server Introspection and Events Cluster (128 tests)
|
|
|
|
**Max scope: 4 classes, ~128 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs` (21 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` (76 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs` (20 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` (11 tests)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2.
|
|
|
|
**Special note for MonitoringHandlerTests (76 tests)**: This is the second-largest class. Process in sub-batches of ~20 tests. After every 20 tests, run the class-level gate to confirm cumulative pass count is growing and no regressions.
|
|
|
|
**After ALL 4 classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port monitoring/events/server implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Port Concurrency Cluster (20 tests)
|
|
|
|
**Max scope: 2 classes, ~20 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` (18 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` (2 tests)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2, with this addition:
|
|
|
|
**Stability check** — run each class 3 times to detect flaky tests:
|
|
```bash
|
|
for i in 1 2 3; do
|
|
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
|
|
--filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.<ClassName>" \
|
|
--verbosity normal 2>&1 | tail -5
|
|
done
|
|
```
|
|
Only mark verified if all 3 runs pass with the same count.
|
|
|
|
**After BOTH classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port concurrency implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Port JetStream Small/Deterministic Cluster (18 tests)
|
|
|
|
**Max scope: 6 classes, ~18 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamVersioningTests.Impltests.cs` (2 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs` (1 test)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` (4 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamTpmTests.Impltests.cs` (5 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` (6 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/StorageEngineTests.Impltests.cs` (5 tests — note: 5 already verified in prior work)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2.
|
|
|
|
**After ALL 6 classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port jetstream small/deterministic implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: Port JetStream Consumer/Engine Heavy Cluster (125 tests)
|
|
|
|
**Max scope: 3 classes, ~125 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` (35 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` (89 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` (1 test)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2.
|
|
|
|
**Special note for JetStreamEngineTests (89 tests)**: This is the largest class. Process in sub-batches of ~15 tests. After every 15 tests:
|
|
- Run the stub detection check on the whole file
|
|
- Run the class-level gate and confirm cumulative pass count
|
|
- If pass count plateaus (same count after adding 15 tests), STOP — tests are likely stubs
|
|
|
|
**After ALL 3 classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port jetstream consumer/engine implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: Port MQTT Cluster (57 tests)
|
|
|
|
**Max scope: 2 classes, ~57 tests. Work ONE CLASS at a time.**
|
|
|
|
**Files:**
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` (56 tests)
|
|
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttExternalTests.Impltests.cs` (1 test)
|
|
|
|
**For EACH class**: follow the same sequence as Task 2.
|
|
|
|
**Special note for MqttHandlerTests (56 tests)**: Process in sub-batches of ~15 tests with stub detection after each sub-batch.
|
|
|
|
**After BOTH classes**: run **Checkpoint Protocol**, then commit:
|
|
```bash
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db
|
|
git commit -m "test(batch0): port mqtt implementable tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Final Batch 0 Verification and Status Closure
|
|
|
|
**Files:**
|
|
- Modify: `porting.db`
|
|
- Generate: `reports/current.md` (via report generation script)
|
|
|
|
**Step 1: Full regression sweep**
|
|
|
|
```bash
|
|
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal 2>&1 | tail -10
|
|
```
|
|
|
|
Capture and report total pass/fail/skip. **Failed MUST be 0.**
|
|
|
|
**Step 2: Stub audit across ALL ImplBacklog files**
|
|
|
|
```bash
|
|
grep -rn -E "(NotImplementedException|Assert\.True\(true\)|Assert\.Pass|// TODO|// PLACEHOLDER)" \
|
|
dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/
|
|
```
|
|
|
|
**If any matches remain**: fix them or reclassify the affected tests as deferred. Do NOT close the batch with stubs present.
|
|
|
|
**Step 3: Validate remaining implementable deferred count**
|
|
|
|
```bash
|
|
sqlite3 porting.db "
|
|
WITH implementable AS (
|
|
SELECT t.id
|
|
FROM unit_tests t
|
|
WHERE t.status='deferred'
|
|
AND EXISTS (
|
|
SELECT 1 FROM dependencies d
|
|
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
|
|
WHERE d.source_type='unit_test' AND d.source_id=t.id
|
|
)
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM dependencies d
|
|
JOIN features f ON f.id=d.target_id AND d.target_type='feature'
|
|
WHERE d.source_type='unit_test' AND d.source_id=t.id
|
|
AND f.status NOT IN ('verified','complete','n_a')
|
|
)
|
|
)
|
|
SELECT COUNT(*) FROM implementable;
|
|
"
|
|
```
|
|
|
|
Expected: `0` (or only explicitly documented runtime-blocked exceptions with notes).
|
|
|
|
**Step 4: Verify Batch 0 visibility**
|
|
|
|
```bash
|
|
dotnet run --project tools/NatsNet.PortTracker -- batch show 0 --db porting.db
|
|
dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
|
|
```
|
|
|
|
**Step 5: Generate report and final commit**
|
|
|
|
```bash
|
|
./reports/generate-report.sh
|
|
git add dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog porting.db reports/
|
|
git commit -m "test(batch0): complete batch 0 implementable tests verification"
|
|
```
|
|
|
|
---
|
|
|
|
## Batch 0 Working Set (Current)
|
|
|
|
- Total implementable candidates: **553**
|
|
- Primary files: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/*.Impltests.cs`
|
|
- Highest-volume classes:
|
|
- `JetStreamEngineTests` (89)
|
|
- `MonitoringHandlerTests` (76)
|
|
- `MqttHandlerTests` (56)
|
|
- `LeafNodeHandlerTests` (47)
|
|
- `NatsConsumerTests` (35)
|
|
|
|
## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)
|
|
|
|
These rules exist because previous attempts degraded into stub-writing after ~50 tests. **Every rule is mandatory.**
|
|
|
|
### Forbidden Patterns
|
|
|
|
The following patterns in a test body mean the test is a STUB, not a real port. **Tests containing these MUST NOT be marked verified:**
|
|
|
|
| Pattern | Why it's a stub |
|
|
|---------|----------------|
|
|
| `throw new NotImplementedException()` | Placeholder, tests nothing |
|
|
| `Assert.True(true)` | Always passes, tests nothing |
|
|
| `Assert.Pass()` | Always passes, tests nothing |
|
|
| `result.ShouldBe(true);` (alone) | Likely a lazy replacement for real assertion |
|
|
| `// TODO` or `// PLACEHOLDER` in body | Incomplete work marker |
|
|
| Empty method body `{ }` | Tests nothing |
|
|
| Only `ShouldNotBeNull()` on a constructor | Trivial, not behavioral |
|
|
|
|
### Verification Evidence Requirements
|
|
|
|
Before marking ANY test as `verified`, you MUST have:
|
|
|
|
1. **Individual test output** showing `Passed: 1` for that specific test method
|
|
2. **Class-level output** showing total `Passed: N` matches the number of tests you ported in that class
|
|
3. **Stub scan output** showing zero matches for the class file
|
|
4. **Assertion count** showing >= 1.5 assertions per test on average
|
|
|
|
### Hard Limits
|
|
|
|
- **Max 15 test IDs per `batch-update` call** — forces you to think about what you're marking
|
|
- **Max 1 class per status-update cycle** — no cross-class bulk updates
|
|
- **Mandatory commit after each task** — prevents loss of work and creates review points
|
|
- **If pass count plateaus after adding tests, STOP and investigate** — this means tests are stubs or not being discovered
|
|
|
|
### If You Get Stuck
|
|
|
|
If a test requires infrastructure you can't provide (running server, cluster, etc.):
|
|
|
|
1. Keep the test as `deferred` — do NOT write a stub that trivially passes
|
|
2. Add a comment in the test: `// DEFERRED: requires <specific reason>`
|
|
3. Move on to the next test
|
|
4. This is the CORRECT behavior — marking deferred with a reason is better than a fake pass
|
|
|