natsnet/docs/plans/2026-03-01-deferred-integration-tests-design.md

# Deferred Integration Tests Design

**Date**: 2026-03-01
**Status**: Draft
**Scope**: 884 deferred tests across 34 Go test files

## Context

After completing all deferred feature batches (42-47), 884 integration/cluster tests remain deferred. These tests need a running server — cluster creation, multi-server coordination, consumer/producer sessions, monitoring endpoints. The .NET server has all method bodies ported but may not fully boot yet.

Current state: 87.3% complete (6057/6942 items). This work targets bringing the test count from 2066 verified to ~2950 verified.

## Decisions

- **Implementation depth**: Port full Go test logic to idiomatic C#. Tests compile and are structurally correct. Each test has a `Skip` guard so it skips gracefully if the server can't boot.
- **Test harness**: Build shared infrastructure first (Batch 48). All subsequent batches use it.
- **Execution**: Parallel Claude Code Sonnet agents with `isolation: "worktree"`, all 12 test batches concurrent after harness merges.
- **PortTracker updates**: Run audit after each batch merge to promote tests to verified.
- **Test framework**: xUnit + Shouldly + NSubstitute (existing standards). NATS.Client.Core for client connections (already in integration test project).

## Test Harness Design (Batch 48)

Target project: `dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/`

### Helpers/TestServerHelper.cs

Server lifecycle management. Mirrors Go `RunServer`, `RunBasicJetStreamServer`, etc.

```csharp
internal static class TestServerHelper
{
    // Create and start a server with given options
    static (NatsServer Server, ServerOptions Opts) RunServer(ServerOptions opts);

    // Create JS-enabled server with temp store directory
    static NatsServer RunBasicJetStreamServer(ITestOutputHelper output);

    // Parse config file and create server
    static (NatsServer Server, ServerOptions Opts) RunServerWithConfig(string configFile);

    // Check if server can boot (for Skip guards)
    static bool CanBoot();

    // Find available TCP port
    static int GetFreePort();

    // Create temp directory with auto-cleanup
    static string CreateTempDir(string prefix);
}
```

### Helpers/TestCluster.cs

Multi-server cluster infrastructure. Mirrors Go `cluster` struct.

```csharp
internal sealed class TestCluster : IDisposable
{
    NatsServer[] Servers { get; }
    ServerOptions[] Options { get; }
    string Name { get; }

    // Factory methods
    static TestCluster CreateJetStreamCluster(int numServers, string name);
    static TestCluster CreateJetStreamClusterWithTemplate(string template, int numServers, string name);

    // Wait helpers
    void WaitOnClusterReady();
    void WaitOnLeader();
    NatsServer WaitOnStreamLeader(string account, string stream);
    NatsServer WaitOnConsumerLeader(string account, string stream, string consumer);

    // Accessors
    NatsServer StreamLeader(string account, string stream);
    NatsServer ConsumerLeader(string account, string stream, string consumer);
    NatsServer Leader();
    NatsServer RandomServer();
    NatsServer ServerByName(string name);

    // Lifecycle
    void StopAll();
    void RestartAll();
    void Dispose(); // Shutdown all servers
}
```

### Helpers/TestSuperCluster.cs

Multi-cluster with gateways. Mirrors Go `supercluster` struct.

```csharp
internal sealed class TestSuperCluster : IDisposable
{
    TestCluster[] Clusters { get; }

    static TestSuperCluster CreateJetStreamSuperCluster(int numPerCluster, int numClusters);

    NatsServer Leader();
    NatsServer RandomServer();
    NatsServer ServerByName(string name);
    void WaitOnLeader();
    void WaitOnStreamLeader(string account, string stream);
    TestCluster ClusterForName(string name);
    void Dispose();
}
```

### Helpers/NatsTestClient.cs

Client connection wrapper using NATS.Client.Core.

```csharp
internal static class NatsTestClient
{
    // Connect with test defaults (error handler, name, reconnect)
    static INatsConnection Connect(string url, NatsOpts? opts = null);

    // Connect to specific server
    static INatsConnection ConnectToServer(NatsServer server, NatsOpts? opts = null);
}
```

### Helpers/CheckHelper.cs

Retry/polling helpers. Mirrors Go `checkFor`.

```csharp
internal static class CheckHelper
{
    // Retry check function until it succeeds or timeout
    static void CheckFor(TimeSpan timeout, TimeSpan interval, Func<Exception?> check);

    // Verify all servers have route connections
    static void CheckClusterFormed(params NatsServer[] servers);

    // Wait for leaf node connection count
    static void CheckLeafNodeConnectedCount(NatsServer server, int expected);
}
```

### Helpers/ConfigHelper.cs

Config templating and file management.

```csharp
internal static class ConfigHelper
{
    // Standard cluster config template (mirrors Go jsClusterTempl)
    const string JsClusterTemplate = "...";

    // Standard supercluster config template
    const string JsSuperClusterTemplate = "...";

    // Write config content to temp file
    static string CreateConfigFile(string content);
}
```

### Test Base Class

```csharp
[Trait("Category", "Integration")]
public abstract class IntegrationTestBase : IDisposable
{
    protected ITestOutputHelper Output { get; }

    protected IntegrationTestBase(ITestOutputHelper output)
    {
        Skip.If(!TestServerHelper.CanBoot(), "Server cannot boot");
        Output = output;
    }

    public virtual void Dispose() { }
}
```

## Test Porting Conventions

| Go Pattern | C# Pattern |
|-----------|------------|
| `TestFoo(t *testing.T)` | `public void Foo_ShouldSucceed()` or `public async Task Foo_ShouldSucceed()` |
| `t.Fatal("msg")` | `Assert.Fail("msg")` or Shouldly assertion |
| `t.Errorf("fmt", args)` | `result.ShouldBe(expected)` |
| `defer s.Shutdown()` | `using var server = TestServerHelper.RunBasicJetStreamServer(...)` |
| `natsConnect(t, url)` | `NatsTestClient.Connect(url)` |
| `checkFor(t, 10*time.Second, ...)` | `CheckHelper.CheckFor(TimeSpan.FromSeconds(10), ...)` |
| `c := createJetStreamClusterExplicit(t, "R3", 3)` | `using var c = TestCluster.CreateJetStreamCluster(3, "R3")` |
| `//go:build !race` | `[Trait("Category", "NoRace")]` |
| `t.Skip("reason")` | `Skip.If(true, "reason")` |

## Batch Structure

### Batch 48: Test Harness (0 tests, foundation)

Creates all helper files above. No tests ported. Must merge before other batches.

### Batch 49: JetStream Core (126 tests)

- `jetstream_test.go` (70 tests) — snapshots, mirrors, sources, basic JS operations
- `jetstream_consumer_test.go` (56 tests) — consumer state, delivery, ack

**Target files**: `IntegrationTests/JetStream/JetStreamTests.cs`, `IntegrationTests/JetStream/JetStreamConsumerTests.cs`

### Batch 50: JetStream Cluster 1 (118 tests)

- `jetstream_cluster_1_test.go` — cluster formation, stream replication, leader election

**Target file**: `IntegrationTests/JetStream/JetStreamCluster1Tests.cs`

### Batch 51: JetStream Cluster 2 (106 tests)

- `jetstream_cluster_2_test.go` — consumer replication, failover, recovery

**Target file**: `IntegrationTests/JetStream/JetStreamCluster2Tests.cs`

### Batch 52: JetStream Cluster 3 (82 tests)

- `jetstream_cluster_3_test.go` — advanced cluster scenarios

**Target file**: `IntegrationTests/JetStream/JetStreamCluster3Tests.cs`

### Batch 53: JetStream Cluster 4 (75 tests)

- `jetstream_cluster_4_test.go` — busy streams, consumption patterns

**Target file**: `IntegrationTests/JetStream/JetStreamCluster4Tests.cs`

### Batch 54: MQTT (78 tests)

- `mqtt_test.go` (77 tests) — MQTT protocol, sessions, QoS, retained messages
- `mqtt_ex_test_test.go` (1 test)

**Target file**: `IntegrationTests/Mqtt/MqttTests.cs`

### Batch 55: NoRace (75 tests)

- `norace_1_test.go` (51 tests) — concurrency tests without race detector
- `norace_2_test.go` (24 tests)

**Target files**: `IntegrationTests/NoRace/NoRace1Tests.cs`, `IntegrationTests/NoRace/NoRace2Tests.cs`

### Batch 56: Reload + Auth (66 tests)

- `reload_test.go` (44 tests) — config reload
- `accounts_test.go` (5 tests) — route mappings
- `auth_callout_test.go` (5 tests) — external auth
- `jwt_test.go` (11 tests) — JWT validation
- `opts_test.go` (1 test)

**Target files**: `IntegrationTests/Config/ReloadTests.cs`, `IntegrationTests/Auth/AuthIntegrationTests.cs`

### Batch 57: SuperCluster + LeafNode (53 tests)

- `jetstream_super_cluster_test.go` (36 tests) — multi-cluster with gateways
- `jetstream_leafnode_test.go` (3 tests) — JS over leaf nodes
- `leafnode_test.go` (14 tests) — leaf node connections

**Target files**: `IntegrationTests/JetStream/JetStreamSuperClusterTests.cs`, `IntegrationTests/LeafNode/LeafNodeTests.cs`

### Batch 58: JetStream Misc (55 tests)

- `jetstream_batching_test.go` (26 tests)
- `jetstream_benchmark_test.go` (11 tests)
- `jetstream_jwt_test.go` (9 tests)
- `jetstream_versioning_test.go` (2 tests)
- `jetstream_meta_benchmark_test.go` (2 tests)
- `jetstream_cluster_long_test.go` (4 tests)
- `jetstream_sourcing_scaling_test.go` (1 test)

**Target files**: `IntegrationTests/JetStream/JetStreamBatchingIntegrationTests.cs`, `IntegrationTests/JetStream/JetStreamMiscTests.cs`

### Batch 59: Events + Monitor + Misc (50 tests)

- `events_test.go` (13 tests)
- `monitor_test.go` (15 tests)
- `msgtrace_test.go` (7 tests)
- `routes_test.go` (5 tests)
- `filestore_test.go` (6 tests)
- `server_test.go` (1 test)
- `memstore_test.go` (1 test)
- `gateway_test.go` (1 test)
- `websocket_test.go` (1 test)

**Target files**: `IntegrationTests/Events/EventsTests.cs`, `IntegrationTests/Monitor/MonitorIntegrationTests.cs`, `IntegrationTests/MiscTests.cs`

## Execution Plan

### Wave 1

- **Batch 48** (Test Harness) — must complete first

### Wave 2 (all parallel, after Wave 1)

- **Batches 49-59** (12 batches, 884 tests total)

## Post-Execution

After all batches merge:
1. Run `dotnet build dotnet/` to confirm compilation
2. Run `dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/` to confirm tests skip gracefully
3. Run `dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/` to confirm no regressions
4. Reset deferred test statuses in porting.db, re-run audit
5. Generate final report

## Expected Outcome

- Tests: 2066 + 884 = 2950 verified (all unit_tests accounted for)
- 884 tests will compile but Skip until server runtime boots
- Harness ready for future integration testing once server starts