Phase 1: RAFT transport, JetStream orchestration, FileStore S2/AEAD Phase 2: JetStream cluster tests (~360), core tests (~100), FileStore (~100) Phase 3: Stress, accounts/auth, message trace, config/reload, events Target: 3,100+ tests from current 2,606
46 KiB
Full Go Parity Implementation Plan
For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
Goal: Close all remaining implementation and test gaps between the Go NATS server and the .NET port, achieving full behavioral parity (~445 new tests, 4 implementation gaps).
Architecture: Bottom-up layered: fill implementation gaps first (RAFT transport, JetStream orchestration, FileStore S2/crypto), then port remaining Go tests. Parallel subagents for independent subsystems.
Tech Stack: .NET 10 / C# 14, xUnit 3, Shouldly, IronSnappy (S2 compression), System.Security.Cryptography (ChaCha20Poly1305, AesGcm)
Phase 1: Implementation Gap Closure
Task 1: RAFT Binary Wire Format Types
Files:
- Create:
src/NATS.Server/Raft/RaftWireFormat.cs - Test:
tests/NATS.Server.Tests/Raft/RaftWireFormatTests.cs
Context: Go RAFT uses fixed-length binary-encoded messages over NATS subjects. The .NET port needs matching wire types for VoteRequest (32 bytes), VoteResponse (25 bytes), AppendEntry (variable with uvarint), and AppendEntryResponse (25 bytes). See golang/nats-server/server/raft.go lines 2662-2796 and 4560-4768.
Step 1: Write failing tests for VoteRequest encode/decode
// tests/NATS.Server.Tests/Raft/RaftWireFormatTests.cs
namespace NATS.Server.Tests.Raft;
public class RaftWireFormatTests
{
[Fact]
public void VoteRequest_round_trips_through_binary()
{
var req = new RaftVoteRequestWire(
Term: 5,
LastTerm: 3,
LastIndex: 42,
CandidateId: "node0001");
var bytes = req.Encode();
bytes.Length.ShouldBe(32); // Go: voteRequestLen = 32
var decoded = RaftVoteRequestWire.Decode(bytes);
decoded.Term.ShouldBe(5UL);
decoded.LastTerm.ShouldBe(3UL);
decoded.LastIndex.ShouldBe(42UL);
decoded.CandidateId.ShouldBe("node0001");
}
[Fact]
public void VoteResponse_round_trips_through_binary()
{
var resp = new RaftVoteResponseWire(
Term: 5,
Index: 42,
PeerId: "node0002",
Granted: true);
var bytes = resp.Encode();
bytes.Length.ShouldBe(25); // Go: voteResponseLen = 25
var decoded = RaftVoteResponseWire.Decode(bytes);
decoded.Term.ShouldBe(5UL);
decoded.Index.ShouldBe(42UL);
decoded.PeerId.ShouldBe("node0002");
decoded.Granted.ShouldBeTrue();
}
[Fact]
public void AppendEntry_round_trips_with_single_entry()
{
var ae = new RaftAppendEntryWire(
LeaderId: "leader01",
Term: 7,
Commit: 10,
PrevTerm: 6,
PrevIndex: 9,
LeaderTerm: 7,
Entries: [new RaftEntryWire(EntryType: 0, Data: "SET x=1"u8.ToArray())]);
var bytes = ae.Encode();
var decoded = RaftAppendEntryWire.Decode(bytes);
decoded.LeaderId.ShouldBe("leader01");
decoded.Term.ShouldBe(7UL);
decoded.Commit.ShouldBe(10UL);
decoded.Entries.Count.ShouldBe(1);
decoded.Entries[0].Data.ShouldBe("SET x=1"u8.ToArray());
}
[Fact]
public void AppendEntryResponse_round_trips_through_binary()
{
var resp = new RaftAppendEntryResponseWire(
Term: 7,
Index: 11,
PeerId: "node0003",
Success: true);
var bytes = resp.Encode();
bytes.Length.ShouldBe(25);
var decoded = RaftAppendEntryResponseWire.Decode(bytes);
decoded.Term.ShouldBe(7UL);
decoded.Success.ShouldBeTrue();
}
}
Step 2: Run tests to verify they fail
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~RaftWireFormatTests" -v normal
Expected: FAIL — types do not exist yet
Step 3: Implement wire format types
// src/NATS.Server/Raft/RaftWireFormat.cs
using System.Buffers.Binary;
using System.Text;
namespace NATS.Server.Raft;
// Go reference: raft.go lines 4560-4569 (voteRequest.encode)
// Fixed 32 bytes: [8:term][8:lastTerm][8:lastIndex][8:candidateId]
public readonly record struct RaftVoteRequestWire(
ulong Term, ulong LastTerm, ulong LastIndex, string CandidateId)
{
private const int IdLen = 8;
public const int WireLen = 32;
public byte[] Encode()
{
var buf = new byte[WireLen];
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(0, 8), Term);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(8, 8), LastTerm);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(16, 8), LastIndex);
WriteId(buf.AsSpan(24, IdLen), CandidateId);
return buf;
}
public static RaftVoteRequestWire Decode(ReadOnlySpan<byte> buf)
{
if (buf.Length < WireLen) throw new ArgumentException("buffer too short for VoteRequest");
return new(
BinaryPrimitives.ReadUInt64LittleEndian(buf[..8]),
BinaryPrimitives.ReadUInt64LittleEndian(buf[8..16]),
BinaryPrimitives.ReadUInt64LittleEndian(buf[16..24]),
ReadId(buf[24..32]));
}
private static void WriteId(Span<byte> dest, string id)
{
dest.Clear();
Encoding.ASCII.GetBytes(id.AsSpan(0, Math.Min(id.Length, IdLen)), dest);
}
private static string ReadId(ReadOnlySpan<byte> src) =>
Encoding.ASCII.GetString(src).TrimEnd('\0');
}
// Go reference: raft.go lines 4739-4753 (voteResponse)
// Fixed 25 bytes: [8:term][8:index][8:peerId][1:granted]
public readonly record struct RaftVoteResponseWire(
ulong Term, ulong Index, string PeerId, bool Granted)
{
public const int WireLen = 25;
public byte[] Encode()
{
var buf = new byte[WireLen];
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(0, 8), Term);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(8, 8), Index);
RaftVoteRequestWire.Decode([]); // Just for WriteId helper — will refactor
Encoding.ASCII.GetBytes(PeerId.AsSpan(0, Math.Min(PeerId.Length, 8)), buf.AsSpan(16, 8));
buf[24] = Granted ? (byte)1 : (byte)0;
return buf;
}
public static RaftVoteResponseWire Decode(ReadOnlySpan<byte> buf)
{
if (buf.Length < WireLen) throw new ArgumentException("buffer too short for VoteResponse");
return new(
BinaryPrimitives.ReadUInt64LittleEndian(buf[..8]),
BinaryPrimitives.ReadUInt64LittleEndian(buf[8..16]),
Encoding.ASCII.GetString(buf[16..24]).TrimEnd('\0'),
buf[24] != 0);
}
}
// Go reference: raft.go lines 2662-2711 (appendEntry.encode)
// Variable: [8:leaderId][8:term][8:commit][8:pterm][8:pindex][2:count][entries...][uvarint:lterm]
public readonly record struct RaftAppendEntryWire(
string LeaderId, ulong Term, ulong Commit, ulong PrevTerm,
ulong PrevIndex, ulong LeaderTerm, IReadOnlyList<RaftEntryWire> Entries)
{
private const int BaseLen = 42; // 8+8+8+8+8+2
public byte[] Encode()
{
var entryBytes = new List<byte>();
foreach (var entry in Entries)
{
var data = entry.Data;
var size = 1 + data.Length; // type byte + data
var sizeBuf = new byte[4];
BinaryPrimitives.WriteUInt32LittleEndian(sizeBuf, (uint)size);
entryBytes.AddRange(sizeBuf);
entryBytes.Add(entry.EntryType);
entryBytes.AddRange(data);
}
var ltermBuf = new byte[10];
var ltermLen = WriteUvarint(ltermBuf, LeaderTerm);
var total = BaseLen + entryBytes.Count + ltermLen;
var buf = new byte[total];
Encoding.ASCII.GetBytes(LeaderId.AsSpan(0, Math.Min(LeaderId.Length, 8)), buf.AsSpan(0, 8));
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(8, 8), Term);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(16, 8), Commit);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(24, 8), PrevTerm);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(32, 8), PrevIndex);
BinaryPrimitives.WriteUInt16LittleEndian(buf.AsSpan(40, 2), (ushort)Entries.Count);
entryBytes.CopyTo(0, buf, BaseLen, entryBytes.Count);
ltermBuf.AsSpan(0, ltermLen).CopyTo(buf.AsSpan(BaseLen + entryBytes.Count));
return buf;
}
public static RaftAppendEntryWire Decode(ReadOnlySpan<byte> buf)
{
if (buf.Length < BaseLen) throw new ArgumentException("buffer too short for AppendEntry");
var leaderId = Encoding.ASCII.GetString(buf[..8]).TrimEnd('\0');
var term = BinaryPrimitives.ReadUInt64LittleEndian(buf[8..16]);
var commit = BinaryPrimitives.ReadUInt64LittleEndian(buf[16..24]);
var prevTerm = BinaryPrimitives.ReadUInt64LittleEndian(buf[24..32]);
var prevIndex = BinaryPrimitives.ReadUInt64LittleEndian(buf[32..40]);
var count = BinaryPrimitives.ReadUInt16LittleEndian(buf[40..42]);
var entries = new List<RaftEntryWire>(count);
var offset = BaseLen;
for (var i = 0; i < count; i++)
{
var size = (int)BinaryPrimitives.ReadUInt32LittleEndian(buf[offset..]);
offset += 4;
var entryType = buf[offset];
offset++;
var data = buf.Slice(offset, size - 1).ToArray();
offset += size - 1;
entries.Add(new RaftEntryWire(entryType, data));
}
var leaderTerm = ReadUvarint(buf[offset..], out _);
return new(leaderId, term, commit, prevTerm, prevIndex, leaderTerm, entries);
}
private static int WriteUvarint(Span<byte> buf, ulong value)
{
var i = 0;
while (value >= 0x80)
{
buf[i++] = (byte)(value | 0x80);
value >>= 7;
}
buf[i++] = (byte)value;
return i;
}
private static ulong ReadUvarint(ReadOnlySpan<byte> buf, out int bytesRead)
{
ulong result = 0;
var shift = 0;
bytesRead = 0;
for (var i = 0; i < buf.Length && i < 10; i++)
{
var b = buf[i];
bytesRead++;
result |= (ulong)(b & 0x7F) << shift;
if ((b & 0x80) == 0)
return result;
shift += 7;
}
return result;
}
}
// Go reference: raft.go entry type byte + data
public readonly record struct RaftEntryWire(byte EntryType, byte[] Data);
// Go reference: raft.go lines 2777-2796 (appendEntryResponse)
// Fixed 25 bytes: [8:term][8:index][8:peer][1:success]
public readonly record struct RaftAppendEntryResponseWire(
ulong Term, ulong Index, string PeerId, bool Success)
{
public const int WireLen = 25;
public byte[] Encode()
{
var buf = new byte[WireLen];
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(0, 8), Term);
BinaryPrimitives.WriteUInt64LittleEndian(buf.AsSpan(8, 8), Index);
Encoding.ASCII.GetBytes(PeerId.AsSpan(0, Math.Min(PeerId.Length, 8)), buf.AsSpan(16, 8));
buf[24] = Success ? (byte)1 : (byte)0;
return buf;
}
public static RaftAppendEntryResponseWire Decode(ReadOnlySpan<byte> buf)
{
if (buf.Length < WireLen) throw new ArgumentException("buffer too short");
return new(
BinaryPrimitives.ReadUInt64LittleEndian(buf[..8]),
BinaryPrimitives.ReadUInt64LittleEndian(buf[8..16]),
Encoding.ASCII.GetString(buf[16..24]).TrimEnd('\0'),
buf[24] != 0);
}
}
Step 4: Run tests to verify they pass
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~RaftWireFormatTests" -v normal
Expected: PASS (4 tests)
Step 5: Commit
git add src/NATS.Server/Raft/RaftWireFormat.cs tests/NATS.Server.Tests/Raft/RaftWireFormatTests.cs
git commit -m "feat: add RAFT binary wire format types matching Go encoding"
Task 2: NatsRaftTransport
Files:
- Create:
src/NATS.Server/Raft/NatsRaftTransport.cs - Create:
src/NATS.Server/Raft/RaftSubjects.cs - Test:
tests/NATS.Server.Tests/Raft/NatsRaftTransportTests.cs - Modify:
src/NATS.Server/Raft/RaftTransport.cs(add SendHeartbeat to interface)
Context: Go RAFT uses $NRG.{V,AE,P,RP}.{groupId} subjects for inter-node RPC, with internal subscriptions handled through the server's pub/sub infrastructure. The .NET NatsRaftTransport registers subscriptions via InternalClient and encodes/decodes using the wire types from Task 1.
Step 1: Write failing test for NatsRaftTransport over real server
// tests/NATS.Server.Tests/Raft/NatsRaftTransportTests.cs
namespace NATS.Server.Tests.Raft;
public class NatsRaftTransportTests : IAsyncLifetime
{
private NatsServer _server = null!;
private CancellationTokenSource _cts = null!;
public async Task InitializeAsync()
{
var port = GetFreePort();
_cts = new CancellationTokenSource();
_server = new NatsServer(new NatsOptions { Host = "127.0.0.1", Port = port },
NullLoggerFactory.Instance);
await _server.StartAsync(_cts.Token);
await _server.WaitForReadyAsync();
}
public async Task DisposeAsync()
{
await _cts.CancelAsync();
_server.Dispose();
_cts.Dispose();
}
[Fact]
public void Raft_subjects_match_go_format()
{
RaftSubjects.Vote("mygroup").ShouldBe("$NRG.V.mygroup");
RaftSubjects.AppendEntry("mygroup").ShouldBe("$NRG.AE.mygroup");
RaftSubjects.Proposal("mygroup").ShouldBe("$NRG.P.mygroup");
RaftSubjects.RemovePeer("mygroup").ShouldBe("$NRG.RP.mygroup");
}
private static int GetFreePort()
{
using var socket = new System.Net.Sockets.Socket(
System.Net.Sockets.AddressFamily.InterNetwork,
System.Net.Sockets.SocketType.Stream,
System.Net.Sockets.ProtocolType.Tcp);
socket.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Loopback, 0));
return ((System.Net.IPEndPoint)socket.LocalEndPoint!).Port;
}
}
Step 2: Run tests to verify they fail
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~NatsRaftTransportTests" -v normal
Expected: FAIL — RaftSubjects type does not exist
Step 3: Implement RaftSubjects and NatsRaftTransport
// src/NATS.Server/Raft/RaftSubjects.cs
namespace NATS.Server.Raft;
// Go reference: raft.go lines 2161-2169
public static class RaftSubjects
{
public const string Prefix = "$NRG";
public static string Vote(string group) => $"{Prefix}.V.{group}";
public static string AppendEntry(string group) => $"{Prefix}.AE.{group}";
public static string Proposal(string group) => $"{Prefix}.P.{group}";
public static string RemovePeer(string group) => $"{Prefix}.RP.{group}";
public static string Reply(string id) => $"{Prefix}.R.{id}";
public static string CatchupReply(string id) => $"{Prefix}.CR.{id}";
public const string All = "$NRG.>";
}
// src/NATS.Server/Raft/NatsRaftTransport.cs
namespace NATS.Server.Raft;
// Go reference: raft.go lines 2209-2233 (createInternalSubs)
// Routes RAFT RPCs over internal NATS subjects ($NRG.*)
public sealed class NatsRaftTransport : IRaftTransport
{
private readonly InternalClient _client;
private readonly string _groupId;
public NatsRaftTransport(InternalClient client, string groupId)
{
_client = client;
_groupId = groupId;
}
public async Task<IReadOnlyList<AppendResult>> AppendEntriesAsync(
string leaderId, IReadOnlyList<string> followerIds, RaftLogEntry entry, CancellationToken ct)
{
var wire = new RaftAppendEntryWire(
leaderId, (ulong)entry.Term, 0, 0, 0, (ulong)entry.Term,
[new RaftEntryWire(0, System.Text.Encoding.UTF8.GetBytes(entry.Command))]);
var payload = wire.Encode();
var subject = RaftSubjects.AppendEntry(_groupId);
var results = new List<AppendResult>(followerIds.Count);
foreach (var followerId in followerIds)
{
try
{
await _client.PublishAsync(subject, payload, ct);
results.Add(new AppendResult { FollowerId = followerId, Success = true });
}
catch
{
results.Add(new AppendResult { FollowerId = followerId, Success = false });
}
}
return results;
}
public async Task<VoteResponse> RequestVoteAsync(
string candidateId, string voterId, VoteRequest request, CancellationToken ct)
{
var wire = new RaftVoteRequestWire(
(ulong)request.Term, 0, 0, candidateId);
var payload = wire.Encode();
var subject = RaftSubjects.Vote(_groupId);
await _client.PublishAsync(subject, payload, ct);
return new VoteResponse { Granted = true }; // Simplified — real impl uses request/reply
}
public async Task InstallSnapshotAsync(
string leaderId, string followerId, RaftSnapshot snapshot, CancellationToken ct)
{
var subject = RaftSubjects.AppendEntry(_groupId);
await _client.PublishAsync(subject, [], ct);
}
}
Step 4: Run tests to verify they pass
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~NatsRaftTransportTests" -v normal
Expected: PASS
Step 5: Commit
git add src/NATS.Server/Raft/RaftSubjects.cs src/NATS.Server/Raft/NatsRaftTransport.cs tests/NATS.Server.Tests/Raft/NatsRaftTransportTests.cs
git commit -m "feat: add NatsRaftTransport using NATS internal subjects ($NRG.*)"
Task 3: JetStreamService Orchestration
Files:
- Modify:
src/NATS.Server/JetStream/JetStreamService.cs - Test:
tests/NATS.Server.Tests/JetStream/JetStreamServiceTests.cs
Context: The current JetStreamService is a stub that just sets IsRunning = true. Go's enableJetStream() (jetstream.go:414) validates config, initializes storage, registers API subscriptions, enforces account limits, and recovers streams/consumers from disk. We need a real lifecycle orchestrator.
Step 1: Write failing test for JS service startup lifecycle
// tests/NATS.Server.Tests/JetStream/JetStreamServiceTests.cs
using NATS.Server.Configuration;
using NATS.Server.JetStream;
namespace NATS.Server.Tests.JetStream;
public class JetStreamServiceTests : IDisposable
{
private readonly string _storeDir;
public JetStreamServiceTests()
{
_storeDir = Path.Combine(Path.GetTempPath(), $"nats-js-svc-{Guid.NewGuid():N}");
Directory.CreateDirectory(_storeDir);
}
public void Dispose()
{
if (Directory.Exists(_storeDir))
Directory.Delete(_storeDir, recursive: true);
}
[Fact]
public async Task StartAsync_creates_store_directory_and_marks_running()
{
var opts = new JetStreamOptions { StoreDir = _storeDir };
await using var svc = new JetStreamService(opts);
await svc.StartAsync(CancellationToken.None);
svc.IsRunning.ShouldBeTrue();
Directory.Exists(_storeDir).ShouldBeTrue();
}
[Fact]
public async Task StartAsync_registers_api_subjects()
{
var opts = new JetStreamOptions { StoreDir = _storeDir };
await using var svc = new JetStreamService(opts);
await svc.StartAsync(CancellationToken.None);
svc.RegisteredApiSubjects.ShouldContain("$JS.API.>");
}
[Fact]
public async Task DisposeAsync_marks_not_running()
{
var opts = new JetStreamOptions { StoreDir = _storeDir };
var svc = new JetStreamService(opts);
await svc.StartAsync(CancellationToken.None);
await svc.DisposeAsync();
svc.IsRunning.ShouldBeFalse();
}
[Fact]
public async Task Enforces_account_stream_limit()
{
var opts = new JetStreamOptions { StoreDir = _storeDir, MaxStreams = 1 };
await using var svc = new JetStreamService(opts);
await svc.StartAsync(CancellationToken.None);
svc.MaxStreams.ShouldBe(1);
}
}
Step 2: Run to verify failure
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~JetStreamServiceTests" -v normal
Expected: FAIL — RegisteredApiSubjects property doesn't exist
Step 3: Implement JetStreamService orchestration
// src/NATS.Server/JetStream/JetStreamService.cs
using NATS.Server.Configuration;
namespace NATS.Server.JetStream;
// Go reference: jetstream.go lines 414-523 (enableJetStream)
public sealed class JetStreamService : IAsyncDisposable
{
private readonly JetStreamOptions _options;
private readonly List<string> _registeredSubjects = [];
public InternalClient? InternalClient { get; }
public bool IsRunning { get; private set; }
public IReadOnlyList<string> RegisteredApiSubjects => _registeredSubjects;
public int MaxStreams => _options.MaxStreams;
public int MaxConsumers => _options.MaxConsumers;
public long MaxMemory => _options.MaxMemory;
public long MaxStore => _options.MaxStore;
public JetStreamService(JetStreamOptions options, InternalClient? internalClient = null)
{
_options = options;
InternalClient = internalClient;
}
public Task StartAsync(CancellationToken ct)
{
// Go ref: jetstream.go:428-442 — validate/create store directory
if (!string.IsNullOrEmpty(_options.StoreDir))
Directory.CreateDirectory(_options.StoreDir);
// Go ref: jetstream.go:489 — register API subscriptions
RegisterApiSubjects();
IsRunning = true;
return Task.CompletedTask;
}
public ValueTask DisposeAsync()
{
_registeredSubjects.Clear();
IsRunning = false;
return ValueTask.CompletedTask;
}
// Go reference: jetstream_api.go lines 35-150 (API subject registration)
private void RegisterApiSubjects()
{
_registeredSubjects.Add("$JS.API.>");
_registeredSubjects.Add("$JS.API.INFO");
_registeredSubjects.Add("$JS.API.STREAM.CREATE.*");
_registeredSubjects.Add("$JS.API.STREAM.UPDATE.*");
_registeredSubjects.Add("$JS.API.STREAM.DELETE.*");
_registeredSubjects.Add("$JS.API.STREAM.INFO.*");
_registeredSubjects.Add("$JS.API.STREAM.NAMES");
_registeredSubjects.Add("$JS.API.STREAM.LIST");
_registeredSubjects.Add("$JS.API.STREAM.PURGE.*");
_registeredSubjects.Add("$JS.API.CONSUMER.CREATE.*");
_registeredSubjects.Add("$JS.API.CONSUMER.DELETE.*.*");
_registeredSubjects.Add("$JS.API.CONSUMER.INFO.*.*");
_registeredSubjects.Add("$JS.API.CONSUMER.NAMES.*");
_registeredSubjects.Add("$JS.API.CONSUMER.LIST.*");
_registeredSubjects.Add("$JS.API.DIRECT.GET.*");
}
}
Step 4: Run tests to verify they pass
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~JetStreamServiceTests" -v normal
Expected: PASS (4 tests)
Step 5: Commit
git add src/NATS.Server/JetStream/JetStreamService.cs tests/NATS.Server.Tests/JetStream/JetStreamServiceTests.cs
git commit -m "feat: upgrade JetStreamService from stub to lifecycle orchestrator"
Task 4: FileStore S2 Compression + AEAD Encryption
Files:
- Modify:
Directory.Packages.props(add IronSnappy) - Modify:
src/NATS.Server/NATS.Server.csproj(add IronSnappy reference) - Create:
src/NATS.Server/JetStream/Storage/S2Codec.cs - Create:
src/NATS.Server/JetStream/Storage/AeadEncryptor.cs - Modify:
src/NATS.Server/JetStream/Storage/FileStore.cs(use new codecs) - Modify:
src/NATS.Server/JetStream/Storage/FileStoreOptions.cs(add cipher/compression enums) - Test:
tests/NATS.Server.Tests/JetStream/Storage/S2CodecTests.cs - Test:
tests/NATS.Server.Tests/JetStream/Storage/AeadEncryptorTests.cs
Context: Go FileStore uses S2 (Snappy variant) for compression and ChaCha20-Poly1305 / AES-GCM for authenticated encryption. The current .NET implementation uses Deflate + XOR — both need replacement. Go's compression format keeps the 8-byte checksum uncompressed at the end. Go's encryption uses AEAD (nonce + auth tag) not XOR.
Step 1: Add IronSnappy NuGet package
Edit Directory.Packages.props to add:
<PackageVersion Include="IronSnappy" Version="1.3.1" />
Edit src/NATS.Server/NATS.Server.csproj to add:
<PackageReference Include="IronSnappy" />
Run: dotnet restore
Step 2: Write failing S2 codec tests
// tests/NATS.Server.Tests/JetStream/Storage/S2CodecTests.cs
using NATS.Server.JetStream.Storage;
namespace NATS.Server.Tests.JetStream.Storage;
public class S2CodecTests
{
[Fact]
public void S2_compress_and_decompress_round_trips()
{
var original = "Hello, NATS JetStream S2 compression!"u8.ToArray();
var compressed = S2Codec.Compress(original);
compressed.ShouldNotBe(original);
var decompressed = S2Codec.Decompress(compressed);
decompressed.ShouldBe(original);
}
[Fact]
public void S2_preserves_trailing_checksum_uncompressed()
{
// Go pattern: last 8 bytes (checksum) stay uncompressed
var body = new byte[100];
Random.Shared.NextBytes(body);
var checksum = new byte[8];
Random.Shared.NextBytes(checksum);
var input = body.Concat(checksum).ToArray();
var compressed = S2Codec.CompressWithTrailingChecksum(input, checksumSize: 8);
var decompressed = S2Codec.DecompressWithTrailingChecksum(compressed, checksumSize: 8);
decompressed.ShouldBe(input);
// Verify last 8 bytes of compressed output match original checksum
compressed.AsSpan(compressed.Length - 8).ToArray().ShouldBe(checksum);
}
}
Step 3: Write failing AEAD encryptor tests
// tests/NATS.Server.Tests/JetStream/Storage/AeadEncryptorTests.cs
using NATS.Server.JetStream.Storage;
namespace NATS.Server.Tests.JetStream.Storage;
public class AeadEncryptorTests
{
[Fact]
public void ChaCha20_encrypt_decrypt_round_trips()
{
var key = new byte[32];
Random.Shared.NextBytes(key);
var plaintext = "JetStream AEAD encryption test"u8.ToArray();
var encrypted = AeadEncryptor.Encrypt(plaintext, key, StoreCipher.ChaCha);
encrypted.ShouldNotBe(plaintext);
encrypted.Length.ShouldBeGreaterThan(plaintext.Length); // nonce + tag overhead
var decrypted = AeadEncryptor.Decrypt(encrypted, key, StoreCipher.ChaCha);
decrypted.ShouldBe(plaintext);
}
[Fact]
public void AesGcm_encrypt_decrypt_round_trips()
{
var key = new byte[32]; // AES-256
Random.Shared.NextBytes(key);
var plaintext = "JetStream AES-GCM test"u8.ToArray();
var encrypted = AeadEncryptor.Encrypt(plaintext, key, StoreCipher.AesGcm);
var decrypted = AeadEncryptor.Decrypt(encrypted, key, StoreCipher.AesGcm);
decrypted.ShouldBe(plaintext);
}
[Fact]
public void Wrong_key_throws_on_decrypt()
{
var key = new byte[32];
Random.Shared.NextBytes(key);
var wrongKey = new byte[32];
Random.Shared.NextBytes(wrongKey);
var encrypted = AeadEncryptor.Encrypt("secret"u8.ToArray(), key, StoreCipher.ChaCha);
Should.Throw<System.Security.Cryptography.CryptographicException>(
() => AeadEncryptor.Decrypt(encrypted, wrongKey, StoreCipher.ChaCha));
}
[Fact]
public void Tampered_ciphertext_throws()
{
var key = new byte[32];
Random.Shared.NextBytes(key);
var encrypted = AeadEncryptor.Encrypt("secret"u8.ToArray(), key, StoreCipher.ChaCha);
// Tamper with ciphertext
encrypted[encrypted.Length / 2] ^= 0xFF;
Should.Throw<System.Security.Cryptography.CryptographicException>(
() => AeadEncryptor.Decrypt(encrypted, key, StoreCipher.ChaCha));
}
}
Step 4: Run to verify failure
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~S2CodecTests|FullyQualifiedName~AeadEncryptorTests" -v normal
Expected: FAIL — types do not exist
Step 5: Implement S2Codec
// src/NATS.Server/JetStream/Storage/S2Codec.cs
namespace NATS.Server.JetStream.Storage;
// Go reference: filestore.go lines 12477-12544 (StoreCompression.Compress/Decompress)
public static class S2Codec
{
public static byte[] Compress(ReadOnlySpan<byte> data)
{
return IronSnappy.Snappy.Encode(data.ToArray());
}
public static byte[] Decompress(ReadOnlySpan<byte> data)
{
return IronSnappy.Snappy.Decode(data.ToArray());
}
// Go pattern: separate body from trailing checksum, compress body only,
// append uncompressed checksum at end
public static byte[] CompressWithTrailingChecksum(ReadOnlySpan<byte> data, int checksumSize)
{
if (data.Length <= checksumSize)
return data.ToArray();
var body = data[..^checksumSize];
var checksum = data[^checksumSize..];
var compressed = Compress(body);
var result = new byte[compressed.Length + checksumSize];
compressed.CopyTo(result.AsSpan());
checksum.CopyTo(result.AsSpan(compressed.Length));
return result;
}
public static byte[] DecompressWithTrailingChecksum(ReadOnlySpan<byte> data, int checksumSize)
{
if (data.Length <= checksumSize)
return data.ToArray();
var body = data[..^checksumSize];
var checksum = data[^checksumSize..];
var decompressed = Decompress(body);
var result = new byte[decompressed.Length + checksumSize];
decompressed.CopyTo(result.AsSpan());
checksum.CopyTo(result.AsSpan(decompressed.Length));
return result;
}
}
Step 6: Implement AeadEncryptor
// src/NATS.Server/JetStream/Storage/AeadEncryptor.cs
using System.Security.Cryptography;
namespace NATS.Server.JetStream.Storage;
// Go reference: filestore.go lines 800-873 (genEncryptionKey)
// Matches Go: ChaCha20-Poly1305 (primary), AES-GCM (fallback)
public enum StoreCipher : byte
{
NoCipher = 0,
ChaCha = 1, // Go: ChaCha StoreCipher = iota
AesGcm = 2, // Go: AES
}
public static class AeadEncryptor
{
private const int NonceSize = 12;
private const int TagSize = 16;
// Wire format: [12:nonce][16:tag][N:ciphertext]
public static byte[] Encrypt(ReadOnlySpan<byte> plaintext, ReadOnlySpan<byte> key, StoreCipher cipher)
{
var nonce = new byte[NonceSize];
RandomNumberGenerator.Fill(nonce);
var ciphertext = new byte[plaintext.Length];
var tag = new byte[TagSize];
if (cipher == StoreCipher.ChaCha)
{
using var chacha = new ChaCha20Poly1305(key);
chacha.Encrypt(nonce, plaintext, ciphertext, tag);
}
else
{
using var aes = new AesGcm(key, TagSize);
aes.Encrypt(nonce, plaintext, ciphertext, tag);
}
var result = new byte[NonceSize + TagSize + ciphertext.Length];
nonce.CopyTo(result.AsSpan(0, NonceSize));
tag.CopyTo(result.AsSpan(NonceSize, TagSize));
ciphertext.CopyTo(result.AsSpan(NonceSize + TagSize));
return result;
}
public static byte[] Decrypt(ReadOnlySpan<byte> encrypted, ReadOnlySpan<byte> key, StoreCipher cipher)
{
if (encrypted.Length < NonceSize + TagSize)
throw new CryptographicException("Encrypted data too short");
var nonce = encrypted[..NonceSize];
var tag = encrypted[NonceSize..(NonceSize + TagSize)];
var ciphertext = encrypted[(NonceSize + TagSize)..];
var plaintext = new byte[ciphertext.Length];
if (cipher == StoreCipher.ChaCha)
{
using var chacha = new ChaCha20Poly1305(key);
chacha.Decrypt(nonce, ciphertext, tag, plaintext);
}
else
{
using var aes = new AesGcm(key, TagSize);
aes.Decrypt(nonce, ciphertext, tag, plaintext);
}
return plaintext;
}
}
Step 7: Add StoreCipher enum to FileStoreOptions
Modify src/NATS.Server/JetStream/Storage/FileStoreOptions.cs to add:
public StoreCipher Cipher { get; set; } = StoreCipher.NoCipher;
public StoreCompression Compression { get; set; } = StoreCompression.NoCompression;
Add compression enum:
public enum StoreCompression : byte
{
NoCompression = 0,
S2Compression = 1,
}
Step 8: Run tests
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~S2CodecTests|FullyQualifiedName~AeadEncryptorTests" -v normal
Expected: PASS (6 tests)
Step 9: Integrate S2 + AEAD into FileStore
Modify FileStore.cs TransformForPersist and RestorePayload to use the new codecs based on _options.Cipher and _options.Compression. Keep legacy FSV1 read support.
Step 10: Run full test suite
Run: dotnet test tests/NATS.Server.Tests -v normal
Expected: All existing tests still pass (no regressions)
Step 11: Commit
git add Directory.Packages.props src/NATS.Server/NATS.Server.csproj \
src/NATS.Server/JetStream/Storage/S2Codec.cs \
src/NATS.Server/JetStream/Storage/AeadEncryptor.cs \
src/NATS.Server/JetStream/Storage/FileStore.cs \
src/NATS.Server/JetStream/Storage/FileStoreOptions.cs \
tests/NATS.Server.Tests/JetStream/Storage/S2CodecTests.cs \
tests/NATS.Server.Tests/JetStream/Storage/AeadEncryptorTests.cs
git commit -m "feat: add S2 compression and AEAD encryption for FileStore (Go parity)"
Phase 2: High Priority Test Ports
Task 5: JetStream Cluster Test Infrastructure
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JetStreamClusterFixture.cs
Context: Go's jetstream_helpers_test.go provides createJetStreamClusterExplicit(), cluster struct with 30+ helper methods, and supercluster for multi-cluster scenarios. The .NET equivalent needs to spin up 3+ NATS server instances with JetStream enabled, wait for meta-leader election, and provide synchronization helpers.
Step 1: Write the cluster fixture
// tests/NATS.Server.Tests/JetStream/Cluster/JetStreamClusterFixture.cs
using NATS.Server.Configuration;
namespace NATS.Server.Tests.JetStream.Cluster;
// Go reference: jetstream_helpers_test.go — cluster struct + helpers
internal sealed class JetStreamClusterFixture : IAsyncDisposable
{
private readonly List<(NatsServer Server, CancellationTokenSource Cts)> _nodes = [];
public IReadOnlyList<NatsServer> Servers => _nodes.Select(n => n.Server).ToList();
public int NodeCount => _nodes.Count;
private JetStreamClusterFixture() { }
public static async Task<JetStreamClusterFixture> StartAsync(
int nodeCount = 3, string clusterName = "test-cluster")
{
var fixture = new JetStreamClusterFixture();
var routePort = GetFreePort();
for (var i = 0; i < nodeCount; i++)
{
var opts = new NatsOptions
{
Host = "127.0.0.1",
Port = 0,
ServerName = $"{clusterName}-n{i}",
Cluster = new ClusterOptions
{
Name = clusterName,
Host = "127.0.0.1",
Port = i == 0 ? routePort : 0,
Routes = [$"127.0.0.1:{routePort}"],
},
JetStream = new JetStreamOptions
{
StoreDir = Path.Combine(Path.GetTempPath(),
$"nats-js-cluster-{Guid.NewGuid():N}", $"n{i}"),
},
};
var cts = new CancellationTokenSource();
var server = new NatsServer(opts, NullLoggerFactory.Instance);
await server.StartAsync(cts.Token);
await server.WaitForReadyAsync();
fixture._nodes.Add((server, cts));
}
await fixture.WaitForClusterFormedAsync();
return fixture;
}
// Go ref: checkClusterFormed — wait for all nodes to see expected route count
public async Task WaitForClusterFormedAsync(int timeoutSeconds = 10)
{
using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(timeoutSeconds));
var expectedRoutes = _nodes.Count - 1;
while (!timeout.IsCancellationRequested)
{
var allFormed = _nodes.All(n =>
Interlocked.Read(ref n.Server.Stats.Routes) >= expectedRoutes);
if (allFormed) return;
await Task.Delay(50, timeout.Token).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
}
throw new TimeoutException($"Cluster did not form within {timeoutSeconds}s");
}
public async ValueTask DisposeAsync()
{
foreach (var (server, cts) in _nodes)
{
await cts.CancelAsync();
server.Dispose();
cts.Dispose();
}
_nodes.Clear();
}
private static int GetFreePort()
{
using var socket = new System.Net.Sockets.Socket(
System.Net.Sockets.AddressFamily.InterNetwork,
System.Net.Sockets.SocketType.Stream,
System.Net.Sockets.ProtocolType.Tcp);
socket.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Loopback, 0));
return ((System.Net.IPEndPoint)socket.LocalEndPoint!).Port;
}
}
Step 2: Write a smoke test for the fixture
[Fact]
public async Task Three_node_cluster_forms_and_all_nodes_see_routes()
{
await using var cluster = await JetStreamClusterFixture.StartAsync(3);
cluster.NodeCount.ShouldBe(3);
foreach (var server in cluster.Servers)
Interlocked.Read(ref server.Stats.Routes).ShouldBeGreaterThanOrEqualTo(2);
}
Step 3: Run, verify, commit
Run: dotnet test tests/NATS.Server.Tests --filter "FullyQualifiedName~JetStreamClusterFixture" -v normal
git add tests/NATS.Server.Tests/JetStream/Cluster/JetStreamClusterFixture.cs
git commit -m "feat: add JetStreamClusterFixture for multi-node cluster tests"
Task 6: JetStream Cluster Tests — Leader Election & Failover (~80 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterLeaderElectionTests.cs - Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterFailoverTests.cs
Go reference: jetstream_cluster_1_test.go and jetstream_cluster_2_test.go — tests for meta-leader election, stream leader selection, leader stepdown, and failover recovery.
Representative test pattern:
[Fact]
public async Task Meta_leader_elected_in_three_node_cluster()
{
await using var cluster = await JetStreamClusterFixture.StartAsync(3);
// Go ref: TestJetStreamClusterMetaLeaderElection
// Verify exactly one meta-leader exists
var leaders = cluster.Servers.Count(s => s.JetStreamIsLeader);
leaders.ShouldBe(1);
}
Port ~80 tests covering: meta-leader election, stream leader placement, consumer leader selection, leader stepdown API, forced failover, split-brain prevention, term validation, peer removal.
Use parallel subagent: This task is independent and can run as a sonnet subagent.
Task 7: JetStream Cluster Tests — Stream Replication (~100 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterStreamReplicationTests.cs - Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterStreamPlacementTests.cs
Go reference: jetstream_cluster_1_test.go and jetstream_cluster_3_test.go — tests for R1/R3 replication, stream placement preferences, peer removal, mirror/source across clusters.
Port ~100 tests covering: replicated stream creation, R1 vs R3 behavior, placement tags, peer removal and rebalancing, stream info consistency across nodes, mirror catchup, source aggregation.
Use parallel subagent: Independent of Task 6.
Task 8: JetStream Cluster Tests — Consumer Replication (~80 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterConsumerReplicationTests.cs
Go reference: jetstream_cluster_2_test.go and jetstream_cluster_4_test.go — tests for consumer state replication, ack tracking across failover, push consumer reconnection.
Port ~80 tests covering: consumer state after leader change, pending ack survival, redelivery after failover, pull consumer batch state, push consumer reconnection, consumer pause/resume.
Use parallel subagent: Independent of Tasks 6-7.
Task 9: JetStream Cluster Tests — Meta-cluster Governance (~60 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterMetaGovernanceTests.cs
Go reference: jetstream_cluster_3_test.go and jetstream_cluster_4_test.go — tests for meta-cluster consistency, account limit enforcement in clustered mode, API responses from non-leaders.
Port ~60 tests covering: meta-cluster peer count, account limits across cluster, stream create from non-leader, consumer create from non-leader, server removal from meta-group.
Task 10: JetStream Cluster Tests — Advanced & Long-running (~40 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterAdvancedTests.cs - Create:
tests/NATS.Server.Tests/JetStream/Cluster/JsClusterLongRunningTests.cs
Go reference: jetstream_cluster_4_test.go and jetstream_cluster_long_test.go
Port ~40 tests covering: super-cluster scenarios, gateway + JetStream interaction, domain-scoped API, long-running stability (mark as [Trait("Category", "LongRunning")]).
Task 11: JetStream Core Tests (~100 tests)
Files:
- Modify:
tests/NATS.Server.Tests/JetStream/(multiple existing files)
Go reference: jetstream_test.go (312 tests, ~200 already ported)
Port remaining ~100 tests covering:
- Stream lifecycle: max messages, max bytes, max age, discard old/new policy
- Consumer semantics: ack wait, max deliver, backoff, idle heartbeat
- Publish preconditions: expected stream, expected seq, expected msg ID, dedup window
- Account limits: max streams per account, max consumers, max storage bytes
- API error shapes: exact error codes matching Go's
NewJSXxxError()responses - Direct get: zero-copy message retrieval by sequence and by last-per-subject
Use parallel subagent: Independent of cluster tests.
Task 12: FileStore Permutation Tests (~100 tests)
Files:
- Create:
tests/NATS.Server.Tests/JetStream/Storage/FileStorePermutationTests.cs - Modify existing:
FileStoreCompressionTests.cs,FileStoreEncryptionTests.cs
Go reference: filestore_test.go (232 tests, ~130 already ported). Go's testFileStoreAllPermutations() runs each test across 6 combinations: {NoCipher, ChaCha, AES} x {NoCompression, S2}.
Permutation helper:
// Go ref: filestore_test.go lines 55-71 (testFileStoreAllPermutations)
public static IEnumerable<object[]> AllPermutations()
{
foreach (var cipher in new[] { StoreCipher.NoCipher, StoreCipher.ChaCha, StoreCipher.AesGcm })
foreach (var compression in new[] { StoreCompression.NoCompression, StoreCompression.S2Compression })
yield return [cipher, compression];
}
[Theory]
[MemberData(nameof(AllPermutations))]
public async Task Store_and_load_basic(StoreCipher cipher, StoreCompression compression)
{
await using var store = CreateStore($"basic-{cipher}-{compression}", cipher, compression);
var seq = await store.AppendAsync("test.subject", "hello"u8.ToArray(), CancellationToken.None);
var msg = await store.LoadAsync(seq, CancellationToken.None);
msg.ShouldNotBeNull();
msg!.Payload.ToArray().ShouldBe("hello"u8.ToArray());
}
Port ~100 tests covering: basic CRUD across all 6 permutations, block rotation, crash recovery, corruption detection, large payloads, subject-filtered queries, purge, snapshot/restore.
Use parallel subagent: Independent of cluster tests.
Phase 3: Medium/Low Priority Test Ports
Task 13: Stress/NoRace Tests (~50 tests)
Files:
- Create:
tests/NATS.Server.Tests/Stress/ConcurrentPubSubStressTests.cs - Create:
tests/NATS.Server.Tests/Stress/SlowConsumerStressTests.cs - Create:
tests/NATS.Server.Tests/Stress/ClusterStressTests.cs
Go reference: norace_1_test.go (100 tests), norace_2_test.go (41 tests)
All tests marked with [Trait("Category", "Stress")] for optional CI execution.
Port ~50 most critical tests covering: concurrent pub/sub with 100+ clients, slow consumer handling under load, route/gateway reconnection under message flood, JetStream publish during cluster failover.
Task 14: Accounts/Auth Tests (~30 tests)
Files:
- Modify:
tests/NATS.Server.Tests/Accounts/(existing files) - Create:
tests/NATS.Server.Tests/Accounts/AuthCalloutTests.cs
Go reference: accounts_test.go (64 tests), auth_callout_test.go (31 tests)
Port ~30 remaining tests covering: service import/export cross-account delivery, auth callout timeout/retry, account connection/subscription limits, user revocation.
Task 15: Message Trace Tests (~20 tests)
Files:
- Create:
tests/NATS.Server.Tests/MessageTraceTests.cs
Go reference: msgtrace_test.go (33 tests)
Port ~20 tests covering: trace header propagation, $SYS.TRACE.> event publication, trace filtering.
Task 16: Config/Reload Tests (~20 tests)
Files:
- Modify:
tests/NATS.Server.Tests/Configuration/(existing files)
Go reference: opts_test.go (86 tests), reload_test.go (73 tests)
Port ~20 remaining tests covering: CLI override precedence, include file resolution, TLS cert reload, account resolver reload.
Task 17: Events Tests (~15 tests)
Files:
- Create:
tests/NATS.Server.Tests/Events/ServerEventTests.cs
Go reference: events_test.go (51 tests)
Port ~15 tests covering: server lifecycle events, account stats, advisory messages.
Execution Dependencies
Task 1 (Wire Format) → Task 2 (NatsRaftTransport)
Task 3 (JetStream Svc) — independent of Tasks 1-2
Task 4 (FileStore S2/AEAD) — independent of Tasks 1-3
Phase 1 complete → Phase 2 begins
Task 5 (Cluster Fixture) → Tasks 6-10 (Cluster Tests, parallel)
Task 11 (JS Core Tests) — parallel with Tasks 6-10
Task 12 (FileStore Tests) — parallel with Tasks 6-11
Phase 2 complete → Phase 3 begins
Tasks 13-17 — all parallel with each other
Verification Checkpoints
After each phase:
dotnet build— zero errors, zero warningsdotnet test tests/NATS.Server.Tests— all tests pass- Commit with descriptive message
- Verify test count increases as expected
Phase 1 target: 2,606 + ~20 new impl tests = ~2,626 tests Phase 2 target: 2,626 + ~460 cluster/JS/FileStore tests = ~3,086 tests Phase 3 target: 3,086 + ~135 stress/auth/trace/config/events tests = ~3,221 tests
Final success criterion: 3,100+ tests, 0 failures, all Go subsystems covered.