commit 3615dcc78a3885a3e5a417d3700f4091556e815d Author: Joseph Doherty Date: Thu Feb 26 05:56:58 2026 -0500 Add porting tracker design document Design for SQLite-based tracking system to manage the Go-to-.NET port of nats-server. Includes DB schema (modules, features, unit_tests, dependencies, library_mappings), Go AST analyzer for codebase decomposition, .NET CLI tool for ongoing management, and 7 phased workflow guides. diff --git a/docs/plans/2026-02-26-porting-tracker-design.md b/docs/plans/2026-02-26-porting-tracker-design.md new file mode 100644 index 0000000..b0d9c1e --- /dev/null +++ b/docs/plans/2026-02-26-porting-tracker-design.md @@ -0,0 +1,402 @@ +# Porting Tracker Design + +A system for tracking the port of NATS server from Go to .NET 10 C#. Consists of an SQLite database, a Go AST analyzer for initial codebase decomposition, a .NET CLI tool for ongoing management, and 7 phase instruction guides. + +## Context + +- **Source**: `golang/nats-server/` — ~130K LOC across 109 non-test Go files, 85 test files (~217K test LOC) +- **Target**: .NET 10 C# solution following conventions defined in `documentation_rules.md` +- **Dependencies**: 10+ Go external packages (klauspost/compress, nats-io/jwt, nats-io/nkeys, nats-io/nuid, golang.org/x/crypto, etc.) + +## Architecture + +Two tools + one database: + +1. **Go AST Analyzer** (`tools/go-analyzer/`) — Parses Go source using `go/ast`, `go/parser`, `go/token`, and `golang.org/x/tools/go/callgraph` to extract packages, functions, types, call graphs, LOC counts. Populates the SQLite DB. Used during Phases 1-2 only. +2. **.NET PortTracker CLI** (`tools/NatsNet.PortTracker/`) — Manages the DB after initial population: update status, map .NET items, run reports, export markdown. Used during all phases. +3. **SQLite Database** (`porting.db`) — Single source of truth for porting progress. + +## Database Schema + +### modules + +Tracks top-level Go packages/logical groupings. + +```sql +CREATE TABLE modules ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + description TEXT, + go_package TEXT, + go_file TEXT, + go_line_start INTEGER, + go_line_count INTEGER, + status TEXT NOT NULL DEFAULT 'not_started' + CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')), + dotnet_project TEXT, + dotnet_namespace TEXT, + dotnet_class TEXT, + notes TEXT, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### features + +Tracks methods/functions/features within a module. + +```sql +CREATE TABLE features ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + module_id INTEGER NOT NULL REFERENCES modules(id), + name TEXT NOT NULL, + description TEXT, + go_file TEXT, + go_class TEXT, + go_method TEXT, + go_line_number INTEGER, + go_line_count INTEGER, + status TEXT NOT NULL DEFAULT 'not_started' + CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')), + dotnet_project TEXT, + dotnet_class TEXT, + dotnet_method TEXT, + notes TEXT, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### unit_tests + +Tracks test functions. + +```sql +CREATE TABLE unit_tests ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + module_id INTEGER NOT NULL REFERENCES modules(id), + feature_id INTEGER REFERENCES features(id), + name TEXT NOT NULL, + description TEXT, + go_file TEXT, + go_class TEXT, + go_method TEXT, + go_line_number INTEGER, + go_line_count INTEGER, + status TEXT NOT NULL DEFAULT 'not_started' + CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')), + dotnet_project TEXT, + dotnet_class TEXT, + dotnet_method TEXT, + notes TEXT, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### dependencies + +Polymorphic dependency relationships between any tracked items. Used to determine porting order (port dependencies first). + +```sql +CREATE TABLE dependencies ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_type TEXT NOT NULL CHECK (source_type IN ('module', 'feature', 'unit_test')), + source_id INTEGER NOT NULL, + target_type TEXT NOT NULL CHECK (target_type IN ('module', 'feature', 'unit_test')), + target_id INTEGER NOT NULL, + dependency_kind TEXT DEFAULT 'calls', + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + UNIQUE (source_type, source_id, target_type, target_id) +); +``` + +### library_mappings + +Maps Go imports to .NET equivalents. + +```sql +CREATE TABLE library_mappings ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + go_import_path TEXT NOT NULL UNIQUE, + go_library_name TEXT, + go_usage_description TEXT, + dotnet_package TEXT, + dotnet_namespace TEXT, + dotnet_usage_notes TEXT, + status TEXT NOT NULL DEFAULT 'not_mapped' + CHECK (status IN ('not_mapped', 'mapped', 'verified')), + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### Indexes + +```sql +CREATE INDEX idx_features_module ON features(module_id); +CREATE INDEX idx_features_status ON features(status); +CREATE INDEX idx_unit_tests_module ON unit_tests(module_id); +CREATE INDEX idx_unit_tests_feature ON unit_tests(feature_id); +CREATE INDEX idx_unit_tests_status ON unit_tests(status); +CREATE INDEX idx_deps_source ON dependencies(source_type, source_id); +CREATE INDEX idx_deps_target ON dependencies(target_type, target_id); +CREATE INDEX idx_library_status ON library_mappings(status); +CREATE INDEX idx_modules_status ON modules(status); +``` + +## Go AST Analyzer + +Located at `tools/go-analyzer/`. A Go program that: + +1. **Parses all `.go` files** in `golang/nats-server/server/` and subdirectories using `go/ast` and `go/parser` +2. **Extracts per-file**: package name, struct types, functions/methods with receiver types, line numbers, LOC counts +3. **Builds call graph**: using `golang.org/x/tools/go/callgraph/cha` (Class Hierarchy Analysis) for whole-program analysis +4. **Extracts test functions**: parses `*_test.go` files separately, links to tested functions by name convention and call graph +5. **Extracts imports**: all imported packages per file, distinguishing stdlib vs external +6. **Writes to SQLite**: directly populates `modules`, `features`, `unit_tests`, `dependencies`, and partially populates `library_mappings` (go-side fields only) + +### Module Grouping Strategy + +- Each Go file in `server/` that represents a major subsystem maps to a module (e.g., `jetstream.go` + `jetstream_api.go` + `jetstream_cluster.go` -> "jetstream") +- Subdirectories (`stree/`, `pse/`, `certidp/`, etc.) each become their own module +- Groupings are auto-generated then user-adjustable via the PortTracker CLI + +### Usage + +```bash +cd tools/go-analyzer +go run . --source ../../golang/nats-server --db ../../porting.db +``` + +## .NET PortTracker CLI + +Located at `tools/NatsNet.PortTracker/`. A .NET 10 console app using `System.CommandLine` and `Microsoft.Data.Sqlite`. + +### Command Structure + +``` +porttracker +├── init # Create/reset the DB schema +├── module +│ ├── list [--status ] # List modules with filters +│ ├── show # Show module detail + features + deps +│ ├── update --status # Update status +│ ├── map --project

--class # Map to .NET +│ └── set-na --reason # Mark as N/A with reason +├── feature +│ ├── list [--module ] [--status ] +│ ├── show +│ ├── update --status +│ ├── map --project

--class --method +│ └── set-na --reason +├── test +│ ├── list [--module ] [--status ] +│ ├── show +│ ├── update --status +│ └── map --project

--class --method +├── library +│ ├── list [--status ] +│ ├── map --package --namespace --notes +│ └── suggest # Show unmapped libraries +├── dependency +│ ├── show # Show deps for an item +│ ├── blocked # Items with unported dependencies +│ └── ready # Items whose deps are all ported +├── report +│ ├── summary # Overall progress stats +│ ├── phase # Progress for a specific phase +│ ├── export --format md # Export markdown status report +│ └── blocking # Critical path / blocking chains +└── phase + ├── list # Show all phases and status + └── check # Run verification for phase N +``` + +### Design Decisions + +- **Raw SQL over EF Core**: Simpler, fewer dependencies. This is a tools app, not a production service. +- **Status stored as text**: Readable in DB tools and queries. +- **Batch operations**: Commands accept `--all-in-module ` for bulk status updates. +- **DB location**: `porting.db` at repo root, overridable with `--db` flag. + +## Phase Instructions + +Seven markdown guides in `docs/plans/phases/`. Each is a standalone step-by-step document. + +### Phase 1: Go Codebase Decomposition + +**Objective**: Parse the Go source into the SQLite DB — modules, features, unit tests, and call dependencies. + +**Steps**: +1. Run `porttracker init` to create the database +2. Build and run the Go AST analyzer: `cd tools/go-analyzer && go run . --source ../../golang/nats-server --db ../../porting.db` +3. Review auto-generated module groupings: `porttracker module list` +4. Adjust module groupings as needed via manual SQL or future CLI commands +5. Spot-check feature extraction against Go source files +6. Verify test function extraction: `porttracker test list` +7. Review dependency graph: `porttracker dependency show module ` for key modules + +**Completion criteria**: All Go source files accounted for. All public functions extracted. Test functions linked to features. Call dependencies populated. + +### Phase 2: Verification of Captured Items + +**Objective**: Ensure nothing was missed and dependencies are accurate. + +**Steps**: +1. Run `porttracker report summary` — compare counts against baselines: + - File count: `find golang/nats-server/server -name "*.go" ! -name "*_test.go" | wc -l` + - Function count: `grep -r "^func " golang/nats-server/server/ --include="*.go" --exclude="*_test.go" | wc -l` + - Test count: `grep -r "^func Test" golang/nats-server/server/ --include="*_test.go" | wc -l` +2. Cross-check: every `*_test.go` function appears in `unit_tests` +3. Cross-check: every non-test `.go` file has a corresponding module +4. Check for orphaned nodes: `porttracker dependency blocked` — items with no path to a root +5. Identify and resolve circular dependencies +6. Manual review of the 10 largest modules for completeness + +**Completion criteria**: Counts match baselines. No orphaned nodes. No missing files. Dependency graph is valid. + +### Phase 3: Library Mapping + +**Objective**: Map every Go external dependency to its .NET equivalent. + +**Steps**: +1. Run `porttracker library suggest` to list all unmapped Go imports +2. For each Go import, determine the .NET equivalent: + - **Go stdlib** -> .NET BCL (e.g., `encoding/json` -> `System.Text.Json`, `sync` -> `System.Threading`, `net/http` -> `Microsoft.AspNetCore` or `System.Net.Http`) + - **External packages** (e.g., `klauspost/compress` -> evaluate `System.IO.Compression` or a NuGet package) + - **NATS-specific** (e.g., `nats-io/nkeys` -> .NET NKeys library or custom implementation) +3. Record each mapping: `porttracker library map --package --namespace --notes ` +4. Flag libraries with no clear .NET equivalent — these need design decisions + +**Completion criteria**: `porttracker library list --status not_mapped` returns empty. All ambiguous cases documented. + +### Phase 4: .NET Solution Design + +**Objective**: Design the target .NET 10 solution structure and map every Go item to its .NET counterpart. + +**Steps**: +1. Define solution structure following .NET conventions: + - `src/NATS.Server/` — main server library + - `src/NATS.Server.Host/` — host/entry point + - Additional projects as needed per module grouping + - `tests/NATS.Server.Tests/` — unit tests + - `tests/NATS.Server.IntegrationTests/` — integration tests +2. For each module: `porttracker module map --project --class ` +3. For each feature: `porttracker feature map --project --class --method ` +4. For each test: `porttracker test map --project --class --method ` +5. For items not applicable in .NET (e.g., Go-specific platform code, custom logger replaced by Serilog): + `porttracker module set-na --reason "Replaced by Serilog"` (or feature/test equivalent) + +**Completion criteria**: Every item has a .NET mapping or is set to N/A with a reason. + +### Phase 5: Mapping Verification + +**Objective**: Verify all Go items are mapped to .NET targets. + +**Steps**: +1. Run `porttracker report summary` — confirm 0 unmapped items (excluding N/A) +2. Review all N/A items — each must have a justification: + `porttracker module list --status n_a`, `porttracker feature list --status n_a` +3. Verify .NET naming follows conventions: + - PascalCase for classes and methods + - Proper namespace hierarchy + - No collisions (two features mapped to the same .NET class+method) +4. Generate full mapping report: `porttracker report export --format md` +5. Review the exported report for completeness + +**Completion criteria**: Zero unmapped items. All N/A items justified. Naming conventions validated. No collisions. + +### Phase 6: Initial Porting + +**Objective**: Port Go code to .NET, working through the dependency graph bottom-up. + +**Steps**: +1. Run `porttracker dependency ready` to find items with all dependencies already ported +2. For each ready item: + a. `porttracker feature update --status stub` — when creating the class/method skeleton + b. Implement the logic, referencing the Go source at the line numbers stored in the DB + c. `porttracker feature update --status complete` — when implementation is done + d. Run the corresponding unit tests for that feature +3. After completing items, re-run `porttracker dependency ready` to find newly unblocked items +4. Periodically run `porttracker report summary` to track overall progress +5. For batch scaffolding: `porttracker feature update --all-in-module --status stub` + +**DB update workflow**: +```bash +# Starting a feature +porttracker feature update 42 --status stub + +# Feature implemented +porttracker feature update 42 --status complete + +# Corresponding test implemented +porttracker test update 87 --status complete + +# Check what's unblocked now +porttracker dependency ready +``` + +**Completion criteria**: All non-N/A items at status `complete` or better. + +### Phase 7: Verification of Ported Items + +**Objective**: Verify all ported code works correctly through targeted testing. + +**Steps**: +1. For each module, identify its mapped tests: `porttracker test list --module ` +2. Run targeted tests per module using `dotnet test --filter`: + ```bash + dotnet test --filter "FullyQualifiedName~NATS.Server.Tests.Protocol" + ``` +3. For each passing module, update status: + `porttracker module update --status verified` + `porttracker feature update --all-in-module --status verified` +4. For failures: investigate, fix, and re-run the targeted tests +5. Run integration tests for cross-module behavior +6. Compare behavior: run Go server and .NET server with same workload, verify equivalent behavior +7. Final report: `porttracker report summary` — all items `verified` or `n_a` + +**Completion criteria**: All targeted tests pass. All items at `verified` or `n_a`. Behavioral equivalence confirmed for key scenarios. + +## Project Layout + +``` +natsnet/ +├── golang/nats-server/ # Existing Go source (reference) +├── tools/ +│ ├── go-analyzer/ # Go AST analyzer (Phases 1-2) +│ │ ├── go.mod +│ │ ├── main.go +│ │ ├── analyzer.go +│ │ ├── grouper.go +│ │ └── sqlite.go +│ └── NatsNet.PortTracker/ # .NET CLI tool (all phases) +│ ├── NatsNet.PortTracker.csproj +│ ├── Program.cs +│ ├── Commands/ +│ ├── Data/ +│ └── Reporting/ +├── docs/ +│ └── plans/ +│ └── phases/ +│ ├── phase-1-decomposition.md +│ ├── phase-2-verification.md +│ ├── phase-3-library-mapping.md +│ ├── phase-4-dotnet-design.md +│ ├── phase-5-mapping-verification.md +│ ├── phase-6-porting.md +│ └── phase-7-porting-verification.md +├── porting.db # SQLite database (gitignored) +├── porting-schema.sql # DDL reference (committed) +└── documentation_rules.md # Existing +``` + +### Notes + +- `porting.db` is `.gitignore`d (contains local state); `porting-schema.sql` is committed +- `porttracker init` creates the DB from the embedded schema +- Go analyzer writes directly to SQLite — no intermediate format +- Phase docs reference `porttracker` commands by name +- Run the .NET tool from anywhere: `dotnet run --project tools/NatsNet.PortTracker/`