Add porting tracker design document
Design for SQLite-based tracking system to manage the Go-to-.NET port of nats-server. Includes DB schema (modules, features, unit_tests, dependencies, library_mappings), Go AST analyzer for codebase decomposition, .NET CLI tool for ongoing management, and 7 phased workflow guides.
This commit is contained in:
Joseph Doherty
402
docs/plans/2026-02-26-porting-tracker-design.md
Normal file
402
docs/plans/2026-02-26-porting-tracker-design.md
Normal file
@@ -0,0 +1,402 @@
|
||||
# Porting Tracker Design
|
||||
|
||||
A system for tracking the port of NATS server from Go to .NET 10 C#. Consists of an SQLite database, a Go AST analyzer for initial codebase decomposition, a .NET CLI tool for ongoing management, and 7 phase instruction guides.
|
||||
|
||||
## Context
|
||||
|
||||
- **Source**: `golang/nats-server/` — ~130K LOC across 109 non-test Go files, 85 test files (~217K test LOC)
|
||||
- **Target**: .NET 10 C# solution following conventions defined in `documentation_rules.md`
|
||||
- **Dependencies**: 10+ Go external packages (klauspost/compress, nats-io/jwt, nats-io/nkeys, nats-io/nuid, golang.org/x/crypto, etc.)
|
||||
|
||||
## Architecture
|
||||
|
||||
Two tools + one database:
|
||||
|
||||
1. **Go AST Analyzer** (`tools/go-analyzer/`) — Parses Go source using `go/ast`, `go/parser`, `go/token`, and `golang.org/x/tools/go/callgraph` to extract packages, functions, types, call graphs, LOC counts. Populates the SQLite DB. Used during Phases 1-2 only.
|
||||
2. **.NET PortTracker CLI** (`tools/NatsNet.PortTracker/`) — Manages the DB after initial population: update status, map .NET items, run reports, export markdown. Used during all phases.
|
||||
3. **SQLite Database** (`porting.db`) — Single source of truth for porting progress.
|
||||
|
||||
## Database Schema
|
||||
|
||||
### modules
|
||||
|
||||
Tracks top-level Go packages/logical groupings.
|
||||
|
||||
```sql
|
||||
CREATE TABLE modules (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL,
|
||||
description TEXT,
|
||||
go_package TEXT,
|
||||
go_file TEXT,
|
||||
go_line_start INTEGER,
|
||||
go_line_count INTEGER,
|
||||
status TEXT NOT NULL DEFAULT 'not_started'
|
||||
CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')),
|
||||
dotnet_project TEXT,
|
||||
dotnet_namespace TEXT,
|
||||
dotnet_class TEXT,
|
||||
notes TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### features
|
||||
|
||||
Tracks methods/functions/features within a module.
|
||||
|
||||
```sql
|
||||
CREATE TABLE features (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
module_id INTEGER NOT NULL REFERENCES modules(id),
|
||||
name TEXT NOT NULL,
|
||||
description TEXT,
|
||||
go_file TEXT,
|
||||
go_class TEXT,
|
||||
go_method TEXT,
|
||||
go_line_number INTEGER,
|
||||
go_line_count INTEGER,
|
||||
status TEXT NOT NULL DEFAULT 'not_started'
|
||||
CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')),
|
||||
dotnet_project TEXT,
|
||||
dotnet_class TEXT,
|
||||
dotnet_method TEXT,
|
||||
notes TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### unit_tests
|
||||
|
||||
Tracks test functions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE unit_tests (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
module_id INTEGER NOT NULL REFERENCES modules(id),
|
||||
feature_id INTEGER REFERENCES features(id),
|
||||
name TEXT NOT NULL,
|
||||
description TEXT,
|
||||
go_file TEXT,
|
||||
go_class TEXT,
|
||||
go_method TEXT,
|
||||
go_line_number INTEGER,
|
||||
go_line_count INTEGER,
|
||||
status TEXT NOT NULL DEFAULT 'not_started'
|
||||
CHECK (status IN ('not_started', 'stub', 'complete', 'verified', 'n_a')),
|
||||
dotnet_project TEXT,
|
||||
dotnet_class TEXT,
|
||||
dotnet_method TEXT,
|
||||
notes TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### dependencies
|
||||
|
||||
Polymorphic dependency relationships between any tracked items. Used to determine porting order (port dependencies first).
|
||||
|
||||
```sql
|
||||
CREATE TABLE dependencies (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_type TEXT NOT NULL CHECK (source_type IN ('module', 'feature', 'unit_test')),
|
||||
source_id INTEGER NOT NULL,
|
||||
target_type TEXT NOT NULL CHECK (target_type IN ('module', 'feature', 'unit_test')),
|
||||
target_id INTEGER NOT NULL,
|
||||
dependency_kind TEXT DEFAULT 'calls',
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE (source_type, source_id, target_type, target_id)
|
||||
);
|
||||
```
|
||||
|
||||
### library_mappings
|
||||
|
||||
Maps Go imports to .NET equivalents.
|
||||
|
||||
```sql
|
||||
CREATE TABLE library_mappings (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
go_import_path TEXT NOT NULL UNIQUE,
|
||||
go_library_name TEXT,
|
||||
go_usage_description TEXT,
|
||||
dotnet_package TEXT,
|
||||
dotnet_namespace TEXT,
|
||||
dotnet_usage_notes TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'not_mapped'
|
||||
CHECK (status IN ('not_mapped', 'mapped', 'verified')),
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### Indexes
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_features_module ON features(module_id);
|
||||
CREATE INDEX idx_features_status ON features(status);
|
||||
CREATE INDEX idx_unit_tests_module ON unit_tests(module_id);
|
||||
CREATE INDEX idx_unit_tests_feature ON unit_tests(feature_id);
|
||||
CREATE INDEX idx_unit_tests_status ON unit_tests(status);
|
||||
CREATE INDEX idx_deps_source ON dependencies(source_type, source_id);
|
||||
CREATE INDEX idx_deps_target ON dependencies(target_type, target_id);
|
||||
CREATE INDEX idx_library_status ON library_mappings(status);
|
||||
CREATE INDEX idx_modules_status ON modules(status);
|
||||
```
|
||||
|
||||
## Go AST Analyzer
|
||||
|
||||
Located at `tools/go-analyzer/`. A Go program that:
|
||||
|
||||
1. **Parses all `.go` files** in `golang/nats-server/server/` and subdirectories using `go/ast` and `go/parser`
|
||||
2. **Extracts per-file**: package name, struct types, functions/methods with receiver types, line numbers, LOC counts
|
||||
3. **Builds call graph**: using `golang.org/x/tools/go/callgraph/cha` (Class Hierarchy Analysis) for whole-program analysis
|
||||
4. **Extracts test functions**: parses `*_test.go` files separately, links to tested functions by name convention and call graph
|
||||
5. **Extracts imports**: all imported packages per file, distinguishing stdlib vs external
|
||||
6. **Writes to SQLite**: directly populates `modules`, `features`, `unit_tests`, `dependencies`, and partially populates `library_mappings` (go-side fields only)
|
||||
|
||||
### Module Grouping Strategy
|
||||
|
||||
- Each Go file in `server/` that represents a major subsystem maps to a module (e.g., `jetstream.go` + `jetstream_api.go` + `jetstream_cluster.go` -> "jetstream")
|
||||
- Subdirectories (`stree/`, `pse/`, `certidp/`, etc.) each become their own module
|
||||
- Groupings are auto-generated then user-adjustable via the PortTracker CLI
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
cd tools/go-analyzer
|
||||
go run . --source ../../golang/nats-server --db ../../porting.db
|
||||
```
|
||||
|
||||
## .NET PortTracker CLI
|
||||
|
||||
Located at `tools/NatsNet.PortTracker/`. A .NET 10 console app using `System.CommandLine` and `Microsoft.Data.Sqlite`.
|
||||
|
||||
### Command Structure
|
||||
|
||||
```
|
||||
porttracker
|
||||
├── init # Create/reset the DB schema
|
||||
├── module
|
||||
│ ├── list [--status <s>] # List modules with filters
|
||||
│ ├── show <id> # Show module detail + features + deps
|
||||
│ ├── update <id> --status <s> # Update status
|
||||
│ ├── map <id> --project <p> --class <c> # Map to .NET
|
||||
│ └── set-na <id> --reason <r> # Mark as N/A with reason
|
||||
├── feature
|
||||
│ ├── list [--module <id>] [--status <s>]
|
||||
│ ├── show <id>
|
||||
│ ├── update <id> --status <s>
|
||||
│ ├── map <id> --project <p> --class <c> --method <m>
|
||||
│ └── set-na <id> --reason <r>
|
||||
├── test
|
||||
│ ├── list [--module <id>] [--status <s>]
|
||||
│ ├── show <id>
|
||||
│ ├── update <id> --status <s>
|
||||
│ └── map <id> --project <p> --class <c> --method <m>
|
||||
├── library
|
||||
│ ├── list [--status <s>]
|
||||
│ ├── map <id> --package <pkg> --namespace <ns> --notes <n>
|
||||
│ └── suggest # Show unmapped libraries
|
||||
├── dependency
|
||||
│ ├── show <type> <id> # Show deps for an item
|
||||
│ ├── blocked # Items with unported dependencies
|
||||
│ └── ready # Items whose deps are all ported
|
||||
├── report
|
||||
│ ├── summary # Overall progress stats
|
||||
│ ├── phase <n> # Progress for a specific phase
|
||||
│ ├── export --format md # Export markdown status report
|
||||
│ └── blocking # Critical path / blocking chains
|
||||
└── phase
|
||||
├── list # Show all phases and status
|
||||
└── check <n> # Run verification for phase N
|
||||
```
|
||||
|
||||
### Design Decisions
|
||||
|
||||
- **Raw SQL over EF Core**: Simpler, fewer dependencies. This is a tools app, not a production service.
|
||||
- **Status stored as text**: Readable in DB tools and queries.
|
||||
- **Batch operations**: Commands accept `--all-in-module <id>` for bulk status updates.
|
||||
- **DB location**: `porting.db` at repo root, overridable with `--db` flag.
|
||||
|
||||
## Phase Instructions
|
||||
|
||||
Seven markdown guides in `docs/plans/phases/`. Each is a standalone step-by-step document.
|
||||
|
||||
### Phase 1: Go Codebase Decomposition
|
||||
|
||||
**Objective**: Parse the Go source into the SQLite DB — modules, features, unit tests, and call dependencies.
|
||||
|
||||
**Steps**:
|
||||
1. Run `porttracker init` to create the database
|
||||
2. Build and run the Go AST analyzer: `cd tools/go-analyzer && go run . --source ../../golang/nats-server --db ../../porting.db`
|
||||
3. Review auto-generated module groupings: `porttracker module list`
|
||||
4. Adjust module groupings as needed via manual SQL or future CLI commands
|
||||
5. Spot-check feature extraction against Go source files
|
||||
6. Verify test function extraction: `porttracker test list`
|
||||
7. Review dependency graph: `porttracker dependency show module <id>` for key modules
|
||||
|
||||
**Completion criteria**: All Go source files accounted for. All public functions extracted. Test functions linked to features. Call dependencies populated.
|
||||
|
||||
### Phase 2: Verification of Captured Items
|
||||
|
||||
**Objective**: Ensure nothing was missed and dependencies are accurate.
|
||||
|
||||
**Steps**:
|
||||
1. Run `porttracker report summary` — compare counts against baselines:
|
||||
- File count: `find golang/nats-server/server -name "*.go" ! -name "*_test.go" | wc -l`
|
||||
- Function count: `grep -r "^func " golang/nats-server/server/ --include="*.go" --exclude="*_test.go" | wc -l`
|
||||
- Test count: `grep -r "^func Test" golang/nats-server/server/ --include="*_test.go" | wc -l`
|
||||
2. Cross-check: every `*_test.go` function appears in `unit_tests`
|
||||
3. Cross-check: every non-test `.go` file has a corresponding module
|
||||
4. Check for orphaned nodes: `porttracker dependency blocked` — items with no path to a root
|
||||
5. Identify and resolve circular dependencies
|
||||
6. Manual review of the 10 largest modules for completeness
|
||||
|
||||
**Completion criteria**: Counts match baselines. No orphaned nodes. No missing files. Dependency graph is valid.
|
||||
|
||||
### Phase 3: Library Mapping
|
||||
|
||||
**Objective**: Map every Go external dependency to its .NET equivalent.
|
||||
|
||||
**Steps**:
|
||||
1. Run `porttracker library suggest` to list all unmapped Go imports
|
||||
2. For each Go import, determine the .NET equivalent:
|
||||
- **Go stdlib** -> .NET BCL (e.g., `encoding/json` -> `System.Text.Json`, `sync` -> `System.Threading`, `net/http` -> `Microsoft.AspNetCore` or `System.Net.Http`)
|
||||
- **External packages** (e.g., `klauspost/compress` -> evaluate `System.IO.Compression` or a NuGet package)
|
||||
- **NATS-specific** (e.g., `nats-io/nkeys` -> .NET NKeys library or custom implementation)
|
||||
3. Record each mapping: `porttracker library map <id> --package <pkg> --namespace <ns> --notes <notes>`
|
||||
4. Flag libraries with no clear .NET equivalent — these need design decisions
|
||||
|
||||
**Completion criteria**: `porttracker library list --status not_mapped` returns empty. All ambiguous cases documented.
|
||||
|
||||
### Phase 4: .NET Solution Design
|
||||
|
||||
**Objective**: Design the target .NET 10 solution structure and map every Go item to its .NET counterpart.
|
||||
|
||||
**Steps**:
|
||||
1. Define solution structure following .NET conventions:
|
||||
- `src/NATS.Server/` — main server library
|
||||
- `src/NATS.Server.Host/` — host/entry point
|
||||
- Additional projects as needed per module grouping
|
||||
- `tests/NATS.Server.Tests/` — unit tests
|
||||
- `tests/NATS.Server.IntegrationTests/` — integration tests
|
||||
2. For each module: `porttracker module map <id> --project <project> --class <class>`
|
||||
3. For each feature: `porttracker feature map <id> --project <project> --class <class> --method <method>`
|
||||
4. For each test: `porttracker test map <id> --project <project> --class <class> --method <method>`
|
||||
5. For items not applicable in .NET (e.g., Go-specific platform code, custom logger replaced by Serilog):
|
||||
`porttracker module set-na <id> --reason "Replaced by Serilog"` (or feature/test equivalent)
|
||||
|
||||
**Completion criteria**: Every item has a .NET mapping or is set to N/A with a reason.
|
||||
|
||||
### Phase 5: Mapping Verification
|
||||
|
||||
**Objective**: Verify all Go items are mapped to .NET targets.
|
||||
|
||||
**Steps**:
|
||||
1. Run `porttracker report summary` — confirm 0 unmapped items (excluding N/A)
|
||||
2. Review all N/A items — each must have a justification:
|
||||
`porttracker module list --status n_a`, `porttracker feature list --status n_a`
|
||||
3. Verify .NET naming follows conventions:
|
||||
- PascalCase for classes and methods
|
||||
- Proper namespace hierarchy
|
||||
- No collisions (two features mapped to the same .NET class+method)
|
||||
4. Generate full mapping report: `porttracker report export --format md`
|
||||
5. Review the exported report for completeness
|
||||
|
||||
**Completion criteria**: Zero unmapped items. All N/A items justified. Naming conventions validated. No collisions.
|
||||
|
||||
### Phase 6: Initial Porting
|
||||
|
||||
**Objective**: Port Go code to .NET, working through the dependency graph bottom-up.
|
||||
|
||||
**Steps**:
|
||||
1. Run `porttracker dependency ready` to find items with all dependencies already ported
|
||||
2. For each ready item:
|
||||
a. `porttracker feature update <id> --status stub` — when creating the class/method skeleton
|
||||
b. Implement the logic, referencing the Go source at the line numbers stored in the DB
|
||||
c. `porttracker feature update <id> --status complete` — when implementation is done
|
||||
d. Run the corresponding unit tests for that feature
|
||||
3. After completing items, re-run `porttracker dependency ready` to find newly unblocked items
|
||||
4. Periodically run `porttracker report summary` to track overall progress
|
||||
5. For batch scaffolding: `porttracker feature update --all-in-module <module_id> --status stub`
|
||||
|
||||
**DB update workflow**:
|
||||
```bash
|
||||
# Starting a feature
|
||||
porttracker feature update 42 --status stub
|
||||
|
||||
# Feature implemented
|
||||
porttracker feature update 42 --status complete
|
||||
|
||||
# Corresponding test implemented
|
||||
porttracker test update 87 --status complete
|
||||
|
||||
# Check what's unblocked now
|
||||
porttracker dependency ready
|
||||
```
|
||||
|
||||
**Completion criteria**: All non-N/A items at status `complete` or better.
|
||||
|
||||
### Phase 7: Verification of Ported Items
|
||||
|
||||
**Objective**: Verify all ported code works correctly through targeted testing.
|
||||
|
||||
**Steps**:
|
||||
1. For each module, identify its mapped tests: `porttracker test list --module <id>`
|
||||
2. Run targeted tests per module using `dotnet test --filter`:
|
||||
```bash
|
||||
dotnet test --filter "FullyQualifiedName~NATS.Server.Tests.Protocol"
|
||||
```
|
||||
3. For each passing module, update status:
|
||||
`porttracker module update <id> --status verified`
|
||||
`porttracker feature update --all-in-module <id> --status verified`
|
||||
4. For failures: investigate, fix, and re-run the targeted tests
|
||||
5. Run integration tests for cross-module behavior
|
||||
6. Compare behavior: run Go server and .NET server with same workload, verify equivalent behavior
|
||||
7. Final report: `porttracker report summary` — all items `verified` or `n_a`
|
||||
|
||||
**Completion criteria**: All targeted tests pass. All items at `verified` or `n_a`. Behavioral equivalence confirmed for key scenarios.
|
||||
|
||||
## Project Layout
|
||||
|
||||
```
|
||||
natsnet/
|
||||
├── golang/nats-server/ # Existing Go source (reference)
|
||||
├── tools/
|
||||
│ ├── go-analyzer/ # Go AST analyzer (Phases 1-2)
|
||||
│ │ ├── go.mod
|
||||
│ │ ├── main.go
|
||||
│ │ ├── analyzer.go
|
||||
│ │ ├── grouper.go
|
||||
│ │ └── sqlite.go
|
||||
│ └── NatsNet.PortTracker/ # .NET CLI tool (all phases)
|
||||
│ ├── NatsNet.PortTracker.csproj
|
||||
│ ├── Program.cs
|
||||
│ ├── Commands/
|
||||
│ ├── Data/
|
||||
│ └── Reporting/
|
||||
├── docs/
|
||||
│ └── plans/
|
||||
│ └── phases/
|
||||
│ ├── phase-1-decomposition.md
|
||||
│ ├── phase-2-verification.md
|
||||
│ ├── phase-3-library-mapping.md
|
||||
│ ├── phase-4-dotnet-design.md
|
||||
│ ├── phase-5-mapping-verification.md
|
||||
│ ├── phase-6-porting.md
|
||||
│ └── phase-7-porting-verification.md
|
||||
├── porting.db # SQLite database (gitignored)
|
||||
├── porting-schema.sql # DDL reference (committed)
|
||||
└── documentation_rules.md # Existing
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- `porting.db` is `.gitignore`d (contains local state); `porting-schema.sql` is committed
|
||||
- `porttracker init` creates the DB from the embedded schema
|
||||
- Go analyzer writes directly to SQLite — no intermediate format
|
||||
- Phase docs reference `porttracker` commands by name
|
||||
- Run the .NET tool from anywhere: `dotnet run --project tools/NatsNet.PortTracker/`
|
||||
Reference in New Issue
Block a user