docs: add phase 1-3 instruction guides
This commit is contained in:
Joseph Doherty
291
docs/plans/phases/phase-1-decomposition.md
Normal file
291
docs/plans/phases/phase-1-decomposition.md
Normal file
@@ -0,0 +1,291 @@
|
||||
# Phase 1: Go Codebase Decomposition
|
||||
|
||||
## Objective
|
||||
|
||||
Parse the Go NATS server source code into a structured SQLite database, extracting
|
||||
modules, features (functions/methods), unit tests, external library imports, and
|
||||
inter-module dependencies. This database becomes the single source of truth that
|
||||
drives all subsequent porting phases.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | Version / Notes |
|
||||
|---|---|
|
||||
| Go | 1.25+ (required by `tools/go-analyzer/go.mod`) |
|
||||
| .NET SDK | 10.0+ |
|
||||
| SQLite3 CLI | 3.x (optional, for manual inspection) |
|
||||
| CGO | Must be enabled (`CGO_ENABLED=1`); the Go analyzer uses `github.com/mattn/go-sqlite3` |
|
||||
| Go source | Cloned at `golang/nats-server/` relative to the repo root |
|
||||
|
||||
Verify prerequisites before starting:
|
||||
|
||||
```bash
|
||||
go version # should print go1.25 or later
|
||||
dotnet --version # should print 10.x
|
||||
sqlite3 --version # optional, any 3.x
|
||||
echo $CGO_ENABLED # should print 1 (or be unset; we set it explicitly below)
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Initialize the porting database
|
||||
|
||||
Create a fresh SQLite database with the porting tracker schema. This creates
|
||||
`porting.db` in the repository root with all tables, indexes, and triggers.
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- init --db porting.db --schema porting-schema.sql
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
Database initialized at porting.db
|
||||
```
|
||||
|
||||
If the database already exists, this command is idempotent -- it applies
|
||||
`CREATE TABLE IF NOT EXISTS` statements and will not destroy existing data.
|
||||
|
||||
### Step 2: Build the Go analyzer
|
||||
|
||||
The Go analyzer is a standalone tool that uses `go/ast` to parse Go source files
|
||||
and writes results directly into the SQLite database.
|
||||
|
||||
```bash
|
||||
cd tools/go-analyzer && CGO_ENABLED=1 go build -o go-analyzer . && cd ../..
|
||||
```
|
||||
|
||||
This produces the binary `tools/go-analyzer/go-analyzer`. If the build fails,
|
||||
see the Troubleshooting section below.
|
||||
|
||||
### Step 3: Run the Go analyzer
|
||||
|
||||
Point the analyzer at the NATS server source and the porting database:
|
||||
|
||||
```bash
|
||||
./tools/go-analyzer/go-analyzer \
|
||||
--source golang/nats-server \
|
||||
--db porting.db \
|
||||
--schema porting-schema.sql
|
||||
```
|
||||
|
||||
Expected output (counts will vary with the NATS server version):
|
||||
|
||||
```
|
||||
Analysis complete:
|
||||
Modules: <N>
|
||||
Features: <N>
|
||||
Unit Tests: <N>
|
||||
Dependencies: <N>
|
||||
Imports: <N>
|
||||
```
|
||||
|
||||
The analyzer does the following:
|
||||
1. Walks `golang/nats-server/server/` for all `.go` files (skipping `configs/`
|
||||
and `testdata/` directories).
|
||||
2. Groups files into logical modules by directory.
|
||||
3. Parses each non-test file, extracting every `func` and method as a feature.
|
||||
4. Parses each `_test.go` file, extracting `Test*` and `Benchmark*` functions.
|
||||
5. Infers module-level dependencies from cross-package imports.
|
||||
6. Collects all import paths and classifies them as stdlib or external.
|
||||
7. Writes modules, features, unit_tests, dependencies, and library_mappings to
|
||||
the database.
|
||||
|
||||
### Step 4: Review module groupings
|
||||
|
||||
List all modules that were created to confirm the grouping makes sense:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- module list --db porting.db
|
||||
```
|
||||
|
||||
This shows every module with its ID, name, status, Go package, and line count.
|
||||
Verify that the major areas of the NATS server are represented (e.g., core,
|
||||
jetstream, client, auth, protocol, route, gateway, leafnode, mqtt, websocket,
|
||||
monitoring, logging, errors, subscriptions, tls, events, raft, config, accounts).
|
||||
|
||||
### Step 5: Spot-check individual modules
|
||||
|
||||
Inspect a few modules to verify that features and tests were correctly extracted:
|
||||
|
||||
```bash
|
||||
# Check the first module
|
||||
dotnet run --project tools/NatsNet.PortTracker -- module show 1 --db porting.db
|
||||
|
||||
# Check a few more
|
||||
dotnet run --project tools/NatsNet.PortTracker -- module show 2 --db porting.db
|
||||
dotnet run --project tools/NatsNet.PortTracker -- module show 3 --db porting.db
|
||||
```
|
||||
|
||||
For each module, confirm that:
|
||||
- Features list functions and methods from the corresponding Go files.
|
||||
- Each feature has a `go_file`, `go_method`, and `go_line_number`.
|
||||
- Tests are listed under the module and have `go_file` and `go_method` populated.
|
||||
- Dependencies point to other valid modules.
|
||||
|
||||
### Step 6: Verify test extraction
|
||||
|
||||
List all extracted tests to confirm test files were parsed:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- test list --db porting.db
|
||||
```
|
||||
|
||||
Spot-check a few individual tests for detail:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- test show 1 --db porting.db
|
||||
```
|
||||
|
||||
Verify that each test has a module assignment and that the `feature_id` link is
|
||||
populated where the analyzer could infer the connection from naming conventions
|
||||
(e.g., `TestConnect` links to a feature named `Connect`).
|
||||
|
||||
### Step 7: Review inter-module dependencies
|
||||
|
||||
Check the dependency graph for a representative module:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- dependency show module 1 --db porting.db
|
||||
```
|
||||
|
||||
Also view the full blocked-items report to see the overall dependency shape:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- dependency blocked --db porting.db
|
||||
```
|
||||
|
||||
And check which items have no unported dependencies and are ready to start:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- dependency ready --db porting.db
|
||||
```
|
||||
|
||||
### Step 8: Review extracted library mappings
|
||||
|
||||
List all external Go libraries that were detected:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library list --db porting.db
|
||||
```
|
||||
|
||||
All entries should have status `not_mapped` at this point. Library mapping is
|
||||
handled in Phase 3.
|
||||
|
||||
### Step 9: Generate a baseline summary report
|
||||
|
||||
Create a summary snapshot to use as a reference for Phase 2 verification:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
|
||||
```
|
||||
|
||||
Optionally export a full markdown report for archival:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- report export --format md --output docs/reports/phase-1-baseline.md --db porting.db
|
||||
```
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Phase 1 is complete when ALL of the following are true:
|
||||
|
||||
- [ ] `porting.db` exists and contains data in all five tables (modules, features,
|
||||
unit_tests, dependencies, library_mappings).
|
||||
- [ ] All Go source files under `golang/nats-server/server/` are accounted for
|
||||
(no files silently skipped without a logged warning).
|
||||
- [ ] All public and private functions/methods are extracted as features.
|
||||
- [ ] All `Test*` and `Benchmark*` functions are extracted as unit_tests.
|
||||
- [ ] Test-to-feature links are populated where naming conventions allow inference.
|
||||
- [ ] Module-level dependencies are recorded in the dependencies table.
|
||||
- [ ] External import paths are recorded in the library_mappings table.
|
||||
- [ ] `phase check 1` shows all checklist items marked `[x]`:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- phase check 1 --db porting.db
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```
|
||||
Phase 1: Analysis & Schema
|
||||
Run Go AST analyzer, populate DB schema, map libraries
|
||||
|
||||
Phase 1 Checklist:
|
||||
[x] Modules populated: <N>
|
||||
[x] Features populated: <N>
|
||||
[x] Unit tests populated: <N>
|
||||
[x] Dependencies mapped: <N>
|
||||
[x] Libraries identified: <N>
|
||||
[ ] All libraries mapped: 0/<N>
|
||||
```
|
||||
|
||||
Note: The "All libraries mapped" item will be unchecked -- that is expected.
|
||||
Library mapping is the concern of Phase 3.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### CGO_ENABLED not set or build fails with "gcc not found"
|
||||
|
||||
The `go-sqlite3` driver requires C compilation. Make sure you have a C compiler
|
||||
installed and CGO is enabled:
|
||||
|
||||
```bash
|
||||
# macOS -- Xcode command line tools
|
||||
xcode-select --install
|
||||
|
||||
# Then build with explicit CGO
|
||||
CGO_ENABLED=1 go build -o go-analyzer .
|
||||
```
|
||||
|
||||
### "cannot find package" errors during Go build
|
||||
|
||||
The analyzer depends on `github.com/mattn/go-sqlite3`. Run:
|
||||
|
||||
```bash
|
||||
cd tools/go-analyzer
|
||||
go mod download
|
||||
go mod verify
|
||||
```
|
||||
|
||||
### Wrong source path
|
||||
|
||||
The `--source` flag must point to the root of the cloned nats-server repository
|
||||
(the directory that contains the `server/` subdirectory). If you see
|
||||
"discovering files: no such file or directory", verify:
|
||||
|
||||
```bash
|
||||
ls golang/nats-server/server/
|
||||
```
|
||||
|
||||
### Database locked errors
|
||||
|
||||
If you run the analyzer while another process has `porting.db` open, SQLite may
|
||||
report a lock error. Close any other connections (including `sqlite3` CLI
|
||||
sessions) and retry. The schema enables WAL mode to reduce lock contention:
|
||||
|
||||
```sql
|
||||
PRAGMA journal_mode=WAL;
|
||||
```
|
||||
|
||||
### Analyzer prints warnings but continues
|
||||
|
||||
Warnings like "Warning: skipping server/foo.go: <parse error>" mean an individual
|
||||
file could not be parsed. The analyzer continues with remaining files. Investigate
|
||||
any warnings -- they may indicate a Go version mismatch or syntax not yet
|
||||
supported by the `go/ast` parser at your Go version.
|
||||
|
||||
### Empty database after analyzer runs
|
||||
|
||||
If the analyzer prints zeros for all counts, verify that:
|
||||
1. The `--source` path is correct and contains Go files.
|
||||
2. The `--schema` path points to a valid `porting-schema.sql`.
|
||||
3. The `--db` path is writable.
|
||||
|
||||
You can inspect the database directly:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM modules;"
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM features;"
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM unit_tests;"
|
||||
```
|
||||
315
docs/plans/phases/phase-2-verification.md
Normal file
315
docs/plans/phases/phase-2-verification.md
Normal file
@@ -0,0 +1,315 @@
|
||||
# Phase 2: Verification of Captured Items
|
||||
|
||||
## Objective
|
||||
|
||||
Verify that the Phase 1 decomposition captured every Go source file, function,
|
||||
test, and dependency accurately. Compare database counts against independent
|
||||
baselines derived directly from the filesystem. Identify and fix any gaps before
|
||||
proceeding to library mapping and porting.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Phase 1 is complete (`porting.db` is populated).
|
||||
- The Go source at `golang/nats-server/` has not changed since the Phase 1
|
||||
analyzer run. If the source was updated, re-run the Phase 1 analyzer first.
|
||||
- `dotnet`, `sqlite3`, `find`, `grep`, and `wc` are available on your PATH.
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Generate the summary report
|
||||
|
||||
Start with a high-level view of what the database contains:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
|
||||
```
|
||||
|
||||
Record the counts for modules, features, unit tests, and library mappings. These
|
||||
are the numbers you will verify in subsequent steps.
|
||||
|
||||
### Step 2: Count Go source files on disk
|
||||
|
||||
Count non-test `.go` files under the server directory (the scope of the analyzer):
|
||||
|
||||
```bash
|
||||
find golang/nats-server/server -name "*.go" ! -name "*_test.go" ! -path "*/configs/*" ! -path "*/testdata/*" | wc -l
|
||||
```
|
||||
|
||||
This should produce approximately 109 files. Compare this count against the
|
||||
number of distinct `go_file` values in the features table:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(DISTINCT go_file) FROM features;"
|
||||
```
|
||||
|
||||
If the database count is lower, some source files may have been skipped. Check
|
||||
the analyzer stderr output for warnings, or list the missing files:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT DISTINCT go_file FROM features ORDER BY go_file;" > /tmp/db_files.txt
|
||||
find golang/nats-server/server -name "*.go" ! -name "*_test.go" ! -path "*/configs/*" ! -path "*/testdata/*" -exec realpath --relative-to=golang/nats-server {} \; | sort > /tmp/disk_files.txt
|
||||
diff /tmp/db_files.txt /tmp/disk_files.txt
|
||||
```
|
||||
|
||||
### Step 3: Count Go test files on disk
|
||||
|
||||
```bash
|
||||
find golang/nats-server/server -name "*_test.go" ! -path "*/configs/*" ! -path "*/testdata/*" | wc -l
|
||||
```
|
||||
|
||||
This should produce approximately 85 files. Compare against distinct test files
|
||||
in the database:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(DISTINCT go_file) FROM unit_tests;"
|
||||
```
|
||||
|
||||
### Step 4: Compare function counts
|
||||
|
||||
Count all exported and unexported functions in source files on disk:
|
||||
|
||||
```bash
|
||||
grep -r "^func " golang/nats-server/server/ --include="*.go" --exclude="*_test.go" | grep -v "/configs/" | grep -v "/testdata/" | wc -l
|
||||
```
|
||||
|
||||
Compare against the features count from the database:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM features;"
|
||||
```
|
||||
|
||||
The numbers should be close. Small discrepancies can occur because:
|
||||
- The `grep` approach counts lines starting with `func` which may miss functions
|
||||
with preceding comments on the same line or multi-line signatures.
|
||||
- The AST parser used by the analyzer is more accurate; it finds all `func`
|
||||
declarations regardless of formatting.
|
||||
|
||||
If the database count is significantly lower (more than 5% off), investigate.
|
||||
|
||||
### Step 5: Compare test function counts
|
||||
|
||||
Count test functions on disk:
|
||||
|
||||
```bash
|
||||
grep -r "^func Test" golang/nats-server/server/ --include="*_test.go" | wc -l
|
||||
```
|
||||
|
||||
Also count benchmarks:
|
||||
|
||||
```bash
|
||||
grep -r "^func Benchmark" golang/nats-server/server/ --include="*_test.go" | wc -l
|
||||
```
|
||||
|
||||
Compare the combined total against the unit_tests table:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM unit_tests;"
|
||||
```
|
||||
|
||||
### Step 6: Run the phase check command
|
||||
|
||||
The PortTracker has a built-in Phase 1 checklist that verifies all tables are
|
||||
populated:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- phase check 1 --db porting.db
|
||||
```
|
||||
|
||||
All items except "All libraries mapped" should show `[x]`.
|
||||
|
||||
### Step 7: Check for orphaned items
|
||||
|
||||
Look for features that are not linked to any module (should be zero):
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM features WHERE module_id NOT IN (SELECT id FROM modules);"
|
||||
```
|
||||
|
||||
Look for tests that are not linked to any module (should be zero):
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM unit_tests WHERE module_id NOT IN (SELECT id FROM modules);"
|
||||
```
|
||||
|
||||
Look for test-to-feature links that point to non-existent features:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM unit_tests WHERE feature_id IS NOT NULL AND feature_id NOT IN (SELECT id FROM features);"
|
||||
```
|
||||
|
||||
Look for dependencies that reference non-existent source or target items:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "
|
||||
SELECT COUNT(*) FROM dependencies
|
||||
WHERE (source_type = 'module' AND source_id NOT IN (SELECT id FROM modules))
|
||||
OR (target_type = 'module' AND target_id NOT IN (SELECT id FROM modules))
|
||||
OR (source_type = 'feature' AND source_id NOT IN (SELECT id FROM features))
|
||||
OR (target_type = 'feature' AND target_id NOT IN (SELECT id FROM features))
|
||||
OR (source_type = 'unit_test' AND source_id NOT IN (SELECT id FROM unit_tests))
|
||||
OR (target_type = 'unit_test' AND target_id NOT IN (SELECT id FROM unit_tests));
|
||||
"
|
||||
```
|
||||
|
||||
All of these queries should return 0.
|
||||
|
||||
### Step 8: Review the largest modules
|
||||
|
||||
The largest modules are the most likely to have issues. List modules sorted by
|
||||
feature count:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "
|
||||
SELECT m.id, m.name, m.go_line_count,
|
||||
COUNT(f.id) as feature_count
|
||||
FROM modules m
|
||||
LEFT JOIN features f ON f.module_id = m.id
|
||||
GROUP BY m.id
|
||||
ORDER BY feature_count DESC
|
||||
LIMIT 10;
|
||||
"
|
||||
```
|
||||
|
||||
For each of the top 3 modules, do a manual spot-check:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- module show <id> --db porting.db
|
||||
```
|
||||
|
||||
Scroll through the features list and verify that the functions look correct
|
||||
(check a few against the actual Go source file).
|
||||
|
||||
### Step 9: Validate the dependency graph
|
||||
|
||||
Check for any circular module dependencies (modules that depend on each other):
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "
|
||||
SELECT d1.source_id, d1.target_id
|
||||
FROM dependencies d1
|
||||
JOIN dependencies d2
|
||||
ON d1.source_type = d2.target_type AND d1.source_id = d2.target_id
|
||||
AND d1.target_type = d2.source_type AND d1.target_id = d2.source_id
|
||||
WHERE d1.source_type = 'module' AND d1.target_type = 'module';
|
||||
"
|
||||
```
|
||||
|
||||
Circular dependencies are not necessarily wrong (Go packages can have them via
|
||||
interfaces), but they should be reviewed.
|
||||
|
||||
Check which items are blocked by unported dependencies:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- dependency blocked --db porting.db
|
||||
```
|
||||
|
||||
And confirm that at least some items are ready to port (have no unported deps):
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- dependency ready --db porting.db
|
||||
```
|
||||
|
||||
### Step 10: Verify library import completeness
|
||||
|
||||
Ensure every external import found in the source is tracked:
|
||||
|
||||
```bash
|
||||
sqlite3 porting.db "SELECT COUNT(*) FROM library_mappings;"
|
||||
```
|
||||
|
||||
Cross-check against a manual count of unique non-stdlib imports:
|
||||
|
||||
```bash
|
||||
grep -rh "\"" golang/nats-server/server/ --include="*.go" | \
|
||||
grep -oP '"\K[^"]+' | \
|
||||
grep '\.' | \
|
||||
sort -u | \
|
||||
wc -l
|
||||
```
|
||||
|
||||
This is an approximate check. The AST-based analyzer is more accurate than grep
|
||||
for import extraction, but the numbers should be in the same ballpark.
|
||||
|
||||
### Step 11: Export a verification snapshot
|
||||
|
||||
Save the current state as a markdown report for your records:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- report export \
|
||||
--format md \
|
||||
--output docs/reports/phase-2-verification.md \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Phase 2 is complete when ALL of the following are true:
|
||||
|
||||
- [ ] Source file counts on disk match distinct `go_file` counts in the database
|
||||
(within a small margin for intentionally excluded directories).
|
||||
- [ ] Feature counts from `grep` are within 5% of the database count (AST is the
|
||||
authoritative source).
|
||||
- [ ] Test function counts from `grep` match the database count closely.
|
||||
- [ ] No orphaned features (all linked to valid modules).
|
||||
- [ ] No orphaned tests (all linked to valid modules).
|
||||
- [ ] No broken test-to-feature links.
|
||||
- [ ] No dangling dependency references.
|
||||
- [ ] Dependency graph is reviewed -- circular deps (if any) are acknowledged.
|
||||
- [ ] `dependency ready` returns at least one item (the graph has valid roots).
|
||||
- [ ] Library mappings table contains all external imports.
|
||||
- [ ] `phase check 1` passes with all items except "All libraries mapped" checked.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### File count mismatch is large
|
||||
|
||||
If the disk file count exceeds the database count by more than a few files,
|
||||
re-run the analyzer with stderr visible:
|
||||
|
||||
```bash
|
||||
./tools/go-analyzer/go-analyzer \
|
||||
--source golang/nats-server \
|
||||
--db porting.db \
|
||||
--schema porting-schema.sql 2>&1 | tee /tmp/analyzer.log
|
||||
```
|
||||
|
||||
Search for warnings:
|
||||
|
||||
```bash
|
||||
grep "Warning" /tmp/analyzer.log
|
||||
```
|
||||
|
||||
Common causes:
|
||||
- Files with build tags that prevent parsing (e.g., `//go:build ignore`).
|
||||
- Files in excluded directories (`configs/`, `testdata/`).
|
||||
- Syntax errors in Go files that the parser cannot handle.
|
||||
|
||||
### Feature count is significantly different
|
||||
|
||||
The AST parser counts every `func` declaration, including unexported helper
|
||||
functions. The `grep` baseline only matches lines starting with `func `. If
|
||||
features that have multiline signatures like:
|
||||
|
||||
```go
|
||||
func (s *Server) handleConnection(
|
||||
conn net.Conn,
|
||||
) {
|
||||
```
|
||||
|
||||
...they will be missed by grep but found by the AST parser. Trust the database
|
||||
count as authoritative.
|
||||
|
||||
### Orphaned records found
|
||||
|
||||
If orphaned records exist, the analyzer may have a bug or the database was
|
||||
partially populated from a prior run. The safest fix is to:
|
||||
|
||||
1. Delete the database: `rm porting.db`
|
||||
2. Re-run Phase 1 from Step 1.
|
||||
|
||||
### Tests not linked to features
|
||||
|
||||
The analyzer uses naming conventions to link tests to features (e.g.,
|
||||
`TestConnect` maps to a feature containing `Connect`). If many tests show
|
||||
`feature_id = NULL`, this is expected for tests whose names do not follow the
|
||||
convention. These links can be manually added later if needed.
|
||||
283
docs/plans/phases/phase-3-library-mapping.md
Normal file
283
docs/plans/phases/phase-3-library-mapping.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Phase 3: Library Mapping
|
||||
|
||||
## Objective
|
||||
|
||||
Map every external Go dependency detected by the analyzer to its .NET equivalent.
|
||||
This includes Go standard library packages, well-known third-party libraries, and
|
||||
NATS ecosystem packages. When Phase 3 is complete, the `library suggest` command
|
||||
returns an empty list and every import has a documented .NET migration path.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Phases 1 and 2 are complete (database is populated and verified).
|
||||
- Familiarity with both the Go standard library and .NET BCL / NuGet ecosystem.
|
||||
- A working `dotnet` CLI for running PortTracker commands.
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: List all unmapped libraries
|
||||
|
||||
Start by seeing everything that needs attention:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library suggest --db porting.db
|
||||
```
|
||||
|
||||
This shows all library_mappings entries with status `not_mapped`, sorted by import
|
||||
path. The output includes the Go import path, the library name, and a usage
|
||||
description.
|
||||
|
||||
If the list is empty, all libraries are already mapped and Phase 3 is complete.
|
||||
|
||||
### Step 2: Review the full library list
|
||||
|
||||
To see both mapped and unmapped libraries in one view:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library list --db porting.db
|
||||
```
|
||||
|
||||
You can also filter by status:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library list --status not_mapped --db porting.db
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library list --status mapped --db porting.db
|
||||
```
|
||||
|
||||
### Step 3: Map each library
|
||||
|
||||
For each unmapped library, determine the appropriate .NET equivalent using the
|
||||
reference table below, then record the mapping:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library map <id> \
|
||||
--package "<NuGet package or BCL>" \
|
||||
--namespace "<.NET namespace>" \
|
||||
--notes "<migration notes>" \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
**Example -- mapping `encoding/json`:**
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library map 1 \
|
||||
--package "System.Text.Json" \
|
||||
--namespace "System.Text.Json" \
|
||||
--notes "Use JsonSerializer. Consider source generators for AOT." \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
**Example -- mapping `github.com/klauspost/compress`:**
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library map 12 \
|
||||
--package "System.IO.Compression" \
|
||||
--namespace "System.IO.Compression" \
|
||||
--notes "S2/Snappy codec needs evaluation; may need custom impl or IronSnappy NuGet." \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
Repeat for every entry in the `library suggest` output.
|
||||
|
||||
### Step 4: Handle libraries that need custom implementations
|
||||
|
||||
Some Go libraries have no direct .NET equivalent and will require custom code.
|
||||
For these, record the mapping with a descriptive note:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library map <id> \
|
||||
--package "Custom" \
|
||||
--namespace "NatsNet.Internal" \
|
||||
--notes "No direct equivalent; requires custom implementation. See <details>." \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
### Step 5: Verify all libraries are mapped
|
||||
|
||||
Run the suggest command again -- it should return an empty list:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library suggest --db porting.db
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
All libraries have been mapped!
|
||||
```
|
||||
|
||||
Also verify via the full list:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- library list --db porting.db
|
||||
```
|
||||
|
||||
Every entry should show status `mapped` (or `verified` if you have already
|
||||
validated the mapping in code).
|
||||
|
||||
### Step 6: Run the phase check
|
||||
|
||||
Confirm Phase 1 now shows full completion including library mappings:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- phase check 1 --db porting.db
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```
|
||||
Phase 1 Checklist:
|
||||
[x] Modules populated: <N>
|
||||
[x] Features populated: <N>
|
||||
[x] Unit tests populated: <N>
|
||||
[x] Dependencies mapped: <N>
|
||||
[x] Libraries identified: <N>
|
||||
[x] All libraries mapped: <N>/<N>
|
||||
```
|
||||
|
||||
### Step 7: Export the mapping report
|
||||
|
||||
Save the complete library mapping state for reference during porting:
|
||||
|
||||
```bash
|
||||
dotnet run --project tools/NatsNet.PortTracker -- report export \
|
||||
--format md \
|
||||
--output docs/reports/phase-3-library-mapping.md \
|
||||
--db porting.db
|
||||
```
|
||||
|
||||
## Common Go to .NET Mappings Reference
|
||||
|
||||
Use this table as a starting point when determining .NET equivalents. Adapt based
|
||||
on the specific usage patterns found in the NATS server source.
|
||||
|
||||
### Go Standard Library
|
||||
|
||||
| Go Package | .NET Equivalent | Notes |
|
||||
|---|---|---|
|
||||
| `encoding/json` | `System.Text.Json` | Use `JsonSerializer`; consider source generators for performance |
|
||||
| `encoding/binary` | `System.Buffers.Binary.BinaryPrimitives` | For endian-aware reads/writes |
|
||||
| `encoding/base64` | `System.Convert` | `Convert.ToBase64String` / `Convert.FromBase64String` |
|
||||
| `encoding/hex` | `System.Convert` | `Convert.ToHexString` (.NET 5+) |
|
||||
| `encoding/pem` | `System.Security.Cryptography.PemEncoding` | .NET 5+ |
|
||||
| `sync` | `System.Threading` | `Mutex` -> `lock` / `Monitor`; `RWMutex` -> `ReaderWriterLockSlim`; `WaitGroup` -> `CountdownEvent`; `Once` -> `Lazy<T>` |
|
||||
| `sync/atomic` | `System.Threading.Interlocked` | `Interlocked.Increment`, `CompareExchange`, etc. |
|
||||
| `net` | `System.Net.Sockets` | `TcpListener`, `TcpClient`, `Socket` |
|
||||
| `net/http` | `System.Net.Http` / `Microsoft.AspNetCore` | Client: `HttpClient`; Server: Kestrel / minimal APIs |
|
||||
| `net/url` | `System.Uri` | `Uri`, `UriBuilder` |
|
||||
| `crypto/tls` | `System.Net.Security.SslStream` | Wrap `NetworkStream` with `SslStream` |
|
||||
| `crypto/x509` | `System.Security.Cryptography.X509Certificates` | `X509Certificate2` |
|
||||
| `crypto/sha256` | `System.Security.Cryptography.SHA256` | `SHA256.HashData()` (.NET 8+) |
|
||||
| `crypto/ed25519` | `System.Security.Cryptography` | `Ed25519` support in .NET 9+ |
|
||||
| `crypto/rand` | `System.Security.Cryptography.RandomNumberGenerator` | `RandomNumberGenerator.Fill()` |
|
||||
| `time` | `System.TimeSpan` / `System.Threading.Timer` | `time.Duration` -> `TimeSpan`; `time.Ticker` -> `PeriodicTimer`; `time.After` -> `Task.Delay` |
|
||||
| `time` (parsing) | `System.DateTime` / `System.DateTimeOffset` | `DateTime.Parse`, custom formats |
|
||||
| `fmt` | String interpolation / `String.Format` | `$"..."` for most cases; `String.Format` for dynamic |
|
||||
| `io` | `System.IO` | `Reader` -> `Stream`; `Writer` -> `Stream`; `io.Copy` -> `Stream.CopyTo` |
|
||||
| `io/fs` | `System.IO` | `Directory`, `File`, `FileInfo` |
|
||||
| `bufio` | `System.IO.BufferedStream` | Or `StreamReader` / `StreamWriter` |
|
||||
| `bytes` | `System.Buffers` / `MemoryStream` | `bytes.Buffer` -> `MemoryStream` or `ArrayBufferWriter<byte>` |
|
||||
| `strings` | `System.String` / `System.Text.StringBuilder` | Most methods have direct equivalents |
|
||||
| `strconv` | `int.Parse`, `double.Parse`, etc. | Or `Convert` class |
|
||||
| `context` | `CancellationToken` | `context.Context` -> `CancellationToken`; `context.WithCancel` -> `CancellationTokenSource` |
|
||||
| `os` | `System.Environment` / `System.IO` | `os.Exit` -> `Environment.Exit`; file ops -> `File` class |
|
||||
| `os/signal` | `System.Runtime.Loader.AssemblyLoadContext` | `UnloadingEvent` or `Console.CancelKeyPress` |
|
||||
| `path/filepath` | `System.IO.Path` | `Path.Combine`, `Path.GetDirectoryName`, etc. |
|
||||
| `sort` | `System.Linq` / `Array.Sort` | LINQ `.OrderBy()` or in-place `Array.Sort` |
|
||||
| `math` | `System.Math` | Direct equivalent |
|
||||
| `math/rand` | `System.Random` | `Random.Shared` for thread-safe usage (.NET 6+) |
|
||||
| `regexp` | `System.Text.RegularExpressions.Regex` | Consider source generators for compiled patterns |
|
||||
| `errors` | `System.Exception` | Go errors -> .NET exceptions; `errors.Is` -> pattern matching |
|
||||
| `log` | `Serilog` | Project choice: Serilog via `Microsoft.Extensions.Logging` |
|
||||
| `testing` | `xUnit` | `testing.T` -> xUnit `[Fact]`/`[Theory]`; `testing.B` -> BenchmarkDotNet |
|
||||
| `flag` | `System.CommandLine` | Or `Microsoft.Extensions.Configuration` |
|
||||
| `embed` | Embedded resources | `.csproj` `<EmbeddedResource>` items |
|
||||
| `runtime` | `System.Runtime` / `System.Environment` | `runtime.GOOS` -> `RuntimeInformation.IsOSPlatform` |
|
||||
|
||||
### NATS Ecosystem Libraries
|
||||
|
||||
| Go Package | .NET Equivalent | Notes |
|
||||
|---|---|---|
|
||||
| `github.com/nats-io/jwt/v2` | Custom / evaluate existing | JWT claims for NATS auth; may need custom implementation matching NATS JWT spec |
|
||||
| `github.com/nats-io/nkeys` | Custom implementation | Ed25519 key pairs for NATS authentication; use `System.Security.Cryptography` Ed25519 |
|
||||
| `github.com/nats-io/nuid` | Custom / `System.Guid` | NATS unique IDs; simple custom implementation or adapt to `Guid` if format is flexible |
|
||||
| `github.com/nats-io/nats.go` | `NATS.Net` (official) | Only used in tests; the official .NET NATS client |
|
||||
|
||||
### Third-Party Libraries
|
||||
|
||||
| Go Package | .NET Equivalent | Notes |
|
||||
|---|---|---|
|
||||
| `github.com/klauspost/compress` | `System.IO.Compression` | General compression: `GZipStream`, `DeflateStream`. S2/Snappy: evaluate IronSnappy NuGet or custom port |
|
||||
| `github.com/minio/highwayhash` | Custom / NuGet | HighwayHash implementation; search NuGet or port the algorithm |
|
||||
| `golang.org/x/crypto` | `System.Security.Cryptography` | `bcrypt` -> `Rfc2898DeriveBytes` or BCrypt.Net NuGet; `argon2` -> Konscious.Security NuGet |
|
||||
| `golang.org/x/sys` | `System.Runtime.InteropServices` | Platform-specific syscalls -> P/Invoke or `RuntimeInformation` |
|
||||
| `golang.org/x/time` | `System.Threading.RateLimiting` | `rate.Limiter` -> `RateLimiter` (.NET 7+); `TokenBucketRateLimiter` or `SlidingWindowRateLimiter` |
|
||||
| `golang.org/x/text` | `System.Globalization` | Unicode normalization, encoding detection |
|
||||
|
||||
## Mapping Decision Guidelines
|
||||
|
||||
When choosing a .NET equivalent, follow these priorities:
|
||||
|
||||
1. **BCL first**: Prefer built-in .NET Base Class Library types over NuGet packages.
|
||||
2. **Official packages second**: If BCL does not cover it, prefer
|
||||
`Microsoft.*` or `System.*` NuGet packages.
|
||||
3. **Well-maintained NuGet third**: Choose packages with active maintenance,
|
||||
high download counts, and compatible licenses.
|
||||
4. **Custom implementation last**: Only write custom code when no suitable
|
||||
package exists. Document the rationale in the mapping notes.
|
||||
|
||||
For each mapping, consider:
|
||||
- **API surface**: Does the .NET equivalent cover all methods used in the Go code?
|
||||
- **Performance**: Are there performance-critical paths that need benchmarking?
|
||||
- **Thread safety**: Go's concurrency model differs from .NET. Note any
|
||||
synchronization concerns.
|
||||
- **Platform support**: Does the .NET package work on all target platforms
|
||||
(Linux, macOS, Windows)?
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Phase 3 is complete when ALL of the following are true:
|
||||
|
||||
- [ ] `library suggest` returns "All libraries have been mapped!"
|
||||
- [ ] Every entry in `library list` shows status `mapped` or `verified`.
|
||||
- [ ] Each mapping includes a `--package` (the NuGet package or BCL assembly),
|
||||
a `--namespace` (the .NET namespace to use), and `--notes` (migration
|
||||
guidance).
|
||||
- [ ] `phase check 1` shows all items checked including "All libraries mapped".
|
||||
- [ ] A mapping report has been exported for reference.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Library <id> not found"
|
||||
|
||||
The ID you passed to `library map` does not exist. Run `library suggest` to get
|
||||
the current list of IDs and their import paths.
|
||||
|
||||
### Unsure which .NET package to use
|
||||
|
||||
For unfamiliar Go packages:
|
||||
1. Check what the package does in the Go source (look at the import usage in the
|
||||
files listed by the analyzer).
|
||||
2. Search NuGet.org for equivalent functionality.
|
||||
3. Check if the Go package is a thin wrapper around a well-known algorithm that
|
||||
.NET implements natively.
|
||||
4. When in doubt, map it as "Custom" with detailed notes and revisit during
|
||||
the porting phase.
|
||||
|
||||
### Multiple .NET options for one Go package
|
||||
|
||||
When there are several valid .NET equivalents (e.g., `Newtonsoft.Json` vs
|
||||
`System.Text.Json`), prefer the one that:
|
||||
- Is part of the BCL or a Microsoft package.
|
||||
- Has better performance characteristics.
|
||||
- Has source generator support for AOT compilation.
|
||||
|
||||
Record the alternatives in the `--notes` field so the decision can be revisited.
|
||||
|
||||
### Stdlib packages showing as unmapped
|
||||
|
||||
The analyzer classifies imports as stdlib vs external based on whether the first
|
||||
path component contains a dot. Standard library packages like `encoding/json`,
|
||||
`net/http`, etc. should still be recorded in the library_mappings table so that
|
||||
every import path has a documented .NET migration path. Map them using the
|
||||
reference table above.
|
||||
Reference in New Issue
Block a user