- Move file size read after streams are disposed to get accurate compressed size
- Clean up definition files to use working example queries
- Add .gitignore for output directory
Add Verifier class that reads exported .pb.zstd files and:
- Deserializes protobuf data through zstd decompression
- Extracts schema information (column names and types)
- Counts rows for verification
- Optionally computes SHA256 hash and compares against sidecar file
Adds DatabaseExporter class that exports query results to compressed
protobuf format. Supports SQL Server and Oracle providers with streaming
compression and SHA256 hash verification.
- Add optional third parameter for parallelism (default: 8)
- Use Parallel.ForEachAsync for concurrent file processing
- Thread-safe console output with lock
- Thread-safe size counters with Interlocked
Usage: dotnet run -- <cache-dir> <scripts-dir> [parallelism]
- Read schema from SQL CREATE TABLE scripts instead of inferring from JSON
- Stream JSON records using Utf8JsonReader instead of loading all into memory
- Write protobuf output in batches of 10000 rows to reduce memory usage
- Add mapping from cache file names to SQL scripts and table names
- Map SQL types (VARCHAR, BIGINT, DECIMAL, DATETIME2, BIT) to .NET types
- Update usage to require scripts directory as second argument
Joseph Doherty