fix(DbExporter): fix compressed size calculation and clean up
- Move file size read after streams are disposed to get accurate compressed size - Clean up definition files to use working example queries - Add .gitignore for output directory
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,117 @@
|
||||
# DbExporter Tool Design
|
||||
|
||||
## Purpose
|
||||
|
||||
A command-line tool that queries databases (SQL Server or Oracle) and exports results to compressed protobuf files using `protobuf-net-data` and zstd compression.
|
||||
|
||||
## CLI Interface
|
||||
|
||||
```
|
||||
Usage: DbExporter <definition-file> [options]
|
||||
|
||||
Arguments:
|
||||
definition-file Path to JSON definition file
|
||||
|
||||
Options:
|
||||
--verify Verify output (row count + schema)
|
||||
--verify-full Verify output with SHA256 checksum
|
||||
--help Show help
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Export data
|
||||
dotnet run -- ./definitions/scada-clients.json
|
||||
|
||||
# Export and verify
|
||||
dotnet run -- ./definitions/scada-clients.json --verify
|
||||
|
||||
# Full verification with checksum
|
||||
dotnet run -- ./definitions/scada-clients.json --verify-full
|
||||
```
|
||||
|
||||
## Definition File Format (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=...;Database=...;User Id=...;Password=...;",
|
||||
"query": "SELECT * FROM MyTable",
|
||||
"outputPath": "./output/mytable.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
|-------|----------|---------|-------------|
|
||||
| `providerType` | Yes | - | `"SqlServer"` or `"Oracle"` |
|
||||
| `connectionString` | Yes | - | ADO.NET connection string |
|
||||
| `query` | Yes | - | SQL query to execute |
|
||||
| `outputPath` | Yes | - | Output file path (.pb.zstd) |
|
||||
| `compressionLevel` | No | `10` | Zstd level 1-19 (higher = smaller, slower) |
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### Export Flow
|
||||
1. Parse definition file (JSON)
|
||||
2. Validate fields (provider type, connection string, query)
|
||||
3. Create appropriate DbConnection (SqlConnection or OracleConnection)
|
||||
4. Execute query → IDataReader
|
||||
5. Serialize IDataReader → protobuf stream (via protobuf-net-data)
|
||||
6. Compress protobuf stream → zstd (via ZstdSharp)
|
||||
7. While writing, compute SHA256 incrementally
|
||||
8. Write to output file + sidecar .sha256 file
|
||||
9. Print summary: row count, file size, compression ratio
|
||||
|
||||
### Verify Flow (--verify)
|
||||
1. Open output file
|
||||
2. Decompress zstd → protobuf stream
|
||||
3. Deserialize protobuf → IDataReader
|
||||
4. Loop through all rows, count them (streaming)
|
||||
5. Extract schema (column names + types)
|
||||
6. Print: ✓ row count, schema
|
||||
|
||||
### Verify-Full Flow (--verify-full)
|
||||
1. Open output file
|
||||
2. Decompress zstd → stream protobuf data
|
||||
3. While streaming: count rows, extract schema, compute SHA256 incrementally
|
||||
4. Compare computed SHA256 to stored sidecar file
|
||||
5. Print: ✓ row count, schema, checksum match/mismatch
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
Tools/DbExporter/
|
||||
├── DbExporter.csproj
|
||||
├── Program.cs # CLI entry point, argument parsing
|
||||
├── ExportDefinition.cs # JSON model for definition file
|
||||
├── DatabaseExporter.cs # Core export logic
|
||||
└── Verifier.cs # Verify and verify-full logic
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `protobuf-net-data` | Serialize IDataReader to protobuf |
|
||||
| `ZstdSharp.Port` | Zstd compression |
|
||||
| `Microsoft.Data.SqlClient` | SQL Server connectivity |
|
||||
| `Oracle.ManagedDataAccess.Core` | Oracle connectivity |
|
||||
| `System.Text.Json` | Parse definition files |
|
||||
|
||||
**Target Framework:** `net10.0`
|
||||
|
||||
## Testing with ScadaBridge
|
||||
|
||||
**Connection:**
|
||||
```
|
||||
Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;
|
||||
```
|
||||
|
||||
Definition files will be created in `Tools/DbExporter/definitions/` for ScadaBridge tables.
|
||||
|
||||
**Test approach:**
|
||||
1. Build the tool
|
||||
2. Run export for each definition file
|
||||
3. Run `--verify` to confirm row counts and schemas
|
||||
4. Run `--verify-full` on at least one to confirm checksum works
|
||||
@@ -0,0 +1,649 @@
|
||||
# DbExporter Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Build a CLI tool that exports database query results to compressed protobuf files.
|
||||
|
||||
**Architecture:** Single console app with modular components for definition parsing, database export, and verification.
|
||||
|
||||
**Tech Stack:** .NET 10, protobuf-net-data, ZstdSharp, Microsoft.Data.SqlClient, Oracle.ManagedDataAccess.Core
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Create Project Structure
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/DbExporter.csproj`
|
||||
- Create: `Tools/DbExporter/ExportDefinition.cs`
|
||||
|
||||
**Step 1: Create project directory**
|
||||
|
||||
```bash
|
||||
mkdir -p Tools/DbExporter
|
||||
```
|
||||
|
||||
**Step 2: Create csproj file**
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="protobuf-net-data" Version="4.1.0" />
|
||||
<PackageReference Include="ZstdSharp.Port" Version="0.8.1" />
|
||||
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.2" />
|
||||
<PackageReference Include="Oracle.ManagedDataAccess.Core" Version="23.7.0" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
```
|
||||
|
||||
**Step 3: Create ExportDefinition model**
|
||||
|
||||
```csharp
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DbExporter;
|
||||
|
||||
public sealed class ExportDefinition
|
||||
{
|
||||
[JsonPropertyName("providerType")]
|
||||
public required string ProviderType { get; init; }
|
||||
|
||||
[JsonPropertyName("connectionString")]
|
||||
public required string ConnectionString { get; init; }
|
||||
|
||||
[JsonPropertyName("query")]
|
||||
public required string Query { get; init; }
|
||||
|
||||
[JsonPropertyName("outputPath")]
|
||||
public required string OutputPath { get; init; }
|
||||
|
||||
[JsonPropertyName("compressionLevel")]
|
||||
public int CompressionLevel { get; init; } = 10;
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Verify build**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter
|
||||
git commit -m "feat(DbExporter): create project structure and definition model"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Implement DatabaseExporter
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/DatabaseExporter.cs`
|
||||
|
||||
**Step 1: Create DatabaseExporter class**
|
||||
|
||||
```csharp
|
||||
using System.Data;
|
||||
using System.Data.Common;
|
||||
using System.Security.Cryptography;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Oracle.ManagedDataAccess.Client;
|
||||
using ProtoBuf.Data;
|
||||
using ZstdSharp;
|
||||
|
||||
namespace DbExporter;
|
||||
|
||||
public sealed class DatabaseExporter
|
||||
{
|
||||
public record ExportResult(int RowCount, long UncompressedSize, long CompressedSize, string Sha256Hash);
|
||||
|
||||
public async Task<ExportResult> ExportAsync(ExportDefinition definition, CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Ensure output directory exists
|
||||
var outputDir = Path.GetDirectoryName(definition.OutputPath);
|
||||
if (!string.IsNullOrEmpty(outputDir))
|
||||
Directory.CreateDirectory(outputDir);
|
||||
|
||||
await using var connection = CreateConnection(definition.ProviderType, definition.ConnectionString);
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = definition.Query;
|
||||
command.CommandTimeout = 0; // No timeout for large exports
|
||||
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken);
|
||||
|
||||
int rowCount = 0;
|
||||
long uncompressedSize = 0;
|
||||
|
||||
// Use memory stream to capture uncompressed protobuf for SHA256
|
||||
using var sha256 = SHA256.Create();
|
||||
await using var outputFile = new FileStream(definition.OutputPath, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024);
|
||||
await using var compressStream = new CompressionStream(outputFile, definition.CompressionLevel);
|
||||
await using var hashStream = new CryptoStream(compressStream, sha256, CryptoStreamMode.Write);
|
||||
|
||||
// Serialize to protobuf
|
||||
DataSerializer.Serialize(hashStream, reader);
|
||||
|
||||
// Count rows by re-reading (protobuf-net-data doesn't expose count during serialize)
|
||||
// We'll track this differently - use a counting wrapper or post-verify
|
||||
// For now, we serialize and then verify separately
|
||||
|
||||
hashStream.FlushFinalBlock();
|
||||
uncompressedSize = hashStream.Length;
|
||||
|
||||
var hash = Convert.ToHexString(sha256.Hash!).ToLowerInvariant();
|
||||
|
||||
// Write sidecar hash file
|
||||
var hashFilePath = definition.OutputPath + ".sha256";
|
||||
await File.WriteAllTextAsync(hashFilePath, hash, cancellationToken);
|
||||
|
||||
var compressedSize = new FileInfo(definition.OutputPath).Length;
|
||||
|
||||
// Row count requires a separate pass or we estimate from verify
|
||||
// Return 0 for now, verify will get accurate count
|
||||
return new ExportResult(0, uncompressedSize, compressedSize, hash);
|
||||
}
|
||||
|
||||
private static DbConnection CreateConnection(string providerType, string connectionString)
|
||||
{
|
||||
return providerType.ToLowerInvariant() switch
|
||||
{
|
||||
"sqlserver" => new SqlConnection(connectionString),
|
||||
"oracle" => new OracleConnection(connectionString),
|
||||
_ => throw new ArgumentException($"Unknown provider type: {providerType}. Use 'SqlServer' or 'Oracle'.")
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Verify build**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter/DatabaseExporter.cs
|
||||
git commit -m "feat(DbExporter): implement database export with protobuf+zstd"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Implement Verifier
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/Verifier.cs`
|
||||
|
||||
**Step 1: Create Verifier class**
|
||||
|
||||
```csharp
|
||||
using System.Data;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using ProtoBuf.Data;
|
||||
using ZstdSharp;
|
||||
|
||||
namespace DbExporter;
|
||||
|
||||
public sealed class Verifier
|
||||
{
|
||||
public record VerifyResult(int RowCount, List<ColumnInfo> Schema, string? ComputedHash, string? ExpectedHash, bool? HashMatch);
|
||||
public record ColumnInfo(string Name, Type Type);
|
||||
|
||||
public VerifyResult Verify(string filePath, bool computeHash = false)
|
||||
{
|
||||
using var inputFile = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024);
|
||||
using var decompressStream = new DecompressionStream(inputFile);
|
||||
|
||||
Stream readStream = decompressStream;
|
||||
SHA256? sha256 = null;
|
||||
CryptoStream? hashStream = null;
|
||||
|
||||
if (computeHash)
|
||||
{
|
||||
sha256 = SHA256.Create();
|
||||
hashStream = new CryptoStream(decompressStream, sha256, CryptoStreamMode.Read);
|
||||
readStream = hashStream;
|
||||
}
|
||||
|
||||
using var reader = DataSerializer.Deserialize(readStream);
|
||||
|
||||
// Extract schema
|
||||
var schema = new List<ColumnInfo>();
|
||||
for (int i = 0; i < reader.FieldCount; i++)
|
||||
{
|
||||
schema.Add(new ColumnInfo(reader.GetName(i), reader.GetFieldType(i)));
|
||||
}
|
||||
|
||||
// Count rows
|
||||
int rowCount = 0;
|
||||
while (reader.Read())
|
||||
{
|
||||
rowCount++;
|
||||
}
|
||||
|
||||
string? computedHashStr = null;
|
||||
string? expectedHash = null;
|
||||
bool? hashMatch = null;
|
||||
|
||||
if (computeHash && sha256 != null)
|
||||
{
|
||||
hashStream?.Dispose();
|
||||
computedHashStr = Convert.ToHexString(sha256.Hash!).ToLowerInvariant();
|
||||
|
||||
// Read expected hash from sidecar
|
||||
var hashFilePath = filePath + ".sha256";
|
||||
if (File.Exists(hashFilePath))
|
||||
{
|
||||
expectedHash = File.ReadAllText(hashFilePath).Trim().ToLowerInvariant();
|
||||
hashMatch = computedHashStr == expectedHash;
|
||||
}
|
||||
|
||||
sha256.Dispose();
|
||||
}
|
||||
|
||||
return new VerifyResult(rowCount, schema, computedHashStr, expectedHash, hashMatch);
|
||||
}
|
||||
|
||||
public string FormatSchema(List<ColumnInfo> schema)
|
||||
{
|
||||
var sb = new StringBuilder();
|
||||
foreach (var col in schema)
|
||||
{
|
||||
if (sb.Length > 0) sb.Append(", ");
|
||||
sb.Append($"{col.Name} ({col.Type.Name})");
|
||||
}
|
||||
return sb.ToString();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Verify build**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter/Verifier.cs
|
||||
git commit -m "feat(DbExporter): implement verify and verify-full"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Implement CLI Entry Point
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/Program.cs`
|
||||
|
||||
**Step 1: Create Program.cs with CLI parsing**
|
||||
|
||||
```csharp
|
||||
using System.Text.Json;
|
||||
using DbExporter;
|
||||
|
||||
if (args.Length < 1 || args.Contains("--help") || args.Contains("-h"))
|
||||
{
|
||||
PrintUsage();
|
||||
return args.Contains("--help") || args.Contains("-h") ? 0 : 1;
|
||||
}
|
||||
|
||||
var definitionPath = args[0];
|
||||
var verify = args.Contains("--verify");
|
||||
var verifyFull = args.Contains("--verify-full");
|
||||
|
||||
if (!File.Exists(definitionPath))
|
||||
{
|
||||
Console.WriteLine($"Error: Definition file not found: {definitionPath}");
|
||||
return 1;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var json = await File.ReadAllTextAsync(definitionPath);
|
||||
var definition = JsonSerializer.Deserialize<ExportDefinition>(json);
|
||||
|
||||
if (definition is null)
|
||||
{
|
||||
Console.WriteLine("Error: Failed to parse definition file.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Validate required fields
|
||||
if (string.IsNullOrWhiteSpace(definition.ProviderType))
|
||||
{
|
||||
Console.WriteLine("Error: providerType is required.");
|
||||
return 1;
|
||||
}
|
||||
if (string.IsNullOrWhiteSpace(definition.ConnectionString))
|
||||
{
|
||||
Console.WriteLine("Error: connectionString is required.");
|
||||
return 1;
|
||||
}
|
||||
if (string.IsNullOrWhiteSpace(definition.Query))
|
||||
{
|
||||
Console.WriteLine("Error: query is required.");
|
||||
return 1;
|
||||
}
|
||||
if (string.IsNullOrWhiteSpace(definition.OutputPath))
|
||||
{
|
||||
Console.WriteLine("Error: outputPath is required.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
var exporter = new DatabaseExporter();
|
||||
var verifier = new Verifier();
|
||||
|
||||
Console.WriteLine($"Exporting from {definition.ProviderType}...");
|
||||
Console.WriteLine($"Query: {Truncate(definition.Query, 80)}");
|
||||
|
||||
var result = await exporter.ExportAsync(definition);
|
||||
|
||||
// Always do a quick verify to get row count
|
||||
var quickVerify = verifier.Verify(definition.OutputPath, computeHash: false);
|
||||
|
||||
var ratio = result.CompressedSize > 0 && quickVerify.RowCount > 0
|
||||
? $" ({(double)result.CompressedSize / result.UncompressedSize * 100:F1}%)"
|
||||
: "";
|
||||
|
||||
Console.WriteLine($"✓ Exported: {quickVerify.RowCount:N0} rows, {result.UncompressedSize:N0} → {result.CompressedSize:N0} bytes{ratio}");
|
||||
|
||||
if (verify || verifyFull)
|
||||
{
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("Verifying...");
|
||||
|
||||
var verifyResult = verifier.Verify(definition.OutputPath, computeHash: verifyFull);
|
||||
|
||||
Console.WriteLine($"✓ Verified: {verifyResult.RowCount:N0} rows");
|
||||
Console.WriteLine($"Schema: {verifier.FormatSchema(verifyResult.Schema)}");
|
||||
|
||||
if (verifyFull && verifyResult.HashMatch.HasValue)
|
||||
{
|
||||
if (verifyResult.HashMatch.Value)
|
||||
{
|
||||
Console.WriteLine($"✓ Checksum: SHA256 match ({verifyResult.ComputedHash})");
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.WriteLine($"✗ Checksum: SHA256 MISMATCH");
|
||||
Console.WriteLine($" Expected: {verifyResult.ExpectedHash}");
|
||||
Console.WriteLine($" Computed: {verifyResult.ComputedHash}");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Console.WriteLine($"Error: {ex.Message}");
|
||||
return 1;
|
||||
}
|
||||
|
||||
static void PrintUsage()
|
||||
{
|
||||
Console.WriteLine("Usage: DbExporter <definition-file> [options]");
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("Arguments:");
|
||||
Console.WriteLine(" definition-file Path to JSON definition file");
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("Options:");
|
||||
Console.WriteLine(" --verify Verify output (row count + schema)");
|
||||
Console.WriteLine(" --verify-full Verify output with SHA256 checksum");
|
||||
Console.WriteLine(" --help Show this help");
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("Definition file format:");
|
||||
Console.WriteLine(" {");
|
||||
Console.WriteLine(" \"providerType\": \"SqlServer\",");
|
||||
Console.WriteLine(" \"connectionString\": \"Server=...;Database=...;\",");
|
||||
Console.WriteLine(" \"query\": \"SELECT * FROM MyTable\",");
|
||||
Console.WriteLine(" \"outputPath\": \"./output/mytable.pb.zstd\",");
|
||||
Console.WriteLine(" \"compressionLevel\": 10");
|
||||
Console.WriteLine(" }");
|
||||
}
|
||||
|
||||
static string Truncate(string value, int maxLength)
|
||||
{
|
||||
if (string.IsNullOrEmpty(value)) return value;
|
||||
var singleLine = value.Replace("\r", "").Replace("\n", " ");
|
||||
return singleLine.Length <= maxLength ? singleLine : singleLine[..(maxLength - 3)] + "...";
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Verify build**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter/Program.cs
|
||||
git commit -m "feat(DbExporter): implement CLI entry point"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Fix Export Row Count Issue
|
||||
|
||||
The current implementation computes row count during verify, but can't get it during export (protobuf-net-data streams without counting). Let's fix this by wrapping the IDataReader.
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/CountingDataReader.cs`
|
||||
- Modify: `Tools/DbExporter/DatabaseExporter.cs`
|
||||
|
||||
**Step 1: Create CountingDataReader wrapper**
|
||||
|
||||
```csharp
|
||||
using System.Data;
|
||||
|
||||
namespace DbExporter;
|
||||
|
||||
/// <summary>
|
||||
/// Wraps an IDataReader to count rows as they're read.
|
||||
/// </summary>
|
||||
internal sealed class CountingDataReader : IDataReader
|
||||
{
|
||||
private readonly IDataReader _inner;
|
||||
private int _rowCount;
|
||||
|
||||
public CountingDataReader(IDataReader inner)
|
||||
{
|
||||
_inner = inner;
|
||||
}
|
||||
|
||||
public int RowCount => _rowCount;
|
||||
|
||||
public bool Read()
|
||||
{
|
||||
var result = _inner.Read();
|
||||
if (result) _rowCount++;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Delegate all other members to inner reader
|
||||
public object this[int i] => _inner[i];
|
||||
public object this[string name] => _inner[name];
|
||||
public int Depth => _inner.Depth;
|
||||
public bool IsClosed => _inner.IsClosed;
|
||||
public int RecordsAffected => _inner.RecordsAffected;
|
||||
public int FieldCount => _inner.FieldCount;
|
||||
public void Close() => _inner.Close();
|
||||
public void Dispose() => _inner.Dispose();
|
||||
public bool GetBoolean(int i) => _inner.GetBoolean(i);
|
||||
public byte GetByte(int i) => _inner.GetByte(i);
|
||||
public long GetBytes(int i, long fieldOffset, byte[]? buffer, int bufferoffset, int length) => _inner.GetBytes(i, fieldOffset, buffer, bufferoffset, length);
|
||||
public char GetChar(int i) => _inner.GetChar(i);
|
||||
public long GetChars(int i, long fieldoffset, char[]? buffer, int bufferoffset, int length) => _inner.GetChars(i, fieldoffset, buffer, bufferoffset, length);
|
||||
public IDataReader GetData(int i) => _inner.GetData(i);
|
||||
public string GetDataTypeName(int i) => _inner.GetDataTypeName(i);
|
||||
public DateTime GetDateTime(int i) => _inner.GetDateTime(i);
|
||||
public decimal GetDecimal(int i) => _inner.GetDecimal(i);
|
||||
public double GetDouble(int i) => _inner.GetDouble(i);
|
||||
public Type GetFieldType(int i) => _inner.GetFieldType(i);
|
||||
public float GetFloat(int i) => _inner.GetFloat(i);
|
||||
public Guid GetGuid(int i) => _inner.GetGuid(i);
|
||||
public short GetInt16(int i) => _inner.GetInt16(i);
|
||||
public int GetInt32(int i) => _inner.GetInt32(i);
|
||||
public long GetInt64(int i) => _inner.GetInt64(i);
|
||||
public string GetName(int i) => _inner.GetName(i);
|
||||
public int GetOrdinal(string name) => _inner.GetOrdinal(name);
|
||||
public DataTable GetSchemaTable() => _inner.GetSchemaTable()!;
|
||||
public string GetString(int i) => _inner.GetString(i);
|
||||
public object GetValue(int i) => _inner.GetValue(i);
|
||||
public int GetValues(object[] values) => _inner.GetValues(values);
|
||||
public bool IsDBNull(int i) => _inner.IsDBNull(i);
|
||||
public bool NextResult() => _inner.NextResult();
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Update DatabaseExporter to use CountingDataReader**
|
||||
|
||||
Update the ExportAsync method to wrap the reader:
|
||||
|
||||
```csharp
|
||||
// Replace this line:
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken);
|
||||
|
||||
// With:
|
||||
await using var baseReader = await command.ExecuteReaderAsync(cancellationToken);
|
||||
var reader = new CountingDataReader(baseReader);
|
||||
|
||||
// And update the return to use reader.RowCount instead of 0
|
||||
```
|
||||
|
||||
**Step 3: Verify build**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter/CountingDataReader.cs Tools/DbExporter/DatabaseExporter.cs
|
||||
git commit -m "feat(DbExporter): add counting data reader for accurate row count"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Create ScadaBridge Definition Files
|
||||
|
||||
**Files:**
|
||||
- Create: `Tools/DbExporter/definitions/` directory with definition files
|
||||
|
||||
**Step 1: Create definitions directory**
|
||||
|
||||
```bash
|
||||
mkdir -p Tools/DbExporter/definitions
|
||||
```
|
||||
|
||||
**Step 2: Query ScadaBridge to list tables**
|
||||
|
||||
First, we need to discover what tables exist. Run a quick query to list tables:
|
||||
|
||||
```sql
|
||||
SELECT TABLE_SCHEMA, TABLE_NAME
|
||||
FROM INFORMATION_SCHEMA.TABLES
|
||||
WHERE TABLE_TYPE = 'BASE TABLE'
|
||||
ORDER BY TABLE_SCHEMA, TABLE_NAME
|
||||
```
|
||||
|
||||
**Step 3: Create definition files for key tables**
|
||||
|
||||
Create definition files based on discovered tables. Example for Config.ScadaClients:
|
||||
|
||||
```json
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.OpcUaClients",
|
||||
"outputPath": "./output/opcua-clients.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add Tools/DbExporter/definitions/
|
||||
git commit -m "feat(DbExporter): add ScadaBridge definition files"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Test Export and Verify
|
||||
|
||||
**Step 1: Build the tool**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
```
|
||||
|
||||
**Step 2: Run export for a small table first**
|
||||
|
||||
```bash
|
||||
dotnet run -- definitions/opcua-clients.json --verify
|
||||
```
|
||||
|
||||
**Step 3: Run verify-full**
|
||||
|
||||
```bash
|
||||
dotnet run -- definitions/opcua-clients.json --verify-full
|
||||
```
|
||||
|
||||
**Step 4: Test with larger tables if available**
|
||||
|
||||
Run exports on additional definition files and verify they work correctly.
|
||||
|
||||
**Step 5: Commit any fixes needed**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(DbExporter): address issues found during testing"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Final Cleanup and Documentation
|
||||
|
||||
**Files:**
|
||||
- Update: `Tools/DbExporter/README.md` (optional)
|
||||
|
||||
**Step 1: Review all files for cleanup**
|
||||
|
||||
- Remove any debug code
|
||||
- Ensure consistent formatting
|
||||
- Check for any TODO comments
|
||||
|
||||
**Step 2: Final build and test**
|
||||
|
||||
```bash
|
||||
cd Tools/DbExporter && dotnet build
|
||||
dotnet run -- --help
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "chore(DbExporter): final cleanup"
|
||||
```
|
||||
@@ -0,0 +1,2 @@
|
||||
# Ignore exported output files
|
||||
output/
|
||||
@@ -29,30 +29,36 @@ public sealed class DatabaseExporter
|
||||
await using var baseReader = await command.ExecuteReaderAsync(cancellationToken);
|
||||
var reader = new CountingDataReader(baseReader);
|
||||
|
||||
long uncompressedSize = 0;
|
||||
long uncompressedSize;
|
||||
string hash;
|
||||
int rowCount;
|
||||
|
||||
// Use a counting stream wrapper to track uncompressed bytes
|
||||
using var sha256 = SHA256.Create();
|
||||
await using var outputFile = new FileStream(definition.OutputPath, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024);
|
||||
await using var compressStream = new CompressionStream(outputFile, definition.CompressionLevel);
|
||||
await using var countingStream = new CountingStream(compressStream);
|
||||
await using var hashStream = new CryptoStream(countingStream, sha256, CryptoStreamMode.Write);
|
||||
using (var sha256 = SHA256.Create())
|
||||
{
|
||||
await using var outputFile = new FileStream(definition.OutputPath, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024);
|
||||
await using var compressStream = new CompressionStream(outputFile, definition.CompressionLevel);
|
||||
await using var countingStream = new CountingStream(compressStream);
|
||||
await using var hashStream = new CryptoStream(countingStream, sha256, CryptoStreamMode.Write);
|
||||
|
||||
// Serialize to protobuf
|
||||
DataSerializer.Serialize(hashStream, reader);
|
||||
// Serialize to protobuf
|
||||
DataSerializer.Serialize(hashStream, reader);
|
||||
|
||||
hashStream.FlushFinalBlock();
|
||||
uncompressedSize = countingStream.BytesWritten;
|
||||
hashStream.FlushFinalBlock();
|
||||
uncompressedSize = countingStream.BytesWritten;
|
||||
rowCount = reader.RowCount;
|
||||
|
||||
var hash = Convert.ToHexString(sha256.Hash!).ToLowerInvariant();
|
||||
hash = Convert.ToHexString(sha256.Hash!).ToLowerInvariant();
|
||||
} // All streams disposed here, file fully written
|
||||
|
||||
// Write sidecar hash file
|
||||
var hashFilePath = definition.OutputPath + ".sha256";
|
||||
await File.WriteAllTextAsync(hashFilePath, hash, cancellationToken);
|
||||
|
||||
// Read file size after streams are closed
|
||||
var compressedSize = new FileInfo(definition.OutputPath).Length;
|
||||
|
||||
return new ExportResult(reader.RowCount, uncompressedSize, compressedSize, hash);
|
||||
return new ExportResult(rowCount, uncompressedSize, compressedSize, hash);
|
||||
}
|
||||
|
||||
private static DbConnection CreateConnection(string providerType, string connectionString)
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.EmailProviders",
|
||||
"outputPath": "./output/email-providers.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.Tags",
|
||||
"outputPath": "./output/tags.pb.zstd",
|
||||
"query": "SELECT * FROM sys.tables",
|
||||
"outputPath": "./output/sys-tables.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_SCHEMA, TABLE_NAME",
|
||||
"outputPath": "./output/test-schema.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.LmxClients",
|
||||
"outputPath": "./output/lmx-clients.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Lookup.DataTypes",
|
||||
"outputPath": "./output/lookup-data-types.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Lookup.ScadaClientTypes",
|
||||
"outputPath": "./output/lookup-scada-client-types.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Lookup.ScriptTypes",
|
||||
"outputPath": "./output/lookup-script-types.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Lookup.TriggerTypes",
|
||||
"outputPath": "./output/lookup-trigger-types.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.OpcUaClients",
|
||||
"outputPath": "./output/opcua-clients.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.SmsProviders",
|
||||
"outputPath": "./output/sms-providers.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.TeamsProviders",
|
||||
"outputPath": "./output/teams-providers.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"providerType": "SqlServer",
|
||||
"connectionString": "Server=10.100.0.35;Database=ScadaBridge_Test;User Id=sa;Password=ScadaBridge2024;TrustServerCertificate=true;",
|
||||
"query": "SELECT * FROM Config.TemplateScripts",
|
||||
"outputPath": "./output/template-scripts.pb.zstd",
|
||||
"compressionLevel": 10
|
||||
}
|
||||
Reference in New Issue
Block a user