- Move file size read after streams are disposed to get accurate compressed size
- Clean up definition files to use working example queries
- Add .gitignore for output directory
Add Verifier class that reads exported .pb.zstd files and:
- Deserializes protobuf data through zstd decompression
- Extracts schema information (column names and types)
- Counts rows for verification
- Optionally computes SHA256 hash and compares against sidecar file
Adds DatabaseExporter class that exports query results to compressed
protobuf format. Supports SQL Server and Oracle providers with streaming
compression and SHA256 hash verification.
Replace JsonZstdFileSource with ProtobufZstdFileSource across all 20
DevEtl pipeline files. This removes the need for manually defined
JsonColumnSchema arrays as protobuf files embed their schema.
Changes per file:
- Remove using JdeScoping.DataSync.Dev.Models
- Remove private JsonColumnSchema[] Schema array
- Change CacheFileName from .json.zstd to .pb.zstd
- Change WithSource from JsonZstdFileSource to ProtobufZstdFileSource
Add centralized SqlObjects class with constants for stored procedure
and function names using usp_/fn_ prefixes. Update LotFinderRepository
and MisQueryBuilder to use the new naming convention.
- Add optional third parameter for parallelism (default: 8)
- Use Parallel.ForEachAsync for concurrent file processing
- Thread-safe console output with lock
- Thread-safe size counters with Interlocked
Usage: dotnet run -- <cache-dir> <scripts-dir> [parallelism]
Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
- Read schema from SQL CREATE TABLE scripts instead of inferring from JSON
- Stream JSON records using Utf8JsonReader instead of loading all into memory
- Write protobuf output in batches of 10000 rows to reduce memory usage
- Add mapping from cache file names to SQL scripts and table names
- Map SQL types (VARCHAR, BIGINT, DECIMAL, DATETIME2, BIT) to .NET types
- Update usage to require scripts directory as second argument
- Remove all List<*FilterEntry> properties and *FilterEnabled computed properties from SearchModel
- Delete TableValuedParameterExtensions.cs
- Delete entire FilterEntries folder and all filter entry model classes
- Delete FilterHandlers folder and all filter handler classes
- Delete IFilterHandler interface and FilterResult model
- Update MisQueryBuilder to use SQL extraction functions instead of model properties
- Update SearchProcessor to get ExtractMisData from database using fn_GetSearchExtractMisData
- Update DependencyInjection to remove filter handler registrations
- Delete obsolete tests for TVP extensions and filter handlers
Filter criteria are now stored as JSON in Search.Criteria column and extracted using SQL functions (fn_GetSearch*) during query execution.
Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Joseph Doherty