Replace JsonZstdFileSource with ProtobufZstdFileSource across all 20
DevEtl pipeline files. This removes the need for manually defined
JsonColumnSchema arrays as protobuf files embed their schema.
Changes per file:
- Remove using JdeScoping.DataSync.Dev.Models
- Remove private JsonColumnSchema[] Schema array
- Change CacheFileName from .json.zstd to .pb.zstd
- Change WithSource from JsonZstdFileSource to ProtobufZstdFileSource
Add centralized SqlObjects class with constants for stored procedure
and function names using usp_/fn_ prefixes. Update LotFinderRepository
and MisQueryBuilder to use the new naming convention.
- Add optional third parameter for parallelism (default: 8)
- Use Parallel.ForEachAsync for concurrent file processing
- Thread-safe console output with lock
- Thread-safe size counters with Interlocked
Usage: dotnet run -- <cache-dir> <scripts-dir> [parallelism]
Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
- Read schema from SQL CREATE TABLE scripts instead of inferring from JSON
- Stream JSON records using Utf8JsonReader instead of loading all into memory
- Write protobuf output in batches of 10000 rows to reduce memory usage
- Add mapping from cache file names to SQL scripts and table names
- Map SQL types (VARCHAR, BIGINT, DECIMAL, DATETIME2, BIT) to .NET types
- Update usage to require scripts directory as second argument
- Remove all List<*FilterEntry> properties and *FilterEnabled computed properties from SearchModel
- Delete TableValuedParameterExtensions.cs
- Delete entire FilterEntries folder and all filter entry model classes
- Delete FilterHandlers folder and all filter handler classes
- Delete IFilterHandler interface and FilterResult model
- Update MisQueryBuilder to use SQL extraction functions instead of model properties
- Update SearchProcessor to get ExtractMisData from database using fn_GetSearchExtractMisData
- Update DependencyInjection to remove filter handler registrations
- Delete obsolete tests for TVP extensions and filter handlers
Filter criteria are now stored as JSON in Search.Criteria column and extracted using SQL functions (fn_GetSearch*) during query execution.
Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Add comprehensive tests for the 5 simple table extraction functions:
- fn_GetSearchWorkOrders
- fn_GetSearchItemNumbers
- fn_GetSearchProfitCenters
- fn_GetSearchWorkCenters
- fn_GetSearchOperatorIDs
Each function tested for:
- Valid array returns all values
- Empty array returns empty
- Missing property returns empty
- Search not found returns empty
- NULL criteria returns empty
- Invalid JSON returns empty
- Bad type values (nulls) filtered out
Additional edge case tests for large arrays and long values.
Total: 38 tests (7 per function + 2 additional edge cases).
Add DatabaseTestBase.cs with xUnit Collection for test isolation:
- DatabaseTestCollection disables parallel execution
- DatabaseTestFixture verifies database connectivity
- DatabaseTestBase provides connection and cleanup helpers
- InsertTestSearchAsync for creating test data with SearchCriteria
- InsertTestSearchWithRawCriteriaAsync for testing invalid JSON scenarios
Also adds required packages: Dapper, FluentAssertions, Microsoft.Data.SqlClient
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Joseph Doherty