Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
- Remove all List<*FilterEntry> properties and *FilterEnabled computed properties from SearchModel
- Delete TableValuedParameterExtensions.cs
- Delete entire FilterEntries folder and all filter entry model classes
- Delete FilterHandlers folder and all filter handler classes
- Delete IFilterHandler interface and FilterResult model
- Update MisQueryBuilder to use SQL extraction functions instead of model properties
- Update SearchProcessor to get ExtractMisData from database using fn_GetSearchExtractMisData
- Update DependencyInjection to remove filter handler registrations
- Delete obsolete tests for TVP extensions and filter handlers
Filter criteria are now stored as JSON in Search.Criteria column and extracted using SQL functions (fn_GetSearch*) during query execution.
Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Add comprehensive tests for the 5 simple table extraction functions:
- fn_GetSearchWorkOrders
- fn_GetSearchItemNumbers
- fn_GetSearchProfitCenters
- fn_GetSearchWorkCenters
- fn_GetSearchOperatorIDs
Each function tested for:
- Valid array returns all values
- Empty array returns empty
- Missing property returns empty
- Search not found returns empty
- NULL criteria returns empty
- Invalid JSON returns empty
- Bad type values (nulls) filtered out
Additional edge case tests for large arrays and long values.
Total: 38 tests (7 per function + 2 additional edge cases).
Add DatabaseTestBase.cs with xUnit Collection for test isolation:
- DatabaseTestCollection disables parallel execution
- DatabaseTestFixture verifies database connectivity
- DatabaseTestBase provides connection and cleanup helpers
- InsertTestSearchAsync for creating test data with SearchCriteria
- InsertTestSearchWithRawCriteriaAsync for testing invalid JSON scenarios
Also adds required packages: Dapper, FluentAssertions, Microsoft.Data.SqlClient
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Add integration tests for SearchApiClient covering:
- GetUserSearchesAsync with authentication returns success
- GetUserSearchesAsync without authentication returns unauthorized
- GetSearchAsync with non-existent ID returns not found
Add comprehensive unit tests for FileApiClient covering:
- Route verification tests for all 4 download endpoints (POST method)
- Route verification tests for 2 upload endpoints (POST method)
- Success tests for downloads returning byte arrays
- Success tests verifying existing data can be passed to downloads
- Success tests for uploads returning IReadOnlyList<T>
- Multipart content-type and filename verification test
- Error tests (404 for download, 400 validation error for upload)
Add unit tests for AuthApiClient covering:
- Route verification tests for GetPublicKey, Login, Logout, GetCurrentUser
- Success tests for all 4 methods
- Proper handling of 204 NoContent for Logout (returns Unit)
- Route verification tests for all 4 methods (FindItems, FindProfitCenters,
FindWorkCenters, FindOperators)
- Query string encoding tests verifying special characters (&, =) and spaces
are properly URL-encoded
- Success tests for all lookup methods verifying deserialization
- Representative error test for 500 status code
Add comprehensive unit tests for SearchApiClient covering:
- Route verification tests for all 6 methods (GetUserSearches, GetQueuedSearches,
GetSearch, CopySearch, CreateSearch, GetResults)
- Success tests verifying deserialization for all methods
- Representative error tests (404 for GetSearch, 401 for GetUserSearches)
Uses MockHttpMessageHandler to verify correct API routes are called
and ApiResult<T> pattern maps responses correctly.
Add 7 edge case tests for ApiClientBase:
- GetAsync_Returns200_EmptyBody_MapsToApiError
- GetAsync_Returns200_InvalidJson_MapsToApiError
- GetAsync_Returns204_ForUnitType_MapsToSuccess
- GetAsync_Returns204_ForNonUnitType_MapsToApiError
- GetAsync_NetworkException_MapsToApiError
- GetAsync_Timeout_MapsToApiError
- GetAsync_Returns400_WithoutValidationFormat_MapsToApiError
Also fix bug in ApiClientBase where 204 NoContent for Unit type
failed due to incorrect type casting. The implicit conversion from
Unit to ApiResult<Unit> must be applied before the runtime cast.
- Add GetAsync_Returns404_MapsToNotFound test
- Add GetAsync_Returns400_WithValidationErrors_MapsToValidationError test
- Add GetAsync_Returns401_MapsToUnauthorized test
- Add GetAsync_Returns403_MapsToForbidden test
- Add GetAsync_Returns500_MapsToApiError test
- Add FrameworkReference to Microsoft.AspNetCore.App for ValidationProblemDetails
Added using statement for JdeScoping.Core.ApiContracts. The existing
routes (/api/file/...) are left unchanged as they test legacy download
by cache key endpoints which differ from the new FileIO routes.
Add @using JdeScoping.Client.Extensions to _Imports.razor so that
extension methods (like ViewModelMappingExtensions) are available
globally in Razor components without explicit using statements.
Delete old service interfaces and implementations that have been
replaced by the new typed API clients:
- ISearchService/SearchService -> ISearchApiClient/SearchApiClient
- ILookupService/LookupService -> ILookupApiClient/LookupApiClient
- IFileService/FileService -> IFileApiClient/FileApiClient
Also remove unused IFileService injection from SearchEdit.razor
(download functionality now uses SearchApi.GetResultsAsync).
Joseph Doherty