Commit Graph

47 Commits

Author SHA1 Message Date
Joseph Doherty 7ad4e3ec1c feat(client): configure HttpClient with AuthRedirectHandler
- Add using statement for JdeScoping.Client.Http namespace
- Register AuthRedirectHandler as a scoped service
- Replace simple HttpClient registration with handler pipeline
- AuthRedirectHandler intercepts 401 responses and redirects to login
- Keep all existing service registrations for backward compatibility
2026-01-06 10:15:48 -05:00
Joseph Doherty ba88450feb feat(client): add ViewModelMappingExtensions for Core<->Client mapping
Add extension methods to convert between Core and Client view models:
- SearchViewModel: Core (enum Status) <-> Client (string Status)
- SearchCriteria: Core (primitive lists) <-> Client (full ViewModels)
- JdeUserViewModel -> OperatorViewModel

Handles structural differences in criteria where Core uses primitive
lists (List<long>, List<string>) and Client uses full objects.
2026-01-06 10:14:09 -05:00
Joseph Doherty 0b50c03e44 feat(client): add AuthRedirectHandler for global 401 redirect
Add HTTP message handler that intercepts 401 Unauthorized responses
and redirects to the login page with return URL preserved.
2026-01-06 10:11:43 -05:00
Joseph Doherty badc6a43f3 fix(datasync): throw on cancellation in DevEtlRegistry.RunAllAsync 2026-01-03 16:32:29 -05:00
Joseph Doherty 6bd2b3c285 feat(datasync): add DevEtlRegistry for managing development ETL pipelines 2026-01-03 16:28:35 -05:00
Joseph Doherty fd1e5454da feat(datasync): add BranchDevEtl pipeline for Branch table dev loading 2026-01-03 16:24:34 -05:00
Joseph Doherty 2629cb26e0 fix(datasync): add guards and exception safety to JsonZstdFileSource 2026-01-03 16:21:59 -05:00
Joseph Doherty 57a44e0f3a feat(datasync): add JsonZstdFileSource for reading zstd-compressed JSON files 2026-01-03 16:19:28 -05:00
Joseph Doherty 6d2d8134cb fix(datasync): dispose JsonDocument in JsonStreamingDataReader.Read() 2026-01-03 16:17:38 -05:00
Joseph Doherty bd1c2fd656 feat(datasync): add JsonStreamingDataReader for streaming JSON array parsing 2026-01-03 16:14:31 -05:00
Joseph Doherty bf7cfe9bf1 feat(datasync): add JsonColumnSchema record for ETL column metadata 2026-01-03 16:11:00 -05:00
Joseph Doherty 9ff21958bb feat(datasync): add ZstdSharp.Port package for zstd decompression 2026-01-03 16:08:13 -05:00
Joseph Doherty 7dcbacd5ca fix(etl): address Codex MCP review findings for Phase 2
- Filter MERGE SQL columns to only include columns that exist in destination
  (allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
  instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
  destinations)
- Add tests for FormatQualifiedTableName and output column collision
2026-01-03 11:27:07 -05:00
Joseph Doherty fcd8b660fa feat(etl): add WithCommandTimeout to EtlPipelineBuilder with validation 2026-01-03 11:09:28 -05:00
Joseph Doherty 3145fca371 feat(etl): add column mapping to destinations (intersect with dest schema) 2026-01-03 11:06:38 -05:00
Joseph Doherty 0b317c1ffc feat(etl): add commandTimeoutSeconds to destinations 2026-01-03 11:01:12 -05:00
Joseph Doherty 0e07a76438 feat(etl): add ParseTableName and QUOTENAME to CommonScripts
- Add ParseTableName method to parse table names with optional schema
  - Supports: "Table", "dbo.Table", "[dbo].[Table]"
  - Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
  - Use QUOTENAME() for SQL injection protection
  - Pass schema and table as parameters via SqlScriptRunner
  - Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
2026-01-03 10:56:05 -05:00
Joseph Doherty 40e458148d feat(etl): add parameters support to SqlScriptRunner 2026-01-03 10:52:33 -05:00
Joseph Doherty 0820a9b024 feat(etl): add collision detection to ColumnRenameTransformer 2026-01-03 10:50:03 -05:00
Joseph Doherty ae84cb3d75 feat(etl): add MapOrdinal and date validation with sentinel to JdeDateTransformer
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
  - Validate date is positive
  - Validate century (0 or 1)
  - Validate year (0-99)
  - Validate day of year (1-366 and respects leap year)
  - Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
2026-01-03 10:46:19 -05:00
Joseph Doherty 577e67ec64 feat(etl): add MapOrdinal override to ColumnDropTransformer 2026-01-03 10:35:54 -05:00
Joseph Doherty 506ba5c61d feat(etl): add binary method overrides to DataTransformerBase
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.

Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
2026-01-03 10:33:13 -05:00
Joseph Doherty f5468d019f feat(etl): add MapOrdinal to IDataTransformer interface
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.

- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
2026-01-03 10:28:49 -05:00
Joseph Doherty 27f84fa3c1 feat(etl): add DI registration for ETL pipeline 2026-01-03 09:33:07 -05:00
Joseph Doherty 4c16e62661 feat(etl): implement EtlPipeline and EtlPipelineBuilder
Add pipeline orchestration for ETL operations:
- EtlPipeline: executes source -> transform -> destination flow
- EtlPipelineBuilder: fluent builder for pipeline configuration
- Supports pre/post scripts, multiple transformers
- Returns PipelineResult with step-by-step timing
2026-01-03 09:31:32 -05:00
Joseph Doherty 644e884b21 feat(etl): implement DbBulkMergeDestination for incremental updates 2026-01-03 09:26:43 -05:00
Joseph Doherty 63a0e7cf7e feat(etl): implement DbBulkImportDestination for full table refresh
Add bulk import destination that truncates and loads data using
SqlBulkCopy with configurable batch sizes and streaming support.
2026-01-03 09:22:57 -05:00
Joseph Doherty 8594baf11d feat(etl): implement DbQuerySource for database queries
Adds DbQuerySource, an IImportSource implementation that executes SQL
queries against the LotFinder database. Supports parameterized queries
using anonymous objects and configurable command timeouts.
2026-01-03 09:18:58 -05:00
Joseph Doherty 74c3f37446 feat(etl): implement JdeDateTransformer for Julian date parsing
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
2026-01-03 09:16:11 -05:00
Joseph Doherty 81cb0df6bf feat(etl): implement ColumnRenameTransformer
Add transformer for renaming columns in the data stream during ETL.
Supports case-insensitive column name matching and multiple renames.
2026-01-03 09:13:02 -05:00
Joseph Doherty f1b7809a45 feat(etl): implement ColumnDropTransformer
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
2026-01-03 09:10:54 -05:00
Joseph Doherty 6e7bcadf68 feat(etl): implement TransformingDataReader and DataTransformerBase
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
  field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
  transformer, enabling on-the-fly data transformations
2026-01-03 09:08:17 -05:00
Joseph Doherty c644b578ba feat(etl): add CommonScripts factory for index and statistics scripts 2026-01-03 09:05:13 -05:00
Joseph Doherty 82573df023 feat(etl): implement SqlScriptRunner
Add SqlScriptRunner class that implements IScriptRunner for executing
SQL scripts against the LotFinderDB cache database. Includes constructor
validation and configurable timeout support (default 1 hour).
2026-01-03 09:03:14 -05:00
Joseph Doherty 5a101a60b3 feat(etl): add core ETL pipeline interfaces
Add the four core interfaces for the ETL pipeline:
- IImportSource: defines data sources for reading
- IDataTransformer: defines data transformation layer
- IImportDestination: defines data write destinations
- IScriptRunner: defines script execution capability
2026-01-03 08:59:57 -05:00
Joseph Doherty dac3d216fd feat(etl): add result models for pipeline execution 2026-01-03 08:58:06 -05:00
Joseph Doherty ec4c8fab87 refactor: relocate options classes to dedicated Options folders
Move configuration options from Core/DataAccess/DataSync/ExcelIO to
dedicated Options folders within each project for better organization.
Update all references and tests accordingly.
2026-01-03 08:55:08 -05:00
Joseph Doherty d59f096cc9 chore(client): register CryptoService in DI 2026-01-03 08:45:10 -05:00
Joseph Doherty 3468402200 feat(client): update AuthService to use encrypted login 2026-01-03 08:39:18 -05:00
Joseph Doherty 30153dcbf8 feat(client): add CryptoService for login encryption
Implements ICryptoService for encrypting login credentials using RSA-OAEP.
Uses JavaScript interop with browser's native SubtleCrypto API instead of
Blazor.SubtleCrypto package (which only supports AES-GCM, not RSA-OAEP).

- ICryptoService interface in JdeScoping.Client.Services namespace
- CryptoService fetches server's public key once, caches it
- interop.js rsaEncrypt function for RSA-OAEP encryption via Web Crypto API
2026-01-03 08:33:10 -05:00
Joseph Doherty 1c1752a4fd chore(client): add Blazor.SubtleCrypto package 2026-01-03 08:30:24 -05:00
Joseph Doherty 6debbe2740 feat(api): update AuthController for encrypted login 2026-01-03 08:24:10 -05:00
Joseph Doherty 5ae634888f feat(infrastructure): register RsaKeyService in DI 2026-01-03 08:19:39 -05:00
Joseph Doherty 97ef3e4345 feat(infrastructure): implement RsaKeyService with tests 2026-01-03 08:16:13 -05:00
Joseph Doherty 4e1523b0c0 feat(core): add IRsaKeyService interface 2026-01-03 08:12:57 -05:00
Joseph Doherty 15b292a6f7 feat(core): add shared auth models for encrypted login 2026-01-03 08:11:41 -05:00
Joseph Doherty 26ff8d9b4f Initial commit: JDE Scoping Tool migration project
Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.
2026-01-02 07:43:29 -05:00