Joseph Doherty
258 lines
20 KiB
Markdown
258 lines
20 KiB
Markdown
# Transport
|
|
|
|
The Transport component provides file-based, encrypted bundle export and import of central configuration artifacts between ScadaBridge environments via the Central UI. It is purely central — no site nodes are touched, no runtime state moves, and no site-scoped artifacts travel in a bundle.
|
|
|
|
## Overview
|
|
|
|
Transport (#24) is a central-only component that lives in `src/ZB.MOM.WW.ScadaBridge.Transport/`, split into three functional areas:
|
|
|
|
- `Export/` — `BundleExporter`, `DependencyResolver`, `ResolvedExport`. The export pipeline resolves artifact dependencies, serializes entities to wire-shaped DTOs, optionally encrypts the content, and produces a ZIP-formatted `.scadabundle` stream.
|
|
- `Import/` — `BundleImporter`, `BundleSessionStore`, `BundleSessionEvictionService`, `ArtifactDiff`, `BundleUnlockRateLimiter`. The import pipeline validates the bundle envelope, decrypts it, diffs against the target environment, and applies operator-chosen conflict resolutions in a single EF transaction.
|
|
- `Serialization/` and `Encryption/` — `BundleSerializer`, `EntitySerializer`, `ManifestBuilder`, `ManifestValidator`, `BundleSecretEncryptor`, `BundleManifestAad`. Stateless helpers that handle ZIP packing/unpacking, DTO projection, SHA-256 hashing, and AES-256-GCM authenticated encryption.
|
|
|
|
The single DI entry point is `ServiceCollectionExtensions.AddTransport`, registered by `Host` for central roles only. Stateless helpers are singletons; the exporter and importer are scoped because they reach into per-request EF Core scopes and audited repositories.
|
|
|
|
## Key Concepts
|
|
|
|
### Bundle format
|
|
|
|
A `.scadabundle` file is a ZIP archive with exactly two entries:
|
|
|
|
```
|
|
bundle.scadabundle
|
|
├── manifest.json # always plaintext; never encrypted
|
|
└── content.json # plaintext artifact data (no passphrase)
|
|
OR content.enc # AES-256-GCM ciphertext (passphrase supplied)
|
|
```
|
|
|
|
`manifest.json` is always plaintext so the import wizard can display source provenance and artifact counts before the operator supplies a passphrase. `BundleManifest` carries: `BundleFormatVersion`, `SchemaVersion`, `CreatedAtUtc`, `SourceEnvironment`, `ExportedBy`, `ScadaBridgeVersion`, `ContentHash` (`sha256:<hex>` of the raw content bytes), optional `Encryption` metadata, a `Summary` (artifact counts by type), and a `Contents` list (one `ManifestContentEntry` per artifact with its `dependsOn` edges).
|
|
|
|
`BundleFormatVersion` is an integer gate: the importer hard-refuses any value higher than `TransportOptions.SchemaVersionMajor` (default `1`). Unknown entity types in `Contents` produce a preview-row classification of "unsupported" rather than aborting the whole import.
|
|
|
|
### Encrypted vs plaintext bundles
|
|
|
|
When a passphrase is supplied, `BundleSecretEncryptor` derives a 256-bit key via PBKDF2-SHA256 (default 600,000 iterations, configurable) from a fresh 16-byte random salt, then encrypts the UTF-8 JSON content using AES-256-GCM with a fresh 12-byte nonce. Output format is `ciphertext ‖ GCM-tag` (16-byte tag appended). The salt and nonce are stored in `manifest.json`; the passphrase is never persisted. An unencrypted export is permitted but produces an `UnencryptedBundleExport` audit event rather than `BundleExported`.
|
|
|
|
### AAD binding (T-005)
|
|
|
|
`BundleManifestAad.Compute` produces AES-GCM Associated Authenticated Data by SHA-256-hashing a canonicalized form of the manifest with `ContentHash` zeroed and `Encryption` nulled. This binds `SourceEnvironment`, `ExportedBy`, `Summary`, and `Contents` to the GCM authentication tag. Tampering with any of those fields on a stolen bundle yields an `AuthenticationTagMismatchException` on decryption, making the Step-4 "type the source environment to confirm" gate tamper-evident.
|
|
|
|
### BundleSession and session lifecycle
|
|
|
|
After `LoadAsync` validates and decrypts a bundle, the plaintext content is stored in a `BundleSession` held by the singleton `BundleSessionStore` (a `ConcurrentDictionary<Guid, BundleSession>`). Sessions have a 30-minute TTL (`BundleSessionTtlMinutes`). `BundleSessionEvictionService` sweeps the store every minute so abandoned sessions — and the secrets they carry — are released without waiting for the next `Get` call. `ApplyAsync` explicitly zeros and removes the session on both success and failure (T-007).
|
|
|
|
### Conflict resolution and `BundleImportId` correlation
|
|
|
|
`PreviewAsync` compares each bundle artifact against the target environment using `ArtifactDiff`, classifying items as `Identical`, `Modified`, `New`, or `Blocker`. The operator assigns a `ResolutionAction` (`Add`, `Overwrite`, `Skip`, `Rename`) per item. `ApplyAsync` honours those resolutions in a single EF transaction and threads a new `BundleImportId` GUID through every per-entity audit row via the scoped `IAuditCorrelationContext`. This makes every configuration row written by a bundle import queryable as a group from the audit log.
|
|
|
|
## Architecture
|
|
|
|
### Export pipeline
|
|
|
|
`BundleExporter.ExportAsync` orchestrates these steps in sequence:
|
|
|
|
1. `DependencyResolver.ResolveAsync` — expands the operator's `ExportSelection` to the full transitive closure, then topologically sorts templates (base-before-derived via Kahn's algorithm).
|
|
2. `EntitySerializer.ToBundleContent` — projects EF entity POCOs to wire-shaped DTOs (`BundleContentDto`), carving secret fields (connection strings, credentials, OAuth tokens) into per-entity `SecretsBlock` records.
|
|
3. `ManifestBuilder.Build` — stamps `BundleFormatVersion`, `SchemaVersion`, SHA-256 `ContentHash`, `Summary`, and `Contents` into a `BundleManifest`.
|
|
4. `BundleSerializer.Pack` — serializes the manifest and content into a ZIP stream. When a passphrase is present, `BundleSecretEncryptor.Encrypt` runs with a fresh salt and nonce; `Pack` re-stamps `ContentHash` and `Encryption` in the manifest against the ciphertext it actually writes.
|
|
5. Audit — `IAuditService.LogAsync` writes one `BundleExported` (or `UnencryptedBundleExport`) row with the SHA-256 of the full ZIP stream as `EntityId`.
|
|
|
|
```csharp
|
|
public sealed class BundleExporter : IBundleExporter
|
|
{
|
|
public async Task<Stream> ExportAsync(
|
|
ExportSelection selection,
|
|
string user,
|
|
string sourceEnvironment,
|
|
string? passphrase,
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
var resolved = await _resolver.ResolveAsync(selection, cancellationToken);
|
|
var aggregate = new EntityAggregate(/* resolved collections */);
|
|
var contentDto = _entitySerializer.ToBundleContent(aggregate);
|
|
var summary = new BundleSummary(/* counts from resolved */);
|
|
EncryptionMetadata? encryptionSeed = passphrase is null ? null
|
|
: new EncryptionMetadata("AES-256-GCM", "PBKDF2-SHA256",
|
|
_options.Value.Pbkdf2Iterations, string.Empty, string.Empty);
|
|
var manifest = _manifestBuilder.Build(sourceEnvironment, user, assemblyVersion,
|
|
encryptionSeed, summary, resolved.ContentManifest,
|
|
_bundleSerializer.SerializeContentBytes(contentDto));
|
|
var zipStream = _bundleSerializer.Pack(contentDto, manifest, passphrase, _encryptor);
|
|
var bundleHash = ComputeStreamSha256(zipStream);
|
|
await _auditService.LogAsync(user,
|
|
passphrase is null ? "UnencryptedBundleExport" : "BundleExported",
|
|
"Bundle", bundleHash, sourceEnvironment, /* afterState */, cancellationToken);
|
|
zipStream.Position = 0;
|
|
return zipStream;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Dependency expansion
|
|
|
|
`DependencyResolver` walks five dependency edge types when `ExportSelection.IncludeDependencies` is true:
|
|
|
|
| Edge | Mechanism |
|
|
|------|-----------|
|
|
| Template composes Template | `TemplateComposition.ComposedTemplateId` (BFS over composition graph) |
|
|
| Template references SharedScript | Name-scan of `TemplateScript.Code` and `TemplateAttribute.Value` |
|
|
| Template references ExternalSystem | Name-scan of `TemplateScript.Code` and `TemplateAttribute.DataSourceReference` |
|
|
| ApiMethod references SharedScript | Name-scan of `ApiMethod.Script` |
|
|
| Template folder ancestor chain | Always included regardless of `IncludeDependencies` |
|
|
|
|
`ExternalSystemMethod` records always travel with their parent `ExternalSystemDefinition`. Templates are emitted in topological order (base-before-derived) so the importer can `Apply*` them in sequence without forward-reference gaps.
|
|
|
|
Inbound API keys are explicitly excluded — per re-architecture C4, keys are environment-specific and must be re-issued on the target cluster. `ApiMethod` definitions travel without key bindings.
|
|
|
|
### Import pipeline
|
|
|
|
`BundleImporter` is a three-phase service:
|
|
|
|
**Phase 1 — `LoadAsync`**: copies the upload stream to a seekable `MemoryStream`, enforces the bundle size cap (`MaxBundleSizeMb`), validates the ZIP envelope (entry count, per-entry decompressed size, compression ratio — all before decompression), reads and validates `manifest.json` (format version, SHA-256 content hash), and decrypts `content.enc` when `Encryption` is present. The decrypted bytes are stored in a new `BundleSession`.
|
|
|
|
Passphrase lockout operates at two levels: per-bundle (3-strike counter in `BundleSessionStore`, keyed by `ContentHash` so a second browser tab sharing the same bundle bytes cannot reset the counter) and per-IP-per-hour (`BundleUnlockRateLimiter`, default 10 attempts). A successful decrypt clears the per-bundle counter.
|
|
|
|
```csharp
|
|
// From BundleImporter.LoadAsync — decrypt path (simplified)
|
|
var aad = Encryption.BundleManifestAad.Compute(manifest);
|
|
try
|
|
{
|
|
decryptedContent = _encryptor.Decrypt(contentBytes, manifest.Encryption, passphrase, aad);
|
|
}
|
|
catch (CryptographicException)
|
|
{
|
|
var newCount = _sessionStore.IncrementUnlockFailureCount(manifest.ContentHash);
|
|
if (newCount >= maxAttempts)
|
|
throw new BundleLockedException(manifest.ContentHash, newCount);
|
|
throw;
|
|
}
|
|
_sessionStore.ClearUnlockFailures(manifest.ContentHash);
|
|
```
|
|
|
|
**Phase 2 — `PreviewAsync`**: deserializes the session's plaintext bytes to `BundleContentDto` and calls `ArtifactDiff.Compare*` methods for each entity type. Diff results use `ConflictKind` (`Identical`, `Modified`, `New`, `Blocker`). `Modified` items carry a `FieldDiffJson` payload with changed-field names and old/new values; script bodies record a line-count delta rather than full text to keep the diff compact. `DetectBlockersAsync` scans script bodies for unresolvable `SharedScript` or `ExternalSystem` name references.
|
|
|
|
**Phase 3 — `ApplyAsync`**: runs semantic validation first (a name-resolution scan plus the full `SemanticValidator` from `TemplateEngine`), then applies all resolutions inside one EF transaction. The correlation GUID is set on `IAuditCorrelationContext.BundleImportId` before any writes so that every `IAuditService.LogAsync` call during the apply picks it up automatically. A two-pass flush handles forward references: the first `SaveChangesAsync` materializes identity values for new rows; `ResolveAlarmScriptLinksAsync` and `ResolveCompositionEdgesAsync` run afterward inside the same transaction. On failure, the transaction rolls back, `BundleImportId` is cleared, and a `BundleImportFailed` row is written outside the rolled-back transaction before the exception propagates.
|
|
|
|
```csharp
|
|
// From BundleImporter.ApplyAsync — correlation + transaction pattern
|
|
_correlationContext.BundleImportId = bundleImportId;
|
|
await using var tx = await _dbContext.Database.BeginTransactionAsync(ct);
|
|
try
|
|
{
|
|
var errors = await RunSemanticValidationAsync(content, resolutionMap, ct);
|
|
if (errors.Count > 0) throw new SemanticValidationException(errors);
|
|
|
|
await ApplyTemplateFoldersAsync(/* ... */);
|
|
await ApplyTemplatesAsync(/* ... */);
|
|
// ... other entity types ...
|
|
await _dbContext.SaveChangesAsync(ct); // flush for FK resolution
|
|
await ResolveAlarmScriptLinksAsync(/* ... */);
|
|
await ResolveCompositionEdgesAsync(/* ... */);
|
|
|
|
await _auditService.LogAsync(user, "BundleImported", "Bundle",
|
|
bundleImportId.ToString(), session.Manifest.SourceEnvironment, /* afterState */, ct);
|
|
await _dbContext.SaveChangesAsync(ct);
|
|
await tx.CommitAsync(ct);
|
|
ZeroDecryptedContent(session);
|
|
_sessionStore.Remove(sessionId);
|
|
return new ImportResult(BundleImportId: bundleImportId, /* counts */);
|
|
}
|
|
catch
|
|
{
|
|
await tx.RollbackAsync(ct);
|
|
_correlationContext.BundleImportId = null;
|
|
await _auditService.LogAsync(user, "BundleImportFailed", /* ... */);
|
|
ZeroDecryptedContent(session);
|
|
_sessionStore.Remove(sessionId);
|
|
throw;
|
|
}
|
|
```
|
|
|
|
## Usage
|
|
|
|
The Central UI surfaces Transport through two wizard pages (see [Central UI](./CentralUI.md)):
|
|
|
|
- **Export** (`/design/transport/export`, Design role): a 4-step wizard — select artifacts, review resolved dependencies, set a passphrase, download the `.scadabundle` file.
|
|
- **Import** (`/design/transport/import`, Admin role): a 5-step wizard — upload bundle, enter passphrase, review the diff and set per-artifact resolutions, confirm (operator types the source environment name), view the result and navigate to the Deployments page for any newly stale instances.
|
|
|
|
The same operations are available via the CLI:
|
|
|
|
```bash
|
|
scadabridge bundle export --output FILE --passphrase X [--templates A,B] \
|
|
[--include-dependencies] [--source-environment NAME]
|
|
|
|
scadabridge bundle preview --input FILE --passphrase X
|
|
|
|
scadabridge bundle import --input FILE --passphrase X [--on-conflict skip|overwrite|rename]
|
|
```
|
|
|
|
CLI commands route through `ManagementActor` handlers (`ExportBundleCommand`, `PreviewBundleCommand`, `ImportBundleCommand`), which delegate to the same `IBundleExporter` / `IBundleImporter` scoped services. Bundle bytes ride the existing `/management` JSON envelope as base64.
|
|
|
|
After import, template changes propagate to deployed instances through revision-hash drift detection in `DeploymentService.CompareAsync`. Transport does not write a stale marker — the existing Deployments page surfaces affected instances automatically.
|
|
|
|
## Configuration
|
|
|
|
`TransportOptions` is bound from the `ScadaBridge:Transport` section.
|
|
|
|
| Key | Default | Description |
|
|
|-----|---------|-------------|
|
|
| `SourceEnvironment` | `"scadabridge"` | Environment label stamped in `manifest.json` and used in export filenames. |
|
|
| `SchemaVersionMajor` | `1` | Maximum `bundleFormatVersion` this node accepts at import. |
|
|
| `BundleSessionTtlMinutes` | `30` | TTL for an in-progress import session. |
|
|
| `MaxBundleSizeMb` | `100` | Upload size cap; enforced before any decompression. |
|
|
| `MaxBundleEntryCount` | `4` | Maximum ZIP entries (a valid bundle has exactly 2). |
|
|
| `MaxBundleEntryDecompressedMb` | `200` | Per-entry decompressed size cap (ZIP-bomb defence). |
|
|
| `MaxBundleEntryCompressionRatio` | `50` | Per-entry compression ratio cap (ZIP-bomb defence). |
|
|
| `MaxUnlockAttemptsPerSession` | `3` | Per-bundle passphrase strike limit. |
|
|
| `MaxUnlockAttemptsPerIpPerHour` | `10` | Per-IP trailing-hour unlock attempts. |
|
|
| `Pbkdf2Iterations` | `600000` | PBKDF2-SHA256 iteration count for key derivation. |
|
|
|
|
`SourceEnvironment` should be set per environment (e.g., `dev-cluster`, `prod-cluster`) so the import wizard's confirmation gate works correctly.
|
|
|
|
## Dependencies & Interactions
|
|
|
|
- [Commons (#16)](./Commons.md) — owns `BundleManifest`, `ExportSelection`, `ImportPreview`, `ImportResolution`, `ImportResult`, `BundleSession`, `EncryptionMetadata`, `BundleSummary`, `ManifestContentEntry`, `ConflictKind`, and the `IBundleExporter`, `IBundleImporter`, `IBundleSessionStore`, `IAuditCorrelationContext` interfaces. Transport implementations bind to these contracts; Commons defines nothing Transport-specific beyond the DTOs and interfaces.
|
|
- [Configuration Database (#17)](./ConfigurationDatabase.md) — supplies all repository implementations (`ITemplateEngineRepository`, `IExternalSystemRepository`, `INotificationRepository`, `IInboundApiRepository`), `IAuditService` for per-entity audit rows, the `IAuditCorrelationContext` implementation (`AuditCorrelationContext`) registered as scoped, the `ScadaBridgeDbContext` used for the import transaction, and the EF migration that adds `BundleImportId uniqueidentifier NULL` (with index `IX_AuditLogEntries_BundleImportId`) to `AuditLogEntries`.
|
|
- [Template Engine (#1)](./TemplateEngine.md) — provides `SemanticValidator`, invoked inside `ApplyAsync` before the transaction commits. The importer feeds each imported `TemplateDto` through the validator alongside the combined in-bundle + pre-existing `SharedScript` catalog; validation errors surface as `SemanticValidationException` and roll back the entire import.
|
|
- [Audit Log / Configuration Audit](./AuditLog.md) — every export produces a `BundleExported` or `UnencryptedBundleExport` row; every import produces a `BundleImported` summary row (or `BundleImportFailed` on rollback). Per-entity rows written by `Apply*` helpers carry `BundleImportId` so operators can query all configuration changes from a single import as a group. `BundleImportUnlockFailed` rows are written on passphrase failures. Warning rows `BundleImportAlarmScriptUnresolved` and `BundleImportCompositionUnresolved` are written when second-pass FK rewire cannot resolve a name.
|
|
- [Central UI](./CentralUI.md) — hosts the Export Bundle page under the Design nav group and the Import Bundle page under the Admin nav group; the import result page links to the Deployments page and to the filtered Configuration Audit Log Viewer pre-populated with the completed `BundleImportId`.
|
|
- [Security & Auth (#10)](./Security.md) — enforces `RequireDesign` on export and `RequireAdmin` on import, both at the Razor page layer and inside the `IBundleExporter` / `IBundleImporter` service entrypoints (defense in depth).
|
|
- [Deployment Manager (#2)](./DeploymentManager.md) — not directly called by Transport; template overwrites naturally change the flattened-config hash that `DeploymentService.CompareAsync` reads, causing affected instances to surface as stale on the Deployments page.
|
|
|
|
## Troubleshooting
|
|
|
|
### Bundle upload rejected at format version check
|
|
|
|
`LoadAsync` throws `NotSupportedException` when `manifest.json` carries a `bundleFormatVersion` higher than `TransportOptions.SchemaVersionMajor`. The bundle was exported from a newer ScadaBridge version. Upgrade the target cluster or re-export from a compatible version.
|
|
|
|
### Content hash mismatch on upload
|
|
|
|
`LoadAsync` throws `InvalidDataException("Bundle content hash does not match manifest — file may be corrupt.")`. The ZIP was corrupted in transit. Compare the SHA-256 shown in the export wizard's Step 4 against the downloaded file and re-export if they differ.
|
|
|
|
### Session expired between diff and apply
|
|
|
|
`PreviewAsync` or `ApplyAsync` throws when the `BundleSession` is not found. The 30-minute TTL elapsed while the operator was reviewing the diff. Re-upload the bundle to start a new session.
|
|
|
|
### Apply rolls back with SemanticValidationException
|
|
|
|
A template's scripts reference a `SharedScript` or `ExternalSystem` that exists neither in the bundle nor in the target environment, or a type mismatch exists in a call argument. The exception lists per-template errors. Either re-export with the missing dependency included, or pre-create the missing artifact in the target environment before importing.
|
|
|
|
### Passphrase lockout
|
|
|
|
After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash`), `BundleImporter.LoadAsync` throws `BundleLockedException`. The session is unusable. Re-upload the bundle file to get a new session with a fresh counter. A `BundleImportUnlockFailed` audit row is written on each failed attempt.
|
|
|
|
### BundleImportAlarmScriptUnresolved / BundleImportCompositionUnresolved warnings
|
|
|
|
These audit rows appear when the second-pass rewire (`ResolveAlarmScriptLinksAsync` / `ResolveCompositionEdgesAsync`) cannot match a name to a persisted row. The import commits — the FK is left null / the composition row is skipped — but the warning signals an incomplete import. Re-examine the bundle's dependency graph and re-export with the missing artifacts included.
|
|
|
|
## Related Documentation
|
|
|
|
- [Transport design specification](../requirements/Component-Transport.md)
|
|
- [Central UI](./CentralUI.md)
|
|
- [Audit Log](./AuditLog.md)
|
|
- [Template Engine](./TemplateEngine.md)
|
|
- [Configuration Database](./ConfigurationDatabase.md)
|
|
- [Commons](./Commons.md)
|
|
- [Security](./Security.md)
|
|
- [Deployment Manager](./DeploymentManager.md)
|