docs: Component-Transport.md (component #24)
This commit is contained in:
Joseph Doherty
356
docs/requirements/Component-Transport.md
Normal file
356
docs/requirements/Component-Transport.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Component: Transport
|
||||
|
||||
## Purpose
|
||||
|
||||
The Transport component provides a file-based, encrypted, environment-agnostic way to promote configuration artifacts from one ScadaLink cluster to another through the Central UI. A user with the Design role on the source cluster exports a selected set of templates and supporting artifacts to a `.scadabundle` file. A user with the Admin role on the target cluster uploads the bundle, reviews a diff, resolves conflicts per artifact, and applies it. Import is config-only: it updates the central configuration database; affected instances surface as stale on the existing Deployments page and the user redeploys via the standard flow. Transport does not touch site nodes, does not move runtime state, and does not move site-scoped artifacts.
|
||||
|
||||
## Location
|
||||
|
||||
- New project: `src/ScadaLink.Transport/`
|
||||
- New tests: `tests/ScadaLink.Transport.Tests/`, `tests/ScadaLink.Transport.IntegrationTests/`
|
||||
- Central UI pages: `src/ScadaLink.CentralUI/Components/Pages/Design/TransportExport.razor`, `TransportImport.razor`
|
||||
- EF migration in `src/ScadaLink.ConfigurationDatabase/Migrations/` (adds `BundleImportId` column to `AuditLogEntries`)
|
||||
- This design doc: `docs/requirements/Component-Transport.md`
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Define and own the `.scadabundle` file format (ZIP container, `manifest.json`, `content.json` / `content.enc`).
|
||||
- Resolve artifact dependencies at export time: base templates, shared scripts, external systems, template folders, notification lists, SMTP configs, API keys, API methods.
|
||||
- Serialize and deserialize all transportable entity types to/from bundle DTOs, carving secret fields into an isolated `SecretsBlock`.
|
||||
- Encrypt content with AES-256-GCM + PBKDF2-SHA256 (600 000 iterations) when a passphrase is supplied; leave content plaintext with a UI warning and an `UnencryptedBundleExport` audit event when none is given.
|
||||
- Validate `manifest.json` on upload: format version gating, SHA-256 content hash verification.
|
||||
- Manage in-memory `BundleSession` objects: 30-minute TTL, 3-strike passphrase lockout per session.
|
||||
- Compute a per-artifact diff between bundle contents and the target environment, classifying each artifact as Identical, Modified, New, or a Blocker.
|
||||
- Apply user-supplied conflict resolutions (Add, Overwrite, Skip, Rename) in a single EF transaction, running the pre-deployment semantic validator before committing.
|
||||
- Emit `BundleExported`, `BundleImported`, `BundleImportFailed`, `UnencryptedBundleExport`, and `BundleImportUnlockFailed` audit events via `IAuditService`.
|
||||
- Thread a `BundleImportId` correlation GUID through every per-entity `AuditLogEntry` written during `ApplyAsync` via a scoped `IAuditCorrelationContext`.
|
||||
- Enforce `RequireDesign` on export and `RequireAdmin` on import both at the Razor page layer and inside the service entrypoints (defense in depth).
|
||||
|
||||
## Bundle Format
|
||||
|
||||
### File Layout
|
||||
|
||||
`.scadabundle` is a renamed `.zip`:
|
||||
|
||||
```
|
||||
bundle.scadabundle
|
||||
├── manifest.json # required, NOT encrypted
|
||||
├── content.json # plaintext artifact data (when no passphrase)
|
||||
├── content.enc # encrypted artifact data (when passphrase set)
|
||||
└── scripts/ # optional: large script bodies as files
|
||||
├── template-{id}-{name}.cs
|
||||
└── shared-{id}-{name}.cs
|
||||
```
|
||||
|
||||
Exactly one of `content.json` or `content.enc` is present.
|
||||
|
||||
### `manifest.json` (plaintext)
|
||||
|
||||
```json
|
||||
{
|
||||
"bundleFormatVersion": 1,
|
||||
"schemaVersion": "1.0",
|
||||
"createdAtUtc": "2026-05-24T12:34:56Z",
|
||||
"sourceEnvironment": "dev-cluster-a",
|
||||
"exportedBy": "alice@corp.example",
|
||||
"scadaLinkVersion": "1.4.2",
|
||||
"contentHash": "sha256:...",
|
||||
"encryption": {
|
||||
"algorithm": "AES-256-GCM",
|
||||
"kdf": "PBKDF2-SHA256",
|
||||
"iterations": 600000,
|
||||
"saltB64": "...",
|
||||
"ivB64": "..."
|
||||
},
|
||||
"summary": {
|
||||
"templates": 12, "templateFolders": 3, "sharedScripts": 4,
|
||||
"externalSystems": 2, "dbConnections": 1,
|
||||
"notificationLists": 1, "smtpConfigs": 0, "apiKeys": 2, "apiMethods": 5
|
||||
},
|
||||
"contents": [
|
||||
{ "type": "Template", "name": "Pump", "version": 5,
|
||||
"dependsOn": ["SharedScript:PumpUtils"] },
|
||||
{ "type": "Template", "name": "Pump.WaterPump", "version": 3,
|
||||
"dependsOn": ["Template:Pump", "ExternalSystem:HistorianAPI"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The manifest is plaintext so the import wizard can preview bundle contents and source provenance before the user supplies a passphrase.
|
||||
|
||||
### `content.json` / `content.enc`
|
||||
|
||||
- One top-level array per entity type, POCO shapes serialized via `System.Text.Json`.
|
||||
- Secret fields (API key hashes, SMTP password, external system credentials, DB connection passwords) live in a nested `secrets` block on each affected entity.
|
||||
- The whole `content` blob is AES-256-GCM encrypted with a key derived via PBKDF2-SHA256 (600 000 iterations) from the user's passphrase, with per-bundle random salt and per-encryption random IV.
|
||||
- Unencrypted bundles are allowed but the UI warns and audit-tags them `UnencryptedBundleExport`.
|
||||
|
||||
### Forward Compatibility
|
||||
|
||||
- Unknown top-level entity types in `contents[]` surface in the import preview as "skipped — unsupported in this version" rather than failing the whole import.
|
||||
- `bundleFormatVersion` newer than what the importer supports produces a hard refusal at upload with a clear upgrade message.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
ScadaLink.Transport
|
||||
├── IBundleExporter
|
||||
│ ExportAsync(ExportSelection, Passphrase?, ct) → Stream
|
||||
├── IBundleImporter
|
||||
│ LoadAsync(stream, Passphrase?, ct) → BundleSession
|
||||
│ PreviewAsync(sessionId, ct) → ImportPreview
|
||||
│ ApplyAsync(sessionId, resolutions, ct) → ImportResult
|
||||
├── DependencyResolver
|
||||
├── BundleSerializer (manifest + content JSON; ZIP packer)
|
||||
├── BundleSecretEncryptor (AES-256-GCM + PBKDF2)
|
||||
├── BundleSessionStore (in-memory, TTL'd)
|
||||
└── ManifestValidator (schema/version gating, hash check)
|
||||
```
|
||||
|
||||
The component is central-only. It is registered in `ScadaLink.Host` for central roles only, never for site roles. All persistence flows through existing audited repository interfaces in `ScadaLink.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session.
|
||||
|
||||
## Export Flow
|
||||
|
||||
### UI — 4-Step Wizard (Design nav group)
|
||||
|
||||
**Step 1 — Select artifacts.** Templates are rendered as a tree matching the existing Templates page (the `TemplateFolderTree.razor` shared component, used in its new checkbox-selection mode). Tri-state checkboxes on folders (`☑` all, `☐` none, `▣` partial). Search filters the tree in place. Other artifact groups (shared scripts, external systems, notification lists, SMTP configs, API keys, API methods) are flat checkbox lists.
|
||||
|
||||
**Step 2 — Review dependencies.** The resolver expands the user's selection along these edges:
|
||||
|
||||
- `Template A` composes `Template B` → include `B`.
|
||||
- `Template` references `SharedScript` (by name) → include the script.
|
||||
- `Template` references `ExternalSystem` → include the definition and its methods.
|
||||
- `ApiMethod` references `SharedScript` → include the script.
|
||||
- `NotificationList` references `SmtpConfiguration` → include the SMTP config.
|
||||
- Any folder containing a selected template is included so the structure is reproducible on import.
|
||||
|
||||
The user can toggle "include all dependencies" off (with a warning that the bundle may produce an invalid import).
|
||||
|
||||
**Step 3 — Encryption.** Passphrase and confirm. Strength meter. Explicit warning lists every secret field being encrypted. An "Export without encryption" path is logged with the `UnencryptedBundleExport` audit flag.
|
||||
|
||||
**Step 4 — Download.** Generated filename pattern: `scadabundle-{sourceEnv}-{yyyy-MM-dd-HHmmss}.scadabundle`. SHA-256 displayed for out-of-band verification.
|
||||
|
||||
### Backend
|
||||
|
||||
```
|
||||
User (Design role) ─► Central UI Export wizard
|
||||
│
|
||||
▼
|
||||
IBundleExporter
|
||||
│
|
||||
├─► DependencyResolver ─► repositories (read)
|
||||
├─► EntitySerializer ─► content.json
|
||||
├─► BundleSecretEncryptor ► content.enc (if passphrase)
|
||||
├─► ManifestBuilder ─► manifest.json
|
||||
▼
|
||||
ZIP packer → temp file → browser download
|
||||
│
|
||||
▼
|
||||
IAuditService.LogAsync(BundleExported …)
|
||||
```
|
||||
|
||||
Audit event: `BundleExported` — caller, artifact count, content hash, encrypted yes/no, bundle filename.
|
||||
|
||||
Authorization: `RequireDesign` on both the Razor page and `IBundleExporter.ExportAsync`.
|
||||
|
||||
## Import Flow
|
||||
|
||||
### UI — 5-Step Wizard (Design nav group)
|
||||
|
||||
**Step 1 — Upload.** Drag-and-drop or browse. On selection, the manifest is parsed and displayed (source env, exporter, timestamp, content count, SHA-256, encrypted yes/no). The manifest hash is validated against the `content` blob.
|
||||
|
||||
**Step 2 — Passphrase** (skipped if the bundle is unencrypted). 3-wrong-attempt lockout invalidates the session.
|
||||
|
||||
**Step 3 — Diff & resolve conflicts.** For each artifact in the bundle, compare to existing by name:
|
||||
|
||||
- **Identical** (field-by-field) → marked, auto-skipped, cannot be selected.
|
||||
- **Modified** → shows `+/-/~` line diff. User picks Skip / Overwrite / Rename.
|
||||
- **New** → marked `+ Add`. User can opt to skip individually.
|
||||
|
||||
Bulk "Apply to all" at the top (Overwrite / Skip / Rename), overridable per row. Summary line at the bottom: "5 add · 2 overwrite · 1 skip · 0 rename".
|
||||
|
||||
Bundle references that cannot be satisfied in either the bundle or the target DB (e.g., a template references a shared script that is neither in the bundle nor pre-existing) appear as **blocker rows** — Apply is disabled until they are resolved (typically by skipping the dependent artifact or re-exporting with dependencies).
|
||||
|
||||
**Step 4 — Confirm.** Final summary plus a "N instances will become stale" warning enumerating affected instances. User types the source environment name to confirm (typo-resistant gate at the prod boundary).
|
||||
|
||||
**Step 5 — Result.** Counts, link to the `BundleImported` audit row, link to the Deployments page filtered to the newly stale instances.
|
||||
|
||||
### Backend
|
||||
|
||||
```
|
||||
User (Admin role) ─► uploads bundle
|
||||
│
|
||||
▼
|
||||
IBundleImporter.LoadAsync
|
||||
· verify SHA-256 (manifest vs content)
|
||||
· check bundleFormatVersion supported
|
||||
· decrypt content.enc with passphrase (if encrypted)
|
||||
· deserialize entities
|
||||
· open BundleSession (30-min TTL)
|
||||
│
|
||||
▼
|
||||
PreviewAsync → diff vs target DB → ImportPreview
|
||||
│
|
||||
▼ (user reviews + resolves conflicts)
|
||||
│
|
||||
ApplyAsync (single EF transaction)
|
||||
· run pre-deployment semantic validator (Template Engine)
|
||||
· apply resolutions (add / overwrite / skip / rename)
|
||||
· upsert TemplateFolder hierarchy
|
||||
· IAuditService.LogAsync(BundleImported …)
|
||||
· commit
|
||||
│
|
||||
▼
|
||||
ImportResult → UI step 5
|
||||
│
|
||||
▼
|
||||
"View on Deployments →" (existing page)
|
||||
```
|
||||
|
||||
Authorization: `RequireAdmin` on both the Razor page and `IBundleImporter.*` entrypoints.
|
||||
|
||||
### Stale-Instance Signaling
|
||||
|
||||
There is no explicit stale-mark write. Overwriting a template during import changes its content, which naturally changes the flattened-config hash that `DeploymentService.CompareAsync` computes at query time. The Deployments page already surfaces any instance whose computed hash differs from the deployed snapshot — no new mechanism is required. The import result enumerates affected instances (by computing the hash drift before committing) so Step 4 can display a preview count and Step 5 can link directly to the Deployments page.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Where | Failure | Surfaced as |
|
||||
|---|---|---|
|
||||
| Upload | Not a zip / missing `manifest.json` | Step 1 error: "Not a valid ScadaLink bundle" |
|
||||
| Upload | `bundleFormatVersion` newer than supported | Step 1 error: "Bundle was created by ScadaLink v{x}; upgrade this cluster" |
|
||||
| Upload | Content hash mismatch | Step 1 error: "Bundle integrity check failed — file may be corrupt" |
|
||||
| Unlock | Wrong passphrase | Step 2 error; 3rd wrong attempt invalidates session, audit `BundleImportUnlockFailed` |
|
||||
| Preview | Bundle references shared script not in bundle and not in target DB | Listed as a blocker row in Step 3; cannot Apply until resolved |
|
||||
| Apply | Semantic validation fails (call target type mismatch, etc.) | Modal: "Validation failed — N errors", per-error list, no DB writes |
|
||||
| Apply | DB transaction fails | Rollback; full transactional guarantee — nothing partial lands; `BundleImportFailed` audit row written outside the rolled-back transaction |
|
||||
| Session | TTL expired | Step 3+ refresh prompts re-upload |
|
||||
|
||||
Imports are all-or-nothing per bundle. A bundle either applies fully or not at all.
|
||||
|
||||
## Security
|
||||
|
||||
- **AES-256-GCM** for content encryption with **PBKDF2-SHA256 / 600 000 iterations** (OWASP 2023+ guidance), per-bundle random salt, random IV per encryption. The GCM auth tag is verified before decryption — a wrong passphrase fails cleanly.
|
||||
- **Passphrase never persisted.** It lives only inside the export/import service call path and is discarded after use. No environment variable, no log line.
|
||||
- **Failed-unlock rate limit:** per-session 3-strike lockout; per-IP-per-hour cap (default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row.
|
||||
- **Bundle size cap** on upload (default 100 MB, configurable) to bound memory.
|
||||
- **In-transit:** existing HTTPS to the Central UI; no new channel.
|
||||
- **Audit trail is the chain of custody.** Every export, every import (including aborted ones at validation), and every unlock failure is audit-logged with source env, content hash, encrypted yes/no, and artifact summary.
|
||||
- **Defense in depth on authorization:** `RequireDesign` (export) and `RequireAdmin` (import) are enforced both on the Razor page and inside `ScadaLink.Transport` service entrypoints. The UI is not the only gate.
|
||||
- **Bundles are not retained server-side** after download (export) or after `ApplyAsync` commits (import).
|
||||
|
||||
## Configuration Audit Trail
|
||||
|
||||
Import flows through the same audited repository methods the UI and CLI use, so every artifact mutated by `ApplyAsync` emits the existing per-entity `AuditLogEntry` row:
|
||||
|
||||
| Action during import | `AuditLogEntries` rows emitted |
|
||||
|---|---|
|
||||
| Template added | `TemplateCreated` + `TemplateAttributeCreated` (×N) + `TemplateScriptCreated` (×N) + … |
|
||||
| Template overwritten | `TemplateUpdated` + per-field rows (`TemplateAttributeAdded`, `TemplateScriptUpdated`, …) |
|
||||
| Template skipped | (no rows) |
|
||||
| Template renamed-on-import | `TemplateCreated` with the new name (existing row untouched) |
|
||||
| External system overwritten | `ExternalSystemDefinitionUpdated` + per-method rows |
|
||||
| Notification list added | `NotificationListCreated` + per-recipient rows |
|
||||
| API key added | `ApiKeyCreated` |
|
||||
|
||||
**Correlation:** every per-entity row written during an import carries a new optional `BundleImportId` column (the GUID of the parent `BundleImported` summary row). The existing Configuration Audit Log Viewer gains a **Bundle Import** filter that surfaces all rows for a given import. The `BundleImported` summary row links to the filtered view.
|
||||
|
||||
**Schema change:** one EF migration adds:
|
||||
|
||||
- `BundleImportId uniqueidentifier NULL` on `AuditLogEntries`.
|
||||
- Non-clustered index `IX_AuditLogEntries_BundleImportId`.
|
||||
|
||||
**Transactional guarantee:** the EF transaction wraps both the entity writes and the audit log writes (existing pattern). A rollback removes both. The `BundleImported` summary row is the final write inside the transaction, so partial audit trails are impossible. A failed import emits a single `BundleImportFailed` row outside the rolled-back transaction.
|
||||
|
||||
## Authorization
|
||||
|
||||
| Operation | Required role | Enforced at |
|
||||
|---|---|---|
|
||||
| Open Export page | `RequireDesign` | Razor page authorize attribute |
|
||||
| `IBundleExporter.ExportAsync` | `RequireDesign` | Service entrypoint |
|
||||
| Open Import page | `RequireAdmin` | Razor page authorize attribute |
|
||||
| `IBundleImporter.LoadAsync` / `PreviewAsync` / `ApplyAsync` | `RequireAdmin` | Service entrypoint |
|
||||
| Configuration Audit Log "Bundle Import" filter | `RequireAdmin` or `Audit` | Existing audit page logic |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **`ScadaLink.Commons`** — Bundle manifest and content DTOs (`BundleManifest`, `ExportSelection`, `ImportPreview`, `ImportResolution`, `ImportResult`, `BundleSession`); transport interface definitions (`IBundleExporter`, `IBundleImporter`, `IBundleSessionStore`, `IAuditCorrelationContext`).
|
||||
- **`ScadaLink.ConfigurationDatabase`** — All repository implementations and `IAuditService` for persistence and per-entity audit emission; `IAuditCorrelationContext` implementation (`AuditCorrelationContext`) registered as a scoped service; EF migration for `BundleImportId`.
|
||||
- **`ScadaLink.TemplateEngine`** — Pre-deployment semantic validator invoked inside `ApplyAsync` before the transaction commits.
|
||||
|
||||
## Interactions
|
||||
|
||||
- **Central UI** — Hosts the Export Bundle (`/design/transport/export`) and Import Bundle (`/design/transport/import`) wizard pages under the Design nav group. The result page links to the Deployments page and to the filtered Configuration Audit Log Viewer.
|
||||
- **Deployment Manager** — Never directly invoked by Transport. Transport-driven template changes propagate to deployed instances through the existing revision-hash drift detection in `DeploymentService.CompareAsync`; the Deployments page surfaces affected instances as stale automatically.
|
||||
- **Security & Auth** — Provides `RequireDesign` and `RequireAdmin` policies from `ScadaLink.Security`, enforced at both the page and service layers.
|
||||
- **Audit Log (Configuration)** — Writes `BundleExported` / `BundleImported` / `BundleImportFailed` / `UnencryptedBundleExport` / `BundleImportUnlockFailed` rows via `IAuditService`; per-entity rows from audited repositories are correlated by `BundleImportId` via `IAuditCorrelationContext`.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Bundle JSON Schema
|
||||
|
||||
The `manifest.json` file is always present in the ZIP root and is never encrypted.
|
||||
|
||||
```json
|
||||
{
|
||||
"bundleFormatVersion": 1,
|
||||
"schemaVersion": "1.0",
|
||||
"createdAtUtc": "2026-05-24T12:34:56Z",
|
||||
"sourceEnvironment": "dev-cluster-a",
|
||||
"exportedBy": "alice@corp.example",
|
||||
"scadaLinkVersion": "1.4.2",
|
||||
"contentHash": "sha256:abc123...",
|
||||
"encryption": {
|
||||
"algorithm": "AES-256-GCM",
|
||||
"kdf": "PBKDF2-SHA256",
|
||||
"iterations": 600000,
|
||||
"saltB64": "<base64-encoded 16-byte random salt>",
|
||||
"ivB64": "<base64-encoded 12-byte random IV>"
|
||||
},
|
||||
"summary": {
|
||||
"templates": 12,
|
||||
"templateFolders": 3,
|
||||
"sharedScripts": 4,
|
||||
"externalSystems": 2,
|
||||
"dbConnections": 1,
|
||||
"notificationLists": 1,
|
||||
"smtpConfigs": 0,
|
||||
"apiKeys": 2,
|
||||
"apiMethods": 5
|
||||
},
|
||||
"contents": [
|
||||
{
|
||||
"type": "Template",
|
||||
"name": "Pump",
|
||||
"version": 5,
|
||||
"dependsOn": ["SharedScript:PumpUtils"]
|
||||
},
|
||||
{
|
||||
"type": "Template",
|
||||
"name": "Pump.WaterPump",
|
||||
"version": 3,
|
||||
"dependsOn": ["Template:Pump", "ExternalSystem:HistorianAPI"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Field descriptions:**
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `bundleFormatVersion` | Integer. Importer hard-refuses any value higher than what its `TransportOptions.SchemaVersionMajor` supports. |
|
||||
| `schemaVersion` | Semver string. Minor increments are additive-only and accepted by older importers. |
|
||||
| `createdAtUtc` | ISO-8601 UTC timestamp of when the export was created. |
|
||||
| `sourceEnvironment` | The `SourceEnvironment` name of the exporting cluster (from `TransportOptions`). Displayed in the import wizard and required to be retyped at the confirm step. |
|
||||
| `exportedBy` | Authenticated username of the person who performed the export. |
|
||||
| `scadaLinkVersion` | Application version of the exporting node. Used for diagnostic display only. |
|
||||
| `contentHash` | `sha256:<hex>` — SHA-256 of the raw `content.json` or `content.enc` bytes (pre-encryption). Verified on upload before any decryption. |
|
||||
| `encryption` | Present only when a passphrase was supplied. Contains the KDF parameters and the per-bundle random salt and IV needed to re-derive the key and decrypt. Omitted for plaintext bundles. |
|
||||
| `encryption.algorithm` | Always `"AES-256-GCM"` in v1. |
|
||||
| `encryption.kdf` | Always `"PBKDF2-SHA256"` in v1. |
|
||||
| `encryption.iterations` | PBKDF2 iteration count. Defaults to 600 000 (configurable via `TransportOptions.Pbkdf2Iterations`). |
|
||||
| `encryption.saltB64` | Base64-encoded 16-byte random salt generated at export time. |
|
||||
| `encryption.ivB64` | Base64-encoded 12-byte (GCM standard) random IV generated at export time. |
|
||||
| `summary` | Artifact count by type, for display in the import wizard's upload step without needing to decrypt content. |
|
||||
| `contents` | Ordered list of all artifacts in the bundle. Order is topological (base templates before derived). Each entry carries the artifact's name, the schema `version` at export time, and its direct `dependsOn` edges for dependency display in the export wizard's Step 2. |
|
||||
Reference in New Issue
Block a user