Files
scadalink-design/docs/plans/2026-05-24-transport-design.md
Joseph Doherty 1b02f33829 docs(plans): add Transport (Component #24) brainstorming design
File-based, encrypted bundle export/import via the Central UI for
promoting templates, system artifacts, and central-only configuration
across environments. Site-scoped artifacts excluded. Per-artifact
conflict resolution; config-only import (user redeploys via existing
Deployments page). Per-entity audit rows correlated by BundleImportId.
2026-05-24 03:32:21 -04:00

424 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Transport — Bundle Export / Import (Component #24) — Design
**Status:** Approved (brainstorming complete, awaiting implementation plan).
**Author session date:** 2026-05-24.
**Target component number:** #24 (Transport).
---
## 1. Purpose
Provide a file-based, encrypted, environment-agnostic way to **promote configuration artifacts from one ScadaLink cluster to another** (e.g., dev → staging → prod) through the Central UI.
A user with the `Design` role on the source cluster exports a selected set of templates and supporting artifacts to a `.scadabundle` file. A user with the `Admin` role on the target cluster uploads the bundle, reviews a diff, resolves conflicts, and applies it. Import is **config-only**: it updates the central configuration database and marks affected instances stale; the user redeploys to sites via the existing Deployments page.
Transport does **not** touch site nodes, does not move runtime state, and does not move site-scoped artifacts.
---
## 2. Location
- New project: `src/ScadaLink.Transport/`
- New tests: `tests/ScadaLink.Transport.Tests/`, `tests/ScadaLink.Transport.IntegrationTests/`
- New design doc: `docs/requirements/Component-Transport.md` (created as part of implementation).
- Central UI pages: `src/ScadaLink.CentralUI/Components/Pages/Design/TransportExport.razor`, `TransportImport.razor`.
- EF migration in `src/ScadaLink.ConfigurationDatabase/Migrations/` (adds `BundleImportId` column to `ConfigurationAuditLog`).
---
## 3. Scope
### 3.1 In scope — transportable artifact groups
| Group | Entities |
|---|---|
| Templates | `Template`, `TemplateAttribute`, `TemplateAlarm`, `TemplateScript`, `TemplateComposition`, `TemplateFolder` |
| System artifacts | `SharedScript`, `ExternalSystemDefinition` + `ExternalSystemMethod`, `DatabaseConnectionDefinition` |
| Central-only | `NotificationList` + recipients, `SmtpConfiguration`, `ApiKey`, `ApiMethod` |
### 3.2 Out of scope (never bundled)
- **Site-scoped:** `Site`, `Area`, `Instance`, `InstanceAttributeOverride`, `InstanceAlarmOverride`, `InstanceConnectionBinding`, `DataConnection`.
- **Operational state:** `DeploymentRecord`, `SystemArtifactDeploymentRecord`, `DeployedConfigSnapshot`, all audit tables, S&F buffers.
- **Cluster identity / auth state:** LDAP mappings, site-scope rules.
Rationale: environments do not share stable site identities (dev "Site-Dev-1" vs. prod "Plant-North"). Migrating instances would require a name-mapping subsystem, deferred until there is concrete user demand.
### 3.3 Non-goals (explicit v1 exclusions)
- Direct cluster-to-cluster pull (file-based only).
- Auto-redeploy on import.
- Bundle signing (separate keypair); manifest SHA-256 is sufficient for v1 tamper detection.
- Differential / incremental bundles — every export is a full snapshot of selected artifacts.
- Server-side bundle library / browsing.
---
## 4. Bundle Format
### 4.1 File layout
`.scadabundle` is a renamed `.zip`:
```
bundle.scadabundle
├── manifest.json # required, NOT encrypted
├── content.json # plaintext artifact data (when no passphrase)
├── content.enc # encrypted artifact data (when passphrase set)
└── scripts/ # optional: large script bodies as files
├── template-{id}-{name}.cs
└── shared-{id}-{name}.cs
```
Exactly one of `content.json` or `content.enc` is present.
### 4.2 `manifest.json` (plaintext)
```json
{
"bundleFormatVersion": 1,
"schemaVersion": "1.0",
"createdAtUtc": "2026-05-24T12:34:56Z",
"sourceEnvironment": "dev-cluster-a",
"exportedBy": "alice@corp.example",
"scadaLinkVersion": "1.4.2",
"contentHash": "sha256:...",
"encryption": {
"algorithm": "AES-256-GCM",
"kdf": "PBKDF2-SHA256",
"iterations": 600000,
"saltB64": "...",
"ivB64": "..."
},
"summary": {
"templates": 12, "templateFolders": 3, "sharedScripts": 4,
"externalSystems": 2, "dbConnections": 1,
"notificationLists": 1, "smtpConfigs": 0, "apiKeys": 2, "apiMethods": 5
},
"contents": [
{ "type": "Template", "name": "Pump", "version": 5,
"dependsOn": ["SharedScript:PumpUtils"] },
{ "type": "Template", "name": "Pump.WaterPump", "version": 3,
"dependsOn": ["Template:Pump", "ExternalSystem:HistorianAPI"] }
]
}
```
The manifest is plaintext so the import wizard can preview bundle contents and source provenance **before** the user supplies a passphrase.
### 4.3 `content.json` / `content.enc`
- One top-level array per entity type, POCO shapes serialized via `System.Text.Json`.
- Secret fields (API key hashes, SMTP password, external system credentials, DB connection passwords) live in a nested `secrets` block on each affected entity.
- The whole `content` blob is AES-256-GCM encrypted with a key derived via PBKDF2-SHA256 (600 000 iterations) from the user's passphrase, with per-bundle random salt and per-encryption random IV.
- Unencrypted bundles are allowed but UI warns and audit-tags them `UnencryptedBundleExport`.
### 4.4 Forward compatibility
- Unknown top-level entity types in `contents[]` surface in the import preview as "skipped — unsupported in this version" rather than failing the whole import.
- `bundleFormatVersion` newer than what the importer supports → hard refusal at upload with a clear upgrade message.
---
## 5. Architecture (Option A — new component)
```
ScadaLink.Transport
├── IBundleExporter
│ ExportAsync(ExportSelection, Passphrase?, ct) → Stream
├── IBundleImporter
│ LoadAsync(stream, Passphrase?, ct) → BundleSession
│ PreviewAsync(sessionId, ct) → ImportPreview
│ ApplyAsync(sessionId, resolutions, ct) → ImportResult
├── DependencyResolver
├── BundleSerializer (manifest + content JSON; ZIP packer)
├── SecretEncryptor (AES-256-GCM + PBKDF2)
├── BundleSessionStore (in-memory, TTL'd)
└── ManifestValidator (schema/version gating, hash check)
```
- Component is **central-only**. Registered in `ScadaLink.Host` for central roles, never for site roles.
- All persistence goes through **existing audited repository interfaces** in `ScadaLink.ConfigurationDatabase` — no raw `DbContext.SaveChangesAsync` from this component.
- `BundleSessionStore` is in-process on the active central node (matches Blazor Server circuit affinity). 30-minute TTL, GC on expiry, 3-strike passphrase lockout per session.
Rejected alternatives:
- **Option B (extend Template Engine + Deployment Manager):** spreads bundle format and encryption knowledge across components with no shared owner. Cross-artifact dependency resolution has no clean home.
- **Option C (Central-UI-only service):** CLI cannot reuse it (CLI talks to the Management Service). Violates the codebase pattern of keeping UI thin.
---
## 6. Export Flow
### 6.1 UI — 4-step wizard under the Design nav group
**Step 1 — Select artifacts.** Templates rendered as a **tree** matching the existing Templates page (reuse the tree component from `docs/plans/2026-03-23-treeview-component.md`, refactored into a shared `TemplateFolderTree.razor` with a new "checkbox selection" mode). Tri-state checkboxes on folders (`☑` all, `☐` none, `▣` partial). Search filters the tree in place. Other artifact groups (shared scripts, external systems, notification lists, SMTP configs, API keys, API methods) are flat lists — no folder hierarchy in their data model.
**Step 2 — Review dependencies.** The resolver expands the user's selection along these edges:
- `Template A` composes `Template B` → include `B`.
- `Template` references `SharedScript` (by name) → include the script.
- `Template` references `ExternalSystem` → include the def and its methods.
- `ApiMethod` references `SharedScript` → include the script.
- `NotificationList` references `SmtpConfiguration` → include the SMTP config.
- Any folder containing a selected template is included so the structure is reproducible on import.
User can toggle "include all dependencies" off (with warning that the bundle may produce an invalid import).
**Step 3 — Encryption.** Passphrase + confirm. Strength meter. Explicit warning lists every secret field being encrypted. Optional "Export without encryption" path is logged with the `UnencryptedBundleExport` audit flag.
**Step 4 — Download.** Generated filename pattern: `scadabundle-{sourceEnv}-{yyyy-MM-dd-HHmmss}.scadabundle`. SHA-256 displayed for out-of-band verification.
### 6.2 Backend
```
User (Design role) ─► Central UI Export wizard
IBundleExporter
├─► DependencyResolver ─► repositories (read)
├─► EntitySerializer ─► content.json
├─► SecretEncryptor ─► content.enc (if passphrase)
├─► ManifestBuilder ─► manifest.json
ZIP packer → temp file → browser download
IAuditService.LogAsync(BundleExported …)
```
Audit event: `BundleExported` — caller, artifact count, content hash, encrypted yes/no, bundle filename.
Authorization: `RequireDesign` on both the Razor page and `IBundleExporter.ExportAsync` (defense in depth).
---
## 7. Import Flow
### 7.1 UI — 5-step wizard under the Design nav group
**Step 1 — Upload.** Drag-and-drop or browse. On selection, manifest is parsed and displayed (source env, exporter, timestamp, content count, SHA-256, encrypted yes/no). Manifest hash validated against `content` blob.
**Step 2 — Passphrase** (skipped if bundle is unencrypted). 3-wrong-attempt lockout invalidates the session.
**Step 3 — Diff & resolve conflicts.** For each artifact in the bundle, compare to existing by name:
- **Identical** (field-by-field) → marked, auto-skipped, cannot be selected.
- **Modified** → shows side-by-side or `+/-/~` line diff. User picks **Skip / Overwrite / Rename**.
- **New** → marked `+ Add`. User can opt to skip individually.
Bulk "Apply to all" at the top (Overwrite / Skip / Rename), overridable per row. Summary line at bottom: "5 add · 2 overwrite · 1 skip · 0 rename".
Bundle references that cannot be satisfied in either the bundle or the target DB (e.g., a template references a shared script that's neither in the bundle nor pre-existing) appear as **blocker rows** — Apply is disabled until they are resolved (typically by skipping the dependent artifact or re-exporting with dependencies).
**Step 4 — Confirm.** Final summary plus a **"3 instances will become stale"** warning enumerating affected instances. User types the **source environment name** to confirm (typo-resistant gate at the prod boundary).
**Step 5 — Result.** Counts, link to `BundleImported` audit row, link to Deployments page filtered to the newly stale instances.
### 7.2 Backend
```
User (Admin role) ─► uploads bundle
IBundleImporter.LoadAsync
· verify SHA-256 (manifest vs content)
· check bundleFormatVersion supported
· decrypt content.enc with passphrase (if encrypted)
· deserialize entities
· open BundleSession (30-min TTL)
PreviewAsync → diff vs target DB → ImportPreview
▼ (user reviews + resolves conflicts)
ApplyAsync (single EF transaction)
· run pre-deployment semantic validator (Template Engine)
· apply resolutions (add / overwrite / skip / rename)
· upsert TemplateFolder hierarchy
· stale-mark affected Instances
· IAuditService.LogAsync(BundleImported …)
· commit
ImportResult → UI step 5
"View on Deployments →" (existing page)
```
Authorization: `RequireAdmin` on both the Razor page and `IBundleImporter.*` entrypoints.
### 7.3 Stale-instance signaling
When an applied conflict overwrites a template (or a template composed by other templates), every `Instance` whose direct or composed template was overwritten gets its `DeployedRevisionHash` set to `stale` (reusing the existing revision-drift mechanism in Deployment Manager). The Deployments page already surfaces stale instances and offers redeploy; no new UI is needed.
---
## 8. Error Handling
| Where | Failure | Surfaced as |
|---|---|---|
| Upload | Not a zip / missing `manifest.json` | Step 1 error: "Not a valid ScadaLink bundle" |
| Upload | `bundleFormatVersion` newer than supported | Step 1 error: "Bundle was created by ScadaLink v{x}; upgrade this cluster" |
| Upload | Content hash mismatch | Step 1 error: "Bundle integrity check failed — file may be corrupt" |
| Unlock | Wrong passphrase | Step 2 error; 3rd wrong attempt invalidates session, audit `BundleImportUnlockFailed` |
| Preview | Bundle references shared script not in bundle and not in target DB | Listed as a blocker row in Step 3; cannot Apply until resolved |
| Apply | Semantic validation fails (call target type mismatch, etc.) | Modal: "Validation failed — N errors", per-error list, no DB writes |
| Apply | DB transaction fails | Rollback; full transactional guarantee — nothing partial lands; `BundleImportFailed` audit row written outside the rolled-back transaction |
| Session | TTL expired | Step 3+ refresh prompts re-upload |
Imports are **all-or-nothing per bundle.** A bundle either applies fully or not at all.
---
## 9. Security
- **AES-256-GCM** for content encryption with **PBKDF2-SHA256 / 600 000 iterations** (OWASP 2023+ guidance), per-bundle random salt, random IV per encryption. GCM auth tag verified before decryption — wrong passphrase fails cleanly.
- **Passphrase never persisted.** Lives only inside the export/import service call path; discarded after use. No environment variable, no log line.
- **Failed-unlock rate limit:** per-session 3-strike lockout; per-IP-per-hour cap (default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row.
- **Bundle size cap** on upload (default 100 MB, configurable) to bound memory.
- **In-transit:** existing HTTPS to Central UI; no new channel.
- **Audit trail is the chain of custody.** Every export, every import (including aborted ones at validation), every unlock failure is audit-logged with source env, content hash, encrypted yes/no, and artifact summary.
- **Defense in depth on authorization:** `RequireDesign` (export) and `RequireAdmin` (import) enforced both on the Razor page and inside `ScadaLink.Transport` service entrypoints. UI is not the only gate.
- **Bundles are not retained server-side** after download (export) or after `ApplyAsync` commits (import).
---
## 10. Configuration Audit Trail
Import flows through the **same audited repository methods** the UI and CLI use, so every artifact mutated by `ApplyAsync` emits the existing per-entity `ConfigurationAuditLog` row:
| Action during import | `ConfigurationAuditLog` rows emitted |
|---|---|
| Template added | `TemplateCreated` + `TemplateAttributeCreated` (×N) + `TemplateScriptCreated` (×N) + … |
| Template overwritten | `TemplateUpdated` + per-field rows (`TemplateAttributeAdded`, `TemplateScriptUpdated`, …) |
| Template skipped | (no rows) |
| Template renamed-on-import | `TemplateCreated` with the new name (existing row untouched) |
| External system overwritten | `ExternalSystemDefinitionUpdated` + per-method rows |
| Notification list added | `NotificationListCreated` + per-recipient rows |
| API key added | `ApiKeyCreated` |
| Instance marked stale | `InstanceMarkedStale` (existing event) |
**Correlation:** every per-entity row written during an import carries a new optional `BundleImportId` column (the GUID of the parent `BundleImported` summary row). The existing `ConfigurationAuditLog` Central UI page gains a **Bundle Import** filter that surfaces all rows for a given import. The `BundleImported` summary row links to the filtered view.
**Schema change:** one EF migration adds:
- `BundleImportId uniqueidentifier NULL` on `ConfigurationAuditLog`.
- Non-clustered index `IX_ConfigurationAuditLog_BundleImportId`.
**Transactional guarantee:** the EF transaction wraps both the entity writes and the audit log writes (existing pattern). A rollback removes both. The `BundleImported` summary row is the final write inside the transaction, so partial audit trails are impossible. A failed import emits a single `BundleImportFailed` row outside the rolled-back transaction.
---
## 11. Observability
- Structured log events: `transport.export.completed`, `transport.import.applied`, `transport.import.failed`, `transport.import.unlock_failed`.
- Counters: bundle size (bytes), artifact count, decryption failures per hour, sessions opened/expired.
- New `BundleImported` and `BundleImportFailed` audit row types appear in the existing audit log page filter.
---
## 12. Authorization Summary
| Operation | Required role | Enforced at |
|---|---|---|
| Open Export page | `RequireDesign` | Razor page authorize attribute |
| `IBundleExporter.ExportAsync` | `RequireDesign` | Service entrypoint |
| Open Import page | `RequireAdmin` | Razor page authorize attribute |
| `IBundleImporter.LoadAsync` / `PreviewAsync` / `ApplyAsync` | `RequireAdmin` | Service entrypoint |
| `ConfigurationAuditLog` "Bundle Import" filter | `RequireAdmin` or `Audit` | Existing audit page logic |
---
## 13. CLI (Deferred)
The `ScadaLink.Transport` library is callable from both Razor pages and `ScadaLink.CLI`. CLI commands are **not** built in v1 but the design leaves a clean path:
```
scadalink transport export \
--templates Pump,Pump.WaterPump \
--shared-scripts PumpUtils \
--out bundle.scadabundle \
--passphrase-file /run/secrets/p
scadalink transport import bundle.scadabundle \
--passphrase-file /run/secrets/p \
--on-conflict overwrite|skip|rename \
--dry-run
```
Same auth model via the Management API.
---
## 14. Test Plan
| Layer | Project | Coverage |
|---|---|---|
| Unit | `tests/ScadaLink.Transport.Tests` (new) | `BundleSerializer` round-trip, `DependencyResolver` topology + cycle handling, `SecretEncryptor` (encrypt → decrypt round-trip, wrong passphrase fails, tampered ciphertext fails GCM auth tag), `ManifestBuilder` schema-version gating, `ImportPreview` diff for each entity type (identical / modified / new), per-conflict resolution merge logic, stale-instance cascade through composed templates |
| Integration | `tests/ScadaLink.Transport.IntegrationTests` (new) | End-to-end against in-memory + real SQL Server: export → import on same DB (no-op idempotency), export → wipe → import (full restore), export → modify-target → import under each conflict resolution, validation failure rolls back cleanly, audit rows correlated by `BundleImportId` |
| UI | `tests/ScadaLink.CentralUI.Tests` | Razor page authorization (`RequireDesign` for export, `RequireAdmin` for import), wizard step navigation, session TTL expiry, tri-state checkbox tree behavior |
| Manual | docker cluster | Cross-cluster: bring up two clusters, export from one, import to the other, verify stale instances surface on Deployments page, redeploy works |
---
## 15. Deployment
- New component registered in `ScadaLink.Host` for central roles only.
- No new ports, no new database (uses existing central MS SQL).
- One EF migration: `BundleImportId` nullable `uniqueidentifier` column on `ConfigurationAuditLog` + supporting index.
- `bash docker/deploy.sh` picks up the change with the standard rebuild + restart — no compose changes.
---
## 16. Documentation Deliverables
- **New:** `docs/requirements/Component-Transport.md` — full component design doc following the standard structure (Purpose, Location, Responsibilities, detailed design sections, Dependencies, Interactions).
- **Updated:** `README.md` component table — add row #24: "Transport — Bundle export/import for templates, shared scripts, external systems, central-only artifacts; AES-256-GCM encryption; per-conflict resolution on import; correlated audit trail."
- **Updated cross-references in:**
- `Component-TemplateEngine.md` — mention Transport as a consumer of template + composition + script entities.
- `Component-DeploymentManager.md` — mention stale-instance signaling from Transport.
- `Component-SecurityAuth.md` — record the role mapping (`RequireDesign` export, `RequireAdmin` import).
- `Component-ConfigurationDatabase.md` — record the new `BundleImportId` column + index.
- `Component-CentralUI.md` — record the two new pages under the Design nav group.
- `Component-Transport.md` includes the bundle JSON schema in an appendix for downstream tooling.
---
## 17. Rollout
- Ship in **one PR / branch**: design doc + component code + UI pages + EF migration + tests + README + cross-reference updates.
- Backward compatible — no behavior change for users who never open the new pages.
- **No feature flag** (per `CLAUDE.md` guidance — don't add flags for hypothetical toggling).
---
## 18. Open Questions / Future Work
- **Site-scoped artifact transport** (Instances, Areas, bindings, DataConnections). Requires a name-mapping subsystem (source-env-site → target-env-site). Deferred until concrete demand.
- **Direct cluster-to-cluster pull** as an alternative to file handoff. Same `ScadaLink.Transport` library can back it; needs cross-env auth design.
- **Bundle signing** with an asymmetric keypair (separate from passphrase encryption) for stronger non-repudiation. Manifest content hash is sufficient for v1 tamper detection.
- **CLI commands** (`scadalink transport export/import`). Shape pre-decided in §13; not built in v1.
- **Differential bundles** ("only what changed since last export"). YAGNI for v1.
---
## 19. Brainstorming Decisions Captured
(For traceability to the brainstorming session that produced this doc.)
| Question | Decision |
|---|---|
| Transport model | File-based export/import (.scadabundle) |
| Artifact groups | Templates + composition; system artifacts; central-only — site-scoped excluded |
| Secrets | Encrypted in bundle via passphrase (AES-256-GCM + PBKDF2 600k) |
| Selection granularity | Per-artifact with auto-included dependencies |
| Selection UI | Tree view for templates (reuse Templates-page tree); flat lists for other groups |
| Conflict resolution | Interactive per-artifact diff + Skip / Overwrite / Rename |
| Post-import behavior | Config-only; user redeploys via existing Deployments page |
| Authorization | `Design` to export, `Admin` to import |
| Audit | Per-entity rows via existing audited repositories, correlated by `BundleImportId` |
| Architecture | New component `ScadaLink.Transport` (Option A) |