docs(plans): add Transport (Component #24) brainstorming design

File-based, encrypted bundle export/import via the Central UI for
promoting templates, system artifacts, and central-only configuration
across environments. Site-scoped artifacts excluded. Per-artifact
conflict resolution; config-only import (user redeploys via existing
Deployments page). Per-entity audit rows correlated by BundleImportId.
This commit is contained in:
Joseph Doherty
2026-05-24 03:32:21 -04:00
parent d630e2646b
commit 1b02f33829

View File

@@ -0,0 +1,423 @@
# Transport — Bundle Export / Import (Component #24) — Design
**Status:** Approved (brainstorming complete, awaiting implementation plan).
**Author session date:** 2026-05-24.
**Target component number:** #24 (Transport).
---
## 1. Purpose
Provide a file-based, encrypted, environment-agnostic way to **promote configuration artifacts from one ScadaLink cluster to another** (e.g., dev → staging → prod) through the Central UI.
A user with the `Design` role on the source cluster exports a selected set of templates and supporting artifacts to a `.scadabundle` file. A user with the `Admin` role on the target cluster uploads the bundle, reviews a diff, resolves conflicts, and applies it. Import is **config-only**: it updates the central configuration database and marks affected instances stale; the user redeploys to sites via the existing Deployments page.
Transport does **not** touch site nodes, does not move runtime state, and does not move site-scoped artifacts.
---
## 2. Location
- New project: `src/ScadaLink.Transport/`
- New tests: `tests/ScadaLink.Transport.Tests/`, `tests/ScadaLink.Transport.IntegrationTests/`
- New design doc: `docs/requirements/Component-Transport.md` (created as part of implementation).
- Central UI pages: `src/ScadaLink.CentralUI/Components/Pages/Design/TransportExport.razor`, `TransportImport.razor`.
- EF migration in `src/ScadaLink.ConfigurationDatabase/Migrations/` (adds `BundleImportId` column to `ConfigurationAuditLog`).
---
## 3. Scope
### 3.1 In scope — transportable artifact groups
| Group | Entities |
|---|---|
| Templates | `Template`, `TemplateAttribute`, `TemplateAlarm`, `TemplateScript`, `TemplateComposition`, `TemplateFolder` |
| System artifacts | `SharedScript`, `ExternalSystemDefinition` + `ExternalSystemMethod`, `DatabaseConnectionDefinition` |
| Central-only | `NotificationList` + recipients, `SmtpConfiguration`, `ApiKey`, `ApiMethod` |
### 3.2 Out of scope (never bundled)
- **Site-scoped:** `Site`, `Area`, `Instance`, `InstanceAttributeOverride`, `InstanceAlarmOverride`, `InstanceConnectionBinding`, `DataConnection`.
- **Operational state:** `DeploymentRecord`, `SystemArtifactDeploymentRecord`, `DeployedConfigSnapshot`, all audit tables, S&F buffers.
- **Cluster identity / auth state:** LDAP mappings, site-scope rules.
Rationale: environments do not share stable site identities (dev "Site-Dev-1" vs. prod "Plant-North"). Migrating instances would require a name-mapping subsystem, deferred until there is concrete user demand.
### 3.3 Non-goals (explicit v1 exclusions)
- Direct cluster-to-cluster pull (file-based only).
- Auto-redeploy on import.
- Bundle signing (separate keypair); manifest SHA-256 is sufficient for v1 tamper detection.
- Differential / incremental bundles — every export is a full snapshot of selected artifacts.
- Server-side bundle library / browsing.
---
## 4. Bundle Format
### 4.1 File layout
`.scadabundle` is a renamed `.zip`:
```
bundle.scadabundle
├── manifest.json # required, NOT encrypted
├── content.json # plaintext artifact data (when no passphrase)
├── content.enc # encrypted artifact data (when passphrase set)
└── scripts/ # optional: large script bodies as files
├── template-{id}-{name}.cs
└── shared-{id}-{name}.cs
```
Exactly one of `content.json` or `content.enc` is present.
### 4.2 `manifest.json` (plaintext)
```json
{
"bundleFormatVersion": 1,
"schemaVersion": "1.0",
"createdAtUtc": "2026-05-24T12:34:56Z",
"sourceEnvironment": "dev-cluster-a",
"exportedBy": "alice@corp.example",
"scadaLinkVersion": "1.4.2",
"contentHash": "sha256:...",
"encryption": {
"algorithm": "AES-256-GCM",
"kdf": "PBKDF2-SHA256",
"iterations": 600000,
"saltB64": "...",
"ivB64": "..."
},
"summary": {
"templates": 12, "templateFolders": 3, "sharedScripts": 4,
"externalSystems": 2, "dbConnections": 1,
"notificationLists": 1, "smtpConfigs": 0, "apiKeys": 2, "apiMethods": 5
},
"contents": [
{ "type": "Template", "name": "Pump", "version": 5,
"dependsOn": ["SharedScript:PumpUtils"] },
{ "type": "Template", "name": "Pump.WaterPump", "version": 3,
"dependsOn": ["Template:Pump", "ExternalSystem:HistorianAPI"] }
]
}
```
The manifest is plaintext so the import wizard can preview bundle contents and source provenance **before** the user supplies a passphrase.
### 4.3 `content.json` / `content.enc`
- One top-level array per entity type, POCO shapes serialized via `System.Text.Json`.
- Secret fields (API key hashes, SMTP password, external system credentials, DB connection passwords) live in a nested `secrets` block on each affected entity.
- The whole `content` blob is AES-256-GCM encrypted with a key derived via PBKDF2-SHA256 (600 000 iterations) from the user's passphrase, with per-bundle random salt and per-encryption random IV.
- Unencrypted bundles are allowed but UI warns and audit-tags them `UnencryptedBundleExport`.
### 4.4 Forward compatibility
- Unknown top-level entity types in `contents[]` surface in the import preview as "skipped — unsupported in this version" rather than failing the whole import.
- `bundleFormatVersion` newer than what the importer supports → hard refusal at upload with a clear upgrade message.
---
## 5. Architecture (Option A — new component)
```
ScadaLink.Transport
├── IBundleExporter
│ ExportAsync(ExportSelection, Passphrase?, ct) → Stream
├── IBundleImporter
│ LoadAsync(stream, Passphrase?, ct) → BundleSession
│ PreviewAsync(sessionId, ct) → ImportPreview
│ ApplyAsync(sessionId, resolutions, ct) → ImportResult
├── DependencyResolver
├── BundleSerializer (manifest + content JSON; ZIP packer)
├── SecretEncryptor (AES-256-GCM + PBKDF2)
├── BundleSessionStore (in-memory, TTL'd)
└── ManifestValidator (schema/version gating, hash check)
```
- Component is **central-only**. Registered in `ScadaLink.Host` for central roles, never for site roles.
- All persistence goes through **existing audited repository interfaces** in `ScadaLink.ConfigurationDatabase` — no raw `DbContext.SaveChangesAsync` from this component.
- `BundleSessionStore` is in-process on the active central node (matches Blazor Server circuit affinity). 30-minute TTL, GC on expiry, 3-strike passphrase lockout per session.
Rejected alternatives:
- **Option B (extend Template Engine + Deployment Manager):** spreads bundle format and encryption knowledge across components with no shared owner. Cross-artifact dependency resolution has no clean home.
- **Option C (Central-UI-only service):** CLI cannot reuse it (CLI talks to the Management Service). Violates the codebase pattern of keeping UI thin.
---
## 6. Export Flow
### 6.1 UI — 4-step wizard under the Design nav group
**Step 1 — Select artifacts.** Templates rendered as a **tree** matching the existing Templates page (reuse the tree component from `docs/plans/2026-03-23-treeview-component.md`, refactored into a shared `TemplateFolderTree.razor` with a new "checkbox selection" mode). Tri-state checkboxes on folders (`☑` all, `☐` none, `▣` partial). Search filters the tree in place. Other artifact groups (shared scripts, external systems, notification lists, SMTP configs, API keys, API methods) are flat lists — no folder hierarchy in their data model.
**Step 2 — Review dependencies.** The resolver expands the user's selection along these edges:
- `Template A` composes `Template B` → include `B`.
- `Template` references `SharedScript` (by name) → include the script.
- `Template` references `ExternalSystem` → include the def and its methods.
- `ApiMethod` references `SharedScript` → include the script.
- `NotificationList` references `SmtpConfiguration` → include the SMTP config.
- Any folder containing a selected template is included so the structure is reproducible on import.
User can toggle "include all dependencies" off (with warning that the bundle may produce an invalid import).
**Step 3 — Encryption.** Passphrase + confirm. Strength meter. Explicit warning lists every secret field being encrypted. Optional "Export without encryption" path is logged with the `UnencryptedBundleExport` audit flag.
**Step 4 — Download.** Generated filename pattern: `scadabundle-{sourceEnv}-{yyyy-MM-dd-HHmmss}.scadabundle`. SHA-256 displayed for out-of-band verification.
### 6.2 Backend
```
User (Design role) ─► Central UI Export wizard
IBundleExporter
├─► DependencyResolver ─► repositories (read)
├─► EntitySerializer ─► content.json
├─► SecretEncryptor ─► content.enc (if passphrase)
├─► ManifestBuilder ─► manifest.json
ZIP packer → temp file → browser download
IAuditService.LogAsync(BundleExported …)
```
Audit event: `BundleExported` — caller, artifact count, content hash, encrypted yes/no, bundle filename.
Authorization: `RequireDesign` on both the Razor page and `IBundleExporter.ExportAsync` (defense in depth).
---
## 7. Import Flow
### 7.1 UI — 5-step wizard under the Design nav group
**Step 1 — Upload.** Drag-and-drop or browse. On selection, manifest is parsed and displayed (source env, exporter, timestamp, content count, SHA-256, encrypted yes/no). Manifest hash validated against `content` blob.
**Step 2 — Passphrase** (skipped if bundle is unencrypted). 3-wrong-attempt lockout invalidates the session.
**Step 3 — Diff & resolve conflicts.** For each artifact in the bundle, compare to existing by name:
- **Identical** (field-by-field) → marked, auto-skipped, cannot be selected.
- **Modified** → shows side-by-side or `+/-/~` line diff. User picks **Skip / Overwrite / Rename**.
- **New** → marked `+ Add`. User can opt to skip individually.
Bulk "Apply to all" at the top (Overwrite / Skip / Rename), overridable per row. Summary line at bottom: "5 add · 2 overwrite · 1 skip · 0 rename".
Bundle references that cannot be satisfied in either the bundle or the target DB (e.g., a template references a shared script that's neither in the bundle nor pre-existing) appear as **blocker rows** — Apply is disabled until they are resolved (typically by skipping the dependent artifact or re-exporting with dependencies).
**Step 4 — Confirm.** Final summary plus a **"3 instances will become stale"** warning enumerating affected instances. User types the **source environment name** to confirm (typo-resistant gate at the prod boundary).
**Step 5 — Result.** Counts, link to `BundleImported` audit row, link to Deployments page filtered to the newly stale instances.
### 7.2 Backend
```
User (Admin role) ─► uploads bundle
IBundleImporter.LoadAsync
· verify SHA-256 (manifest vs content)
· check bundleFormatVersion supported
· decrypt content.enc with passphrase (if encrypted)
· deserialize entities
· open BundleSession (30-min TTL)
PreviewAsync → diff vs target DB → ImportPreview
▼ (user reviews + resolves conflicts)
ApplyAsync (single EF transaction)
· run pre-deployment semantic validator (Template Engine)
· apply resolutions (add / overwrite / skip / rename)
· upsert TemplateFolder hierarchy
· stale-mark affected Instances
· IAuditService.LogAsync(BundleImported …)
· commit
ImportResult → UI step 5
"View on Deployments →" (existing page)
```
Authorization: `RequireAdmin` on both the Razor page and `IBundleImporter.*` entrypoints.
### 7.3 Stale-instance signaling
When an applied conflict overwrites a template (or a template composed by other templates), every `Instance` whose direct or composed template was overwritten gets its `DeployedRevisionHash` set to `stale` (reusing the existing revision-drift mechanism in Deployment Manager). The Deployments page already surfaces stale instances and offers redeploy; no new UI is needed.
---
## 8. Error Handling
| Where | Failure | Surfaced as |
|---|---|---|
| Upload | Not a zip / missing `manifest.json` | Step 1 error: "Not a valid ScadaLink bundle" |
| Upload | `bundleFormatVersion` newer than supported | Step 1 error: "Bundle was created by ScadaLink v{x}; upgrade this cluster" |
| Upload | Content hash mismatch | Step 1 error: "Bundle integrity check failed — file may be corrupt" |
| Unlock | Wrong passphrase | Step 2 error; 3rd wrong attempt invalidates session, audit `BundleImportUnlockFailed` |
| Preview | Bundle references shared script not in bundle and not in target DB | Listed as a blocker row in Step 3; cannot Apply until resolved |
| Apply | Semantic validation fails (call target type mismatch, etc.) | Modal: "Validation failed — N errors", per-error list, no DB writes |
| Apply | DB transaction fails | Rollback; full transactional guarantee — nothing partial lands; `BundleImportFailed` audit row written outside the rolled-back transaction |
| Session | TTL expired | Step 3+ refresh prompts re-upload |
Imports are **all-or-nothing per bundle.** A bundle either applies fully or not at all.
---
## 9. Security
- **AES-256-GCM** for content encryption with **PBKDF2-SHA256 / 600 000 iterations** (OWASP 2023+ guidance), per-bundle random salt, random IV per encryption. GCM auth tag verified before decryption — wrong passphrase fails cleanly.
- **Passphrase never persisted.** Lives only inside the export/import service call path; discarded after use. No environment variable, no log line.
- **Failed-unlock rate limit:** per-session 3-strike lockout; per-IP-per-hour cap (default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row.
- **Bundle size cap** on upload (default 100 MB, configurable) to bound memory.
- **In-transit:** existing HTTPS to Central UI; no new channel.
- **Audit trail is the chain of custody.** Every export, every import (including aborted ones at validation), every unlock failure is audit-logged with source env, content hash, encrypted yes/no, and artifact summary.
- **Defense in depth on authorization:** `RequireDesign` (export) and `RequireAdmin` (import) enforced both on the Razor page and inside `ScadaLink.Transport` service entrypoints. UI is not the only gate.
- **Bundles are not retained server-side** after download (export) or after `ApplyAsync` commits (import).
---
## 10. Configuration Audit Trail
Import flows through the **same audited repository methods** the UI and CLI use, so every artifact mutated by `ApplyAsync` emits the existing per-entity `ConfigurationAuditLog` row:
| Action during import | `ConfigurationAuditLog` rows emitted |
|---|---|
| Template added | `TemplateCreated` + `TemplateAttributeCreated` (×N) + `TemplateScriptCreated` (×N) + … |
| Template overwritten | `TemplateUpdated` + per-field rows (`TemplateAttributeAdded`, `TemplateScriptUpdated`, …) |
| Template skipped | (no rows) |
| Template renamed-on-import | `TemplateCreated` with the new name (existing row untouched) |
| External system overwritten | `ExternalSystemDefinitionUpdated` + per-method rows |
| Notification list added | `NotificationListCreated` + per-recipient rows |
| API key added | `ApiKeyCreated` |
| Instance marked stale | `InstanceMarkedStale` (existing event) |
**Correlation:** every per-entity row written during an import carries a new optional `BundleImportId` column (the GUID of the parent `BundleImported` summary row). The existing `ConfigurationAuditLog` Central UI page gains a **Bundle Import** filter that surfaces all rows for a given import. The `BundleImported` summary row links to the filtered view.
**Schema change:** one EF migration adds:
- `BundleImportId uniqueidentifier NULL` on `ConfigurationAuditLog`.
- Non-clustered index `IX_ConfigurationAuditLog_BundleImportId`.
**Transactional guarantee:** the EF transaction wraps both the entity writes and the audit log writes (existing pattern). A rollback removes both. The `BundleImported` summary row is the final write inside the transaction, so partial audit trails are impossible. A failed import emits a single `BundleImportFailed` row outside the rolled-back transaction.
---
## 11. Observability
- Structured log events: `transport.export.completed`, `transport.import.applied`, `transport.import.failed`, `transport.import.unlock_failed`.
- Counters: bundle size (bytes), artifact count, decryption failures per hour, sessions opened/expired.
- New `BundleImported` and `BundleImportFailed` audit row types appear in the existing audit log page filter.
---
## 12. Authorization Summary
| Operation | Required role | Enforced at |
|---|---|---|
| Open Export page | `RequireDesign` | Razor page authorize attribute |
| `IBundleExporter.ExportAsync` | `RequireDesign` | Service entrypoint |
| Open Import page | `RequireAdmin` | Razor page authorize attribute |
| `IBundleImporter.LoadAsync` / `PreviewAsync` / `ApplyAsync` | `RequireAdmin` | Service entrypoint |
| `ConfigurationAuditLog` "Bundle Import" filter | `RequireAdmin` or `Audit` | Existing audit page logic |
---
## 13. CLI (Deferred)
The `ScadaLink.Transport` library is callable from both Razor pages and `ScadaLink.CLI`. CLI commands are **not** built in v1 but the design leaves a clean path:
```
scadalink transport export \
--templates Pump,Pump.WaterPump \
--shared-scripts PumpUtils \
--out bundle.scadabundle \
--passphrase-file /run/secrets/p
scadalink transport import bundle.scadabundle \
--passphrase-file /run/secrets/p \
--on-conflict overwrite|skip|rename \
--dry-run
```
Same auth model via the Management API.
---
## 14. Test Plan
| Layer | Project | Coverage |
|---|---|---|
| Unit | `tests/ScadaLink.Transport.Tests` (new) | `BundleSerializer` round-trip, `DependencyResolver` topology + cycle handling, `SecretEncryptor` (encrypt → decrypt round-trip, wrong passphrase fails, tampered ciphertext fails GCM auth tag), `ManifestBuilder` schema-version gating, `ImportPreview` diff for each entity type (identical / modified / new), per-conflict resolution merge logic, stale-instance cascade through composed templates |
| Integration | `tests/ScadaLink.Transport.IntegrationTests` (new) | End-to-end against in-memory + real SQL Server: export → import on same DB (no-op idempotency), export → wipe → import (full restore), export → modify-target → import under each conflict resolution, validation failure rolls back cleanly, audit rows correlated by `BundleImportId` |
| UI | `tests/ScadaLink.CentralUI.Tests` | Razor page authorization (`RequireDesign` for export, `RequireAdmin` for import), wizard step navigation, session TTL expiry, tri-state checkbox tree behavior |
| Manual | docker cluster | Cross-cluster: bring up two clusters, export from one, import to the other, verify stale instances surface on Deployments page, redeploy works |
---
## 15. Deployment
- New component registered in `ScadaLink.Host` for central roles only.
- No new ports, no new database (uses existing central MS SQL).
- One EF migration: `BundleImportId` nullable `uniqueidentifier` column on `ConfigurationAuditLog` + supporting index.
- `bash docker/deploy.sh` picks up the change with the standard rebuild + restart — no compose changes.
---
## 16. Documentation Deliverables
- **New:** `docs/requirements/Component-Transport.md` — full component design doc following the standard structure (Purpose, Location, Responsibilities, detailed design sections, Dependencies, Interactions).
- **Updated:** `README.md` component table — add row #24: "Transport — Bundle export/import for templates, shared scripts, external systems, central-only artifacts; AES-256-GCM encryption; per-conflict resolution on import; correlated audit trail."
- **Updated cross-references in:**
- `Component-TemplateEngine.md` — mention Transport as a consumer of template + composition + script entities.
- `Component-DeploymentManager.md` — mention stale-instance signaling from Transport.
- `Component-SecurityAuth.md` — record the role mapping (`RequireDesign` export, `RequireAdmin` import).
- `Component-ConfigurationDatabase.md` — record the new `BundleImportId` column + index.
- `Component-CentralUI.md` — record the two new pages under the Design nav group.
- `Component-Transport.md` includes the bundle JSON schema in an appendix for downstream tooling.
---
## 17. Rollout
- Ship in **one PR / branch**: design doc + component code + UI pages + EF migration + tests + README + cross-reference updates.
- Backward compatible — no behavior change for users who never open the new pages.
- **No feature flag** (per `CLAUDE.md` guidance — don't add flags for hypothetical toggling).
---
## 18. Open Questions / Future Work
- **Site-scoped artifact transport** (Instances, Areas, bindings, DataConnections). Requires a name-mapping subsystem (source-env-site → target-env-site). Deferred until concrete demand.
- **Direct cluster-to-cluster pull** as an alternative to file handoff. Same `ScadaLink.Transport` library can back it; needs cross-env auth design.
- **Bundle signing** with an asymmetric keypair (separate from passphrase encryption) for stronger non-repudiation. Manifest content hash is sufficient for v1 tamper detection.
- **CLI commands** (`scadalink transport export/import`). Shape pre-decided in §13; not built in v1.
- **Differential bundles** ("only what changed since last export"). YAGNI for v1.
---
## 19. Brainstorming Decisions Captured
(For traceability to the brainstorming session that produced this doc.)
| Question | Decision |
|---|---|
| Transport model | File-based export/import (.scadabundle) |
| Artifact groups | Templates + composition; system artifacts; central-only — site-scoped excluded |
| Secrets | Encrypted in bundle via passphrase (AES-256-GCM + PBKDF2 600k) |
| Selection granularity | Per-artifact with auto-included dependencies |
| Selection UI | Tree view for templates (reuse Templates-page tree); flat lists for other groups |
| Conflict resolution | Interactive per-artifact diff + Skip / Overwrite / Rename |
| Post-import behavior | Config-only; user redeploys via existing Deployments page |
| Authorization | `Design` to export, `Admin` to import |
| Audit | Per-entity rows via existing audited repositories, correlated by `BundleImportId` |
| Architecture | New component `ScadaLink.Transport` (Option A) |