docs(transport): design addendum for cluster-to-cluster pull and differential bundles (ScadaBridge#45)
Neither feature is specified well enough to build safely, so this records the open questions, the options and a recommendation instead of code. - Pull: the existing CLI can already orchestrate export from one cluster and import into another with per-cluster auth and audit. A server-side pull needs decisions on the machine credential, source-side authorisation, network direction and passphrase handling. Recommendation: orchestrate it with the CLI and RequireSignedBundles; write no code now. - Differential: imports already auto-skip Identical rows, so a full bundle applies only what changed. The baseline, deletions and target drift are undefined. Recommendation: keep YAGNI; if it is ever needed, use per-entry content hashes plus --since, and never tombstones. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,124 @@
|
||||
# Transport addendum — cluster-to-cluster pull and differential bundles (ScadaBridge#45)
|
||||
|
||||
**Date:** 2026-11-09 · **Status:** design addendum. Not implemented, and not scheduled.
|
||||
**Origin:** `2026-05-24-transport-design.md` §18 and deferred-work register row 19 (now Gitea #45).
|
||||
The third deferred item, bundle signing, **is** implemented. See "Bundle Signing" in
|
||||
`docs/requirements/Component-Transport.md`.
|
||||
|
||||
#45 grouped three deferred Transport features. Signing could be built safely because it is
|
||||
off by default and adds only an optional zip entry. The other two are not specified well
|
||||
enough to build:
|
||||
|
||||
- **Pull** opens a new network surface whose authentication and network model nobody has
|
||||
decided.
|
||||
- **Differential** has no defined baseline and no defined deletion semantics.
|
||||
|
||||
This addendum records the questions each one has to answer, the options, and a
|
||||
recommendation, so either feature can be picked up when there is a concrete need for it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Cluster-to-cluster pull
|
||||
|
||||
### What exists today
|
||||
|
||||
A pull can already be **orchestrated** without any new server code. The CLI speaks to any
|
||||
central cluster's `POST /management` endpoint. It authenticates with a bearer token from
|
||||
`/auth/token`, and the server does the LDAP bind and role check. So an operator or a CI job
|
||||
can run:
|
||||
|
||||
```bash
|
||||
scadabridge --url https://central-dev bundle export --all --passphrase "$P" --output b.scadabundle
|
||||
scadabridge --url https://central-prod bundle import --input b.scadabundle --passphrase "$P" --on-conflict skip
|
||||
```
|
||||
|
||||
Each leg runs under its own cluster's role gate (`RequireDesign` to export, `RequireAdmin` to
|
||||
import) with a human's or service account's credentials, and each leg is audited. With
|
||||
signing on, the target can also require that the bundle came from a trusted source key.
|
||||
None of this adds a server-to-server path.
|
||||
|
||||
### What a server-side pull would add, and the questions it raises
|
||||
|
||||
A server-side pull means the target central calls the source central itself, for example
|
||||
from an Import wizard option such as "Pull from environment X".
|
||||
|
||||
| # | Question | Why it matters |
|
||||
|---|---|---|
|
||||
| P1 | **Credential.** What identity does the target present to the source? | Management auth is LDAP-backed and per-user. A pull would need either (a) a stored service-account credential for env A kept in env B's secret store, or (b) a new machine credential such as an export-scoped API key, mTLS client certificate, or PSK. Inbound-API keys (`ZB.MOM.WW.Auth.ApiKeys`) are scoped to inbound API *methods* and are deliberately non-transportable, so they cannot be reused as they are. |
|
||||
| P2 | **Authorisation on the source.** Which role does a remote pull run under, and can it be limited to a selection? | `RequireDesign` is a human authoring role. A machine pull that holds it could export everything. It would need either a narrower `RequireExport` permission or a source-side allow-list of the environments that may pull. |
|
||||
| P3 | **Network direction.** Which side opens the connection? | The environments are usually segmented, for example DEV vd03 and QA vq03/vq04. Production → dev egress, or dev → production ingress, is a firewall and change-control decision, not a code decision. A *push* from source to target has the mirror-image problem. |
|
||||
| P4 | **Secrets in flight.** Where does the passphrase come from? | Bundles carry secrets: DataConnection credentials, SMTP and SMS tokens. An automated pull needs a passphrase that both sides know, which becomes yet another shared secret. The alternative is per-pull ECDH key agreement, which is a new crypto protocol. TLS alone protects the transport but not the bundle at rest in the target's staging directory. |
|
||||
| P5 | **Provenance.** How does the target know the bundle came from env A? | Answered by #45 signing. A pull should **require** a Verified signature from a key pinned to that source environment. |
|
||||
| P6 | **Review.** Does a pull skip the diff/confirm gate? | It must not. The import wizard's preview, Map step and environment-name confirmation are the safety gate at the production boundary. So a pull would only replace the file upload; it would not be "sync". |
|
||||
|
||||
### Options
|
||||
|
||||
- **A — Keep orchestration in the CLI (recommended).** Document the two-command recipe above,
|
||||
and ask operators to enable signing with `RequireSignedBundles` on the receiving side.
|
||||
This needs no new surface, no stored cross-environment credential and no firewall change.
|
||||
The cost is that automation needs credentials for both clusters, which a CI system holds anyway.
|
||||
- **B — Target-initiated pull over the existing `/management` endpoint.** The target stores an
|
||||
env-A service account in its secret store (P1a), calls `ExportBundleCommand` remotely, and
|
||||
feeds the result into the normal wizard at the Upload step, so P6 is kept. This needs
|
||||
decisions on P1–P4, a `Transport:PullSources` configuration (URL, credential secret name,
|
||||
pinned signing key), a source-side allow-list (P2) and a firewall rule (P3).
|
||||
- **C — A dedicated gRPC export service with mTLS.** This is the most robust machine-to-machine
|
||||
option and the most work: new contract, certificate management and a new port. It is
|
||||
justified only if pulls become frequent and unattended.
|
||||
|
||||
**Recommendation:** A, and **no code now**. Revisit B when a concrete requirement names a pair
|
||||
of environments and a network path that change control has approved. Settle P1–P4 in a
|
||||
design session before writing any of it.
|
||||
|
||||
---
|
||||
|
||||
## 2. Differential bundles
|
||||
|
||||
### What exists today
|
||||
|
||||
Every export is a full snapshot of the selected artifacts plus, optionally, their dependency
|
||||
closure. On import, every artifact that matches the target field for field is classified
|
||||
**Identical** and auto-skipped. A full bundle therefore already *applies* differentially. Only
|
||||
the changed and new rows are written, audited, and make instances stale. A differential bundle
|
||||
would gain only:
|
||||
|
||||
- a smaller file;
|
||||
- a shorter preview for the operator to review.
|
||||
|
||||
### Open questions
|
||||
|
||||
| # | Question | Notes |
|
||||
|---|---|---|
|
||||
| D1 | **Baseline.** "Changed since *what*?" | The candidates are a previous bundle file, a timestamp, or the target's actual state. Entities carry no reliable last-modified stamp to diff against. The manifest's per-entry `version` is a constant `1` today: it is reserved, not populated. |
|
||||
| D2 | **Deletions.** How does an artifact deleted on the source reach the target? | The importer never deletes, and the bundle has no tombstone concept. A differential bundle without tombstones silently diverges. Adding tombstones would make import destructive, which needs its own review UX and its own audit. |
|
||||
| D3 | **Dependency closure.** Should an unchanged dependency be left out? | If it is, the target must already hold it. The importer's blocker scan already accepts a reference the target satisfies, so this works as long as the baseline really was applied to that target. If it was not, the result is a blocker the operator cannot fix without a full export. |
|
||||
| D4 | **Baseline drift.** What if the target was edited after the baseline was applied? | A differential bundle computed against the *source's* last export cannot see target-side edits. The full-bundle diff can. |
|
||||
|
||||
### Smallest sound design, if one is ever needed
|
||||
|
||||
1. Make the manifest additive: give each `contents[]` entry a `contentHash` (SHA-256 of that
|
||||
artifact's canonical DTO JSON). Older importers ignore unknown members, so this is not a
|
||||
format break.
|
||||
2. Export: add `--since <baseline.scadabundle>` in the CLI and wizard. Read the baseline's
|
||||
manifest, which needs no passphrase because the manifest is plaintext, and leave out every
|
||||
selected artifact whose hash is unchanged. Keep dependencies by reference only. Do not add
|
||||
tombstones.
|
||||
3. Manifest: record `baselineContentHash`, so the import preview can warn when the target never
|
||||
imported that baseline (D4).
|
||||
|
||||
**Recommendation: do not build it yet (YAGNI stands).** The import-side Identical auto-skip
|
||||
already gives operators the differential *effect*. The remaining benefits, file size and
|
||||
review noise, have not been reported as a problem. Revisit only if bundle size or preview
|
||||
length turns into a real complaint. Build the hash-only variant above, never tombstones,
|
||||
unless D2 is designed on its own.
|
||||
|
||||
---
|
||||
|
||||
## Decisions this addendum leaves open
|
||||
|
||||
1. Pull: is any cross-environment network path acceptable to change control (P3)? Nothing else
|
||||
can be decided until this is.
|
||||
2. Pull: which machine identity, a stored service account or a new export-scoped credential (P1/P2)?
|
||||
3. Differential: is propagating deletions in scope at all (D2)?
|
||||
4. Signing, which the pull depends on: see the open decisions in Component-Transport.md
|
||||
"Bundle Signing".
|
||||
Reference in New Issue
Block a user