ScadaBridge/docs/plans/2026-11-09-transport-pull-and-differential-addendum.md

Transport addendum — cluster-to-cluster pull and differential bundles (ScadaBridge#45)

Date: 2026-11-09 · Status: design addendum. Owner decisions 2026-11-09 (ScadaBridge#110): pull — decided NOT to build; differential deletions — deferred. Neither is implemented. Origin: 2026-05-24-transport-design.md §18 and deferred-work register row 19 (now Gitea #45). The third deferred item, bundle signing, is implemented. See "Bundle Signing" in docs/requirements/Component-Transport.md.

#45 grouped three deferred Transport features. Signing could be built safely because it is off by default and adds only an optional zip entry. The other two are not specified well enough to build:

• Pull opens a new network surface whose authentication and network model nobody has decided.
• Differential has no defined baseline and no defined deletion semantics.

This addendum records the questions each one has to answer, the options, and a recommendation, so either feature can be picked up when there is a concrete need for it.

---

1. Cluster-to-cluster pull

DECIDED — NOT TO BUILD (owner, 2026-11-09, ScadaBridge#110). No cluster-to-cluster pull, no cross-environment network path, no stored cross-environment credential. The CLI export → import recipe below (Option A) is the supported path, with the receiving side requiring signed bundles (docs/operations/bundle-signing-runbook.md). The questions and options that follow are kept as a historical record only; do not build B or C without a new owner decision.

What exists today

A pull can already be orchestrated without any new server code. The CLI speaks to any central cluster's POST /management endpoint. It authenticates with a bearer token from /auth/token, and the server does the LDAP bind and role check. So an operator or a CI job can run:

scadabridge --url https://central-dev  bundle export --all --passphrase "$P" --output b.scadabundle
scadabridge --url https://central-prod bundle import --input b.scadabundle --passphrase "$P" --on-conflict skip

Each leg runs under its own cluster's role gate (RequireDesign to export, RequireAdmin to import) with a human's or service account's credentials, and each leg is audited. With signing on, the target can also require that the bundle came from a trusted source key. None of this adds a server-to-server path.

What a server-side pull would add, and the questions it raises

A server-side pull means the target central calls the source central itself, for example from an Import wizard option such as "Pull from environment X".

#	Question	Why it matters
P1	Credential. What identity does the target present to the source?	Management auth is LDAP-backed and per-user. A pull would need either (a) a stored service-account credential for env A kept in env B's secret store, or (b) a new machine credential such as an export-scoped API key, mTLS client certificate, or PSK. Inbound-API keys (ZB.MOM.WW.Auth.ApiKeys) are scoped to inbound API methods and are deliberately non-transportable, so they cannot be reused as they are.
P2	Authorisation on the source. Which role does a remote pull run under, and can it be limited to a selection?	RequireDesign is a human authoring role. A machine pull that holds it could export everything. It would need either a narrower RequireExport permission or a source-side allow-list of the environments that may pull.
P3	Network direction. Which side opens the connection?	The environments are usually segmented, for example DEV vd03 and QA vq03/vq04. Production → dev egress, or dev → production ingress, is a firewall and change-control decision, not a code decision. A push from source to target has the mirror-image problem.
P4	Secrets in flight. Where does the passphrase come from?	Bundles carry secrets: DataConnection credentials, SMTP and SMS tokens. An automated pull needs a passphrase that both sides know, which becomes yet another shared secret. The alternative is per-pull ECDH key agreement, which is a new crypto protocol. TLS alone protects the transport but not the bundle at rest in the target's staging directory.
P5	Provenance. How does the target know the bundle came from env A?	Answered by #45 signing. A pull should require a Verified signature from a key pinned to that source environment.
P6	Review. Does a pull skip the diff/confirm gate?	It must not. The import wizard's preview, Map step and environment-name confirmation are the safety gate at the production boundary. So a pull would only replace the file upload; it would not be "sync".

Options

• A — Keep orchestration in the CLI (recommended). Document the two-command recipe above, and ask operators to enable signing with RequireSignedBundles on the receiving side. This needs no new surface, no stored cross-environment credential and no firewall change. The cost is that automation needs credentials for both clusters, which a CI system holds anyway.
• B — Target-initiated pull over the existing /management endpoint. The target stores an env-A service account in its secret store (P1a), calls ExportBundleCommand remotely, and feeds the result into the normal wizard at the Upload step, so P6 is kept. This needs decisions on P1–P4, a Transport:PullSources configuration (URL, credential secret name, pinned signing key), a source-side allow-list (P2) and a firewall rule (P3).
• C — A dedicated gRPC export service with mTLS. This is the most robust machine-to-machine option and the most work: new contract, certificate management and a new port. It is justified only if pulls become frequent and unattended.

Recommendation: A, and no code now. Revisit B when a concrete requirement names a pair of environments and a network path that change control has approved. Settle P1–P4 in a design session before writing any of it.

---

2. Differential bundles

DEFERRED (owner, 2026-11-09, ScadaBridge#110). Differential deletions (D2) stay out of scope: the importer never deletes and there are no tombstones. Import-side Identical auto-skip already gives the differential effect. The hash-only variant below remains an option only if file size or preview length becomes a real complaint.

What exists today

Every export is a full snapshot of the selected artifacts plus, optionally, their dependency closure. On import, every artifact that matches the target field for field is classified Identical and auto-skipped. A full bundle therefore already applies differentially. Only the changed and new rows are written, audited, and make instances stale. A differential bundle would gain only:

• a smaller file;
• a shorter preview for the operator to review.

Open questions

#	Question	Notes
D1	Baseline. "Changed since what?"	The candidates are a previous bundle file, a timestamp, or the target's actual state. Entities carry no reliable last-modified stamp to diff against. The manifest's per-entry version is a constant 1 today: it is reserved, not populated.
D2	Deletions. How does an artifact deleted on the source reach the target?	The importer never deletes, and the bundle has no tombstone concept. A differential bundle without tombstones silently diverges. Adding tombstones would make import destructive, which needs its own review UX and its own audit.
D3	Dependency closure. Should an unchanged dependency be left out?	If it is, the target must already hold it. The importer's blocker scan already accepts a reference the target satisfies, so this works as long as the baseline really was applied to that target. If it was not, the result is a blocker the operator cannot fix without a full export.
D4	Baseline drift. What if the target was edited after the baseline was applied?	A differential bundle computed against the source's last export cannot see target-side edits. The full-bundle diff can.

Smallest sound design, if one is ever needed

1. Make the manifest additive: give each contents[] entry a contentHash (SHA-256 of that artifact's canonical DTO JSON). Older importers ignore unknown members, so this is not a format break.
2. Export: add --since <baseline.scadabundle> in the CLI and wizard. Read the baseline's manifest, which needs no passphrase because the manifest is plaintext, and leave out every selected artifact whose hash is unchanged. Keep dependencies by reference only. Do not add tombstones.
3. Manifest: record baselineContentHash, so the import preview can warn when the target never imported that baseline (D4).

Recommendation: do not build it yet (YAGNI stands). The import-side Identical auto-skip already gives operators the differential effect. The remaining benefits, file size and review noise, have not been reported as a problem. Revisit only if bundle size or preview length turns into a real complaint. Build the hash-only variant above, never tombstones, unless D2 is designed on its own.

---

Decisions (resolved 2026-11-09, ScadaBridge#110)

1. Pull: not built. No cross-environment network path (P3) and no machine identity (P1/P2) will be introduced.
2. Differential: deletions (D2) stay deferred; no tombstones.
3. Signing: one key per environment, hand-edited TrustedKeys, ${secret:} key source, enforcement on at the production boundary once every exporter signs — see docs/operations/bundle-signing-runbook.md.