ScadaBridge/docs/operations/bundle-signing-runbook.md

Transport bundle signing — operator runbook (ScadaBridge#45 / #110)

Decisions (owner, 2026-11-09): one signing key per environment (DEV, QA, PROD each have their own); trust is the hand-edited ScadaBridge:Transport:Signing:TrustedKeys list (no Admin UI trust list); the private key lives in the Secrets store and is read through ${secret:} config expansion; RequireSignedBundles is turned on at the production boundary once every exporting cluster signs. Reference: "Bundle Signing" in docs/requirements/Component-Transport.md.

A bundle names its signer by key fingerprint (sha256:<hex of SubjectPublicKeyInfo>). The importing environment trusts a bundle only if that fingerprint matches a public key in its TrustedKeys. There is no key-generation CLI; use openssl.

1. Generate a key for an environment

Run once per environment, on a trusted workstation (not on the application node):

ENV=dev    # dev | qa | prod
openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out "bundle-signing-$ENV.pem"
openssl ec -in "bundle-signing-$ENV.pem" -pubout -out "bundle-signing-$ENV.pub.pem"

bundle-signing-$ENV.pem is the private key (PKCS#8 PEM); *.pub.pem is the public key to distribute. The key must be ECDSA P-256 — anything else fails boot.

2. Store the private key as a secret

On the exporting environment, create a secret (for example SB-BUNDLE-SIGNING-KEY) whose value is the full contents of the private PEM, via /admin/secrets on a central node (or the ZB.MOM.WW.Secrets CLI). Both central nodes share the secret store when Secrets:Store:Provider=SqlServer, and either node may run an export. Then reference it from config — never paste the PEM into appsettings:

"ScadaBridge": { "Transport": { "Signing": {
  "Enabled": true,
  "PrivateKey": "${secret:SB-BUNDLE-SIGNING-KEY}"
} } }

The reference is expanded before startup validation (Program.cs, pre-host); a key that is missing or does not parse as P-256 fails boot, it never silently ships unsigned bundles. The key is not logged: validation messages name only the config key, and the signature entry carries only the public fingerprint. Destroy the local *.pem private file after storing it (keep an offline backup per your key-custody policy).

Restart both central nodes together after changing config.

3. Distribute the public key into each importing environment

On every environment that will import bundles from this one, add the public key to TrustedKeys (hand-edited config; restart both central nodes together):

"Signing": { "TrustedKeys": [
  { "Name": "dev", "PublicKey": "-----BEGIN PUBLIC KEY-----\nMFkw...\n-----END PUBLIC KEY-----" }
] }

Name is shown on a verified import and recorded on the BundleImported audit row. Public keys are not secret, so they may sit in appsettings. After this, a bundle exported from DEV shows Verified (dev) in the QA import wizard; before RequireSignedBundles is on, an untrusted or unsigned bundle still loads and shows Unsigned / UntrustedKey.

4. Rotation

1. Generate the new key pair (step 1) and store the new private key as a new secret name.
2. Add the new public key to TrustedKeys in every importing environment (keep the old one). Restart.
3. On the exporting environment, switch PrivateKey to the new ${secret:} reference. Restart. New exports are signed with the new key and verify everywhere.
4. Once no outstanding bundle signed with the old key still needs importing, remove the old public key from every importing environment's TrustedKeys, and delete the old secret.

Never remove the old trusted key before step 3 is live everywhere (it would refuse in-flight bundles when RequireSignedBundles is on). A compromised key: skip the grace period — do steps 1-3, then remove the old key immediately.

5. Turning on RequireSignedBundles (order matters)

The code default stays off. For the production boundary:

1. Every exporting environment has a key (step 1-2) and Signing:Enabled=true; confirm an export shows Signed on its BundleExported audit row.
2. Every key is in the production environment's TrustedKeys (step 3); confirm a real bundle from each shows Verified in the import wizard preview.
3. On the production environment set ScadaBridge:Transport:Signing:ProductionBoundary=true. While RequireSignedBundles is still false the node logs a startup WARNING ("ProductionBoundary is true but RequireSignedBundles is false") and /health/ready shows the bundle-signing-policy entry as Degraded (HTTP 200 — visible on Overview, not out of rotation). That is the reminder to finish the flip.
4. Set ScadaBridge:Transport:Signing:RequireSignedBundles=true and restart both central nodes together. Unsigned, untrusted-key and badly-signed bundles are now refused at upload; the warning and Degraded entry clear.
5. Rollback: set RequireSignedBundles=false (the warning returns). Pre-signing bundles remain importable whenever it is off.

A signature that is present, names a trusted key and fails to verify is refused regardless of the flag.