test(rig): site secrets artifact-deploy live gate PASS 8/8
This commit is contained in:
Joseph Doherty
Executable
+593
@@ -0,0 +1,593 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# Live gate for SITE SECRETS RIDING THE ARTIFACT DEPLOY (design
|
||||
# docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md, plan task 13). Drives the real
|
||||
# central paths end to end on the running docker rig:
|
||||
#
|
||||
# - secrets are created / rotated / deleted at central through the Central UI's own
|
||||
# /admin/secrets page (ZB.MOM.WW.Secrets.Ui over the central pair's shared SQL store) — the
|
||||
# supported operator path; there is no CLI/management verb for central secrets
|
||||
# - data connections and artifact deploys go through the ScadaBridge CLI
|
||||
# (CLI -> Traefik -> active central -> ArtifactDeploymentService)
|
||||
# - the Artifacts badge is read from the Sites page after pressing Re-check (the pending-updates
|
||||
# evaluation has no CLI/management surface; it is Blazor-interactive, so a headless browser is
|
||||
# the only honest reader)
|
||||
# - each site node's own secret store (the library's SQLite `secret` table in
|
||||
# /app/data/scadabridge-secrets.db) and LocalDb (`deployed_secrets`, `mirrored_secrets`,
|
||||
# `data_connection_definitions` in /app/data/site-localdb.db) are copied out with `docker cp`
|
||||
# (the image carries no sqlite3) and read with the host's python3 sqlite3 module
|
||||
# - the central row is read (never written) from ZbSecretsHub.zbsecrets.secret with sqlcmd
|
||||
#
|
||||
# Ciphertext is compared by SHA-256 only; secret VALUES are random throwaways held in shell
|
||||
# variables and handed to the browser through the environment — never printed, never on argv.
|
||||
#
|
||||
# Checks (each prints PASS/FAIL; the run ends "PASS n/8" and exits non-zero on any FAIL):
|
||||
# 1 secret + secret-referencing MxGateway connection on site-a -> site-a Out of date, b/c Current
|
||||
# 2 deploy site-a -> success, secretsDeployed 1, site-a Current
|
||||
# 3 both site-a nodes hold the secret at the central revision + ciphertext (standby <= 30 s)
|
||||
# 4 rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision
|
||||
# 5 drop the secret: reference -> deploy -> both nodes no longer hold it; an out-of-band secret
|
||||
# seeded on site-a-a through the library's own write path (the ZB.MOM.WW.Secrets `secret` CLI
|
||||
# run against that node's store) survives
|
||||
# 6 reference a non-existent secret -> site-a deploy fails closed with the named message,
|
||||
# site-a stays Out of date, nothing changed on either site-a node
|
||||
# 7 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the KekId
|
||||
# 8 one DeploySiteSecrets audit row per secret-bearing deploy; no plaintext / base64 ciphertext
|
||||
# in any of the 8 nodes' logs
|
||||
#
|
||||
# Teardown (trap EXIT, also run up-front so a rerun starts clean): gate connections deleted,
|
||||
# site-a + site-b redeployed so the removal reaches them, the central gate secret deleted, the
|
||||
# out-of-band secret removed from site-a-a, then a fleet-wide artifact deploy so the rig ends with
|
||||
# every site Current. The secret library tombstones rather than hard-deletes, so deleted gate
|
||||
# names remain as is_deleted=1 rows in the stores — that is the library's contract, not residue.
|
||||
#
|
||||
# Prereqs: rig up (8 nodes ready), `python3 -m playwright` with chromium, the ScadaBridge CLI
|
||||
# built, and scadaproj checked out next to this repo (the `secret` CLI is published from it once
|
||||
# into $SECRETS_CLI_DIR).
|
||||
#
|
||||
# Usage: bash docker/verify-site-secrets.sh
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CLI="${SCADABRIDGE_CLI:-$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge}"
|
||||
URL="${SCADABRIDGE_URL:-http://localhost:9000}"
|
||||
A=(--url "$URL" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||||
SECRETS_CLI_PROJECT="${SECRETS_CLI_PROJECT:-$HOME/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli}"
|
||||
SECRETS_CLI_DIR="${SECRETS_CLI_DIR:-${TMPDIR:-/tmp}/sb-gate-secrets-cli}"
|
||||
STANDBY_WINDOW_S="${STANDBY_WINDOW_S:-30}"
|
||||
|
||||
# DEV-ONLY rig KEK shared by central + site-a (docker/docker-compose.yml, x-secrets-shared-kek-env).
|
||||
SITE_A_KEK="zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
||||
MSSQL_CONTAINER="${MSSQL_CONTAINER:-scadabridge-mssql}"
|
||||
MSSQL_USER="scadabridge_app"; MSSQL_PASS='ScadaBridge_Dev1#'
|
||||
|
||||
SECRET="sb-gate-mxgw"
|
||||
MISSING="sb-gate-missing"
|
||||
OOB="sb-gate-oob"
|
||||
CONN_A="sb-gate-conn-a"
|
||||
CONN_B="sb-gate-conn-b"
|
||||
ACTIVE_A="scadabridge-site-a-a"; STANDBY_A="scadabridge-site-a-b"
|
||||
NODES=(central-a central-b site-a-a site-a-b site-b-a site-b-b site-c-a site-c-b)
|
||||
|
||||
VALUE_1=$(python3 -c 'import secrets; print("gate1-" + secrets.token_hex(16))')
|
||||
VALUE_2=$(python3 -c 'import secrets; print("gate2-" + secrets.token_hex(16))')
|
||||
VALUE_OOB=$(python3 -c 'import secrets; print("gateoob-" + secrets.token_hex(16))')
|
||||
|
||||
CHECKS_PASSED=0; CHECKS_FAILED=0
|
||||
CHECK_OK=1
|
||||
begin_check() { CHECK_OK=1; echo ""; echo "=== Check $1: $2 ==="; }
|
||||
ok() { echo " ok $*"; }
|
||||
bad() { CHECK_OK=0; echo " FAIL $*"; }
|
||||
end_check() {
|
||||
if [ "$CHECK_OK" -eq 1 ]; then CHECKS_PASSED=$((CHECKS_PASSED+1)); echo " PASS check $1"
|
||||
else CHECKS_FAILED=$((CHECKS_FAILED+1)); echo " FAIL check $1"; fi
|
||||
}
|
||||
|
||||
[ -x "$CLI" ] || { echo "CLI not found at $CLI — build it: dotnet build src/ZB.MOM.WW.ScadaBridge.CLI"; exit 1; }
|
||||
python3 -c 'import playwright.sync_api' 2>/dev/null || { echo "python3 playwright missing (pip install playwright && playwright install chromium)"; exit 1; }
|
||||
|
||||
WORK=$(mktemp -d "${TMPDIR:-/tmp}/sb-gate-site-secrets.XXXXXX")
|
||||
HELPER="$WORK/gate.py"
|
||||
|
||||
# ── python helper: browser (UI) + read-only store readers ──────────────────────────────────────
|
||||
cat >"$HELPER" <<'PY'
|
||||
import hashlib, json, os, sqlite3, subprocess, sys, tempfile, time, shutil
|
||||
|
||||
def sha(b):
|
||||
return hashlib.sha256(b if isinstance(b, (bytes, bytearray)) else b.encode()).hexdigest()
|
||||
|
||||
# ---- Central UI -------------------------------------------------------------------------------
|
||||
def _browser(fn):
|
||||
from playwright.sync_api import sync_playwright
|
||||
with sync_playwright() as p:
|
||||
b = p.chromium.launch()
|
||||
try:
|
||||
pg = b.new_page()
|
||||
return fn(pg)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def _wait_circuit(pg):
|
||||
# Blazor Server: handlers only work once the circuit is up.
|
||||
pg.wait_for_function("() => window.Blazor !== undefined", timeout=30000)
|
||||
time.sleep(2)
|
||||
|
||||
def badges(url):
|
||||
def run(pg):
|
||||
pg.goto(url + "/admin/sites")
|
||||
pg.wait_for_selector("button[aria-label='Re-check pending updates']", timeout=30000)
|
||||
_wait_circuit(pg)
|
||||
pg.click("button[aria-label='Re-check pending updates']")
|
||||
time.sleep(1)
|
||||
pg.wait_for_function("() => { const b=document.querySelector(\"button[aria-label='Re-check pending updates']\"); return b && !b.disabled; }", timeout=120000)
|
||||
pg.wait_for_function("() => !document.querySelector('[aria-label=\"Artifacts: checking\"]')", timeout=60000)
|
||||
out = {}
|
||||
for card in pg.query_selector_all(".card"):
|
||||
code = card.query_selector("code"); badge = card.query_selector("[aria-label^='Artifacts:']")
|
||||
if code and badge:
|
||||
out[code.inner_text().strip()] = badge.get_attribute("aria-label").split(": ", 1)[1]
|
||||
return out
|
||||
print(json.dumps(_browser(run)))
|
||||
|
||||
def _row_exists(pg, name):
|
||||
return pg.query_selector(f"button.rotate-secret[data-secret-name='{name}']") is not None
|
||||
|
||||
def _secrets_page(pg, url):
|
||||
pg.goto(url + "/admin/secrets")
|
||||
pg.wait_for_selector("[data-testid='add-secret']", timeout=30000)
|
||||
pg.wait_for_function("() => !document.body.innerText.includes('Loading…')", timeout=30000)
|
||||
_wait_circuit(pg)
|
||||
|
||||
def secret_upsert(url, name, desc):
|
||||
value = os.environ["SB_GATE_VALUE"]
|
||||
def run(pg):
|
||||
_secrets_page(pg, url)
|
||||
mode = "rotate" if _row_exists(pg, name) else "add"
|
||||
for attempt in range(5):
|
||||
if mode == "rotate":
|
||||
pg.click(f"button.rotate-secret[data-secret-name='{name}']")
|
||||
else:
|
||||
pg.click("[data-testid='add-secret']")
|
||||
try:
|
||||
pg.wait_for_selector("[data-testid='editor-submit']", timeout=3000); break
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
if mode == "add":
|
||||
pg.fill("[data-testid='secret-name']", name); pg.press("[data-testid='secret-name']", "Tab")
|
||||
pg.fill("[data-testid='secret-description']", desc); pg.press("[data-testid='secret-description']", "Tab")
|
||||
pg.fill("[data-testid='secret-value']", value); pg.press("[data-testid='secret-value']", "Tab")
|
||||
pg.click("[data-testid='editor-submit']")
|
||||
deadline = time.time() + 20
|
||||
while time.time() < deadline:
|
||||
err = pg.query_selector("[data-testid='editor-error']")
|
||||
if err:
|
||||
return "error: " + err.inner_text()
|
||||
if pg.query_selector("[data-testid='editor-submit']") is None and _row_exists(pg, name):
|
||||
return mode
|
||||
time.sleep(0.3)
|
||||
return "error: editor did not close"
|
||||
print(_browser(run))
|
||||
|
||||
def secret_delete(url, name):
|
||||
def run(pg):
|
||||
_secrets_page(pg, url)
|
||||
if not _row_exists(pg, name):
|
||||
return "absent"
|
||||
for attempt in range(5):
|
||||
pg.click(f"button.delete-secret[data-secret-name='{name}']")
|
||||
try:
|
||||
pg.wait_for_selector("[data-testid='confirm-delete']", timeout=3000); break
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
pg.click("[data-testid='confirm-delete']")
|
||||
deadline = time.time() + 20
|
||||
while time.time() < deadline:
|
||||
err = pg.query_selector("[data-testid='delete-error']")
|
||||
if err:
|
||||
return "error: " + err.inner_text()
|
||||
if not _row_exists(pg, name):
|
||||
return "deleted"
|
||||
time.sleep(0.3)
|
||||
return "error: row still listed"
|
||||
print(_browser(run))
|
||||
|
||||
# ---- read-only store readers ------------------------------------------------------------------
|
||||
def _copy_db(container, path):
|
||||
"""docker cp the db + its WAL (NOT the -shm: SQLite rebuilds the wal-index from the WAL on
|
||||
open, which is safer than trusting a copied live shared-memory file)."""
|
||||
d = tempfile.mkdtemp(prefix="sbgate-")
|
||||
base = os.path.basename(path)
|
||||
for suffix in ("-wal", ""):
|
||||
r = subprocess.run(["docker", "cp", f"{container}:{path}{suffix}", os.path.join(d, base + suffix)],
|
||||
capture_output=True)
|
||||
if r.returncode != 0 and suffix == "":
|
||||
raise SystemExit(f"docker cp {container}:{path} failed: {r.stderr.decode().strip()}")
|
||||
return d, os.path.join(d, base)
|
||||
|
||||
def _query(container, path, sql):
|
||||
# Two consecutive reads must agree — guards against a checkpoint racing the two docker cp's.
|
||||
last = None
|
||||
for _ in range(4):
|
||||
d, db = _copy_db(container, path)
|
||||
try:
|
||||
con = sqlite3.connect(db); rows = con.execute(sql).fetchall(); con.close()
|
||||
finally:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
if rows == last:
|
||||
return rows
|
||||
last = rows
|
||||
return last
|
||||
|
||||
def site_secrets(container):
|
||||
rows = _query(container, "/app/data/scadabridge-secrets.db",
|
||||
"SELECT name, revision, kek_id, is_deleted, ciphertext FROM secret WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
print(json.dumps({n: {"rev": r, "kek": k, "deleted": bool(dl), "sha": sha(c)} for n, r, k, dl, c in rows}, sort_keys=True))
|
||||
|
||||
def site_localdb(container):
|
||||
db = "/app/data/site-localdb.db"
|
||||
dep = _query(container, db, "SELECT name, revision, is_removed, row_json FROM deployed_secrets WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
mir = _query(container, db, "SELECT name, revision FROM mirrored_secrets WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
dcs = _query(container, db, "SELECT name, configuration FROM data_connection_definitions WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
print(json.dumps({
|
||||
"deployed": {n: {"rev": r, "removed": bool(x), "sha": sha(j)} for n, r, x, j in dep},
|
||||
"mirrored": {n: r for n, r in mir},
|
||||
"connections": {n: sha(c or "") for n, c in dcs},
|
||||
}, sort_keys=True))
|
||||
|
||||
def central_secret(name):
|
||||
q = ("SET NOCOUNT ON; SELECT name, revision, kek_id, is_deleted, "
|
||||
"LOWER(CONVERT(varchar(64), HASHBYTES('SHA2_256', ciphertext), 2)), "
|
||||
"CONVERT(varchar(max), ciphertext, 2) "
|
||||
f"FROM zbsecrets.secret WHERE name = N'{name}'")
|
||||
r = subprocess.run(["docker", "exec", os.environ["MSSQL_CONTAINER"], "/opt/mssql-tools18/bin/sqlcmd",
|
||||
"-S", "localhost", "-U", os.environ["MSSQL_USER"], "-P", os.environ["MSSQL_PASS"],
|
||||
"-C", "-d", "ZbSecretsHub", "-h", "-1", "-W", "-s", "|", "-Q", q],
|
||||
capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
raise SystemExit("sqlcmd failed: " + r.stderr.strip())
|
||||
lines = [l for l in r.stdout.splitlines() if l.strip()]
|
||||
if not lines:
|
||||
print(json.dumps(None)); return
|
||||
n, rev, kek, dl, h, hexct = lines[0].split("|")
|
||||
import base64
|
||||
b64 = base64.b64encode(bytes.fromhex(hexct)).decode()
|
||||
if sha(bytes.fromhex(hexct)) != h:
|
||||
raise SystemExit("ciphertext hex truncated by sqlcmd")
|
||||
# b64 is only used by check 8's log grep; it is written to a file by the caller, never printed.
|
||||
print(json.dumps({"rev": int(rev), "kek": kek, "deleted": dl == "1", "sha": h, "b64": b64}))
|
||||
|
||||
cmd, args = sys.argv[1], sys.argv[2:]
|
||||
{"badges": lambda: badges(*args),
|
||||
"secret-upsert": lambda: secret_upsert(*args),
|
||||
"secret-delete": lambda: secret_delete(*args),
|
||||
"site-secrets": lambda: site_secrets(*args),
|
||||
"site-localdb": lambda: site_localdb(*args),
|
||||
"central-secret": lambda: central_secret(*args)}[cmd]()
|
||||
PY
|
||||
export MSSQL_CONTAINER MSSQL_USER MSSQL_PASS
|
||||
gp() { python3 "$HELPER" "$@"; }
|
||||
jget() { python3 -c 'import json,sys
|
||||
d=json.loads(sys.argv[1])
|
||||
for k in sys.argv[2].split("."):
|
||||
if d is None: break
|
||||
d = d.get(k) if isinstance(d, dict) else None
|
||||
print("" if d is None else (json.dumps(d) if isinstance(d,(dict,list,bool)) else d))' "$1" "$2"; }
|
||||
|
||||
UNWRAP='
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
if isinstance(d, dict) and "data" in d and len(d) <= 2:
|
||||
d = d["data"]
|
||||
'
|
||||
|
||||
site_id() {
|
||||
"$CLI" site list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next(str(s["id"]) for s in d if s["siteIdentifier"]==sys.argv[1]))' "$1"
|
||||
}
|
||||
site_hash() {
|
||||
"$CLI" site list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next((s.get("artifactRevisionHash") or "-") for s in d if s["siteIdentifier"]==sys.argv[1]))' "$1"
|
||||
}
|
||||
conn_id() { # conn_id <name> -> id or empty
|
||||
"$CLI" data-connection list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next((str(c["id"]) for c in d if c["name"]==sys.argv[1]), ""))' "$1"
|
||||
}
|
||||
mxgw_config() { # mxgw_config <apiKeyValue>
|
||||
python3 -c 'import json,sys; print(json.dumps({"Endpoint":"http://sb-gate-no-such-gateway:5000","ApiKey":sys.argv[1],"ClientName":"sb-gate","ReadTimeoutMs":5000}))' "$1"
|
||||
}
|
||||
upsert_conn() { # upsert_conn <name> <siteId> <apiKeyValue>
|
||||
local id; id=$(conn_id "$1")
|
||||
if [ -z "$id" ]; then
|
||||
"$CLI" data-connection create "${A[@]}" --site-id "$2" --name "$1" --protocol MxGateway \
|
||||
--primary-config "$(mxgw_config "$3")" >/dev/null
|
||||
else
|
||||
"$CLI" data-connection update "${A[@]}" --id "$id" --name "$1" --protocol MxGateway \
|
||||
--primary-config "$(mxgw_config "$3")" >/dev/null
|
||||
fi
|
||||
}
|
||||
# deploy <siteId|all> -> sets DEPLOY_OUT (raw CLI output) and DEPLOY_RC; never aborts the script
|
||||
DEPLOY_OUT=""; DEPLOY_RC=0
|
||||
deploy() {
|
||||
DEPLOY_RC=0
|
||||
if [ "$1" = "all" ]; then
|
||||
DEPLOY_OUT=$("$CLI" site deploy-artifacts "${A[@]}" 2>&1) || DEPLOY_RC=$?
|
||||
else
|
||||
DEPLOY_OUT=$("$CLI" site deploy-artifacts "${A[@]}" --site-id "$1" 2>&1) || DEPLOY_RC=$?
|
||||
fi
|
||||
}
|
||||
# site_result <siteIdentifier> <field> -> the field from DEPLOY_OUT's siteResults
|
||||
site_result() {
|
||||
printf '%s' "$DEPLOY_OUT" | python3 -c '
|
||||
import json,sys
|
||||
raw=sys.stdin.read()
|
||||
try:
|
||||
d=json.loads(raw[raw.index("{"):])
|
||||
except Exception:
|
||||
print(""); sys.exit(0)
|
||||
r=next((x for x in d.get("siteResults",[]) if x.get("siteId")==sys.argv[1]), None)
|
||||
v=None if r is None else r.get(sys.argv[2])
|
||||
print("" if v is None else (json.dumps(v) if isinstance(v,bool) else v))' "$1" "$2"
|
||||
}
|
||||
badge_of() { jget "$1" "$2"; }
|
||||
|
||||
ensure_secrets_cli() {
|
||||
[ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ] && return 0
|
||||
echo " publishing the ZB.MOM.WW.Secrets 'secret' CLI to $SECRETS_CLI_DIR ..."
|
||||
dotnet publish "$SECRETS_CLI_PROJECT" -c Release -o "$SECRETS_CLI_DIR" >/dev/null
|
||||
}
|
||||
# Runs the library's own `secret` CLI against site-a-a's LOCAL store, in a throwaway container that
|
||||
# shares that node's data volume and carries that node's KEK — the library's normal SQLite write path.
|
||||
site_a_secret_cli() {
|
||||
local verb="$1"; shift
|
||||
docker run --rm --volumes-from "$ACTIVE_A" -v "$SECRETS_CLI_DIR:/cli:ro" -w /cli \
|
||||
-e Secrets__SqlitePath=/data/scadabridge-secrets.db -e "ZB_SECRETS_MASTER_KEY=$SITE_A_KEK" \
|
||||
-e SB_GATE_VALUE --entrypoint sh scadabridge:latest \
|
||||
-c "exec dotnet /cli/ZB.MOM.WW.Secrets.Cli.dll $verb \"\$@\"" _ "$@"
|
||||
}
|
||||
# `secret set <name> <value>` takes the value positionally: hand it over through the container's
|
||||
# environment (SB_GATE_VALUE) so it never appears on this host's argv; the container's inner sh
|
||||
# expands it.
|
||||
site_a_secret_set() { # site_a_secret_set <name> <description> (value in $SB_GATE_VALUE)
|
||||
docker run --rm --volumes-from "$ACTIVE_A" -v "$SECRETS_CLI_DIR:/cli:ro" -w /cli \
|
||||
-e Secrets__SqlitePath=/data/scadabridge-secrets.db -e "ZB_SECRETS_MASTER_KEY=$SITE_A_KEK" \
|
||||
-e SB_GATE_VALUE --entrypoint sh scadabridge:latest \
|
||||
-c 'exec dotnet /cli/ZB.MOM.WW.Secrets.Cli.dll set "$1" "$SB_GATE_VALUE" --description "$2"' _ "$1" "$2"
|
||||
}
|
||||
|
||||
# ── teardown (idempotent; also the pre-clean) ──────────────────────────────────────────────────
|
||||
teardown() {
|
||||
local rc=$?
|
||||
set +e
|
||||
echo ""
|
||||
echo "=== teardown ==="
|
||||
local id
|
||||
for c in "$CONN_A" "$CONN_B"; do
|
||||
id=$(conn_id "$c" 2>/dev/null)
|
||||
if [ -n "$id" ]; then
|
||||
"$CLI" data-connection delete "${A[@]}" --id "$id" >/dev/null 2>&1 \
|
||||
&& echo " removed data connection $c (id=$id)" || echo " WARN could not remove data connection $c"
|
||||
fi
|
||||
done
|
||||
# Removal must reach both sites that ever held the gate secret BEFORE central forgets it.
|
||||
[ -n "${SITE_A_ID:-}" ] && { deploy "$SITE_A_ID"; echo " site-a redeploy rc=$DEPLOY_RC"; }
|
||||
[ -n "${SITE_B_ID:-}" ] && { deploy "$SITE_B_ID"; echo " site-b redeploy rc=$DEPLOY_RC"; }
|
||||
echo " central $SECRET: $(gp secret-delete "$URL" "$SECRET" 2>&1 | tail -1)"
|
||||
if [ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ]; then
|
||||
local live
|
||||
live=$(gp site-secrets "$ACTIVE_A" 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print("1" if d.get(sys.argv[1],{}).get("deleted") is False else "")' "$OOB" 2>/dev/null)
|
||||
if [ -n "$live" ]; then
|
||||
site_a_secret_cli rm "$OOB" >/dev/null 2>&1 && echo " removed out-of-band $OOB from site-a-a" \
|
||||
|| echo " WARN could not remove out-of-band $OOB"
|
||||
fi
|
||||
fi
|
||||
deploy all; echo " fleet-wide artifact redeploy rc=$DEPLOY_RC"
|
||||
echo " badges after teardown: $(gp badges "$URL" 2>&1 | tail -1)"
|
||||
[ "${1:-}" = "pre" ] || rm -rf "$WORK"
|
||||
return $rc
|
||||
}
|
||||
|
||||
echo "=== ScadaBridge site-secrets artifact-deploy live gate ==="
|
||||
echo "management url: $URL"
|
||||
GATE_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
SITE_A_ID=$(site_id site-a); SITE_B_ID=$(site_id site-b); SITE_C_ID=$(site_id site-c)
|
||||
echo "sites: site-a=$SITE_A_ID site-b=$SITE_B_ID site-c=$SITE_C_ID"
|
||||
ensure_secrets_cli
|
||||
|
||||
echo ""
|
||||
echo "=== pre-clean (idempotent rerun) ==="
|
||||
teardown pre || true
|
||||
trap teardown EXIT
|
||||
GATE_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
# Docker log checkpoint for check 8: only lines written during this run are grepped.
|
||||
LOG_SINCE="$GATE_START"
|
||||
|
||||
BASE=$(gp badges "$URL")
|
||||
echo "baseline badges: $BASE"
|
||||
for s in site-a site-b site-c; do
|
||||
[ "$(badge_of "$BASE" "$s")" = "current" ] || { echo "baseline: $s is not Current — the gate needs a Current rig to start"; exit 1; }
|
||||
done
|
||||
|
||||
# ── 1 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 1 "secret + referencing connection on site-a -> site-a Out of date, site-b/c unchanged"
|
||||
R=$(SB_GATE_VALUE="$VALUE_1" gp secret-upsert "$URL" "$SECRET" "site-secrets live gate (throwaway)")
|
||||
[ "$R" = "add" ] && ok "created $SECRET at central via /admin/secrets" || bad "create $SECRET: $R"
|
||||
C1=$(gp central-secret "$SECRET")
|
||||
[ "$(jget "$C1" deleted)" = "false" ] && ok "central row $SECRET r$(jget "$C1" rev) kek $(jget "$C1" kek) sha $(jget "$C1" sha | cut -c1-12)" \
|
||||
|| bad "central row for $SECRET not found live"
|
||||
H_A_BEFORE=$(site_hash site-a)
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$SECRET" && ok "created MxGateway connection $CONN_A on site-a (ApiKey=secret:$SECRET, unbound)" \
|
||||
|| bad "create $CONN_A"
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a Out of date" || bad "site-a is '$(badge_of "$B" site-a)', expected out of date"
|
||||
[ "$(badge_of "$B" site-b)" = "current" ] && ok "site-b Current (unchanged)" || bad "site-b is '$(badge_of "$B" site-b)'"
|
||||
[ "$(badge_of "$B" site-c)" = "current" ] && ok "site-c Current (unchanged)" || bad "site-c is '$(badge_of "$B" site-c)'"
|
||||
end_check 1
|
||||
|
||||
# ── 2 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 2 "deploy artifacts to site-a -> success, secretsDeployed 1, site-a Current"
|
||||
T0=$(python3 -c 'import time; print(time.time())')
|
||||
deploy "$SITE_A_ID"
|
||||
T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
echo " deploy rc=$DEPLOY_RC ($(python3 -c "print(round($T_DEPLOYED-$T0,1))") s): $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "site-a deploy succeeded" || bad "site-a deploy failed (rc=$DEPLOY_RC)"
|
||||
[ "$(site_result site-a secretsDeployed)" = "1" ] && ok "secretsDeployed: 1" || bad "secretsDeployed='$(site_result site-a secretsDeployed)', expected 1"
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "current" ] && ok "site-a Current" || bad "site-a is '$(badge_of "$B" site-a)', expected current"
|
||||
end_check 2
|
||||
|
||||
# wait_node <container> <python predicate over the site-secrets JSON d> -> sets WAIT_S (or empty on timeout)
|
||||
WAIT_S=""
|
||||
wait_node() {
|
||||
local c="$1" pred="$2" start now j
|
||||
start=$(python3 -c 'import time; print(time.time())'); WAIT_S=""
|
||||
while :; do
|
||||
j=$(gp site-secrets "$c")
|
||||
if python3 -c "import json,sys; d=json.loads(sys.argv[1]); sys.exit(0 if ($pred) else 1)" "$j"; then
|
||||
now=$(python3 -c 'import time; print(time.time())')
|
||||
WAIT_S=$(python3 -c "print(round($now-${3:-$start},1))"); LAST_NODE_JSON="$j"; return 0
|
||||
fi
|
||||
now=$(python3 -c 'import time; print(time.time())')
|
||||
python3 -c "import sys; sys.exit(0 if $now-$start < $STANDBY_WINDOW_S else 1)" || { LAST_NODE_JSON="$j"; return 1; }
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
matches_central() { # python predicate: node row == central row C (rev + sha), live
|
||||
echo "d.get('$SECRET') is not None and not d['$SECRET']['deleted'] and d['$SECRET']['rev']==$(jget "$1" rev) and d['$SECRET']['sha']=='$(jget "$1" sha)' and d['$SECRET']['kek']=='$(jget "$1" kek)'"
|
||||
}
|
||||
|
||||
# ── 3 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 3 "both site-a nodes hold $SECRET at the central revision (standby within ${STANDBY_WINDOW_S}s)"
|
||||
C=$(gp central-secret "$SECRET")
|
||||
PRED=$(matches_central "$C")
|
||||
if wait_node "$ACTIVE_A" "$PRED" "$T_DEPLOYED"; then ok "site-a-a (active) r$(jget "$C" rev), ciphertext sha256 $(jget "$C" sha | cut -c1-16)… == central (seen ${WAIT_S}s after the deploy returned)"
|
||||
else bad "site-a-a does not match central: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$PRED" "$T_DEPLOYED"; then STANDBY_3="$WAIT_S"; ok "site-a-b (standby) converged ${WAIT_S}s after the deploy returned"
|
||||
else STANDBY_3="timeout"; bad "site-a-b did not converge within ${STANDBY_WINDOW_S}s: $LAST_NODE_JSON"; fi
|
||||
L=$(gp site-localdb "$STANDBY_A")
|
||||
[ "$(jget "$L" "mirrored.$SECRET")" = "$(jget "$C" rev)" ] && ok "site-a-b mirrored_secrets ledger r$(jget "$C" rev)" || bad "site-a-b ledger: $L"
|
||||
end_check 3
|
||||
|
||||
# ── 4 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 4 "rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision"
|
||||
R=$(SB_GATE_VALUE="$VALUE_2" gp secret-upsert "$URL" "$SECRET" "site-secrets live gate (throwaway, rotated)")
|
||||
[ "$R" = "rotate" ] && ok "rotated $SECRET via /admin/secrets" || bad "rotate: $R"
|
||||
C2=$(gp central-secret "$SECRET")
|
||||
if [ "$(jget "$C2" rev)" -gt "$(jget "$C1" rev)" ] && [ "$(jget "$C2" sha)" != "$(jget "$C1" sha)" ]; then
|
||||
ok "central r$(jget "$C1" rev) -> r$(jget "$C2" rev), ciphertext changed"
|
||||
else bad "central row did not advance: before r$(jget "$C1" rev) after r$(jget "$C2" rev)"; fi
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a Out of date after the rotation" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && [ "$(site_result site-a secretsDeployed)" = "1" ] \
|
||||
&& ok "redeploy succeeded, secretsDeployed 1" || bad "redeploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
PRED=$(matches_central "$C2")
|
||||
if wait_node "$ACTIVE_A" "$PRED" "$T_DEPLOYED"; then ok "site-a-a at r$(jget "$C2" rev) (${WAIT_S}s)"; else bad "site-a-a: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$PRED" "$T_DEPLOYED"; then STANDBY_4="$WAIT_S"; ok "site-a-b at r$(jget "$C2" rev) (${WAIT_S}s after the deploy returned)"
|
||||
else STANDBY_4="timeout"; bad "site-a-b: $LAST_NODE_JSON"; fi
|
||||
B=$(gp badges "$URL")
|
||||
[ "$(badge_of "$B" site-a)" = "current" ] && ok "site-a Current again" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
end_check 4
|
||||
|
||||
# ── 5 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 5 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
|
||||
SB_GATE_VALUE="$VALUE_OOB" site_a_secret_set "$OOB" "site-secrets gate out-of-band" >/dev/null 2>&1 || true
|
||||
J=$(gp site-secrets "$ACTIVE_A")
|
||||
if [ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ]; then
|
||||
ok "seeded out-of-band $OOB on site-a-a through the library's own write path"
|
||||
else bad "out-of-band seed not visible on site-a-a: $J"; fi
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "gate-literal-not-a-secret-ref" && ok "$CONN_A no longer references a secret" || bad "update $CONN_A"
|
||||
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "deploy succeeded (secretsDeployed='$(site_result site-a secretsDeployed)')" \
|
||||
|| bad "deploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
GONE="(d.get('$SECRET') is None or d['$SECRET']['deleted'])"
|
||||
if wait_node "$ACTIVE_A" "$GONE" "$T_DEPLOYED"; then ok "site-a-a no longer holds $SECRET (${WAIT_S}s)"; else bad "site-a-a still holds $SECRET: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$GONE" "$T_DEPLOYED"; then STANDBY_5="$WAIT_S"; ok "site-a-b no longer holds $SECRET (${WAIT_S}s after the deploy returned)"
|
||||
else STANDBY_5="timeout"; bad "site-a-b still holds $SECRET: $LAST_NODE_JSON"; fi
|
||||
J=$(gp site-secrets "$ACTIVE_A")
|
||||
[ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ] \
|
||||
&& ok "out-of-band $OOB still live on site-a-a" || bad "out-of-band $OOB lost: $J"
|
||||
L=$(gp site-localdb "$ACTIVE_A")
|
||||
[ "$(jget "$L" "deployed.$SECRET.removed")" = "true" ] && ok "deployed_secrets.$SECRET is_removed=1 (kept for the standby)" || bad "deployed_secrets: $L"
|
||||
end_check 5
|
||||
|
||||
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 6 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
|
||||
SNAP_BEFORE="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$MISSING" && ok "$CONN_A now references secret:$MISSING" || bad "update $CONN_A"
|
||||
deploy "$SITE_A_ID"
|
||||
ERR=$(site_result site-a errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
EXPECT="Secret '$MISSING' referenced by data connection(s) $CONN_A is not in the central secret store"
|
||||
[ "$(site_result site-a success)" = "false" ] && ok "site-a deploy reported failure" || bad "site-a deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
case "$ERR" in *"$EXPECT"*) ok "named fail-closed message";; *) bad "message does not contain: $EXPECT";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a stays Out of date" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
sleep 3
|
||||
SNAP_AFTER="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
[ "$SNAP_BEFORE" = "$SNAP_AFTER" ] && ok "nothing changed: stamped hash, both nodes' secret rows, deployed_secrets, ledgers and the $CONN_A definition identical" \
|
||||
|| { bad "site state changed"; echo " before: $SNAP_BEFORE"; echo " after: $SNAP_AFTER"; }
|
||||
end_check 6
|
||||
|
||||
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 7 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
|
||||
C=$(gp central-secret "$SECRET")
|
||||
KEK=$(jget "$C" kek)
|
||||
upsert_conn "$CONN_B" "$SITE_B_ID" "secret:$SECRET" && ok "created $CONN_B on site-b (ApiKey=secret:$SECRET)" || bad "create $CONN_B"
|
||||
deploy "$SITE_B_ID"
|
||||
ERR=$(site_result site-b errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
[ "$(site_result site-b success)" = "false" ] && ok "site-b deploy reported failure" || bad "site-b deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
case "$ERR" in *"$SECRET"*"$KEK"*) ok "message names $SECRET and the row's KekId $KEK";; *) bad "message does not name $SECRET + KekId $KEK";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-b)" = "out of date" ] && ok "site-b not stamped (Out of date)" || bad "site-b is '$(badge_of "$B" site-b)'"
|
||||
end_check 7
|
||||
|
||||
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 8 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
|
||||
AUD=$("$CLI" audit-config query "${A[@]}" --action DeploySiteSecrets --entity-type SiteSecrets --from "$GATE_START" --page-size 50 2>/dev/null)
|
||||
ROWS=$(printf '%s' "$AUD" | python3 -c '
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
rows=[]
|
||||
for e in sorted(d.get("entries",[]), key=lambda e: e["timestamp"]):
|
||||
s=json.loads(e.get("afterStateJson") or "{}")
|
||||
names=",".join("%s@r%s" % (x.get("name"), x.get("revision")) for x in (s.get("secrets") or []))
|
||||
miss=",".join(s.get("missing") or [])
|
||||
rows.append("%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", s.get("deploymentId")))
|
||||
print("\n".join(rows))')
|
||||
echo "$ROWS" | sed 's/^/ audit: /'
|
||||
R2=$(jget "$C1" rev); R4=$(jget "$C2" rev)
|
||||
EXPECTED="site-a|True|$SECRET@r$R2|-
|
||||
site-a|True|$SECRET@r$R4|-
|
||||
site-a|False|-|$MISSING
|
||||
site-b|False|$SECRET@r$R4|-"
|
||||
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-4)
|
||||
[ "$GOT" = "$EXPECTED" ] && ok "exactly 4 rows, one per secret-bearing deploy (2, 4, 6-fail-closed, 7-kek-mismatch); the no-secret deploy of check 5 wrote none" \
|
||||
|| { bad "audit rows differ from expected"; echo " expected:"; echo "$EXPECTED" | sed 's/^/ /'; }
|
||||
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f5 | sort -u | grep -c .)" = "4" ] && ok "4 distinct deploymentIds" || bad "deploymentIds not distinct"
|
||||
# Material to hunt for: both plaintext values and both ciphertexts (base64, as they appear in any
|
||||
# JSON serialization). Written to a pattern file so they never reach argv or the output.
|
||||
PAT="$WORK/patterns"; : >"$PAT"; chmod 600 "$PAT"
|
||||
printf '%s\n%s\n%s\n' "$VALUE_1" "$VALUE_2" "$VALUE_OOB" >>"$PAT"
|
||||
jget "$C1" b64 >>"$PAT"; jget "$C2" b64 >>"$PAT"
|
||||
[ "$(grep -c . "$PAT")" = "5" ] && [ "$(awk 'length($0) < 20' "$PAT" | grep -c .)" = "0" ] \
|
||||
&& ok "5 material patterns (3 plaintext, 2 base64 ciphertext) armed" || bad "pattern file malformed"
|
||||
HITS=0
|
||||
for n in "${NODES[@]}"; do
|
||||
h=$(docker logs --since "$LOG_SINCE" "scadabridge-$n" 2>&1 | grep -c -F -f "$PAT" || true)
|
||||
HITS=$((HITS + h)); [ "$h" = "0" ] || echo " scadabridge-$n: $h hit(s)"
|
||||
done
|
||||
AUDH=$(printf '%s' "$AUD" | grep -c -F -f "$PAT" || true)
|
||||
[ "$HITS" = "0" ] && ok "0 hits across all 8 nodes' logs since $LOG_SINCE" || bad "$HITS log line(s) carry secret material"
|
||||
[ "$AUDH" = "0" ] && ok "0 hits in the DeploySiteSecrets audit payloads" || bad "audit payload carries secret material"
|
||||
end_check 8
|
||||
|
||||
echo ""
|
||||
echo "=== Summary ==="
|
||||
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check5=${STANDBY_5:-?}"
|
||||
echo "PASS $CHECKS_PASSED/8"
|
||||
[ "$CHECKS_FAILED" -eq 0 ] || exit 1
|
||||
@@ -0,0 +1,137 @@
|
||||
# Site secrets ride the artifact deploy — live gate (Task 13)
|
||||
|
||||
**Date:** 2026-09-28/29 (UTC 00:28–00:50 on 2026-09-29) · **Branch:** `ScadaBridge-site-secrets` worktree ·
|
||||
**Result:** `bash docker/verify-site-secrets.sh` → **PASS 8/8** (on the image carrying fix `ad19d4b1`).
|
||||
The first run, on the pre-fix image, was **PASS 7/8** — check 5 found a product defect, fixed below.
|
||||
|
||||
Design: [`2026-09-28-site-secrets-artifact-deploy-design.md`](2026-09-28-site-secrets-artifact-deploy-design.md).
|
||||
|
||||
## 1. Method
|
||||
|
||||
### Rig bring-up
|
||||
|
||||
`docker/deploy.sh` was deliberately **not** run from the worktree (its relative bind mounts would rehome
|
||||
to the worktree). Instead:
|
||||
|
||||
1. `bash docker/build.sh` from the worktree → `scadabridge:latest` built from branch code.
|
||||
2. `docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate`
|
||||
— the branch compose (new `Secrets__*` env: central `Store:Provider=SqlServer` + shared KEK, site-a shared KEK,
|
||||
site-b/site-c a deliberately mismatched KEK, no hub keys anywhere) with bind mounts resolved against the primary
|
||||
checkout's `docker/` directory. `git diff 2aede140 HEAD --stat -- docker/` shows only `docker-compose.yml` changed,
|
||||
so the mounted node appsettings are identical to the branch's. `docker compose config` resolved project name
|
||||
`docker`, the running project, so the containers were replaced, not duplicated.
|
||||
3. All 8 nodes answered `/health/ready` 200 (centrals on 9001/9002; sites probed on their internal 8084 from the
|
||||
Traefik container, which is on `scadabridge-net`).
|
||||
|
||||
The same procedure was repeated after the fix (image rebuild + force-recreate); all 8 ready within ~10 s.
|
||||
|
||||
### Boot-log review (secrets)
|
||||
|
||||
- **Central (both):** no secrets-related warning or error. `Secrets:Store:Provider=SqlServer` against the rig's
|
||||
existing `ZbSecretsHub` database came up clean; no leftover-key warning (the rig carries no `Secrets:GrpcHub:*` or
|
||||
`Replication:Mode`).
|
||||
- **site-a:** no secrets-related warning or error. Its local `secrets.db` still holds six rows swept from the retired
|
||||
hub in August (`hub-gate-smoke-2/3`, `shared-gate-*`, KEK `sha256:f94d4ae2c4b1`) — inert, not ledgered by the
|
||||
mirror, so never touched by it.
|
||||
- **site-b / site-c:** booted cleanly with a KEK for the first time; no secrets warnings.
|
||||
- The only secrets line after the second recreate was an expected `ERR` on site-b-a — the mirror's startup pass
|
||||
re-verifying the `sb-gate-mxgw` row left live there by the pre-fix run (the defect's residue); the next gate
|
||||
run's pre-clean removed it.
|
||||
- Unrelated noise present on every boot: singleton-proxy "not found after 30 s" warnings during cluster formation,
|
||||
one `akka-cluster ... Removed` health-check ERR per pair during the recreate, OPC UA auto-accept warnings.
|
||||
|
||||
### How each surface is driven
|
||||
|
||||
| Surface | Tool | Notes |
|
||||
|---|---|---|
|
||||
| Central secret create / rotate / delete | Central UI **`/admin/secrets`** (ZB.MOM.WW.Secrets.Ui), headless Chromium via Python Playwright | The supported operator path; there is no CLI or management verb for central secrets, and the library's `secret` CLI only talks to a SQLite store headless. Value goes through the environment into the masked input, never argv/stdout. |
|
||||
| Data connections, artifact deploys, audit query | ScadaBridge CLI via Traefik (`localhost:9000`) | `data-connection create/update/delete`, `site deploy-artifacts --site-id`, `audit-config query --action DeploySiteSecrets`. |
|
||||
| Artifacts badge (Current / Out of date) | Sites page, **Re-check** pressed, badge `aria-label` read | The pending-updates evaluation has no CLI surface and is Blazor-interactive. |
|
||||
| Central row | `sqlcmd` **read-only** on `ZbSecretsHub.zbsecrets.secret` | Revision, KekId, `HASHBYTES('SHA2_256', ciphertext)`. |
|
||||
| Site node stores | `docker cp` of `/app/data/scadabridge-secrets.db` (library table `secret`) and `/app/data/site-localdb.db` (`deployed_secrets`, `mirrored_secrets`, `data_connection_definitions`) + WAL, read by host python `sqlite3` | The image has no sqlite3. The `-shm` is not copied (SQLite rebuilds the wal-index); each read is repeated until two consecutive copies agree. |
|
||||
| Out-of-band seed on site-a-a (check 5) | The library's own **`secret` CLI** (`ZB.MOM.WW.Secrets.Cli`, published from scadaproj main = 0.6.2) run in a throwaway `scadabridge:latest` container with `--volumes-from scadabridge-site-a-a`, `Secrets__SqlitePath=/data/scadabridge-secrets.db` and site-a's KEK | i.e. the library's normal local SQLite write path on that node's store — not a raw SQL insert. |
|
||||
|
||||
Gate objects: central secret `sb-gate-mxgw`; never-created `sb-gate-missing`; out-of-band `sb-gate-oob` on site-a-a;
|
||||
MxGateway connections `sb-gate-conn-a` (site-a) and `sb-gate-conn-b` (site-b), unbound, endpoint
|
||||
`http://sb-gate-no-such-gateway:5000`. The trap-based teardown (also run up-front as a pre-clean) deletes both
|
||||
connections, redeploys site-a and site-b so the removal reaches them, deletes the central secret, removes the
|
||||
out-of-band secret with the `secret` CLI, then redeploys artifacts fleet-wide; the run ended with all three sites
|
||||
**Current**. The library tombstones rather than hard-deletes, so after teardown the stores hold only
|
||||
`is_deleted = 1` rows for the gate names (verified: site-a-a, site-a-b, site-b-a and central all `deleted`;
|
||||
no gate connection left).
|
||||
|
||||
## 2. Per-check evidence (run 2, PASS 8/8)
|
||||
|
||||
| # | Check | Evidence |
|
||||
|---|---|---|
|
||||
| 1 | Secret + referencing connection → site-a Out of date, b/c unchanged | `sb-gate-mxgw` created via `/admin/secrets` (central r3, KEK `sha256:f94d4ae2c4b1` — the revision continues past run 1's tombstone); after `sb-gate-conn-a` (`"ApiKey":"secret:sb-gate-mxgw"`) badges `site-a: out of date, site-b: current, site-c: current`. |
|
||||
| 2 | Deploy site-a → success, `secretsDeployed: 1`, Current | `{"siteId":"site-a","success":true,"secretsDeployed":1}` (0.2 s); badge `current`. |
|
||||
| 3 | Both nodes at the central revision; standby ≤ 30 s | site-a-a r3, ciphertext SHA-256 `707ae73ba872…` == central (visible 4.0 s after the deploy returned — dominated by the docker-cp read loop; the active node applies synchronously before replying); **site-a-b converged 5.5 s** after the deploy returned; its `mirrored_secrets` ledger at r3. |
|
||||
| 4 | Rotate → Out of date → deploy → both at new revision | central r3 → r4 with new ciphertext; badge `out of date`; redeploy `secretsDeployed 1`; site-a-a r4 (0.3 s), **site-a-b r4 at 6.8 s**; badge back to `current`. |
|
||||
| 5 | Drop the reference → both nodes drop it; out-of-band survives | `sb-gate-oob` seeded on site-a-a via the `secret` CLI; `sb-gate-conn-a` switched to a literal ApiKey; deploy succeeded (no `secretsDeployed`); site-a-a no longer holds `sb-gate-mxgw` (0.2 s), **site-a-b at 9.0 s**; `sb-gate-oob` still live on site-a-a; `deployed_secrets.sb-gate-mxgw.is_removed = 1` (kept for the standby). |
|
||||
| 6 | Missing secret → fail closed, stays Out of date, site untouched | Error `Secret 'sb-gate-missing' referenced by data connection(s) sb-gate-conn-a is not in the central secret store`; badge `out of date`; stamped hash, both nodes' secret rows, `deployed_secrets`, ledgers and the site's `sb-gate-conn-a` definition byte-identical before/after. |
|
||||
| 7 | Same reference on site-b (mismatched KEK) → fails naming KekId | Error `Secret 'sb-gate-mxgw' (kek sha256:f94d4ae2c4b1) did not decrypt on this node — the node's master key differs from central's`; site-b not stamped (`out of date`). |
|
||||
| 8 | Audit rows; no material in logs | Exactly 4 `DeploySiteSecrets`/`SiteSecrets` rows with distinct deploymentIds: `site-a ok sb-gate-mxgw@r3`, `site-a ok @r4`, `site-a fail missing sb-gate-missing`, `site-b fail @r4`; the no-secret deploy of check 5 wrote none. Grep of `docker logs --since <gate start>` on all 8 nodes and of the audit payloads for both plaintext values, the out-of-band value and both base64 ciphertexts: **0 hits**. |
|
||||
|
||||
### Standby convergence (site-a-b, seconds after the deploy call returned)
|
||||
|
||||
| Run | Check 3 (first deploy) | Check 4 (rotation) | Check 5 (removal) |
|
||||
|---|---|---|---|
|
||||
| 1 (pre-fix) | 9.4 | 5.4 | never (defect) |
|
||||
| 2 (fixed) | 5.5 | 6.8 | 9.0 |
|
||||
|
||||
All inside the 30 s window, consistent with LocalDb CDC replication of `deployed_secrets` plus the standby's
|
||||
15 s `DeployedSecretsMirrorInterval`. The figures include up to ~1–2 s of read-loop overhead (docker cp + double
|
||||
read per poll).
|
||||
|
||||
## 3. Defects found
|
||||
|
||||
### 3.1 Dropping a site's last `secret:` reference never removed the secret (fixed, `ad19d4b1`)
|
||||
|
||||
Run 1, check 5: after the removal deploy both site-a nodes still held `sb-gate-mxgw` r1 live,
|
||||
`deployed_secrets.is_removed = 0`, ledger r1. Cause: `ArtifactDeploymentService.BuildDeployArtifactsCommandAsync`
|
||||
shipped `Secrets = null` whenever a site had no references ("null-when-empty"), and the site deliberately treats
|
||||
`null` as "central shipped no secrets" (pre-feature contract) and touches nothing — so the full-set soft-remove
|
||||
never ran. Fix: a secrets-aware central always ships a list (empty when there are no references). Old sites
|
||||
ignore proto field 9; the artifact hash is unchanged (empty and null hash identically); `SecretsDeployed` stays
|
||||
null and no audit row is written when nothing shipped. Regression test
|
||||
`NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes` (red on the old code, green on
|
||||
the fix). Spec updated in `Component-DeploymentManager.md`, `Component-SiteRuntime.md` and the design doc's
|
||||
as-built notes.
|
||||
|
||||
### 3.2 `ManagementActorTests` did not compile a valid proxy (fixed, `1444f859`)
|
||||
|
||||
Four `DeployArtifacts_*` tests built an NSubstitute proxy of `ArtifactDeploymentService` with the pre-Task-6
|
||||
constructor arguments; they now pass a `SiteSecretsCollector`. Test-only.
|
||||
|
||||
## 4. Test suite
|
||||
|
||||
`dotnet build ZB.MOM.WW.ScadaBridge.slnx`: 0 warnings, 0 errors.
|
||||
`dotnet test ZB.MOM.WW.ScadaBridge.slnx` (30 test assemblies, run against the live rig): **10 006 tests, 9 979
|
||||
passed, 27 failed** — 4 were the `ManagementActorTests` proxy failures fixed in `1444f859`
|
||||
(ManagementService.Tests re-run: 668/668), the other 23 are Playwright E2E. After the fix commit
|
||||
DeploymentManager.Tests is 234/234 (one new test). Every non-Playwright assembly is green.
|
||||
|
||||
Playwright residue (23 of 184):
|
||||
|
||||
| Class | Tests | Classification |
|
||||
|---|---|---|
|
||||
| `RoleNavigationTests` | `AdminUser_DoesNotSeeDeploymentSection`, `AdminUser_DoesNotSeeDesignSection`, `AdminUser_SeesHealthDashboard_NotDeploymentMonitoring`, `DeploymentUser_DoesNotSeeAdminSection`, `DeploymentUser_DoesNotSeeDesignSection`, `DeploymentUser_SeesMonitoringButNotConfigurationAuditLog`, `DesignUser_DoesNotSeeAdminSection`, `DesignUser_DoesNotSeeDeploymentSection`, `DesignUser_SeesHealthDashboard_NotDeploymentMonitoringOrAudit` | Known env-only — role-negative under `DisableLogin=true` |
|
||||
| `LoginTests` | `UnauthenticatedUser_RedirectsToLogin` | Known env-only — `DisableLogin` |
|
||||
| `AuditLogPageTests` | `PermissionGating_DesignerWithoutOperationalAudit_IsDeniedAccess` | Known env-only — `DisableLogin` |
|
||||
| `DeploymentActionTests` | `Delete_Instance_RemovesFromTree`, `Deploy_Instance_ShowsOutcomeToast`, `Disable_Instance_ShowsOutcomeToast`, `Enable_Instance_ShowsOutcomeToast` | Known residue (pagination helper — row not found) |
|
||||
| `TopologyAreaTests` | `CreateArea_ViaToolbar_AppearsInTreeAndPersists`, `Diff_OnDeployedInstance_OpensDialog`, `MoveArea_UnderAnotherArea_ShowsMovedToast`, `MoveInstance_ToArea_ShowsMovedToast`, `RenameArea_EnterCommits_PersistsNewName`, `RenameArea_Escape_RevertsToOriginalLabel` | Known residue (SignalR/visibility timeout) |
|
||||
| `AuditLogPageTests` | `ExportCsv_LinkVisibleForAuditExportUserAndTriggersDownload` | **Not on the known list** — `Download` event timeout (30 s), reproducible in isolation. Audit export is untouched by this branch; central logs showed audit-ingest SQL timeouts against the large `AuditLog` table at the time. Judged environmental. |
|
||||
| `TransportImportTests` | `ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` | **Not on the known list** — reproducible in isolation; central logs `Cannot insert duplicate key row in object 'dbo.Sites' ... (site-a)` during the import, i.e. the synthetic bundle's site mapping collides with the rig's existing site-a. Transport and site mapping are untouched by this branch. Judged rig-data related. |
|
||||
|
||||
## 5. Observations (not gated)
|
||||
|
||||
- **Standby of an unreplicated site pair never receives deployed secrets.** `deployed_secrets` reaches the
|
||||
standby only through LocalDb CDC replication; on site-b (deliberately unreplicated on this rig) the gate secret
|
||||
landed in site-b-a's store only — site-b-b's store never saw it. A failover on an unreplicated pair would leave
|
||||
the new active node without the site's secrets until the next artifact deploy. This follows from the design
|
||||
(§3.5) but is worth stating in the topology guide: secrets-bearing sites should run with LocalDb replication on.
|
||||
- The CLI's `site deploy-artifacts` exits 0 even when a site result is `success: false`; the gate reads the JSON
|
||||
`siteResults` rather than the exit code.
|
||||
- Re-adding a name that is tombstoned at central continues its revision (run 2 started at r3 after run 1's r1 +
|
||||
delete), which is why the gate compares revisions to the live central row rather than to fixed numbers.
|
||||
Reference in New Issue
Block a user