Joseph Doherty
83f7bbfb50
First run (docker stop) found two things: SIGTERM is a graceful CoordinatedShutdown handover (takeover 0 s), so the drill now uses docker kill; and inbound API keys are a PER-NODE SQLite store (/app/data/inbound-api-keys.sqlite on each central's own volume, written by whichever node serves the management call), so the key minted through the active node did not exist on the survivor and every post-takeover call answered 401 KeyNotFound. The sync script is a rig-only workaround; the verify script's preflight refuses to start unless both nodes list the key. Second run: A/B/E/F YES, C/G NO, D 503 in the takeover window, takeover 27 s, victim rejoined as standby. Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
47 lines
3.2 KiB
Bash
Executable File
47 lines
3.2 KiB
Bash
Executable File
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
# RIG-ONLY workaround: copy the inbound API key store from one central node to the other.
|
|
#
|
|
# Why this exists: inbound API keys live in a PER-NODE SQLite file
|
|
# (ScadaBridge:InboundApi:ApiKeyStore:SqlitePath, /app/data/inbound-api-keys.sqlite on each
|
|
# central's own data volume) and key admin writes only the node that serves the management call
|
|
# (the active one, via Traefik). A key therefore exists on exactly one central node, and after a
|
|
# central failover the survivor answers 401 to every inbound client — found live 2026-09-02 by
|
|
# docker/verify-delmia-notifier.sh (deferred-work register). Until the store is shared, a 2-node
|
|
# central needs the file mirrored by hand; this script does that for the docker rig.
|
|
#
|
|
# Usage: bash docker/sync-inbound-api-keys.sh [from a|b] [to b|a] (default: a -> b)
|
|
# - The SOURCE stays up (its WAL is checkpointed into the copy on the host with sqlite3).
|
|
# Quiesce key admin while it runs — a create racing the copy may be missed.
|
|
# - The TARGET is stopped for the copy and restarted; if it is the ACTIVE node the peer takes over
|
|
# (graceful CoordinatedShutdown handover) and the target rejoins as standby. Its own key store is
|
|
# REPLACED (both rig nodes share one pepper, so the copied hashes verify on either node).
|
|
|
|
FROM="${1:-a}"; TO="${2:-b}"
|
|
[ "$FROM" != "$TO" ] || { echo "from and to must differ" >&2; exit 2; }
|
|
SRC="scadabridge-central-$FROM"; DST="scadabridge-central-$TO"; VOL="scadabridge-central-node-$TO-data"
|
|
port_of() { [ "$1" = a ] && echo 9001 || echo 9002; }
|
|
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
|
|
|
|
echo "== copying key store out of $SRC (main + WAL + SHM) and checkpointing on the host"
|
|
for f in inbound-api-keys.sqlite inbound-api-keys.sqlite-wal inbound-api-keys.sqlite-shm; do
|
|
docker cp "$SRC:/app/data/$f" "$WORK/$f" 2>/dev/null || true
|
|
done
|
|
[ -s "$WORK/inbound-api-keys.sqlite" ] || { echo "FAIL: no key store on $SRC" >&2; exit 1; }
|
|
sqlite3 "$WORK/inbound-api-keys.sqlite" "PRAGMA wal_checkpoint(TRUNCATE);" > /dev/null
|
|
[ "$(sqlite3 "$WORK/inbound-api-keys.sqlite" "PRAGMA integrity_check;")" = ok ] || { echo "FAIL: copy failed integrity_check" >&2; exit 1; }
|
|
echo " api_keys rows in copy: $(sqlite3 "$WORK/inbound-api-keys.sqlite" "select count(*) from api_keys")"
|
|
|
|
echo "== stopping $DST, replacing its key store on volume $VOL, restarting"
|
|
docker stop "$DST" > /dev/null
|
|
docker run --rm -v "$VOL:/d" -v "$WORK:/src:ro" alpine:3.20 sh -c \
|
|
'cp /src/inbound-api-keys.sqlite /d/inbound-api-keys.sqlite && rm -f /d/inbound-api-keys.sqlite-wal /d/inbound-api-keys.sqlite-shm && chmod 644 /d/inbound-api-keys.sqlite'
|
|
docker start "$DST" > /dev/null
|
|
|
|
P=$(port_of "$TO"); Q=$(port_of "$FROM")
|
|
for i in $(seq 1 120); do curl -sf -o /dev/null "http://localhost:$P/health/ready" && break; sleep 1; done
|
|
for i in $(seq 1 60); do curl -s "http://localhost:$Q/health/ready" | grep -q '"memberCount":2,"unreachableCount":0' && break; sleep 2; done
|
|
echo "== $DST ready; members on $SRC: $(curl -s "http://localhost:$Q/health/ready" | grep -o '"memberCount":[0-9]*,"unreachableCount":[0-9]*')"
|
|
echo " active: central-a=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:9001/health/active) central-b=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:9002/health/active)"
|