chore(docker): both rigs' centrals on the shared SQL Server inbound key store; delete sync-inbound-api-keys.sh (superseded); verify-delmia-notifier.sh preflight is now the proof that both nodes see a key minted on one — drill green with no file copy, takeover 28 s
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
This commit is contained in:
Joseph Doherty
@@ -31,6 +31,12 @@ services:
|
||||
# pepper per the "different per environment" guidance; real deployments inject a
|
||||
# true secret out-of-band, never from source control. Both Central nodes share it.
|
||||
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
|
||||
# Shared inbound API key store (design 2026-09-02, register row 108): both centrals read and
|
||||
# write ONE zbauth schema in the configuration database, so a key minted on either node
|
||||
# survives a central failover and a revoke is cluster-wide. The SqlServer connection string
|
||||
# is deliberately NOT set: it defaults to ScadaBridge__Database__ConfigurationDb below.
|
||||
# Each node's legacy per-node SQLite file is imported once at boot and renamed *.imported.
|
||||
ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer"
|
||||
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
||||
# appsettings.Central.json (mirrors docker/docker-compose.yml's G-4 T5 secrets
|
||||
# cleanup) so that app-config file carries no plaintext credentials. Env overrides
|
||||
@@ -88,6 +94,12 @@ services:
|
||||
# pepper per the "different per environment" guidance; real deployments inject a
|
||||
# true secret out-of-band, never from source control. Both Central nodes share it.
|
||||
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
|
||||
# Shared inbound API key store (design 2026-09-02, register row 108): both centrals read and
|
||||
# write ONE zbauth schema in the configuration database, so a key minted on either node
|
||||
# survives a central failover and a revoke is cluster-wide. The SqlServer connection string
|
||||
# is deliberately NOT set: it defaults to ScadaBridge__Database__ConfigurationDb below.
|
||||
# Each node's legacy per-node SQLite file is imported once at boot and renamed *.imported.
|
||||
ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer"
|
||||
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
||||
# appsettings.Central.json (mirrors docker/docker-compose.yml's G-4 T5 secrets
|
||||
# cleanup) so that app-config file carries no plaintext credentials. Env overrides
|
||||
|
||||
+2
-1
@@ -38,6 +38,8 @@ Local Docker deployment of the full ScadaBridge cluster topology: a 2-node centr
|
||||
|
||||
Runs the web UI (Blazor Server), Template Engine, Deployment Manager, Security, Inbound API, Management Service, and Health Monitoring. Connects to MS SQL for configuration and machine data, LDAP for authentication, and SMTP for notifications.
|
||||
|
||||
Both central nodes run `ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer"` (`docker-compose.yml`), so inbound API keys live in ONE shared `zbauth` schema in `ScadaBridgeConfig` rather than a per-node SQLite file — a key minted through either node survives a central failover and a revoke is cluster-wide (design 2026-09-02, deferred-work register row 108). Each node's legacy `/app/data/inbound-api-keys.sqlite` was imported once at the first boot on this provider and renamed `*.imported`. The former `sync-inbound-api-keys.sh` file-copy workaround is gone; `verify-delmia-notifier.sh`'s preflight now doubles as the proof that both nodes see the same keys.
|
||||
|
||||
### Site Clusters (active/standby each)
|
||||
|
||||
Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, and Site Event Logging. Sites connect to OPC UA for device data and to the central cluster via Akka.NET remoting. Each site node also hosts a gRPC streaming server (port 8083) that central nodes connect to for real-time attribute value and alarm state streams. Deployed configurations and S&F buffers are stored in local SQLite databases per node.
|
||||
@@ -98,7 +100,6 @@ docker/
|
||||
├── seed-delmia-sim.sh # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a
|
||||
├── seed-delmia-api.sh # Seed the DelmiaRecipeDownload inbound method + a scoped API key (token printed once)
|
||||
├── verify-delmia-notifier.sh # DelmiaNotifier end-to-end vs the rig: standby relay, active-central crash/takeover, rejoin
|
||||
├── sync-inbound-api-keys.sh # RIG-ONLY: mirror the per-node inbound API key store a->b (needed before any central failover test)
|
||||
├── teardown.sh # Stop and remove containers
|
||||
├── central-node-a/
|
||||
│ ├── appsettings.Central.json # Central node A configuration
|
||||
|
||||
@@ -99,6 +99,12 @@ services:
|
||||
# out-of-band (env/secret store), never from source control — see
|
||||
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
|
||||
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
|
||||
# Shared inbound API key store (design 2026-09-02, register row 108): both centrals read and
|
||||
# write ONE zbauth schema in the configuration database, so a key minted on either node
|
||||
# survives a central failover and a revoke is cluster-wide. The SqlServer connection string
|
||||
# is deliberately NOT set: it defaults to ScadaBridge__Database__ConfigurationDb below.
|
||||
# Each node's legacy per-node SQLite file is imported once at boot and renamed *.imported.
|
||||
ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer"
|
||||
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
||||
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
|
||||
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
|
||||
@@ -160,6 +166,12 @@ services:
|
||||
# out-of-band (env/secret store), never from source control — see
|
||||
# docs/operations/inbound-api-key-reissue.md. Both Central nodes share one pepper.
|
||||
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-docker-cluster-0001"
|
||||
# Shared inbound API key store (design 2026-09-02, register row 108): both centrals read and
|
||||
# write ONE zbauth schema in the configuration database, so a key minted on either node
|
||||
# survives a central failover and a revoke is cluster-wide. The SqlServer connection string
|
||||
# is deliberately NOT set: it defaults to ScadaBridge__Database__ConfigurationDb below.
|
||||
# Each node's legacy per-node SQLite file is imported once at boot and renamed *.imported.
|
||||
ScadaBridge__InboundApi__ApiKeyStore__Provider: "SqlServer"
|
||||
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
|
||||
# appsettings.Central.json (G-4 T5 secrets cleanup) so that app-config file carries no
|
||||
# plaintext credentials. Env overrides layer over JSON BEFORE the ${secret:} pre-host
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# RIG-ONLY workaround: copy the inbound API key store from one central node to the other.
|
||||
#
|
||||
# Why this exists: inbound API keys live in a PER-NODE SQLite file
|
||||
# (ScadaBridge:InboundApi:ApiKeyStore:SqlitePath, /app/data/inbound-api-keys.sqlite on each
|
||||
# central's own data volume) and key admin writes only the node that serves the management call
|
||||
# (the active one, via Traefik). A key therefore exists on exactly one central node, and after a
|
||||
# central failover the survivor answers 401 to every inbound client — found live 2026-09-02 by
|
||||
# docker/verify-delmia-notifier.sh (deferred-work register). Until the store is shared, a 2-node
|
||||
# central needs the file mirrored by hand; this script does that for the docker rig.
|
||||
#
|
||||
# Usage: bash docker/sync-inbound-api-keys.sh [from a|b] [to b|a] (default: a -> b)
|
||||
# - The SOURCE stays up (its WAL is checkpointed into the copy on the host with sqlite3).
|
||||
# Quiesce key admin while it runs — a create racing the copy may be missed.
|
||||
# - The TARGET is stopped for the copy and restarted; if it is the ACTIVE node the peer takes over
|
||||
# (graceful CoordinatedShutdown handover) and the target rejoins as standby. Its own key store is
|
||||
# REPLACED (both rig nodes share one pepper, so the copied hashes verify on either node).
|
||||
|
||||
FROM="${1:-a}"; TO="${2:-b}"
|
||||
[ "$FROM" != "$TO" ] || { echo "from and to must differ" >&2; exit 2; }
|
||||
SRC="scadabridge-central-$FROM"; DST="scadabridge-central-$TO"; VOL="scadabridge-central-node-$TO-data"
|
||||
port_of() { [ "$1" = a ] && echo 9001 || echo 9002; }
|
||||
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
echo "== copying key store out of $SRC (main + WAL + SHM) and checkpointing on the host"
|
||||
for f in inbound-api-keys.sqlite inbound-api-keys.sqlite-wal inbound-api-keys.sqlite-shm; do
|
||||
docker cp "$SRC:/app/data/$f" "$WORK/$f" 2>/dev/null || true
|
||||
done
|
||||
[ -s "$WORK/inbound-api-keys.sqlite" ] || { echo "FAIL: no key store on $SRC" >&2; exit 1; }
|
||||
sqlite3 "$WORK/inbound-api-keys.sqlite" "PRAGMA wal_checkpoint(TRUNCATE);" > /dev/null
|
||||
[ "$(sqlite3 "$WORK/inbound-api-keys.sqlite" "PRAGMA integrity_check;")" = ok ] || { echo "FAIL: copy failed integrity_check" >&2; exit 1; }
|
||||
echo " api_keys rows in copy: $(sqlite3 "$WORK/inbound-api-keys.sqlite" "select count(*) from api_keys")"
|
||||
|
||||
echo "== stopping $DST, replacing its key store on volume $VOL, restarting"
|
||||
docker stop "$DST" > /dev/null
|
||||
docker run --rm -v "$VOL:/d" -v "$WORK:/src:ro" alpine:3.20 sh -c \
|
||||
'cp /src/inbound-api-keys.sqlite /d/inbound-api-keys.sqlite && rm -f /d/inbound-api-keys.sqlite-wal /d/inbound-api-keys.sqlite-shm && chmod 644 /d/inbound-api-keys.sqlite'
|
||||
docker start "$DST" > /dev/null
|
||||
|
||||
P=$(port_of "$TO"); Q=$(port_of "$FROM")
|
||||
for i in $(seq 1 120); do curl -sf -o /dev/null "http://localhost:$P/health/ready" && break; sleep 1; done
|
||||
for i in $(seq 1 60); do curl -s "http://localhost:$Q/health/ready" | grep -q '"memberCount":2,"unreachableCount":0' && break; sleep 2; done
|
||||
echo "== $DST ready; members on $SRC: $(curl -s "http://localhost:$Q/health/ready" | grep -o '"memberCount":[0-9]*,"unreachableCount":[0-9]*')"
|
||||
echo " active: central-a=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:9001/health/active) central-b=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:9002/health/active)"
|
||||
@@ -6,11 +6,14 @@ set -euo pipefail
|
||||
# graceful handover) + rejoin. Requires: seed-delmia-sim.sh + seed-delmia-api.sh already run, and
|
||||
# DELMIA_API_KEY exported (the token printed by seed-delmia-api.sh).
|
||||
#
|
||||
# PREREQUISITE the first run of this script found the hard way (2026-09-02): inbound API keys are a
|
||||
# PER-NODE SQLite store, so the key minted through the active node does not exist on the standby
|
||||
# and every call after a takeover answers 401 KeyNotFound. Run `bash docker/sync-inbound-api-keys.sh`
|
||||
# after seed-delmia-api.sh (and again after re-minting a key). The preflight below lists the key
|
||||
# via each node's own management API and refuses to start the drill if either node lacks it.
|
||||
# HISTORY (2026-09-02): the first run of this script found that inbound API keys were a PER-NODE
|
||||
# SQLite store, so the key minted through the active node did not exist on the standby and every
|
||||
# call after a takeover answered 401 KeyNotFound (deferred-work register row 108; the rig carried
|
||||
# a sync-inbound-api-keys.sh file-copy workaround). Both rig centrals now run
|
||||
# ScadaBridge__InboundApi__ApiKeyStore__Provider=SqlServer (one shared zbauth schema in the config
|
||||
# DB), so the preflight below — which lists the key via EACH node's own management API — is no
|
||||
# longer a gate that needs a script to satisfy: it is the proof that a key minted on one node is
|
||||
# visible on the other without any file copy.
|
||||
#
|
||||
# The notifier is run as the managed build (dotnet WWNotifier.dll) from a scratch copy so the
|
||||
# API key is written to THAT copy's appsettings.json (ScadaBridge:ApiKey) and never into the
|
||||
@@ -57,12 +60,12 @@ wait_for() { # $1 url $2 path $3 seconds
|
||||
ACT=$(active); STB=$(other "$ACT")
|
||||
echo "== active=central-$ACT standby=central-$STB"
|
||||
|
||||
echo "== preflight: the notifier's key must be present in BOTH nodes' key stores"
|
||||
echo "== preflight/proof: the key minted on the active node must be visible from BOTH nodes (shared store, no file copy)"
|
||||
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
|
||||
KEY_ID="${DELMIA_API_KEY#sbk_}"; KEY_ID="${KEY_ID%%_*}"
|
||||
for n in a b; do
|
||||
if "$CLI" security api-key list --url "$(url_of "$n")" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}" | grep -q "\"keyId\":\"$KEY_ID\""; then echo " central-$n: key $KEY_ID present"
|
||||
else echo "FAIL: central-$n does not hold key $KEY_ID — run: bash docker/sync-inbound-api-keys.sh $ACT $STB" >&2; exit 1; fi
|
||||
else echo "FAIL: central-$n does not see key $KEY_ID — is ScadaBridge__InboundApi__ApiKeyStore__Provider=SqlServer set on both centrals (docker-compose.yml)?" >&2; exit 1; fi
|
||||
done
|
||||
|
||||
echo "== A baseline (active first)"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"; notify A YES
|
||||
|
||||
Reference in New Issue
Block a user