Joseph Doherty
b5f120c370
scadabridge_site_connection_up never counted data connections. It counts open central->site gRPC subscriber streams (Debug View + the site-wide alarm feed), so it correctly reads 0 on a healthy site nobody is watching — and the name invited reading it as device health. That misreading has already cost a live debugging session: register row 48's 'connection_up is 0 on all six site nodes' was healthy steady state, not the fault it was diagnosed as. Two halves, per the row: RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active (helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and the instrument gains the unit:"1" the other gauges already carried. Every reference updated: the gRPC server, both test suites, Component-Communication.md, components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test now pins the new name AND asserts the old one is gone, so it cannot creep back. NEW. scadabridge.site.data_connections.connected / .total — the first exported data-connection signal (the state existed on the health report and in the CLI, but never as a metric). connected < total is the alert condition; neither number alone separates 'a device is down' from 'nothing is configured', which is why the denominator ships with the numerator. Sourcing is deliberate on two counts. It is fed by the DCL at its four state transitions rather than derived from the health report, so the metric survives a dead report pipeline — the exact failure the 'metrics stale' flag exists to catch, and a derived gauge would go blind at the same moment. And it uses a static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened precedent) rather than a DI-injected holder: the DCL actors already take ISiteHealthCollector through their constructors, and threading a second dependency through both would ripple into every test that builds them. It also keeps SiteHealthCollector untouched, which matters right now — another agent is concurrently editing that file for the tag-resolution path. Removal is wired too: an undeployed connection leaves the registry, so it stops counting toward the total instead of latching connected < total on forever. Both gauges read 0 on the standby (connections open on the active node only) — healthy, documented, and called out in the docs so it does not become the next site_connection_up. No ObservedMeters allowlist change: all three instruments ride the existing ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.) Tests: Commons 711 passed (9 new, delta-based against a baseline with unique per-test connection names, so they do not inherit the register row 32 QueueDepthGaugeTests static-state flake); Communication 717; DCL 319.
126 lines
6.3 KiB
Bash
Executable File
126 lines
6.3 KiB
Bash
Executable File
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
# Seed ONE deployable, live-data instance on site-a so the rig always has real
|
|
# stream traffic for live probes.
|
|
#
|
|
# Why this exists: the rig repeatedly ends up with instances that cannot deploy
|
|
# (deferred-work register row 48 — the seven `soak-*` rows fail pre-deployment
|
|
# validation because their bound data connections no longer exist on the sites).
|
|
# With nothing deployed, `debug snapshot` returns `instanceNotFound`, no gRPC site
|
|
# stream is ever opened, and the R2 batching histogram
|
|
# `scadabridge.site.stream.batch_size` has nothing to record — so any live probe of
|
|
# the streaming path is unverifiable. This script rebuilds the minimum working set
|
|
# from scratch, using only the CLI (no DB edits).
|
|
#
|
|
# What it creates (all on site-a / site id 1, against data connection id 1
|
|
# "OPC PLC Simulator" -> opc.tcp://scadabridge-opcua:50000):
|
|
# - Template `RigProbe`
|
|
# * Spike (Double) <- ns=3;s=SpikeData — opc-plc built-in, changes ~1/s
|
|
# * RandomInt (Int32) <- ns=3;s=RandomSignedInt32 — opc-plc built-in, changes ~1/s
|
|
# * Heartbeat (Int32) — static, bumped every 5s by the Heartbeat script
|
|
# * Heartbeat script — Interval 5000ms, trust-policy-clean
|
|
# - Instance `rig-probe-1`, bound + deployed
|
|
#
|
|
# IMPORTANT — pick self-changing tags. The custom tags in `infra/opcua/nodes.json`
|
|
# (Motor.*, Tank.*, Valve.*, TestChildObject.*) are declared but NOT simulated by
|
|
# opc-plc: they read 0 / false / epoch and never change on their own, so binding to
|
|
# them yields a healthy connection with zero event traffic. The built-in simulated
|
|
# nodes are the ones that move: SpikeData, RandomSignedInt32, StepUp, FastUInt1..5
|
|
# (1s) and SlowUInt1..5 (10s).
|
|
#
|
|
# Prerequisites:
|
|
# - Infrastructure services running (infra/docker-compose.yml — needs scadabridge-opcua)
|
|
# - Application containers running (docker/deploy.sh)
|
|
# - Sites seeded (docker/seed-sites.sh) and central healthy
|
|
# curl -sf http://localhost:9000/health/ready
|
|
#
|
|
# Usage:
|
|
# docker/seed-live-probe.sh
|
|
#
|
|
# Idempotency: this script is NOT idempotent — template and instance names must be
|
|
# unique, so a second run fails on the create steps. To re-seed, delete the previous
|
|
# instance and template first:
|
|
# scadabridge instance delete --id <id> && scadabridge template delete --id <id>
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
CLI="dotnet run --project $PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI --"
|
|
AUTH="--username multi-role --password password"
|
|
URL="--url http://localhost:9000"
|
|
|
|
SITE_ID=1 # site-a
|
|
CONNECTION_ID=1 # "OPC PLC Simulator" on site-a
|
|
TEMPLATE_NAME="RigProbe"
|
|
INSTANCE_NAME="rig-probe-1"
|
|
|
|
# Pulls the numeric "id" out of the CLI's JSON reply without needing jq.
|
|
json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
|
|
|
|
echo "=== Seeding live-data probe instance on site-a ==="
|
|
|
|
echo ""
|
|
echo "1/6 Creating template $TEMPLATE_NAME..."
|
|
TEMPLATE_ID=$($CLI $URL $AUTH template create \
|
|
--name "$TEMPLATE_NAME" \
|
|
--description "Minimal live-traffic probe: two OPC UA sim tags that change on their own + a 5s heartbeat script" \
|
|
| json_id)
|
|
echo " template id = $TEMPLATE_ID"
|
|
|
|
echo ""
|
|
echo "2/6 Adding data-bound attributes..."
|
|
$CLI $URL $AUTH template attribute add \
|
|
--template-id "$TEMPLATE_ID" --name Spike --data-type Double \
|
|
--data-source "ns=3;s=SpikeData" \
|
|
--description "opc-plc SpikeData — Double, changes every ~1s on its own" > /dev/null
|
|
$CLI $URL $AUTH template attribute add \
|
|
--template-id "$TEMPLATE_ID" --name RandomInt --data-type Int32 \
|
|
--data-source "ns=3;s=RandomSignedInt32" \
|
|
--description "opc-plc RandomSignedInt32 — Int32, changes every ~1s on its own" > /dev/null
|
|
|
|
echo ""
|
|
echo "3/6 Adding the static heartbeat attribute + its interval script..."
|
|
$CLI $URL $AUTH template attribute add \
|
|
--template-id "$TEMPLATE_ID" --name Heartbeat --data-type Int32 --value 0 \
|
|
--description "Static attribute bumped every 5s by the Heartbeat script — script-sourced traffic" > /dev/null
|
|
$CLI $URL $AUTH template script add \
|
|
--template-id "$TEMPLATE_ID" --name Heartbeat \
|
|
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
|
|
--code 'var current = await Instance.GetAttribute("Heartbeat"); var next = (int.TryParse(current?.ToString(), out var n) ? n : 0) + 1; await Instance.SetAttribute("Heartbeat", next.ToString());' > /dev/null
|
|
|
|
echo ""
|
|
echo "4/6 Creating instance $INSTANCE_NAME..."
|
|
INSTANCE_ID=$($CLI $URL $AUTH instance create \
|
|
--name "$INSTANCE_NAME" --template-id "$TEMPLATE_ID" --site-id "$SITE_ID" | json_id)
|
|
echo " instance id = $INSTANCE_ID"
|
|
|
|
echo ""
|
|
echo "5/6 Binding the data-sourced attributes to connection $CONNECTION_ID..."
|
|
$CLI $URL $AUTH instance set-bindings --id "$INSTANCE_ID" \
|
|
--bindings "[[\"Spike\",$CONNECTION_ID],[\"RandomInt\",$CONNECTION_ID]]" > /dev/null
|
|
|
|
echo ""
|
|
echo "6/6 Deploying..."
|
|
$CLI $URL $AUTH instance deploy --id "$INSTANCE_ID"
|
|
|
|
echo ""
|
|
echo "=== Done. Verify with: ==="
|
|
echo " scadabridge debug snapshot --id $INSTANCE_ID"
|
|
echo " scadabridge health site --identifier site-a # dataConnectionEndpoints / dataConnectionTagQuality"
|
|
echo " scadabridge debug stream --id $INSTANCE_ID # opens a gRPC site stream; ~2 events/s"
|
|
echo ""
|
|
echo "While a stream is open, the site node serving it reports both signals"
|
|
echo "(metrics are on container port 8084, and the containers have no curl/wget):"
|
|
echo " docker exec scadabridge-site-a-a bash -c 'exec 3<>/dev/tcp/localhost/8084 && printf \"GET /metrics HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n\" >&3 && cat <&3' \\"
|
|
echo " | grep -E 'stream_subscribers_active|stream_batch_size|data_connections'"
|
|
echo ""
|
|
echo "NOTE: scadabridge_site_stream_subscribers_active (renamed from the misleading"
|
|
echo "scadabridge_site_connection_up, register row 54) counts OPEN CENTRAL->SITE gRPC"
|
|
echo "SUBSCRIBER STREAMS, not OPC UA data connections. It reads 0 on a perfectly healthy"
|
|
echo "site with no subscriber attached — that is correct, not a fault."
|
|
echo ""
|
|
echo "For real data-connection health use the gauges added by the same row:"
|
|
echo " scadabridge_site_data_connections_connected / _total"
|
|
echo "connected < total is the alert condition. Both read 0 on the STANDBY node, since"
|
|
echo "connections are opened by the deployment path on the active node only."
|