Files
scadaproj/infra/otel-collector
Joseph Doherty 71b8595742 feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:10:15 -05:00
..

Shared dev OpenTelemetry collector (zb-otel-collector)

One OpenTelemetry Collector (contrib) that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the shared Docker host 10.100.0.35, next to the other shared fixtures (zb-shared-glauth, the OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: components/observability/2026-11-12-otlp-end-to-end-evidence.md.

Scope: dev/test only. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54): wonder/.

Endpoints

Port What Use
4317 OTLP gRPC receiver The family default. Every app's OtlpExporter uses the gRPC protocol, so point apps here.
4318 OTLP HTTP/protobuf receiver For clients that only speak OTLP/HTTP (/v1/traces, /v1/metrics, /v1/logs).
8889 Prometheus exporter Every metric received over OTLP, re-exposed for scraping, with job=<service.namespace>/<service.name>, instance=<service.instance.id> and the identity labels (service_name, site_id, node_role, host_name). Series expire 15 min after the last update.
13133 health_check curl -s http://10.100.0.35:13133/ → {"status":"Server available",...}

Reachable from the Mac and from windev (10.100.0.48) — both verified.

Pointing an app at it

OTLP is opt-in and additive: Prometheus /metrics keeps working when it is switched on.

App Config keys (env form)
OtOpcUa OtOpcUa__Telemetry__Exporter=Otlp, OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
ScadaBridge ScadaBridge__Telemetry__Exporter=Otlp, ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
MxAccessGateway MxGateway__Telemetry__Exporter=Otlp, MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
HistorianGateway Telemetry__Exporter=Otlp, Telemetry__OtlpEndpoint=http://10.100.0.35:4317 (root-level section)
  • Always include the scheme: http://10.100.0.35:4317. Telemetry 0.3.0 (what the apps pin since 2026-11-12) rejects a scheme-less host:port at startup when Exporter=Otlp.
  • The SDK's default metric export interval is 60 s. For a quick check set OTEL_METRIC_EXPORT_INTERVAL=10000 (ms) on the process.
  • What travels: metrics, traces and logs (Telemetry 0.3.0+, scadaproj#55): with Exporter=Otlp the Serilog logs go to the same endpoint with the same Resource. <prefix>:Telemetry:ExportLogs=false keeps them local. Every app's Resource carries service.version (git SHA), site.id and node.role; mxgw and HistorianGateway report node.role=standalone and take site.id from <prefix>:Telemetry:SiteId.

Reading what arrived

ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector                  # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py     # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua                            # one service
python3 ../otelidentity.py otopcua :12345                # Resource per signal + log samples, one process (instance-id suffix)
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head

data/{metrics,traces,logs}.jsonl are OTLP-JSON, one export request per line, rotated at 50 MB (3 backups). otelsum.py is the summariser used for the evidence doc; otelidentity.py prints the Resource each signal arrived with, the log count by severity and sample log records, filtered to one process (scadaproj#55).

Deploy / update

# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
  • The image runs as uid 10001, so data/ must be writable by it (chmod 777 above, or sudo chown 10001:10001 data).
  • The image is pinned to an exact release (0.162.0). Bump deliberately, re-run the health check and one app, and update this README. resource_to_telemetry_conversion is deprecated in this release; the config uses its replacement resource_constant_labels.
  • Host directory: ~/zb-otel-collector/ (compose project zb-otel-collector). It shares no network or volume with any other container. Tear down with docker compose down there.