Files
scadaproj/infra/otel-collector/README.md
T
Joseph Doherty e1a2df83f7 feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)
- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 —
  OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889,
  health :13133; README runbook + otelsum.py summariser. Deployed.
- components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces
  received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway;
  OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996).
- GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups
  #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133
  (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:48:36 -05:00

4.1 KiB

Shared dev OpenTelemetry collector (zb-otel-collector)

One OpenTelemetry Collector (contrib) that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the shared Docker host 10.100.0.35, next to the other shared fixtures (zb-shared-glauth, the OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: components/observability/2026-11-12-otlp-end-to-end-evidence.md.

Scope: dev/test only. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA vq03/vq04) has a collector yet — that is scadaproj#54.

Endpoints

Port What Use
4317 OTLP gRPC receiver The family default. Every app's OtlpExporter uses the gRPC protocol, so point apps here.
4318 OTLP HTTP/protobuf receiver For clients that only speak OTLP/HTTP (/v1/traces, /v1/metrics, /v1/logs).
8889 Prometheus exporter Every metric received over OTLP, re-exposed for scraping, with job=<service.namespace>/<service.name>, instance=<service.instance.id> and the identity labels (service_name, site_id, node_role, host_name). Series expire 15 min after the last update.
13133 health_check curl -s http://10.100.0.35:13133/ → {"status":"Server available",...}

Reachable from the Mac and from windev (10.100.0.48) — both verified.

Pointing an app at it

OTLP is opt-in and additive: Prometheus /metrics keeps working when it is switched on.

App Config keys (env form)
OtOpcUa OtOpcUa__Telemetry__Exporter=Otlp, OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
ScadaBridge ScadaBridge__Telemetry__Exporter=Otlp, ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
MxAccessGateway MxGateway__Telemetry__Exporter=Otlp, MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317
HistorianGateway Telemetry__Exporter=Otlp, Telemetry__OtlpEndpoint=http://10.100.0.35:4317 (root-level section)
  • Always include the scheme: http://10.100.0.35:4317. Telemetry 0.2.1+ rejects a scheme-less host:port at startup; 0.1.0/0.2.0 (what the apps pin today) do not catch it there.
  • The SDK's default metric export interval is 60 s. For a quick check set OTEL_METRIC_EXPORT_INTERVAL=10000 (ms) on the process.
  • What travels: metrics + traces. Logs do not, in any app today — AddZbSerilog is called without the exporter settings (scadaproj#55).

Reading what arrived

ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector                  # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py     # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua                            # one service
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head

data/{metrics,traces,logs}.jsonl are OTLP-JSON, one export request per line, rotated at 50 MB (3 backups). otelsum.py is the summariser used for the evidence doc.

Deploy / update

# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
  • The image runs as uid 10001, so data/ must be writable by it (chmod 777 above, or sudo chown 10001:10001 data).
  • The image is pinned to an exact release (0.162.0). Bump deliberately, re-run the health check and one app, and update this README. resource_to_telemetry_conversion is deprecated in this release; the config uses its replacement resource_constant_labels.
  • Host directory: ~/zb-otel-collector/ (compose project zb-otel-collector). It shares no network or volume with any other container. Tear down with docker compose down there.