# Shared dev OpenTelemetry collector (`zb-otel-collector`) One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib) that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md). > Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA > vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54): > [`wonder/`](wonder/). ## Endpoints | Port | What | Use | |---|---|---| | `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. | | `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). | | `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=/`, `instance=` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. | | `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` | Reachable from the Mac and from windev (`10.100.0.48`) — both verified. ## Pointing an app at it OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on. | App | Config keys (env form) | |---|---| | OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | | ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | | MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | | HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) | - Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.3.0` (what the apps pin since 2026-11-12) rejects a scheme-less `host:port` at startup when `Exporter=Otlp`. - The SDK's default metric export interval is 60 s. For a quick check set `OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process. - What travels: **metrics, traces and logs** (Telemetry `0.3.0`+, scadaproj#55): with `Exporter=Otlp` the Serilog logs go to the same endpoint with the same Resource. `:Telemetry:ExportLogs=false` keeps them local. Every app's Resource carries `service.version` (git SHA), `site.id` and `node.role`; mxgw and HistorianGateway report `node.role=standalone` and take `site.id` from `:Telemetry:SiteId`. ## Reading what arrived ```bash ssh dohertj2@10.100.0.35 docker logs --tail 20 zb-otel-collector # one debug line per received batch cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names python3 ../otelsum.py otopcua # one service python3 ../otelidentity.py otopcua :12345 # Resource per signal + log samples, one process (instance-id suffix) curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head ``` `data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB (3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc; [`otelidentity.py`](otelidentity.py) prints the Resource each signal arrived with, the log count by severity and sample log records, filtered to one process (scadaproj#55). ## Deploy / update ```bash # from this directory on the Mac ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data' scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/ ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate' ``` - The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or `sudo chown 10001:10001 data`). - The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this release; the config uses its replacement `resource_constant_labels`. - Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no network or volume with any other container. Tear down with `docker compose down` there.