Files
scadaproj/infra/otel-collector/README.md
T
Joseph Doherty 71b8595742 feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:10:15 -05:00

76 lines
4.7 KiB
Markdown

# Shared dev OpenTelemetry collector (`zb-otel-collector`)
One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib)
that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the
shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
> Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA
> vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54):
> [`wonder/`](wonder/).
## Endpoints
| Port | What | Use |
|---|---|---|
| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. |
| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). |
| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=<service.namespace>/<service.name>`, `instance=<service.instance.id>` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. |
| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` |
Reachable from the Mac and from windev (`10.100.0.48`) — both verified.
## Pointing an app at it
OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on.
| App | Config keys (env form) |
|---|---|
| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.3.0` (what the apps pin since
2026-11-12) rejects a scheme-less `host:port` at startup when `Exporter=Otlp`.
- The SDK's default metric export interval is 60 s. For a quick check set
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
- What travels: **metrics, traces and logs** (Telemetry `0.3.0`+, scadaproj#55): with `Exporter=Otlp` the
Serilog logs go to the same endpoint with the same Resource. `<prefix>:Telemetry:ExportLogs=false` keeps them
local. Every app's Resource carries `service.version` (git SHA), `site.id` and `node.role`; mxgw and
HistorianGateway report `node.role=standalone` and take `site.id` from `<prefix>:Telemetry:SiteId`.
## Reading what arrived
```bash
ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua # one service
python3 ../otelidentity.py otopcua :12345 # Resource per signal + log samples, one process (instance-id suffix)
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
```
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc;
[`otelidentity.py`](otelidentity.py) prints the Resource each signal arrived with, the log count by severity
and sample log records, filtered to one process (scadaproj#55).
## Deploy / update
```bash
# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
```
- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or
`sudo chown 10001:10001 data`).
- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check
and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this
release; the config uses its replacement `resource_constant_labels`.
- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no
network or volume with any other container. Tear down with `docker compose down` there.