71b8595742
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
76 lines
4.7 KiB
Markdown
76 lines
4.7 KiB
Markdown
# Shared dev OpenTelemetry collector (`zb-otel-collector`)
|
|
|
|
One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib)
|
|
that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the
|
|
shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the
|
|
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
|
|
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
|
|
|
|
> Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA
|
|
> vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54):
|
|
> [`wonder/`](wonder/).
|
|
|
|
## Endpoints
|
|
|
|
| Port | What | Use |
|
|
|---|---|---|
|
|
| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. |
|
|
| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). |
|
|
| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=<service.namespace>/<service.name>`, `instance=<service.instance.id>` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. |
|
|
| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` |
|
|
|
|
Reachable from the Mac and from windev (`10.100.0.48`) — both verified.
|
|
|
|
## Pointing an app at it
|
|
|
|
OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on.
|
|
|
|
| App | Config keys (env form) |
|
|
|---|---|
|
|
| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
|
| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
|
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
|
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
|
|
|
|
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.3.0` (what the apps pin since
|
|
2026-11-12) rejects a scheme-less `host:port` at startup when `Exporter=Otlp`.
|
|
- The SDK's default metric export interval is 60 s. For a quick check set
|
|
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
|
|
- What travels: **metrics, traces and logs** (Telemetry `0.3.0`+, scadaproj#55): with `Exporter=Otlp` the
|
|
Serilog logs go to the same endpoint with the same Resource. `<prefix>:Telemetry:ExportLogs=false` keeps them
|
|
local. Every app's Resource carries `service.version` (git SHA), `site.id` and `node.role`; mxgw and
|
|
HistorianGateway report `node.role=standalone` and take `site.id` from `<prefix>:Telemetry:SiteId`.
|
|
|
|
## Reading what arrived
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35
|
|
docker logs --tail 20 zb-otel-collector # one debug line per received batch
|
|
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
|
|
python3 ../otelsum.py otopcua # one service
|
|
python3 ../otelidentity.py otopcua :12345 # Resource per signal + log samples, one process (instance-id suffix)
|
|
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
|
|
```
|
|
|
|
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
|
|
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc;
|
|
[`otelidentity.py`](otelidentity.py) prints the Resource each signal arrived with, the log count by severity
|
|
and sample log records, filtered to one process (scadaproj#55).
|
|
|
|
## Deploy / update
|
|
|
|
```bash
|
|
# from this directory on the Mac
|
|
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
|
|
scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/
|
|
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
|
|
```
|
|
|
|
- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or
|
|
`sudo chown 10001:10001 data`).
|
|
- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check
|
|
and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this
|
|
release; the config uses its replacement `resource_constant_labels`.
|
|
- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no
|
|
network or volume with any other container. Tear down with `docker compose down` there.
|