feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)
- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 — OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889, health :13133; README runbook + otelsum.py summariser. Deployed. - components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway; OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996). - GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133 (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -0,0 +1 @@
|
||||
data/
|
||||
@@ -0,0 +1,69 @@
|
||||
# Shared dev OpenTelemetry collector (`zb-otel-collector`)
|
||||
|
||||
One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib)
|
||||
that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the
|
||||
shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the
|
||||
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
|
||||
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
|
||||
|
||||
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA
|
||||
> vq03/vq04) has a collector yet — that is scadaproj#54.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Port | What | Use |
|
||||
|---|---|---|
|
||||
| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. |
|
||||
| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). |
|
||||
| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=<service.namespace>/<service.name>`, `instance=<service.instance.id>` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. |
|
||||
| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` |
|
||||
|
||||
Reachable from the Mac and from windev (`10.100.0.48`) — both verified.
|
||||
|
||||
## Pointing an app at it
|
||||
|
||||
OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on.
|
||||
|
||||
| App | Config keys (env form) |
|
||||
|---|---|
|
||||
| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
||||
| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
||||
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
|
||||
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
|
||||
|
||||
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less
|
||||
`host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there.
|
||||
- The SDK's default metric export interval is 60 s. For a quick check set
|
||||
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
|
||||
- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called
|
||||
without the exporter settings (scadaproj#55).
|
||||
|
||||
## Reading what arrived
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35
|
||||
docker logs --tail 20 zb-otel-collector # one debug line per received batch
|
||||
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
|
||||
python3 ../otelsum.py otopcua # one service
|
||||
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
|
||||
```
|
||||
|
||||
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
|
||||
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc.
|
||||
|
||||
## Deploy / update
|
||||
|
||||
```bash
|
||||
# from this directory on the Mac
|
||||
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
|
||||
scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/
|
||||
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
|
||||
```
|
||||
|
||||
- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or
|
||||
`sudo chown 10001:10001 data`).
|
||||
- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check
|
||||
and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this
|
||||
release; the config uses its replacement `resource_constant_labels`.
|
||||
- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no
|
||||
network or volume with any other container. Tear down with `docker compose down` there.
|
||||
@@ -0,0 +1,65 @@
|
||||
# zb-otel-collector — dev/test collector for the ZB.MOM.WW family (scadaproj#13).
|
||||
# Receives OTLP (gRPC 4317 + HTTP 4318) from OtOpcUa, ScadaBridge, MxAccessGateway and
|
||||
# HistorianGateway and proves receipt three ways:
|
||||
# * debug exporter -> `docker logs zb-otel-collector` (one summary line per batch)
|
||||
# * file exporters -> /data/{metrics,traces,logs}.jsonl (OTLP-JSON, rotated)
|
||||
# * prometheus exporter-> :8889/metrics, every received metric re-exposed with
|
||||
# job/instance + site_id/node_role/host_name labels (resource_constant_labels)
|
||||
# No auth on the receivers: dev network only. Do not expose this config to a production tier.
|
||||
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 0.0.0.0:13133
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_mib: 512
|
||||
spike_limit_mib: 128
|
||||
batch: {}
|
||||
|
||||
exporters:
|
||||
debug:
|
||||
verbosity: basic
|
||||
file/metrics:
|
||||
path: /data/metrics.jsonl
|
||||
rotation: { max_megabytes: 50, max_backups: 3 }
|
||||
file/traces:
|
||||
path: /data/traces.jsonl
|
||||
rotation: { max_megabytes: 50, max_backups: 3 }
|
||||
file/logs:
|
||||
path: /data/logs.jsonl
|
||||
rotation: { max_megabytes: 50, max_backups: 3 }
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:8889
|
||||
# service.name/namespace become job/instance automatically; also lift the family identity triple.
|
||||
resource_constant_labels:
|
||||
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name]
|
||||
metric_expiration: 15m
|
||||
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
logs:
|
||||
level: info
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [debug, file/metrics, prometheus]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [debug, file/traces]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [debug, file/logs]
|
||||
@@ -0,0 +1,31 @@
|
||||
# Shared dev OpenTelemetry collector for the ZB.MOM.WW family (scadaproj#13).
|
||||
# Deploy on the shared Docker host 10.100.0.35:
|
||||
# mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data, copy docker-compose.yml,
|
||||
# config.yaml and otelsum.py into ~/zb-otel-collector/ (README "Deploy / update")
|
||||
# cd ~/zb-otel-collector && docker compose up -d
|
||||
# Verify: curl -s http://10.100.0.35:13133/ -> {"status":"Server available",...}
|
||||
# curl -s http://10.100.0.35:8889/metrics -> received metrics, re-exposed for Prometheus
|
||||
# tail -f ~/zb-otel-collector/data/traces.jsonl (also metrics.jsonl / logs.jsonl)
|
||||
name: zb-otel-collector
|
||||
services:
|
||||
otel-collector:
|
||||
# Pinned exact release (published 2026-10-05). Bump deliberately; never :latest.
|
||||
image: otel/opentelemetry-collector-contrib:0.162.0
|
||||
container_name: zb-otel-collector
|
||||
restart: unless-stopped
|
||||
# The image runs as uid 10001; the bind-mounted data dir must be writable by it
|
||||
# (README: `sudo chown 10001:10001 data` or `chmod 777 data`).
|
||||
command: ["--config=/etc/otelcol-contrib/config.yaml"]
|
||||
ports:
|
||||
- "4317:4317" # OTLP gRPC (family default: <App>:Telemetry:OtlpEndpoint=http://10.100.0.35:4317)
|
||||
- "4318:4318" # OTLP HTTP/protobuf
|
||||
- "8889:8889" # Prometheus exporter: everything received over OTLP, re-exposed for scraping
|
||||
- "13133:13133" # health_check extension
|
||||
volumes:
|
||||
- ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
||||
- ./data:/data
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "20m"
|
||||
max-file: "3"
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env python3
|
||||
# Summarise what zb-otel-collector received: per service.name, the Resource attributes, the
|
||||
# metric names per instrumentation scope, and span counts per (scope, name).
|
||||
# Usage on the collector host: cd ~/zb-otel-collector/data && python3 ../otelsum.py [service.name]
|
||||
import json,sys,collections
|
||||
svc_filter=sys.argv[1] if len(sys.argv)>1 else None
|
||||
def attrs(r): return {a["key"]:list(a["value"].values())[0] if a["value"] else None for a in r.get("attributes",[])}
|
||||
res={}; mets=collections.defaultdict(set); spans=collections.Counter()
|
||||
for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")):
|
||||
try: lines=open(f)
|
||||
except FileNotFoundError: continue
|
||||
for l in lines:
|
||||
d=json.loads(l)
|
||||
for r in d.get(key,[]):
|
||||
a=attrs(r["resource"]); s=a.get("service.name")
|
||||
if svc_filter and s!=svc_filter: continue
|
||||
res[s]={k:v for k,v in a.items() if not k.startswith("telemetry.sdk")} | {"_sdk":a.get("telemetry.sdk.version")}
|
||||
if key=="resourceMetrics":
|
||||
for sm in r["scopeMetrics"]:
|
||||
for m in sm["metrics"]: mets[(s,sm["scope"]["name"])].add(m["name"])
|
||||
elif key=="resourceSpans":
|
||||
for ss in r["scopeSpans"]:
|
||||
for sp in ss["spans"]: spans[(s,ss["scope"]["name"],sp["name"])]+=1
|
||||
else:
|
||||
for sl in r["scopeLogs"]: spans[(s,"<logs>",sl["scope"].get("name",""))]+=len(sl["logRecords"])
|
||||
for s,a in res.items(): print("RESOURCE",s,json.dumps(a))
|
||||
for (s,sc),names in sorted(mets.items()): print("METRICS",s,sc,len(names),sorted(names))
|
||||
for k,v in sorted(spans.items()): print("SPANS",v,k)
|
||||
Reference in New Issue
Block a user