feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)

- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 —
  OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889,
  health :13133; README runbook + otelsum.py summariser. Deployed.
- components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces
  received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway;
  OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996).
- GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups
  #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133
  (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 20:48:36 -05:00
parent f98ba08910
commit e1a2df83f7
12 changed files with 409 additions and 7 deletions
+1
View File
@@ -0,0 +1 @@
data/
+69
View File
@@ -0,0 +1,69 @@
# Shared dev OpenTelemetry collector (`zb-otel-collector`)
One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib)
that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the
shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA
> vq03/vq04) has a collector yet — that is scadaproj#54.
## Endpoints
| Port | What | Use |
|---|---|---|
| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. |
| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). |
| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=<service.namespace>/<service.name>`, `instance=<service.instance.id>` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. |
| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` |
Reachable from the Mac and from windev (`10.100.0.48`) — both verified.
## Pointing an app at it
OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on.
| App | Config keys (env form) |
|---|---|
| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less
`host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there.
- The SDK's default metric export interval is 60 s. For a quick check set
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called
without the exporter settings (scadaproj#55).
## Reading what arrived
```bash
ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua # one service
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
```
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc.
## Deploy / update
```bash
# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
```
- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or
`sudo chown 10001:10001 data`).
- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check
and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this
release; the config uses its replacement `resource_constant_labels`.
- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no
network or volume with any other container. Tear down with `docker compose down` there.
+65
View File
@@ -0,0 +1,65 @@
# zb-otel-collector — dev/test collector for the ZB.MOM.WW family (scadaproj#13).
# Receives OTLP (gRPC 4317 + HTTP 4318) from OtOpcUa, ScadaBridge, MxAccessGateway and
# HistorianGateway and proves receipt three ways:
# * debug exporter -> `docker logs zb-otel-collector` (one summary line per batch)
# * file exporters -> /data/{metrics,traces,logs}.jsonl (OTLP-JSON, rotated)
# * prometheus exporter-> :8889/metrics, every received metric re-exposed with
# job/instance + site_id/node_role/host_name labels (resource_constant_labels)
# No auth on the receivers: dev network only. Do not expose this config to a production tier.
extensions:
health_check:
endpoint: 0.0.0.0:13133
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 5s
limit_mib: 512
spike_limit_mib: 128
batch: {}
exporters:
debug:
verbosity: basic
file/metrics:
path: /data/metrics.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
file/traces:
path: /data/traces.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
file/logs:
path: /data/logs.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
prometheus:
endpoint: 0.0.0.0:8889
# service.name/namespace become job/instance automatically; also lift the family identity triple.
resource_constant_labels:
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name]
metric_expiration: 15m
service:
extensions: [health_check]
telemetry:
logs:
level: info
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/metrics, prometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/traces]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/logs]
+31
View File
@@ -0,0 +1,31 @@
# Shared dev OpenTelemetry collector for the ZB.MOM.WW family (scadaproj#13).
# Deploy on the shared Docker host 10.100.0.35:
# mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data, copy docker-compose.yml,
# config.yaml and otelsum.py into ~/zb-otel-collector/ (README "Deploy / update")
# cd ~/zb-otel-collector && docker compose up -d
# Verify: curl -s http://10.100.0.35:13133/ -> {"status":"Server available",...}
# curl -s http://10.100.0.35:8889/metrics -> received metrics, re-exposed for Prometheus
# tail -f ~/zb-otel-collector/data/traces.jsonl (also metrics.jsonl / logs.jsonl)
name: zb-otel-collector
services:
otel-collector:
# Pinned exact release (published 2026-10-05). Bump deliberately; never :latest.
image: otel/opentelemetry-collector-contrib:0.162.0
container_name: zb-otel-collector
restart: unless-stopped
# The image runs as uid 10001; the bind-mounted data dir must be writable by it
# (README: `sudo chown 10001:10001 data` or `chmod 777 data`).
command: ["--config=/etc/otelcol-contrib/config.yaml"]
ports:
- "4317:4317" # OTLP gRPC (family default: <App>:Telemetry:OtlpEndpoint=http://10.100.0.35:4317)
- "4318:4318" # OTLP HTTP/protobuf
- "8889:8889" # Prometheus exporter: everything received over OTLP, re-exposed for scraping
- "13133:13133" # health_check extension
volumes:
- ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
- ./data:/data
logging:
driver: json-file
options:
max-size: "20m"
max-file: "3"
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env python3
# Summarise what zb-otel-collector received: per service.name, the Resource attributes, the
# metric names per instrumentation scope, and span counts per (scope, name).
# Usage on the collector host: cd ~/zb-otel-collector/data && python3 ../otelsum.py [service.name]
import json,sys,collections
svc_filter=sys.argv[1] if len(sys.argv)>1 else None
def attrs(r): return {a["key"]:list(a["value"].values())[0] if a["value"] else None for a in r.get("attributes",[])}
res={}; mets=collections.defaultdict(set); spans=collections.Counter()
for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")):
try: lines=open(f)
except FileNotFoundError: continue
for l in lines:
d=json.loads(l)
for r in d.get(key,[]):
a=attrs(r["resource"]); s=a.get("service.name")
if svc_filter and s!=svc_filter: continue
res[s]={k:v for k,v in a.items() if not k.startswith("telemetry.sdk")} | {"_sdk":a.get("telemetry.sdk.version")}
if key=="resourceMetrics":
for sm in r["scopeMetrics"]:
for m in sm["metrics"]: mets[(s,sm["scope"]["name"])].add(m["name"])
elif key=="resourceSpans":
for ss in r["scopeSpans"]:
for sp in ss["spans"]: spans[(s,ss["scope"]["name"],sp["name"])]+=1
else:
for sl in r["scopeLogs"]: spans[(s,"<logs>",sl["scope"].get("name",""))]+=len(sl["logRecords"])
for s,a in res.items(): print("RESOURCE",s,json.dumps(a))
for (s,sc),names in sorted(mets.items()): print("METRICS",s,sc,len(names),sorted(names))
for k,v in sorted(spans.items()): print("SPANS",v,k)