feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)

ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:10:15 -05:00
parent 85654d7b43
commit 71b8595742
12 changed files with 649 additions and 8 deletions
@@ -163,7 +163,7 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
(`logs.jsonl` stayed empty).
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers.
- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
@@ -211,7 +211,7 @@ scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack)
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
wonder tier has a collector yet (#54), so no deployed box can trip it.
wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
+7 -3
View File
@@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
> (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12.
>
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
@@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
Prometheus-only. → #54
- ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a
per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1,
file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview**
has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds
are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the