feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)

ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:10:15 -05:00
parent 85654d7b43
commit 71b8595742
12 changed files with 649 additions and 8 deletions
+7 -3
View File
@@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
> (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12.
>
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
@@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
Prometheus-only. → #54
- ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a
per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1,
file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview**
has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds
are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the