feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -163,7 +163,7 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew
|
||||
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
|
||||
(`logs.jsonl` stayed empty).
|
||||
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
|
||||
- **scadaproj#54:** collector on the wonder tiers.
|
||||
- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
|
||||
|
||||
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
|
||||
|
||||
@@ -211,7 +211,7 @@ scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack)
|
||||
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
|
||||
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
|
||||
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
|
||||
wonder tier has a collector yet (#54), so no deployed box can trip it.
|
||||
wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
|
||||
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
|
||||
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
|
||||
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
|
||||
|
||||
@@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
|
||||
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
|
||||
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
|
||||
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
|
||||
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
|
||||
> (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12.
|
||||
>
|
||||
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
|
||||
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
|
||||
@@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
|
||||
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
|
||||
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
|
||||
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
|
||||
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
|
||||
Prometheus-only. → #54
|
||||
- ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a
|
||||
per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1,
|
||||
file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
|
||||
HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview**
|
||||
has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds
|
||||
are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
|
||||
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
|
||||
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
|
||||
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the
|
||||
|
||||
Reference in New Issue
Block a user