[Telemetry] Deploy an OTel collector on the wonder tiers (DEV vd03, QA vq03/vq04) and switch the apps to OTLP #54
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #13. The OTLP path is now proven end to end against the shared dev collector
zb-otel-collectoron 10.100.0.35 (infra/otel-collector/, evidencecomponents/observability/2026-11-12-otlp-end-to-end-evidence.md), but no collector exists on DEV or QA, so every wonder box still runs Prometheus-only and OtOpcUa/HistorianGateway spans go nowhere there.Do: decide the topology (one collector per tier on the app box, or a central one), the backend (keep the file/Prometheus exporters, or add Tempo/Jaeger + Prometheus), a Windows service shape (otelcol-contrib has a Windows build; NSSM like the other apps), receiver auth/TLS (the dev config has none), retention. Then set
<App>:Telemetry:Exporter=Otlp+:OtlpEndpoint=http://<collector>:4317per app (HistorianGateway's keys are root-levelTelemetry:*). Deploys are deployment activities under the winrm scope rules.Prometheus
/metricsstays served under Otlp (OTLP is additive), so nothing scraping today breaks.Done 2026-11-12, 21:38–21:56 EST, on DEV vd03 and QA vq03 + vq04. Infra and config only: no binary, appsettings or Telemetry pin changed. Record and runbook are in
infra/otel-collector/wonder/(71b8595).Design (owner defaults)
ZbOtelCollector, otelcol-contrib 0.162.0 windows_amd64. The tarball SHA-256 matches the release.sha256asset; this release has no combinedchecksums.txt.NT SERVICE\ZbOtelCollector, installed atE:\ApiInstall\OtelCollector\.fileexporters (OTLP-JSON, 100 MB × 5 backups, max 7 days, ≤ ~1.8 GB per box) + a loopback Prometheus exporter. No Tempo/Jaeger yet (future option). Each signal is stamped withdeployment.environment.name=DEV|QA.BUILTIN\Usersremoved; the service account has RX, plus Modify ondata\andlogs\. The worker deny (lmxopcua#891) is kept: on vd03 as the copiedE:\ApiInstalldeny, on QA by re-running the isolation installer withE:\ApiInstall\OtelCollectorappended to-DenyPaths(11 → 12). OnlyDenyPathRecordand the new folder changed, the re-apply was allunchanged, and the OtOpcUaHost env hash was unchanged.Per-app change
<Prefix>Telemetry__Exporter=Otlpand<Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317, written straight to the registry. Prefixes:OtOpcUa__,ScadaBridge__,MxGateway__; HG uses rootTelemetry__..regbackup of each service was taken first into the protectedE:\psrem-stage\otel-54\backup-<COMPUTER>\.0d01abcc1fe5, env block3f283e9a1d57before).AddZbTelemetrybinds no exporter key, so it needs a code change./health/ready200; ServiceLevel 250; SB site MxGatewayConnected810/810, good 398 / bad 412 (= baseline); 1 gateway processQA ScadaBridge → MxGateway went 0 resolved after the vq03 gateway restart and took ~4 min to recover (21:49:34 → 21:53:38). That is longer than the ~1.5 min seen on 2026-09-10. It recovered unaided; I waited for 2196 Good before touching the active nodes. After the site failover it recovered in ~1 min. Overview: QA 12/12 Up on both dashboards, DEV 6/6.
Data at the collectors. Every process on every box sends metrics and traces with the right
host.name/node.role/ tier, including the ScadaBridge Central role (node.role=Central) on vd03, vq03 and vq04. Receivers refused 0. Prometheus:8889series per job:Every app's own
/metricsstill answers 200 (OTLP is additive). OTLP logs won't flow until the Telemetry 0.3.0 builds (#55) are deployed; the key names are unchanged, so no further config is needed.Retention and disk. At the day-one rate, size binds before
max_days: the cap holds roughly 16 h to 3.7 days per signal (vq03 traces are the fastest at ~860 MB/day). Disk stays ≤ ~1.8 GB per box. E: free: vd03 179 GB, vq03 215 GB, vq04 230 GB.Rollback
Switch-AppsToOtlp.ps1 -Rollback(orreg importthe backup) + restart, standby-first on QA.Install-OtelCollector.ps1 -Uninstall; on QA also drop the path from-DenyPaths.E:\psrem-stage\otel-54\on each box.Follow-ups, not filed: Overview exporter binding; raise
max_backupsif a longer local window is wanted.