[Telemetry] Deploy an OTel collector on the wonder tiers (DEV vd03, QA vq03/vq04) and switch the apps to OTLP #54

Closed
opened 2026-11-12 20:45:10 -05:00 by dohertj2 · 1 comment
Owner

Follow-up to #13. The OTLP path is now proven end to end against the shared dev collector zb-otel-collector on 10.100.0.35 (infra/otel-collector/, evidence components/observability/2026-11-12-otlp-end-to-end-evidence.md), but no collector exists on DEV or QA, so every wonder box still runs Prometheus-only and OtOpcUa/HistorianGateway spans go nowhere there.

Do: decide the topology (one collector per tier on the app box, or a central one), the backend (keep the file/Prometheus exporters, or add Tempo/Jaeger + Prometheus), a Windows service shape (otelcol-contrib has a Windows build; NSSM like the other apps), receiver auth/TLS (the dev config has none), retention. Then set <App>:Telemetry:Exporter=Otlp + :OtlpEndpoint=http://<collector>:4317 per app (HistorianGateway's keys are root-level Telemetry:*). Deploys are deployment activities under the winrm scope rules.

Prometheus /metrics stays served under Otlp (OTLP is additive), so nothing scraping today breaks.

Follow-up to #13. The OTLP path is now proven end to end against the shared dev collector `zb-otel-collector` on 10.100.0.35 (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`), but no collector exists on DEV or QA, so every wonder box still runs Prometheus-only and OtOpcUa/HistorianGateway spans go nowhere there. **Do:** decide the topology (one collector per tier on the app box, or a central one), the backend (keep the file/Prometheus exporters, or add Tempo/Jaeger + Prometheus), a Windows service shape (otelcol-contrib has a Windows build; NSSM like the other apps), receiver auth/TLS (the dev config has none), retention. Then set `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint=http://<collector>:4317` per app (HistorianGateway's keys are root-level `Telemetry:*`). Deploys are deployment activities under the winrm scope rules. Prometheus `/metrics` stays served under Otlp (OTLP is additive), so nothing scraping today breaks.
dohertj2 added the type:followuptype:infrapriority:p3cross-repo labels 2026-11-12 20:45:10 -05:00
Author
Owner

Done 2026-11-12, 21:38–21:56 EST, on DEV vd03 and QA vq03 + vq04. Infra and config only: no binary, appsettings or Telemetry pin changed. Record and runbook are in infra/otel-collector/wonder/ (71b8595).

Design (owner defaults)

  • One collector per box: ZbOtelCollector, otelcol-contrib 0.162.0 windows_amd64. The tarball SHA-256 matches the release .sha256 asset; this release has no combined checksums.txt.
  • Runs under NSSM as the virtual account NT SERVICE\ZbOtelCollector, installed at E:\ApiInstall\OtelCollector\.
  • All listeners are on 127.0.0.1: 4317/4318 OTLP, 8889 Prometheus exporter, 8888 self-metrics, 13133 health. That is why the receivers have no auth or TLS. The trade-off is no tier-wide view; a central backend would be added as an exporter on these collectors.
  • Backend: file exporters (OTLP-JSON, 100 MB × 5 backups, max 7 days, ≤ ~1.8 GB per box) + a loopback Prometheus exporter. No Tempo/Jaeger yet (future option). Each signal is stamped with deployment.environment.name=DEV|QA.
  • ACL: the root is protected, BUILTIN\Users removed; the service account has RX, plus Modify on data\ and logs\. The worker deny (lmxopcua#891) is kept: on vd03 as the copied E:\ApiInstall deny, on QA by re-running the isolation installer with E:\ApiInstall\OtelCollector appended to -DenyPaths (11 → 12). Only DenyPathRecord and the new folder changed, the re-apply was all unchanged, and the OtOpcUaHost env hash was unchanged.

Per-app change

  • Each service env gets <Prefix>Telemetry__Exporter=Otlp and <Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317, written straight to the registry. Prefixes: OtOpcUa__, ScadaBridge__, MxGateway__; HG uses root Telemetry__.
  • A .reg backup of each service was taken first into the protected E:\psrem-stage\otel-54\backup-<COMPUTER>\.
  • Every pre-existing entry was hash-identical on readback, and the pepper SHA prefixes were unchanged (vd03 mxgw 0d01abcc1fe5, env block 3f283e9a1d57 before).
  • Overview is not switched. Its AddZbTelemetry binds no exporter key, so it needs a code change.
Box Restarts (stop→ready) Result
vd03 HG 5 s, OtOpcUa 8 s, SB Central 13 s, SB Site 9 s, mxgw 5 s all /health/ready 200; ServiceLevel 250; SB site MxGateway Connected 810/810, good 398 / bad 412 (= baseline); 1 gateway process
vq04 (1) HG 5 s, mxgw 6 s, OtOpcUa Secondary 10 s (stop 32 s) 240 back; site 2196/2196 Good
vq03 HG, mxgw, SB Central + Site standby, then OtOpcUa Primary OtOpcUa failover → vq04 250 / vq03 240
vq04 (2) SB Central + Site active failover → SB Central + Site active on vq03, 2 nodes online each

QA ScadaBridge → MxGateway went 0 resolved after the vq03 gateway restart and took ~4 min to recover (21:49:34 → 21:53:38). That is longer than the ~1.5 min seen on 2026-09-10. It recovered unaided; I waited for 2196 Good before touching the active nodes. After the site failover it recovered in ~1 min. Overview: QA 12/12 Up on both dashboards, DEV 6/6.

Data at the collectors. Every process on every box sends metrics and traces with the right host.name / node.role / tier, including the ScadaBridge Central role (node.role=Central) on vd03, vq03 and vq04. Receivers refused 0. Prometheus :8889 series per job:

  • vd03: HG 289, mxgw 376, OtOpcUa 304, SB 680
  • vq03: 306 / 644 / 396 / 771
  • vq04: 248 / 235 / 472 / 438

Every app's own /metrics still answers 200 (OTLP is additive). OTLP logs won't flow until the Telemetry 0.3.0 builds (#55) are deployed; the key names are unchanged, so no further config is needed.

Retention and disk. At the day-one rate, size binds before max_days: the cap holds roughly 16 h to 3.7 days per signal (vq03 traces are the fastest at ~860 MB/day). Disk stays ≤ ~1.8 GB per box. E: free: vd03 179 GB, vq03 215 GB, vq04 230 GB.

Rollback

  • Per app: Switch-AppsToOtlp.ps1 -Rollback (or reg import the backup) + restart, standby-first on QA.
  • Collector: Install-OtelCollector.ps1 -Uninstall; on QA also drop the path from -DenyPaths.
  • Staging + scripts stay at E:\psrem-stage\otel-54\ on each box.

Follow-ups, not filed: Overview exporter binding; raise max_backups if a longer local window is wanted.

Done 2026-11-12, 21:38–21:56 EST, on DEV vd03 and QA vq03 + vq04. Infra and config only: no binary, appsettings or Telemetry pin changed. Record and runbook are in [`infra/otel-collector/wonder/`](https://gitea.dohertylan.com/dohertj2/scadaproj/src/branch/main/infra/otel-collector/wonder) (`71b8595`). **Design (owner defaults)** - One collector per box: `ZbOtelCollector`, otelcol-contrib **0.162.0** windows_amd64. The tarball SHA-256 matches the release `.sha256` asset; this release has no combined `checksums.txt`. - Runs under NSSM as the virtual account `NT SERVICE\ZbOtelCollector`, installed at `E:\ApiInstall\OtelCollector\`. - **All listeners are on 127.0.0.1**: 4317/4318 OTLP, 8889 Prometheus exporter, 8888 self-metrics, 13133 health. That is why the receivers have no auth or TLS. The trade-off is no tier-wide view; a central backend would be added as an exporter on these collectors. - Backend: `file` exporters (OTLP-JSON, 100 MB × 5 backups, max 7 days, ≤ ~1.8 GB per box) + a loopback Prometheus exporter. No Tempo/Jaeger yet (future option). Each signal is stamped with `deployment.environment.name=DEV|QA`. - ACL: the root is protected, `BUILTIN\Users` removed; the service account has RX, plus Modify on `data\` and `logs\`. The worker deny (lmxopcua#891) is kept: on vd03 as the copied `E:\ApiInstall` deny, on QA by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` (11 → 12). Only `DenyPathRecord` and the new folder changed, the re-apply was all `unchanged`, and the OtOpcUaHost env hash was unchanged. **Per-app change** - Each service env gets `<Prefix>Telemetry__Exporter=Otlp` and `<Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317`, written straight to the registry. Prefixes: `OtOpcUa__`, `ScadaBridge__`, `MxGateway__`; HG uses root `Telemetry__`. - A `.reg` backup of each service was taken first into the protected `E:\psrem-stage\otel-54\backup-<COMPUTER>\`. - Every pre-existing entry was hash-identical on readback, and the pepper SHA prefixes were unchanged (vd03 mxgw `0d01abcc1fe5`, env block `3f283e9a1d57` before). - **Overview is not switched.** Its `AddZbTelemetry` binds no exporter key, so it needs a code change. | Box | Restarts (stop→ready) | Result | |---|---|---| | vd03 | HG 5 s, OtOpcUa 8 s, SB Central 13 s, SB Site 9 s, mxgw 5 s | all `/health/ready` 200; ServiceLevel 250; SB site MxGateway `Connected` 810/810, good 398 / bad 412 (= baseline); 1 gateway process | | vq04 (1) | HG 5 s, mxgw 6 s, OtOpcUa **Secondary** 10 s (stop 32 s) | 240 back; site 2196/2196 Good | | vq03 | HG, mxgw, SB Central + Site **standby**, then OtOpcUa **Primary** | OtOpcUa failover → **vq04 250 / vq03 240** | | vq04 (2) | SB Central + Site **active** | failover → **SB Central + Site active on vq03**, 2 nodes online each | QA ScadaBridge → MxGateway went 0 resolved after the vq03 gateway restart and **took ~4 min to recover** (21:49:34 → 21:53:38). That is longer than the ~1.5 min seen on 2026-09-10. It recovered unaided; I waited for 2196 Good before touching the active nodes. After the site failover it recovered in ~1 min. Overview: QA **12/12 Up** on both dashboards, DEV 6/6. **Data at the collectors.** Every process on every box sends metrics and traces with the right `host.name` / `node.role` / tier, including the **ScadaBridge Central role** (`node.role=Central`) on vd03, vq03 and vq04. Receivers refused 0. Prometheus `:8889` series per job: - vd03: HG 289, mxgw 376, OtOpcUa 304, SB 680 - vq03: 306 / 644 / 396 / 771 - vq04: 248 / 235 / 472 / 438 Every app's own `/metrics` still answers 200 (OTLP is additive). **OTLP logs won't flow until the Telemetry 0.3.0 builds (#55) are deployed**; the key names are unchanged, so no further config is needed. **Retention and disk.** At the day-one rate, size binds before `max_days`: the cap holds roughly 16 h to 3.7 days per signal (vq03 traces are the fastest at ~860 MB/day). Disk stays ≤ ~1.8 GB per box. E: free: vd03 179 GB, vq03 215 GB, vq04 230 GB. **Rollback** - Per app: `Switch-AppsToOtlp.ps1 -Rollback` (or `reg import` the backup) + restart, standby-first on QA. - Collector: `Install-OtelCollector.ps1 -Uninstall`; on QA also drop the path from `-DenyPaths`. - Staging + scripts stay at `E:\psrem-stage\otel-54\` on each box. Follow-ups, not filed: Overview exporter binding; raise `max_backups` if a longer local window is wanted.
Sign in to join this conversation.