Files
scadaproj/infra/otel-collector/wonder/README.md
T
Joseph Doherty 71b8595742 feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:10:15 -05:00

7.7 KiB

ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)

One OpenTelemetry Collector (contrib) per wonder box: wonder-app-vd03 (DEV) and wonder-app-vq03 / vq04 (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on 10.100.0.35 (../README.md) is a separate thing and is unchanged.

Design note and rollout record: 2026-11-12-wonder-otel-collector.md.

What is installed

Item Value
Binary otelcol-contrib 0.162.0 windows_amd64 (the same release as the dev collector). The tarball's SHA-256 b3718291…9e5c3d matches the release's .sha256 asset. The exe SHA-256 is f8f69bd6…e436e. It was repacked on the Mac as a zip (97a675ee…a167) because PS 5.1 Expand-Archive can't read .tar.gz.
Folder E:\ApiInstall\OtelCollector\ containing bin\otelcol-contrib.exe, bin\nssm.exe (a copy of the box's NSSM), config\config.yaml (config.yaml), data\ and logs\
Service ZbOtelCollector (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s
Account Virtual account NT SERVICE\ZbOtelCollector, not LocalSystem. It needs only to bind loopback ports and write data\ + logs\
Service env ZB_TIER=DEV or QA, stamped on every signal as deployment.environment.name. NO_WINDOWS_SERVICE=1 makes otelcol run in console mode under NSSM
ACL The root is protected: the inherited ACEs were copied and BUILTIN\Users was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on data\ and logs\. The NT SERVICE\OtOpcUaScriptWorker deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited E:\ApiInstall deny, and on QA it was added by re-running the isolation installer with E:\ApiInstall\OtelCollector appended to -DenyPaths

Endpoints (all 127.0.0.1, none reachable off-box)

Port What
4317 OTLP gRPC receiver. Apps point here: http://127.0.0.1:4317
4318 OTLP HTTP receiver
8889 Prometheus exporter: everything received, labelled job=<namespace>/<service>, plus site_id, node_role, host_name and deployment_environment_name
8888 The collector's own metrics (otelcol_receiver_accepted_*, otelcol_exporter_send_failed_*)
13133 health_check

Trade-off: loopback-only receivers. Nothing on the network can send to the collector, so the receivers need no auth and no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac, a future Prometheus or Tempo) cannot scrape :8889 or query it. When a central backend is wanted, put it behind an exporter on these collectors (OTLP out with TLS and auth) rather than opening the receivers.

Backend and retention

  • file/metrics, file/traces and file/logs write data\{metrics,traces,logs}.jsonl as OTLP-JSON, one export request per line. Rotation is set to max_megabytes: 100, max_backups: 5 and max_days: 7. Each signal is capped at about 600 MB and all three at about 1.8 GB. At the rate seen on day one that holds 16 h to 3.7 days, depending on the signal and box (size binds before max_days; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and 231 GB (vq04).
  • The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
  • logs\collector.log is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files. At info level the collector writes a few lines at start and then only errors, so this stays small.
  • logs.jsonl stays empty until the apps run Telemetry 0.3.0 (scadaproj#55, OTLP logs). The builds deployed today do not export logs. When 0.3.0 ships, logs start flowing with no collector change.
  • No Tempo or Jaeger. A trace backend is a future option: add an otlp exporter to the traces pipeline.

App settings (service environment)

App Service Where the env lives Entries added
OtOpcUa OtOpcUaHost service key Environment OtOpcUa__Telemetry__Exporter=Otlp, OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
ScadaBridge ScadaBridge-Central, ScadaBridge-Site service key Environment ScadaBridge__Telemetry__Exporter, ScadaBridge__Telemetry__OtlpEndpoint
MxAccessGateway MxAccessGw NSSM Parameters\AppEnvironmentExtra MxGateway__Telemetry__Exporter, MxGateway__Telemetry__OtlpEndpoint
HistorianGateway HistorianGateway NSSM Parameters\AppEnvironmentExtra Telemetry__Exporter, Telemetry__OtlpEndpoint (root-level section)
Overview ZbOverview not changed Its AddZbTelemetry call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change

These key names are unchanged in Telemetry 0.3.0 (<App>:Telemetry section), so the entries carry over to that upgrade. Prometheus /metrics on every app stays served (OTLP is additive).

Scripts

All are run on the box from the stage folder E:\psrem-stage\otel-54\ through the winrm skill (deployment scope only).

Script Does
Install-OtelCollector.ps1 Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. -Uninstall stops and deletes the service and renames the folder to OtelCollector.removed.<ts>
Switch-AppsToOtlp.ps1 Per app, in order: .reg export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through nssm set or install-service.bat. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for /health/ready 200. On failure it restores the in-memory original and restarts. -Rollback removes the two entries and restarts
Test-OtelCollector.ps1 Read-only: service, health, listeners, what each service.name sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's /metrics
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
  'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
  'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220   # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content

On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).

Rollback

  • One app: Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>' removes the two entries and restarts. Alternatively, reg import <dir>\<Service>-pre-otlp-<ts>.reg and restart. On QA, restart Akka-pair nodes standby-first.
  • Collector: roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then run Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall. On QA, also re-run the isolation installer with E:\ApiInstall\OtelCollector removed from -DenyPaths.