ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
One OpenTelemetry Collector (contrib) per wonder box: wonder-app-vd03 (DEV) and wonder-app-vq03 /
vq04 (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
10.100.0.35 (../README.md) is a separate thing and is unchanged.
Design note and rollout record: 2026-11-12-wonder-otel-collector.md.
What is installed
| Item | Value |
|---|---|
| Binary | otelcol-contrib 0.162.0 windows_amd64 (the same release as the dev collector). The tarball's SHA-256 b3718291…9e5c3d matches the release's .sha256 asset. The exe SHA-256 is f8f69bd6…e436e. It was repacked on the Mac as a zip (97a675ee…a167) because PS 5.1 Expand-Archive can't read .tar.gz. |
| Folder | E:\ApiInstall\OtelCollector\ containing bin\otelcol-contrib.exe, bin\nssm.exe (a copy of the box's NSSM), config\config.yaml (config.yaml), data\ and logs\ |
| Service | ZbOtelCollector (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
| Account | Virtual account NT SERVICE\ZbOtelCollector, not LocalSystem. It needs only to bind loopback ports and write data\ + logs\ |
| Service env | ZB_TIER=DEV or QA, stamped on every signal as deployment.environment.name. NO_WINDOWS_SERVICE=1 makes otelcol run in console mode under NSSM |
| ACL | The root is protected: the inherited ACEs were copied and BUILTIN\Users was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on data\ and logs\. The NT SERVICE\OtOpcUaScriptWorker deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited E:\ApiInstall deny, and on QA it was added by re-running the isolation installer with E:\ApiInstall\OtelCollector appended to -DenyPaths |
Endpoints (all 127.0.0.1, none reachable off-box)
| Port | What |
|---|---|
| 4317 | OTLP gRPC receiver. Apps point here: http://127.0.0.1:4317 |
| 4318 | OTLP HTTP receiver |
| 8889 | Prometheus exporter: everything received, labelled job=<namespace>/<service>, plus site_id, node_role, host_name and deployment_environment_name |
| 8888 | The collector's own metrics (otelcol_receiver_accepted_*, otelcol_exporter_send_failed_*) |
| 13133 | health_check |
Trade-off: loopback-only receivers. Nothing on the network can send to the collector, so the receivers need no auth and
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
a future Prometheus or Tempo) cannot scrape :8889 or query it. When a central backend is wanted, put it behind an exporter
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
Backend and retention
file/metrics,file/tracesandfile/logswritedata\{metrics,traces,logs}.jsonlas OTLP-JSON, one export request per line. Rotation is set tomax_megabytes: 100,max_backups: 5andmax_days: 7. Each signal is capped at about 600 MB and all three at about 1.8 GB. At the rate seen on day one that holds 16 h to 3.7 days, depending on the signal and box (size binds beforemax_days; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and 231 GB (vq04).- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
logs\collector.logis NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files. Atinfolevel the collector writes a few lines at start and then only errors, so this stays small.logs.jsonlstays empty until the apps run Telemetry 0.3.0 (scadaproj#55, OTLP logs). The builds deployed today do not export logs. When 0.3.0 ships, logs start flowing with no collector change.- No Tempo or Jaeger. A trace backend is a future option: add an
otlpexporter to thetracespipeline.
App settings (service environment)
| App | Service | Where the env lives | Entries added |
|---|---|---|---|
| OtOpcUa | OtOpcUaHost |
service key Environment |
OtOpcUa__Telemetry__Exporter=Otlp, OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317 |
| ScadaBridge | ScadaBridge-Central, ScadaBridge-Site |
service key Environment |
ScadaBridge__Telemetry__Exporter, ScadaBridge__Telemetry__OtlpEndpoint |
| MxAccessGateway | MxAccessGw |
NSSM Parameters\AppEnvironmentExtra |
MxGateway__Telemetry__Exporter, MxGateway__Telemetry__OtlpEndpoint |
| HistorianGateway | HistorianGateway |
NSSM Parameters\AppEnvironmentExtra |
Telemetry__Exporter, Telemetry__OtlpEndpoint (root-level section) |
| Overview | ZbOverview |
not changed | Its AddZbTelemetry call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
These key names are unchanged in Telemetry 0.3.0 (<App>:Telemetry section), so the entries carry over to that upgrade.
Prometheus /metrics on every app stays served (OTLP is additive).
Scripts
All are run on the box from the stage folder E:\psrem-stage\otel-54\ through the winrm skill (deployment scope only).
| Script | Does |
|---|---|
Install-OtelCollector.ps1 |
Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. -Uninstall stops and deletes the service and renames the folder to OtelCollector.removed.<ts> |
Switch-AppsToOtlp.ps1 |
Per app, in order: .reg export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through nssm set or install-service.bat. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for /health/ready 200. On failure it restores the in-memory original and restarts. -Rollback removes the two entries and restarts |
Test-OtelCollector.ps1 |
Read-only: service, health, listeners, what each service.name sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's /metrics |
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
Rollback
- One app:
Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'removes the two entries and restarts. Alternatively,reg import <dir>\<Service>-pre-otlp-<ts>.regand restart. On QA, restart Akka-pair nodes standby-first. - Collector: roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
run
Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall. On QA, also re-run the isolation installer withE:\ApiInstall\OtelCollectorremoved from-DenyPaths.