Files
scadaproj/infra/otel-collector/wonder/README.md
T
Joseph Doherty 71b8595742 feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:10:15 -05:00

90 lines
7.7 KiB
Markdown

# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 /
vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged.
Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md).
## What is installed
| Item | Value |
|---|---|
| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. |
| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` |
| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` |
| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM |
| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` |
## Endpoints (all 127.0.0.1, none reachable off-box)
| Port | What |
|---|---|
| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** |
| 4318 | OTLP HTTP receiver |
| 8889 | Prometheus exporter: everything received, labelled `job=<namespace>/<service>`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` |
| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) |
| 13133 | `health_check` |
**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
## Backend and retention
- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export
request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at
about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and
231 GB (vq04).
- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files.
At `info` level the collector writes a few lines at start and then only errors, so this stays small.
- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do
not export logs. When 0.3.0 ships, logs start flowing with no collector change.
- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline.
## App settings (service environment)
| App | Service | Where the env lives | Entries added |
|---|---|---|---|
| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` |
| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` |
| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) |
| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
These key names are unchanged in Telemetry 0.3.0 (`<App>:Telemetry` section), so the entries carry over to that upgrade.
Prometheus `/metrics` on every app stays served (OTLP is additive).
## Scripts
All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only).
| Script | Does |
|---|---|
| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.<ts>` |
| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts |
| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` |
```powershell
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
```
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
## Rollback
- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'` removes the two entries and
restarts. Alternatively, `reg import <dir>\<Service>-pre-otlp-<ts>.reg` and restart. On QA, restart Akka-pair nodes standby-first.
- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation
installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`.