71b8595742
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
90 lines
7.7 KiB
Markdown
90 lines
7.7 KiB
Markdown
# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
|
|
|
|
One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 /
|
|
vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
|
|
`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged.
|
|
|
|
Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md).
|
|
|
|
## What is installed
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. |
|
|
| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` |
|
|
| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
|
|
| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` |
|
|
| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM |
|
|
| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` |
|
|
|
|
## Endpoints (all 127.0.0.1, none reachable off-box)
|
|
|
|
| Port | What |
|
|
|---|---|
|
|
| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** |
|
|
| 4318 | OTLP HTTP receiver |
|
|
| 8889 | Prometheus exporter: everything received, labelled `job=<namespace>/<service>`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` |
|
|
| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) |
|
|
| 13133 | `health_check` |
|
|
|
|
**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and
|
|
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
|
|
a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter
|
|
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
|
|
|
|
## Backend and retention
|
|
|
|
- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export
|
|
request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at
|
|
about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and
|
|
231 GB (vq04).
|
|
- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
|
|
- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files.
|
|
At `info` level the collector writes a few lines at start and then only errors, so this stays small.
|
|
- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do
|
|
not export logs. When 0.3.0 ships, logs start flowing with no collector change.
|
|
- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline.
|
|
|
|
## App settings (service environment)
|
|
|
|
| App | Service | Where the env lives | Entries added |
|
|
|---|---|---|---|
|
|
| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` |
|
|
| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
|
|
| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` |
|
|
| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) |
|
|
| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
|
|
|
|
These key names are unchanged in Telemetry 0.3.0 (`<App>:Telemetry` section), so the entries carry over to that upgrade.
|
|
Prometheus `/metrics` on every app stays served (OTLP is additive).
|
|
|
|
## Scripts
|
|
|
|
All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only).
|
|
|
|
| Script | Does |
|
|
|---|---|
|
|
| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.<ts>` |
|
|
| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts |
|
|
| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` |
|
|
|
|
```powershell
|
|
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
|
|
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
|
|
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
|
|
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
|
|
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
|
|
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
|
|
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
|
|
```
|
|
|
|
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
|
|
|
|
## Rollback
|
|
|
|
- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'` removes the two entries and
|
|
restarts. Alternatively, `reg import <dir>\<Service>-pre-otlp-<ts>.reg` and restart. On QA, restart Akka-pair nodes standby-first.
|
|
- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
|
|
run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation
|
|
installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`.
|