feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)

ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:10:15 -05:00
parent 85654d7b43
commit 71b8595742
12 changed files with 649 additions and 8 deletions
+12
View File
@@ -401,6 +401,8 @@ nodes, verified Healthy at setup):
| HistorianGateway | Up | Up (since 2026-08-14) |
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) |
> **Active node as of 2026-11-12 22:00 EST** (after the scadaproj#54 rolling restart, standby-first): OtOpcUa **vq04** Primary (250) / vq03 240; ScadaBridge central **and** site **active on vq03**. The table above is the 2026-08-13 capture.
**Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge
`2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window,
rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because
@@ -488,6 +490,16 @@ Neither install is a scripted in-repo flow — the operational procedures live i
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
those runbooks, or not probed in §7.1, as unverified.
**OpenTelemetry collector on every wonder box (2026-11-12, scadaproj#54):** service `ZbOtelCollector`
(otelcol-contrib 0.162.0, NSSM, virtual account `NT SERVICE\ZbOtelCollector`) at `E:\ApiInstall\OtelCollector\` on
vd03, vq03 and vq04. All of its ports are **loopback only**: OTLP `127.0.0.1:4317`/`4318`, Prometheus exporter `:8889`,
self-metrics `:8888`, health `:13133`, so nothing new is reachable off-box. Data is in `data\{metrics,traces,logs}.jsonl`
(rotated, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway have
`<App>__Telemetry__Exporter=Otlp` + `__OtlpEndpoint=http://127.0.0.1:4317` in their service environment (HG: root `Telemetry__*`);
Overview is unchanged. On QA, `E:\ApiInstall\OtelCollector` is the 12th `-DenyPaths` entry of the OtOpcUa worker isolation.
The rolling restart left **OtOpcUa Primary on vq04** and **ScadaBridge Central + Site active on vq03**. Env backups:
`E:\psrem-stage\otel-54\backup-<COMPUTER>\`. Runbook + record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/).
**Auth posture differs between the two, and QA is the correct one:**
| | VD03 (DEV) | VQ03/VQ04 (QA) |