feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -6,8 +6,9 @@ shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-sha
|
||||
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
|
||||
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
|
||||
|
||||
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA
|
||||
> vq03/vq04) has a collector yet — that is scadaproj#54.
|
||||
> Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA
|
||||
> vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54):
|
||||
> [`wonder/`](wonder/).
|
||||
|
||||
## Endpoints
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# 2026-11-12: OTel collector on the wonder tiers, apps switched to OTLP (scadaproj#54)
|
||||
|
||||
**Status: EXECUTED 2026-11-12, 21:38–21:56 EST.** DEV wonder-app-vd03 and QA wonder-app-vq03 + vq04 each run the
|
||||
`ZbOtelCollector` service. Every app that has an exporter setting now exports OTLP to the collector on its own box.
|
||||
No binary, appsettings file or Telemetry pin was changed. The change consists of a new service plus two service-environment entries per app.
|
||||
Runbook and scripts: [`README.md`](README.md).
|
||||
|
||||
Channel: the `winrm` skill, used only for this deployment's on-box steps (preflight, install, env edits, restarts,
|
||||
verification), with the user's request of 2026-11-12 ("resolve the remaining otel issues"). Files were staged over SMB to
|
||||
`E:\psrem-stage\otel-54\`. Health, ScadaBridge site health, OPC UA ServiceLevel and Overview were checked from the Mac
|
||||
over HTTP / opc.tcp.
|
||||
|
||||
## Design (owner-approved defaults)
|
||||
|
||||
| Decision | Choice | Why / trade-off |
|
||||
|---|---|---|
|
||||
| Topology | One collector **per box**, apps → `http://127.0.0.1:4317` | No new network path and nothing new reachable on the corporate network. On QA each node keeps its own copy, so the two pair nodes' data sits in two places. A tier-wide view needs a backend that the collectors export *to* |
|
||||
| Receiver security | Loopback-only bind on all five ports, no auth or TLS | Only local processes can send. Opening the receivers to the network would require auth + TLS first |
|
||||
| Backend | `file` exporter (OTLP-JSON, rotated) + `prometheus` exporter on `127.0.0.1:8889` | Durable local record plus a scrapeable view, with nothing new to operate. **No Tempo or Jaeger yet.** That is the future option: add an `otlp` exporter to the traces pipeline |
|
||||
| Retention | Each signal file rotates at 100 MB, keeps 5 backups, max 7 days → ≤ ~600 MB per signal, ≤ ~1.8 GB total | E: had 180 / 216 / 231 GB free (vd03 / vq03 / vq04), so the cap is under 1 %. **At the observed rate (§ Disk) the size cap binds, not the 7 days:** about 1.5–2.8 days of metrics and 16 h – 1.5 days of traces |
|
||||
| Version | `otelcol-contrib` 0.162.0 `windows_amd64` | Same release as the dev collector, so `config.yaml` is validated by the same binary. The tarball SHA-256 `b371829181e8…9e5c3d` matches the release's `.sha256` asset (this release publishes per-asset `.sha256` files and no combined `checksums.txt`) |
|
||||
| Service shape | NSSM (a copy of the box's `nssm.exe` in `bin\`), Automatic, restart on exit after 5 s, stdout/stderr → `logs\collector.log` rotated at 10 MB | Same shape as the other apps. `NO_WINDOWS_SERVICE=1` keeps otelcol in console mode under NSSM |
|
||||
| Account | Virtual account `NT SERVICE\ZbOtelCollector` | Least privilege that works: it binds unprivileged loopback ports and writes only `data\` + `logs\`. LocalSystem was not needed |
|
||||
| Install location | `E:\ApiInstall\OtelCollector\{bin,config,data,logs}` | Next to the apps. The root DACL is protected, `BUILTIN\Users` removed, SYSTEM + Administrators full, service account RX / Modify on `data\` + `logs\` |
|
||||
| Worker deny (lmxopcua#891) | vd03: the inherited `E:\ApiInstall` deny, kept as an explicit copy when inheritance was cut. QA: the isolation installer re-run with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` (11 → 12) | The collector holds no secrets: its config has no credentials and the receivers have no auth. Telemetry files can carry URL paths and host names, which is why `Users` was removed and the deny was added anyway. The procedure asks that every new `E:\ApiInstall` folder join the list |
|
||||
|
||||
## Per-app change
|
||||
|
||||
Two entries were added to each service's environment: `<Prefix>Telemetry__Exporter=Otlp` and
|
||||
`<Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317`. The key names were read from each **deployed** commit's source:
|
||||
|
||||
| App (deployed build) | Service | Env store | Prefix | Entries vd03 | Entries vq03 | Entries vq04 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| HistorianGateway (vd03 `ea9f21c`, QA `fa9cf31`) | `HistorianGateway` | NSSM `AppEnvironmentExtra` | *(root `Telemetry:`)* | 7 → 9 | 32 → 34 | 32 → 34 |
|
||||
| OtOpcUa (`fea7de0a`) | `OtOpcUaHost` | service-key `Environment` | `OtOpcUa__` | 26 → 28 | 46 → 48 | 45 → 47 |
|
||||
| ScadaBridge Central (vd03 `424db155`, QA as deployed) | `ScadaBridge-Central` | service-key `Environment` | `ScadaBridge__` | 10 → 12 | 13 → 15 | 13 → 15 |
|
||||
| ScadaBridge Site | `ScadaBridge-Site` | service-key `Environment` | `ScadaBridge__` | 4 → 6 | 6 → 8 | 6 → 8 |
|
||||
| MxAccessGateway (vd03 `76a71bc8`, QA `4f931e4`) | `MxAccessGw` | NSSM `AppEnvironmentExtra` | `MxGateway__` | 7 → 9 | 31 → 33 | 31 → 33 |
|
||||
| Overview | `ZbOverview` | — | — | not changed | not changed | not changed |
|
||||
|
||||
- **Overview cannot be switched by config.** Its `AddZbTelemetry` call sets `ServiceName` and `Meters` only and binds no
|
||||
exporter key, so it stays Prometheus-only. This needs a code change (Telemetry 0.3.0 `ApplyConfiguration`), which is out of scope here.
|
||||
- Every pre-existing entry was hash-identical on readback. Peppers were compared by SHA-256 prefix only, never printed:
|
||||
mxgw vd03 `0d01abcc1fe5` (env block `3f283e9a1d57` before, matching the #229 record), QA `88188ab11b50`; OtOpcUa vd03
|
||||
`a26449ba4e13`, QA `d61fda284aa4`; HistorianGateway vd03 `8f8410157077`, QA `f779032d1cac`; ScadaBridge inbound vd03
|
||||
`2fc0e4846d17`, QA `c358a1ef70c4`. All were unchanged after the edit.
|
||||
- The values were written straight to the registry as REG_MULTI_SZ. `nssm set`, `install-service.bat` and the apps'
|
||||
upgrade scripts were not used, so the vd03 mxgw env-only pepper stayed in place.
|
||||
- These keys are the same in Telemetry 0.3.0 (`<App>:Telemetry` section, scadaproj#55), so they carry over. **OTLP logs will not
|
||||
flow until the 0.3.0 builds are deployed**; `logs.jsonl` is empty until then.
|
||||
|
||||
## Timeline (EST)
|
||||
|
||||
| Time | Box | Step |
|
||||
|---|---|---|
|
||||
| 21:30 | all | Read-only preflight. Collector ports 4317/4318/8888/8889/13133 free on all three boxes, no existing service or folder |
|
||||
| 21:38–21:39 | vd03 | Collector installed and healthy (process started in about 15 s) |
|
||||
| 21:39:33–21:40:43 | vd03 | HG (5 s stop→ready), OtOpcUa (8 s), SB Central (13 s), SB Site (9 s), mxgw (5 s), one at a time |
|
||||
| 21:44–21:46 | vq04, vq03 | Collectors installed and healthy |
|
||||
| 21:46–21:47 | vq04, vq03 | Isolation installer `-WhatIf`, apply (2 `changed`: `DenyPathRecord` + `SensitiveDirDeny E:\ApiInstall\OtelCollector`), re-apply all `unchanged`; OtOpcUaHost env block hash unchanged |
|
||||
| 21:48:06–21:49:05 | vq04 | HG (5 s), mxgw (6 s), OtOpcUa **Secondary** (stop 32 s, 10 s to ready) |
|
||||
| 21:49:17–21:50:20 | vq03 | HG (5 s), mxgw (6 s), SB Central **standby** (6 s), SB Site **standby** (3 s), OtOpcUa **Primary** (failover to vq04) |
|
||||
| 21:50–21:53:38 | — | Waited for the QA site's MxGateway tags to recover (see below) before touching the active ScadaBridge nodes |
|
||||
| 21:53:59–21:55:03 | vq04 | SB Central **active** (6 s, failover to vq03), SB Site **active** (stop 41 s, failover to vq03) |
|
||||
| 21:56:05 | — | QA site back to 2196/2196 Good on vq03 |
|
||||
|
||||
The QA pairs were restarted standby-first, one node at a time, as the 2026-10-22 ScadaBridge QA record does for a
|
||||
config-only change and as the OtOpcUa QA record's Secondary restart + failover drill shows for `fea7de0a`. No pair was
|
||||
ever fully down.
|
||||
|
||||
## Verification
|
||||
|
||||
### Apps (from the Mac)
|
||||
|
||||
- `/health/ready` **200** for OtOpcUa `:9000`, SB Central `:8085`, SB Site `:8084`, mxgw `:5130`, HG `:5220` and Overview
|
||||
`:5320` on all three boxes.
|
||||
- `/metrics` **200** on every app port (checked on the box): Prometheus is still served, since OTLP is additive.
|
||||
- **OtOpcUa ServiceLevel (opc.tcp i=2267):** vd03 **250**. QA before was vq03 250 / vq04 240. After: **vq04 250 (Primary,
|
||||
`/health/active` 200) / vq03 240** (503). Roles swapped by the rolling restart, with no failback, as recorded for `fea7de0a`.
|
||||
- **ScadaBridge active nodes:** QA Central and Site are now **active on vq03** (were vq04). Both pairs report 2 nodes online.
|
||||
- **ScadaBridge → MxGateway (site health report):**
|
||||
- vd03: `Connected`, 810/810 resolved, good 398 / bad 412, **identical to the pre-change report** (the Bad tags are
|
||||
pre-existing idle receivers).
|
||||
- QA: 2196/2196 Good before and after. Two dips: (1) the vq03 gateway restart at 21:49:34 took the site (then Primary on
|
||||
vq04) to 0 resolved until **21:53:38, about 4 min**; (2) the site failover at 21:55 recovered in about 1 min (21:56:05). The
|
||||
first dip is longer than the ~1.5 min seen on 2026-09-10 and is worth knowing for future QA gateway restarts. It recovered unaided.
|
||||
- **mxgw:** exactly one gateway process per box after the restart.
|
||||
- **Overview:** QA **12/12 Up** on both vq03 and vq04 dashboards, DEV **6/6 Up**.
|
||||
|
||||
### Collector (on the box, `Test-OtelCollector.ps1`)
|
||||
|
||||
vd03, about 3 min after the last restart:
|
||||
|
||||
| Signal | What arrived (`service.name`, all `host.name=WONDER-APP-VD03`, `deployment.environment.name=DEV`) |
|
||||
|---|---|
|
||||
| Traces | `historian-gateway` (historian.Connect/Probe, HistoryService/*), `mxgateway` (OpenSession/Invoke), `otopcua` (`otopcua.opcua.address_space_rebuild`, HistorianStatus/Probe), `scadabridge` Central (`site.id=central node.role=Central`: CentralControlService heartbeat/report/reconcile, `/management`), `scadabridge` Site (`site.id=site-a node.role=Site`: MxAccessGateway Invoke/OpenSession client spans) |
|
||||
| Metrics | Prometheus `:8889` series per job: `ZB.MOM.WW/historian-gateway` 289, `mxgateway` 376, `otopcua` 304, `scadabridge` 680 |
|
||||
| Collector counters | accepted 1372 metric points / 3488 spans, **refused 0**; file + prometheus exporters sent all of it |
|
||||
| Logs | none (expected until Telemetry 0.3.0, scadaproj#55) |
|
||||
|
||||
**ScadaBridge Central over OTLP is proven on vd03**: Central-role spans and metrics arrived with `node.role=Central`. That is
|
||||
ScadaBridge#133's question. This record does not close that issue; it is left to its owner.
|
||||
|
||||
QA (22:07–22:08, files pulled over SMB and summarised on the Mac, plus the on-box check):
|
||||
|
||||
| Box | Every process present with metrics + traces (`deployment.environment.name=QA`) | `:8889` series per job | Accepted / refused |
|
||||
|---|---|---|---|
|
||||
| vq03 | `historian-gateway`, `mxgateway`, `otopcua`, `scadabridge` Central (`site.id=central`), `scadabridge` Site (`site.id=site-a`: MxAccessGateway Invoke 2216, LocalDbSync spans) | HG 306, mxgw 644, OtOpcUa 396, SB 771 | 9098 points, 15942 spans / **0** |
|
||||
| vq04 | the same five; OtOpcUa sends `otopcua.script.*` (runner on) | HG 248, mxgw 235, OtOpcUa 472, SB 438 | 7081 points, 4941 spans / **0** |
|
||||
|
||||
Collector working set 115–126 MB. `Test-OtelCollector.ps1`'s per-line `ConvertFrom-Json` is too slow on PS 5.1 for QA-sized
|
||||
export lines (it did not finish in 10 min), so its default `-Tail` is now 20. Run it with `-Tail 1` for the quick
|
||||
service/listener/Prometheus/counter check, and pull `data\*.jsonl` over SMB for the content check.
|
||||
|
||||
## Disk
|
||||
|
||||
E: free after install: vd03 179 GB, vq03 215 GB, vq04 230 GB. The collector binary is about 370 MB.
|
||||
|
||||
Observed write rate in the first 20–30 minutes:
|
||||
|
||||
| Box | metrics.jsonl | traces.jsonl | → metrics retained | → traces retained |
|
||||
|---|---|---|---|---|
|
||||
| vd03 (28 min) | 4.9 MB (~250 MB/day) | 8.0 MB (~410 MB/day) | ~2.4 days | ~1.5 days |
|
||||
| vq03 (21 min) | 3.1 MB (~210 MB/day) | 12.6 MB (~860 MB/day) | ~2.8 days | ~16 h |
|
||||
| vq04 (22 min) | 2.4 MB (~160 MB/day) | 3.5 MB (~230 MB/day) | ~3.7 days | ~2.6 days |
|
||||
|
||||
The 600 MB per-signal cap therefore holds about 16 h to 3.7 days, not 7 days. Disk stays bounded at ~1.8 GB per box. The trace
|
||||
volume is mostly the ScadaBridge Site's MxAccessGateway `Invoke` client spans (the active site) and HistorianGateway's
|
||||
four `Get*InterfaceVersion` handshake spans per historian connect. Raise `max_backups` if a longer window is wanted; there is room.
|
||||
|
||||
## Backups and rollback
|
||||
|
||||
- Registry exports per service, taken before each edit: `E:\psrem-stage\otel-54\backup-<COMPUTER>\<Service>-pre-otlp-<ts>.reg`,
|
||||
plus `<Service>-env-names-pre-<ts>.txt` (names only). The folder ACL is protected, SYSTEM + Administrators only.
|
||||
vq04 has two stamps, one for step 1 (HG/mxgw/OtOpcUa) and one for step 3 (ScadaBridge).
|
||||
- **App rollback:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<entry>'` removes the two entries and restarts,
|
||||
or `reg import` the `.reg` and restart. QA Akka pairs go standby-first.
|
||||
- **Collector rollback:** `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall` stops and deletes the
|
||||
service and renames the folder to `OtelCollector.removed.<ts>`. On QA, re-run the isolation installer without
|
||||
`E:\ApiInstall\OtelCollector` in `-DenyPaths`. With the collector gone, the apps keep running; only OTLP export fails, silently.
|
||||
- Staging `E:\psrem-stage\otel-54\` stays on each box (zip, config, scripts, backups).
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Overview: no exporter config binding (needs Telemetry 0.3.0 `ApplyConfiguration` in its `Program.cs`).
|
||||
- OTLP logs: arrive with the Telemetry 0.3.0 builds (scadaproj#55); no collector change needed.
|
||||
- A trace backend (Tempo/Jaeger) or a central collector, if a cross-box view is wanted.
|
||||
@@ -0,0 +1,19 @@
|
||||
# scadaproj#54, QA only: append E:\ApiInstall\OtelCollector to the recorded OtOpcUaScriptWorker -DenyPaths (lmxopcua#891 procedure).
|
||||
param([switch]$Apply)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$inst = 'E:\psrem-stage\otopcua-890-f75ee7d0\install\Install-ScriptWorkerIsolation.ps1'
|
||||
$recorded = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths)
|
||||
$deny = @($recorded | Where-Object { $_ -ne 'E:\ApiInstall\OtelCollector' }) + 'E:\ApiInstall\OtelCollector'
|
||||
"recorded $($recorded.Count) -> $($deny.Count) paths"
|
||||
function EnvSig { $e = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost' -Name Environment).Environment)
|
||||
$b = [Text.Encoding]::UTF8.GetBytes($e -join "`n"); "$($e.Count) entries, block " + (-join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | % { $_.ToString('x2') })) }
|
||||
"OtOpcUaHost env before: $(EnvSig)"
|
||||
$common = @{ InstallRoot = 'E:\ApiInstall\OtOpcUa\app'; HostServiceAccount = 'LocalSystem'; ConfigureHostService = $true; DenyPaths = $deny }
|
||||
if ($Apply) {
|
||||
& $inst @common 2>&1 | Out-String -Width 300
|
||||
"--- re-apply (expect all unchanged)"
|
||||
& $inst @common 2>&1 | Out-String -Width 300
|
||||
} else { & $inst @common -WhatIf 2>&1 | Out-String -Width 300 }
|
||||
"OtOpcUaHost env after: $(EnvSig)"
|
||||
"recorded now: " + (@((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths) -join '; ')
|
||||
(icacls 'E:\ApiInstall\OtelCollector') | Select-String 'OtOpcUaScriptWorker' | % { "acl $_" }
|
||||
@@ -0,0 +1,116 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54).
|
||||
|
||||
.DESCRIPTION
|
||||
Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\}
|
||||
Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit.
|
||||
ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit
|
||||
ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and
|
||||
Modify on data\ and logs\.
|
||||
Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service
|
||||
wrapper, so otelcol runs in console mode).
|
||||
Every listener in config.yaml binds 127.0.0.1. The script checks that after start.
|
||||
|
||||
.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt.
|
||||
.PARAMETER Tier DEV or QA.
|
||||
.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\.
|
||||
.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed.<ts> (kept).
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$StageDir,
|
||||
[ValidateSet('DEV','QA')] [string]$Tier,
|
||||
[string]$NssmPath,
|
||||
[switch]$Uninstall
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$svcName = 'ZbOtelCollector'
|
||||
$root = 'E:\ApiInstall\OtelCollector'
|
||||
$acct = "NT SERVICE\$svcName"
|
||||
$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
|
||||
if ($Uninstall) {
|
||||
$s = Get-Service $svcName -ErrorAction SilentlyContinue
|
||||
if ($s) {
|
||||
if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) }
|
||||
& sc.exe delete $svcName | Out-Null
|
||||
Say "service $svcName deleted"
|
||||
}
|
||||
if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" }
|
||||
return
|
||||
}
|
||||
|
||||
if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' }
|
||||
if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" }
|
||||
if (Test-Path $root) { throw "$root already exists" }
|
||||
|
||||
# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git).
|
||||
$sums = @{}
|
||||
Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } |
|
||||
ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() }
|
||||
foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') {
|
||||
$h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash
|
||||
if ($h -ne $sums[$f]) { throw "hash mismatch $f" }
|
||||
Say "hash ok $f $($h.Substring(0,12))"
|
||||
}
|
||||
|
||||
# 2. Layout
|
||||
foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null }
|
||||
$tmp = Join-Path $StageDir 'x'
|
||||
if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force }
|
||||
Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp
|
||||
$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash
|
||||
if ($h -ne $exeSha) { throw "exe hash mismatch $h" }
|
||||
Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe')
|
||||
Remove-Item $tmp -Recurse -Force
|
||||
Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe')
|
||||
Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml')
|
||||
Say "layout ok, exe sha $($exeSha.Substring(0,12))"
|
||||
|
||||
# 3. Service
|
||||
$nssm = Join-Path $root 'bin\nssm.exe'
|
||||
& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null
|
||||
$set = @(
|
||||
@('AppParameters', "--config=$root\config\config.yaml"),
|
||||
@('AppDirectory', $root),
|
||||
@('DisplayName', 'ZB OpenTelemetry Collector'),
|
||||
@('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'),
|
||||
@('Start', 'SERVICE_AUTO_START'),
|
||||
@('AppStdout', "$root\logs\collector.log"),
|
||||
@('AppStderr', "$root\logs\collector.log"),
|
||||
@('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'),
|
||||
@('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000'))
|
||||
foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } }
|
||||
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1')
|
||||
& sc.exe config $svcName obj= $acct | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" }
|
||||
Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)"
|
||||
|
||||
# 4. ACL
|
||||
& icacls $root /inheritance:d | Out-Null
|
||||
& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null
|
||||
& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null
|
||||
foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null }
|
||||
(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() }
|
||||
|
||||
# 5. Start + checks
|
||||
Start-Service $svcName
|
||||
$ok = $false
|
||||
for ($i = 0; $i -lt 30; $i++) {
|
||||
Start-Sleep 2
|
||||
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {}
|
||||
}
|
||||
Say "service $((Get-Service $svcName).Status); health_check ok=$ok"
|
||||
if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' }
|
||||
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId
|
||||
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||
$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId
|
||||
$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||
$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' }
|
||||
if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" }
|
||||
$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces'
|
||||
Say "OTLP/HTTP empty POST -> $($r.StatusCode)"
|
||||
$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false
|
||||
Say "collector.log error lines: $(@($err).Count)"
|
||||
Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) }
|
||||
@@ -0,0 +1,89 @@
|
||||
# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
|
||||
|
||||
One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 /
|
||||
vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
|
||||
`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged.
|
||||
|
||||
Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md).
|
||||
|
||||
## What is installed
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. |
|
||||
| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` |
|
||||
| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
|
||||
| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` |
|
||||
| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM |
|
||||
| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` |
|
||||
|
||||
## Endpoints (all 127.0.0.1, none reachable off-box)
|
||||
|
||||
| Port | What |
|
||||
|---|---|
|
||||
| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** |
|
||||
| 4318 | OTLP HTTP receiver |
|
||||
| 8889 | Prometheus exporter: everything received, labelled `job=<namespace>/<service>`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` |
|
||||
| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) |
|
||||
| 13133 | `health_check` |
|
||||
|
||||
**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and
|
||||
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
|
||||
a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter
|
||||
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
|
||||
|
||||
## Backend and retention
|
||||
|
||||
- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export
|
||||
request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at
|
||||
about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and
|
||||
231 GB (vq04).
|
||||
- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
|
||||
- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files.
|
||||
At `info` level the collector writes a few lines at start and then only errors, so this stays small.
|
||||
- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do
|
||||
not export logs. When 0.3.0 ships, logs start flowing with no collector change.
|
||||
- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline.
|
||||
|
||||
## App settings (service environment)
|
||||
|
||||
| App | Service | Where the env lives | Entries added |
|
||||
|---|---|---|---|
|
||||
| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` |
|
||||
| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
|
||||
| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` |
|
||||
| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) |
|
||||
| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
|
||||
|
||||
These key names are unchanged in Telemetry 0.3.0 (`<App>:Telemetry` section), so the entries carry over to that upgrade.
|
||||
Prometheus `/metrics` on every app stays served (OTLP is additive).
|
||||
|
||||
## Scripts
|
||||
|
||||
All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only).
|
||||
|
||||
| Script | Does |
|
||||
|---|---|
|
||||
| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.<ts>` |
|
||||
| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts |
|
||||
| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` |
|
||||
|
||||
```powershell
|
||||
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
|
||||
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
|
||||
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
|
||||
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
|
||||
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
|
||||
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
|
||||
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
|
||||
```
|
||||
|
||||
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
|
||||
|
||||
## Rollback
|
||||
|
||||
- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'` removes the two entries and
|
||||
restarts. Alternatively, `reg import <dir>\<Service>-pre-otlp-<ts>.reg` and restart. On QA, restart Akka-pair nodes standby-first.
|
||||
- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
|
||||
run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation
|
||||
installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`.
|
||||
@@ -0,0 +1,108 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Adds the OTLP exporter settings to app services' environments and restarts them one at a time (scadaproj#54).
|
||||
|
||||
.DESCRIPTION
|
||||
For each -Apps entry, in order:
|
||||
1. reads the service environment (service-key 'Environment' for the .NET-hosted services, NSSM
|
||||
Parameters\AppEnvironmentExtra for the NSSM ones) and exports the service key (.reg) to -BackupDir;
|
||||
2. writes <Prefix>Telemetry__Exporter=Otlp and <Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317
|
||||
(replacing any existing entries of those two names), as REG_MULTI_SZ, never via nssm/install scripts;
|
||||
3. reads it back: every pre-existing entry must be hash-identical and the count must be +2 (or +0 when the
|
||||
entries were already present);
|
||||
4. stops the service (waits for Stopped and for its process to exit), starts it, waits for /health/ready 200.
|
||||
If 3 or 4 fails, the original environment (held in memory) is written back, the service is restarted and the
|
||||
script stops with exit 1. Only entry NAMES and SHA-256 prefixes are printed, never values.
|
||||
|
||||
-Rollback writes back the environment from the .reg backups instead (removes the two entries) and restarts.
|
||||
|
||||
.PARAMETER Apps Entries 'Service|Kind|Prefix|Port', Kind = Env or Nssm, Prefix e.g. 'OtOpcUa__' or '' (HistorianGateway).
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory)] [string[]]$Apps,
|
||||
[Parameter(Mandatory)] [string]$BackupDir,
|
||||
[string]$Endpoint = 'http://127.0.0.1:4317',
|
||||
[switch]$Rollback
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
function Sha([string]$s) { $b = [Text.Encoding]::UTF8.GetBytes($s); -join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | ForEach-Object { $_.ToString('x2') }) }
|
||||
|
||||
if (-not (Test-Path $BackupDir)) {
|
||||
New-Item -ItemType Directory -Path $BackupDir | Out-Null
|
||||
$acl = New-Object Security.AccessControl.DirectorySecurity
|
||||
$acl.SetAccessRuleProtection($true, $false)
|
||||
foreach ($id in 'NT AUTHORITY\SYSTEM','BUILTIN\Administrators') {
|
||||
$acl.AddAccessRule((New-Object Security.AccessControl.FileSystemAccessRule($id, 'FullControl', 'ContainerInherit,ObjectInherit', 'None', 'Allow')))
|
||||
}
|
||||
Set-Acl $BackupDir $acl
|
||||
}
|
||||
|
||||
function Get-EnvLoc($svc, $kind) {
|
||||
if ($kind -eq 'Nssm') { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc\Parameters"; Name = 'AppEnvironmentExtra' } }
|
||||
else { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"; Name = 'Environment' } }
|
||||
}
|
||||
function Read-Env($loc) { @((Get-ItemProperty $loc.Path -Name $loc.Name -ErrorAction SilentlyContinue).($loc.Name) | Where-Object { $_ }) }
|
||||
function Write-Env($loc, [string[]]$lines) { Set-ItemProperty $loc.Path -Name $loc.Name -Type MultiString -Value $lines }
|
||||
|
||||
function Test-Ready($port) {
|
||||
foreach ($h in 'localhost', '127.0.0.1', $env:COMPUTERNAME) {
|
||||
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 "http://${h}:$port/health/ready"; if ($r.StatusCode -eq 200) { return $true } } catch {}
|
||||
}
|
||||
$false
|
||||
}
|
||||
function Restart-App($svc, $port) {
|
||||
$t0 = Get-Date
|
||||
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
|
||||
Stop-Service $svc -NoWait -Force
|
||||
$deadline = (Get-Date).AddSeconds(180)
|
||||
while ((Get-Service $svc).Status -ne 'Stopped') { if ((Get-Date) -gt $deadline) { throw "$svc did not stop" }; Start-Sleep 1 }
|
||||
while ($procId -and (Get-Process -Id $procId -ErrorAction SilentlyContinue)) { if ((Get-Date) -gt $deadline) { throw "$svc pid $procId still alive" }; Start-Sleep 1 }
|
||||
$tStop = Get-Date
|
||||
Start-Service $svc
|
||||
$deadline = (Get-Date).AddSeconds(240)
|
||||
while (-not (Test-Ready $port)) { if ((Get-Date) -gt $deadline) { throw "$svc not ready on :$port" }; Start-Sleep 2 }
|
||||
$newPid = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
|
||||
Say ("{0} restarted: stop {1:N0} s, stop->ready {2:N0} s, pid {3} -> {4}" -f $svc, ($tStop - $t0).TotalSeconds, ((Get-Date) - $tStop).TotalSeconds, $procId, $newPid)
|
||||
}
|
||||
|
||||
$stamp = Get-Date -Format yyyyMMddTHHmmss
|
||||
foreach ($entry in $Apps) {
|
||||
$svc, $kind, $prefix, $port = $entry -split '\|'
|
||||
$loc = Get-EnvLoc $svc $kind
|
||||
$kExp = "${prefix}Telemetry__Exporter"; $kEnd = "${prefix}Telemetry__OtlpEndpoint"
|
||||
$old = Read-Env $loc
|
||||
if ($old.Count -eq 0) { throw "$svc has no environment at $($loc.Path)\$($loc.Name)" }
|
||||
$oldHashes = $old | ForEach-Object { Sha $_ }
|
||||
$pep = $old | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
|
||||
Say "$svc env before: $($old.Count) entries, block $(Sha ($old -join "`n")), pepper [$($pep -join ',')]"
|
||||
|
||||
if ($Rollback) {
|
||||
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
|
||||
} else {
|
||||
$reg = Join-Path $BackupDir "$svc-pre-otlp-$stamp.reg"
|
||||
& reg.exe export ($loc.Path -replace '^HKLM:\\','HKLM\' -replace '\\Parameters$','') $reg /y | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "reg export failed for $svc" }
|
||||
Set-Content (Join-Path $BackupDir "$svc-env-names-pre-$stamp.txt") ($old | ForEach-Object { ($_ -split '=',2)[0] })
|
||||
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + @("$kExp=Otlp", "$kEnd=$Endpoint")
|
||||
}
|
||||
$kept = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
|
||||
try {
|
||||
Write-Env $loc $new
|
||||
$back = Read-Env $loc
|
||||
$backHashes = @($back | ForEach-Object { Sha $_ })
|
||||
foreach ($k in $kept) { if ($backHashes -notcontains (Sha $k)) { throw "$svc readback lost entry $(($k -split '=',2)[0])" } }
|
||||
if ($back.Count -ne $new.Count) { throw "$svc readback count $($back.Count) != $($new.Count)" }
|
||||
$pep2 = $back | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
|
||||
if (($pep -join ',') -ne ($pep2 -join ',')) { throw "$svc pepper changed" }
|
||||
$added = $back | Where-Object { $_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=") }
|
||||
Say "$svc env after: $($back.Count) entries ($($kept.Count) pre-existing hash-identical), pepper [$($pep2 -join ',')], telemetry: $($added -join ' ')"
|
||||
Restart-App $svc $port
|
||||
} catch {
|
||||
Say "FAIL $svc : $($_.Exception.Message) -- restoring the original environment"
|
||||
Write-Env $loc $old
|
||||
try { Restart-App $svc $port } catch { Say "ROLLBACK RESTART FAILED $svc : $($_.Exception.Message)" }
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
Say 'all done'
|
||||
@@ -0,0 +1,65 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent.
|
||||
.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window.
|
||||
.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP).
|
||||
#>
|
||||
param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20)
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$root = 'E:\ApiInstall\OtelCollector'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'"
|
||||
Say "service $($s.State) $($s.StartMode) $($s.StartName)"
|
||||
try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" }
|
||||
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||
if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||
" collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) }
|
||||
Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime }
|
||||
" E: free GB " + [int]((Get-PSDrive E).Free/1GB)
|
||||
|
||||
$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000
|
||||
function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue }
|
||||
foreach ($sig in 'metrics','traces','logs') {
|
||||
$f = "$root\data\$sig.jsonl"
|
||||
if (-not (Test-Path $f)) { Say "${sig}: no file"; continue }
|
||||
$agg = @{}
|
||||
foreach ($line in (Get-Content $f -Tail $Tail)) {
|
||||
try { $o = $line | ConvertFrom-Json } catch { continue }
|
||||
$rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs }
|
||||
foreach ($r in $rs) {
|
||||
$key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role')
|
||||
if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } }
|
||||
$a = $agg[$key]
|
||||
if ($sig -eq 'metrics') {
|
||||
foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1
|
||||
foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } }
|
||||
} elseif ($sig -eq 'traces') {
|
||||
foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } }
|
||||
} else {
|
||||
foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } }
|
||||
}
|
||||
}
|
||||
}
|
||||
Say "$sig (last $Tail export requests):"
|
||||
foreach ($k in ($agg.Keys | Sort-Object)) {
|
||||
$a = $agg[$k]
|
||||
$newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' }
|
||||
$fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' }
|
||||
" $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh"
|
||||
if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') }
|
||||
}
|
||||
}
|
||||
try {
|
||||
$p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n"
|
||||
$jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name
|
||||
Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" }
|
||||
} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" }
|
||||
try {
|
||||
$q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n"
|
||||
Say "collector self-metrics:"
|
||||
$q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') }
|
||||
} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" }
|
||||
foreach ($port in $AppPorts) {
|
||||
$c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} }
|
||||
" app :$port /metrics $c"
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54).
|
||||
# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA),
|
||||
# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector.
|
||||
#
|
||||
# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317,
|
||||
# and nothing is reachable from the network. That is why the receivers have no auth and no TLS.
|
||||
# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md).
|
||||
#
|
||||
# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as
|
||||
# deployment.environment.name, so files copied off two tiers can still be told apart.
|
||||
#
|
||||
# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older
|
||||
# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB.
|
||||
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 127.0.0.1:13133
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 127.0.0.1:4317
|
||||
http:
|
||||
endpoint: 127.0.0.1:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_mib: 256
|
||||
spike_limit_mib: 64
|
||||
resource/tier:
|
||||
attributes:
|
||||
- key: deployment.environment.name
|
||||
value: ${env:ZB_TIER}
|
||||
action: upsert
|
||||
batch: {}
|
||||
|
||||
exporters:
|
||||
file/metrics:
|
||||
path: E:/ApiInstall/OtelCollector/data/metrics.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
file/traces:
|
||||
path: E:/ApiInstall/OtelCollector/data/traces.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
file/logs:
|
||||
path: E:/ApiInstall/OtelCollector/data/logs.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
prometheus:
|
||||
endpoint: 127.0.0.1:8889
|
||||
resource_constant_labels:
|
||||
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name]
|
||||
metric_expiration: 15m
|
||||
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
logs:
|
||||
level: info
|
||||
metrics:
|
||||
level: basic
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 127.0.0.1
|
||||
port: 8888
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/metrics, prometheus]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/traces]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/logs]
|
||||
Reference in New Issue
Block a user