From 71b85957429c6df78ea931a67e5bde339775cf01 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 22:10:15 -0500 Subject: [PATCH] feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54) ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- CLAUDE.md | 2 +- .../2026-11-12-otlp-end-to-end-evidence.md | 4 +- components/observability/GAPS.md | 10 +- deployment.md | 12 ++ infra/otel-collector/README.md | 5 +- .../2026-11-12-wonder-otel-collector.md | 147 ++++++++++++++++++ .../wonder/Add-OtelDenyPath.ps1 | 19 +++ .../wonder/Install-OtelCollector.ps1 | 116 ++++++++++++++ infra/otel-collector/wonder/README.md | 89 +++++++++++ .../wonder/Switch-AppsToOtlp.ps1 | 108 +++++++++++++ .../wonder/Test-OtelCollector.ps1 | 65 ++++++++ infra/otel-collector/wonder/config.yaml | 80 ++++++++++ 12 files changed, 649 insertions(+), 8 deletions(-) create mode 100644 infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md create mode 100644 infra/otel-collector/wonder/Add-OtelDenyPath.ps1 create mode 100644 infra/otel-collector/wonder/Install-OtelCollector.ps1 create mode 100644 infra/otel-collector/wonder/README.md create mode 100644 infra/otel-collector/wonder/Switch-AppsToOtlp.ps1 create mode 100644 infra/otel-collector/wonder/Test-OtelCollector.ps1 create mode 100644 infra/otel-collector/wonder/config.yaml diff --git a/CLAUDE.md b/CLAUDE.md index d5caac7..8a3fe27 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | -| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). Open: wonder-tier collector #54. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | +| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). **Wonder tiers on OTLP since 2026-11-12 (#54, closed):** one loopback-only `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, `NT SERVICE\ZbOtelCollector`, `E:\ApiInstall\OtelCollector`, file + Prometheus `127.0.0.1:8889` exporters, ≤ ~1.8 GB per box) on vd03/vq03/vq04. OtOpcUa, ScadaBridge Central + Site, mxgw and HG export to `http://127.0.0.1:4317` via service env (Overview has no exporter binding). The deployed builds send metrics + traces; logs follow when 0.3.0 is deployed. Record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/). Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | diff --git a/components/observability/2026-11-12-otlp-end-to-end-evidence.md b/components/observability/2026-11-12-otlp-end-to-end-evidence.md index ba2c28c..a58e741 100644 --- a/components/observability/2026-11-12-otlp-end-to-end-evidence.md +++ b/components/observability/2026-11-12-otlp-end-to-end-evidence.md @@ -163,7 +163,7 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew `AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates (`logs.jsonl` stayed empty). - **ScadaBridge#133:** OTLP from the Central role is not yet proven. -- **scadaproj#54:** collector on the wonder tiers. +- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md). ## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below) @@ -211,7 +211,7 @@ scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack) dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no - wonder tier has a collector yet (#54), so no deployed box can trip it. + wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed. - **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an `ExportLogs=false` case with a metrics positive control, and a Prometheus case. diff --git a/components/observability/GAPS.md b/components/observability/GAPS.md index 6f49012..d2d00f1 100644 --- a/components/observability/GAPS.md +++ b/components/observability/GAPS.md @@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p > listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the > "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously > contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there -> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain. +> (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12. > > **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector` > on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived @@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change - ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against `zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central → ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). -- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still - Prometheus-only. → #54 +- ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a + per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1, + file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and + HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview** + has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds + are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md). - ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry `0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP (OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the diff --git a/deployment.md b/deployment.md index 2a72238..b7034c9 100644 --- a/deployment.md +++ b/deployment.md @@ -401,6 +401,8 @@ nodes, verified Healthy at setup): | HistorianGateway | Up | Up (since 2026-08-14) | | Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) | +> **Active node as of 2026-11-12 22:00 EST** (after the scadaproj#54 rolling restart, standby-first): OtOpcUa **vq04** Primary (250) / vq03 240; ScadaBridge central **and** site **active on vq03**. The table above is the 2026-08-13 capture. + **Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge `2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window, rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because @@ -488,6 +490,16 @@ Neither install is a scripted in-repo flow — the operational procedures live i `inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in those runbooks, or not probed in §7.1, as unverified. +**OpenTelemetry collector on every wonder box (2026-11-12, scadaproj#54):** service `ZbOtelCollector` +(otelcol-contrib 0.162.0, NSSM, virtual account `NT SERVICE\ZbOtelCollector`) at `E:\ApiInstall\OtelCollector\` on +vd03, vq03 and vq04. All of its ports are **loopback only**: OTLP `127.0.0.1:4317`/`4318`, Prometheus exporter `:8889`, +self-metrics `:8888`, health `:13133`, so nothing new is reachable off-box. Data is in `data\{metrics,traces,logs}.jsonl` +(rotated, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway have +`__Telemetry__Exporter=Otlp` + `__OtlpEndpoint=http://127.0.0.1:4317` in their service environment (HG: root `Telemetry__*`); +Overview is unchanged. On QA, `E:\ApiInstall\OtelCollector` is the 12th `-DenyPaths` entry of the OtOpcUa worker isolation. +The rolling restart left **OtOpcUa Primary on vq04** and **ScadaBridge Central + Site active on vq03**. Env backups: +`E:\psrem-stage\otel-54\backup-\`. Runbook + record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/). + **Auth posture differs between the two, and QA is the correct one:** | | VD03 (DEV) | VQ03/VQ04 (QA) | diff --git a/infra/otel-collector/README.md b/infra/otel-collector/README.md index 5e44e0e..6fe2b92 100644 --- a/infra/otel-collector/README.md +++ b/infra/otel-collector/README.md @@ -6,8 +6,9 @@ shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-sha OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md). -> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA -> vq03/vq04) has a collector yet — that is scadaproj#54. +> Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA +> vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54): +> [`wonder/`](wonder/). ## Endpoints diff --git a/infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md b/infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md new file mode 100644 index 0000000..f2cb46d --- /dev/null +++ b/infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md @@ -0,0 +1,147 @@ +# 2026-11-12: OTel collector on the wonder tiers, apps switched to OTLP (scadaproj#54) + +**Status: EXECUTED 2026-11-12, 21:38–21:56 EST.** DEV wonder-app-vd03 and QA wonder-app-vq03 + vq04 each run the +`ZbOtelCollector` service. Every app that has an exporter setting now exports OTLP to the collector on its own box. +No binary, appsettings file or Telemetry pin was changed. The change consists of a new service plus two service-environment entries per app. +Runbook and scripts: [`README.md`](README.md). + +Channel: the `winrm` skill, used only for this deployment's on-box steps (preflight, install, env edits, restarts, +verification), with the user's request of 2026-11-12 ("resolve the remaining otel issues"). Files were staged over SMB to +`E:\psrem-stage\otel-54\`. Health, ScadaBridge site health, OPC UA ServiceLevel and Overview were checked from the Mac +over HTTP / opc.tcp. + +## Design (owner-approved defaults) + +| Decision | Choice | Why / trade-off | +|---|---|---| +| Topology | One collector **per box**, apps → `http://127.0.0.1:4317` | No new network path and nothing new reachable on the corporate network. On QA each node keeps its own copy, so the two pair nodes' data sits in two places. A tier-wide view needs a backend that the collectors export *to* | +| Receiver security | Loopback-only bind on all five ports, no auth or TLS | Only local processes can send. Opening the receivers to the network would require auth + TLS first | +| Backend | `file` exporter (OTLP-JSON, rotated) + `prometheus` exporter on `127.0.0.1:8889` | Durable local record plus a scrapeable view, with nothing new to operate. **No Tempo or Jaeger yet.** That is the future option: add an `otlp` exporter to the traces pipeline | +| Retention | Each signal file rotates at 100 MB, keeps 5 backups, max 7 days → ≤ ~600 MB per signal, ≤ ~1.8 GB total | E: had 180 / 216 / 231 GB free (vd03 / vq03 / vq04), so the cap is under 1 %. **At the observed rate (§ Disk) the size cap binds, not the 7 days:** about 1.5–2.8 days of metrics and 16 h – 1.5 days of traces | +| Version | `otelcol-contrib` 0.162.0 `windows_amd64` | Same release as the dev collector, so `config.yaml` is validated by the same binary. The tarball SHA-256 `b371829181e8…9e5c3d` matches the release's `.sha256` asset (this release publishes per-asset `.sha256` files and no combined `checksums.txt`) | +| Service shape | NSSM (a copy of the box's `nssm.exe` in `bin\`), Automatic, restart on exit after 5 s, stdout/stderr → `logs\collector.log` rotated at 10 MB | Same shape as the other apps. `NO_WINDOWS_SERVICE=1` keeps otelcol in console mode under NSSM | +| Account | Virtual account `NT SERVICE\ZbOtelCollector` | Least privilege that works: it binds unprivileged loopback ports and writes only `data\` + `logs\`. LocalSystem was not needed | +| Install location | `E:\ApiInstall\OtelCollector\{bin,config,data,logs}` | Next to the apps. The root DACL is protected, `BUILTIN\Users` removed, SYSTEM + Administrators full, service account RX / Modify on `data\` + `logs\` | +| Worker deny (lmxopcua#891) | vd03: the inherited `E:\ApiInstall` deny, kept as an explicit copy when inheritance was cut. QA: the isolation installer re-run with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` (11 → 12) | The collector holds no secrets: its config has no credentials and the receivers have no auth. Telemetry files can carry URL paths and host names, which is why `Users` was removed and the deny was added anyway. The procedure asks that every new `E:\ApiInstall` folder join the list | + +## Per-app change + +Two entries were added to each service's environment: `Telemetry__Exporter=Otlp` and +`Telemetry__OtlpEndpoint=http://127.0.0.1:4317`. The key names were read from each **deployed** commit's source: + +| App (deployed build) | Service | Env store | Prefix | Entries vd03 | Entries vq03 | Entries vq04 | +|---|---|---|---|---|---|---| +| HistorianGateway (vd03 `ea9f21c`, QA `fa9cf31`) | `HistorianGateway` | NSSM `AppEnvironmentExtra` | *(root `Telemetry:`)* | 7 → 9 | 32 → 34 | 32 → 34 | +| OtOpcUa (`fea7de0a`) | `OtOpcUaHost` | service-key `Environment` | `OtOpcUa__` | 26 → 28 | 46 → 48 | 45 → 47 | +| ScadaBridge Central (vd03 `424db155`, QA as deployed) | `ScadaBridge-Central` | service-key `Environment` | `ScadaBridge__` | 10 → 12 | 13 → 15 | 13 → 15 | +| ScadaBridge Site | `ScadaBridge-Site` | service-key `Environment` | `ScadaBridge__` | 4 → 6 | 6 → 8 | 6 → 8 | +| MxAccessGateway (vd03 `76a71bc8`, QA `4f931e4`) | `MxAccessGw` | NSSM `AppEnvironmentExtra` | `MxGateway__` | 7 → 9 | 31 → 33 | 31 → 33 | +| Overview | `ZbOverview` | — | — | not changed | not changed | not changed | + +- **Overview cannot be switched by config.** Its `AddZbTelemetry` call sets `ServiceName` and `Meters` only and binds no + exporter key, so it stays Prometheus-only. This needs a code change (Telemetry 0.3.0 `ApplyConfiguration`), which is out of scope here. +- Every pre-existing entry was hash-identical on readback. Peppers were compared by SHA-256 prefix only, never printed: + mxgw vd03 `0d01abcc1fe5` (env block `3f283e9a1d57` before, matching the #229 record), QA `88188ab11b50`; OtOpcUa vd03 + `a26449ba4e13`, QA `d61fda284aa4`; HistorianGateway vd03 `8f8410157077`, QA `f779032d1cac`; ScadaBridge inbound vd03 + `2fc0e4846d17`, QA `c358a1ef70c4`. All were unchanged after the edit. +- The values were written straight to the registry as REG_MULTI_SZ. `nssm set`, `install-service.bat` and the apps' + upgrade scripts were not used, so the vd03 mxgw env-only pepper stayed in place. +- These keys are the same in Telemetry 0.3.0 (`:Telemetry` section, scadaproj#55), so they carry over. **OTLP logs will not + flow until the 0.3.0 builds are deployed**; `logs.jsonl` is empty until then. + +## Timeline (EST) + +| Time | Box | Step | +|---|---|---| +| 21:30 | all | Read-only preflight. Collector ports 4317/4318/8888/8889/13133 free on all three boxes, no existing service or folder | +| 21:38–21:39 | vd03 | Collector installed and healthy (process started in about 15 s) | +| 21:39:33–21:40:43 | vd03 | HG (5 s stop→ready), OtOpcUa (8 s), SB Central (13 s), SB Site (9 s), mxgw (5 s), one at a time | +| 21:44–21:46 | vq04, vq03 | Collectors installed and healthy | +| 21:46–21:47 | vq04, vq03 | Isolation installer `-WhatIf`, apply (2 `changed`: `DenyPathRecord` + `SensitiveDirDeny E:\ApiInstall\OtelCollector`), re-apply all `unchanged`; OtOpcUaHost env block hash unchanged | +| 21:48:06–21:49:05 | vq04 | HG (5 s), mxgw (6 s), OtOpcUa **Secondary** (stop 32 s, 10 s to ready) | +| 21:49:17–21:50:20 | vq03 | HG (5 s), mxgw (6 s), SB Central **standby** (6 s), SB Site **standby** (3 s), OtOpcUa **Primary** (failover to vq04) | +| 21:50–21:53:38 | — | Waited for the QA site's MxGateway tags to recover (see below) before touching the active ScadaBridge nodes | +| 21:53:59–21:55:03 | vq04 | SB Central **active** (6 s, failover to vq03), SB Site **active** (stop 41 s, failover to vq03) | +| 21:56:05 | — | QA site back to 2196/2196 Good on vq03 | + +The QA pairs were restarted standby-first, one node at a time, as the 2026-10-22 ScadaBridge QA record does for a +config-only change and as the OtOpcUa QA record's Secondary restart + failover drill shows for `fea7de0a`. No pair was +ever fully down. + +## Verification + +### Apps (from the Mac) + +- `/health/ready` **200** for OtOpcUa `:9000`, SB Central `:8085`, SB Site `:8084`, mxgw `:5130`, HG `:5220` and Overview + `:5320` on all three boxes. +- `/metrics` **200** on every app port (checked on the box): Prometheus is still served, since OTLP is additive. +- **OtOpcUa ServiceLevel (opc.tcp i=2267):** vd03 **250**. QA before was vq03 250 / vq04 240. After: **vq04 250 (Primary, + `/health/active` 200) / vq03 240** (503). Roles swapped by the rolling restart, with no failback, as recorded for `fea7de0a`. +- **ScadaBridge active nodes:** QA Central and Site are now **active on vq03** (were vq04). Both pairs report 2 nodes online. +- **ScadaBridge → MxGateway (site health report):** + - vd03: `Connected`, 810/810 resolved, good 398 / bad 412, **identical to the pre-change report** (the Bad tags are + pre-existing idle receivers). + - QA: 2196/2196 Good before and after. Two dips: (1) the vq03 gateway restart at 21:49:34 took the site (then Primary on + vq04) to 0 resolved until **21:53:38, about 4 min**; (2) the site failover at 21:55 recovered in about 1 min (21:56:05). The + first dip is longer than the ~1.5 min seen on 2026-09-10 and is worth knowing for future QA gateway restarts. It recovered unaided. +- **mxgw:** exactly one gateway process per box after the restart. +- **Overview:** QA **12/12 Up** on both vq03 and vq04 dashboards, DEV **6/6 Up**. + +### Collector (on the box, `Test-OtelCollector.ps1`) + +vd03, about 3 min after the last restart: + +| Signal | What arrived (`service.name`, all `host.name=WONDER-APP-VD03`, `deployment.environment.name=DEV`) | +|---|---| +| Traces | `historian-gateway` (historian.Connect/Probe, HistoryService/*), `mxgateway` (OpenSession/Invoke), `otopcua` (`otopcua.opcua.address_space_rebuild`, HistorianStatus/Probe), `scadabridge` Central (`site.id=central node.role=Central`: CentralControlService heartbeat/report/reconcile, `/management`), `scadabridge` Site (`site.id=site-a node.role=Site`: MxAccessGateway Invoke/OpenSession client spans) | +| Metrics | Prometheus `:8889` series per job: `ZB.MOM.WW/historian-gateway` 289, `mxgateway` 376, `otopcua` 304, `scadabridge` 680 | +| Collector counters | accepted 1372 metric points / 3488 spans, **refused 0**; file + prometheus exporters sent all of it | +| Logs | none (expected until Telemetry 0.3.0, scadaproj#55) | + +**ScadaBridge Central over OTLP is proven on vd03**: Central-role spans and metrics arrived with `node.role=Central`. That is +ScadaBridge#133's question. This record does not close that issue; it is left to its owner. + +QA (22:07–22:08, files pulled over SMB and summarised on the Mac, plus the on-box check): + +| Box | Every process present with metrics + traces (`deployment.environment.name=QA`) | `:8889` series per job | Accepted / refused | +|---|---|---|---| +| vq03 | `historian-gateway`, `mxgateway`, `otopcua`, `scadabridge` Central (`site.id=central`), `scadabridge` Site (`site.id=site-a`: MxAccessGateway Invoke 2216, LocalDbSync spans) | HG 306, mxgw 644, OtOpcUa 396, SB 771 | 9098 points, 15942 spans / **0** | +| vq04 | the same five; OtOpcUa sends `otopcua.script.*` (runner on) | HG 248, mxgw 235, OtOpcUa 472, SB 438 | 7081 points, 4941 spans / **0** | + +Collector working set 115–126 MB. `Test-OtelCollector.ps1`'s per-line `ConvertFrom-Json` is too slow on PS 5.1 for QA-sized +export lines (it did not finish in 10 min), so its default `-Tail` is now 20. Run it with `-Tail 1` for the quick +service/listener/Prometheus/counter check, and pull `data\*.jsonl` over SMB for the content check. + +## Disk + +E: free after install: vd03 179 GB, vq03 215 GB, vq04 230 GB. The collector binary is about 370 MB. + +Observed write rate in the first 20–30 minutes: + +| Box | metrics.jsonl | traces.jsonl | → metrics retained | → traces retained | +|---|---|---|---|---| +| vd03 (28 min) | 4.9 MB (~250 MB/day) | 8.0 MB (~410 MB/day) | ~2.4 days | ~1.5 days | +| vq03 (21 min) | 3.1 MB (~210 MB/day) | 12.6 MB (~860 MB/day) | ~2.8 days | ~16 h | +| vq04 (22 min) | 2.4 MB (~160 MB/day) | 3.5 MB (~230 MB/day) | ~3.7 days | ~2.6 days | + +The 600 MB per-signal cap therefore holds about 16 h to 3.7 days, not 7 days. Disk stays bounded at ~1.8 GB per box. The trace +volume is mostly the ScadaBridge Site's MxAccessGateway `Invoke` client spans (the active site) and HistorianGateway's +four `Get*InterfaceVersion` handshake spans per historian connect. Raise `max_backups` if a longer window is wanted; there is room. + +## Backups and rollback + +- Registry exports per service, taken before each edit: `E:\psrem-stage\otel-54\backup-\-pre-otlp-.reg`, + plus `-env-names-pre-.txt` (names only). The folder ACL is protected, SYSTEM + Administrators only. + vq04 has two stamps, one for step 1 (HG/mxgw/OtOpcUa) and one for step 3 (ScadaBridge). +- **App rollback:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir -Apps ''` removes the two entries and restarts, + or `reg import` the `.reg` and restart. QA Akka pairs go standby-first. +- **Collector rollback:** `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall` stops and deletes the + service and renames the folder to `OtelCollector.removed.`. On QA, re-run the isolation installer without + `E:\ApiInstall\OtelCollector` in `-DenyPaths`. With the collector gone, the apps keep running; only OTLP export fails, silently. +- Staging `E:\psrem-stage\otel-54\` stays on each box (zip, config, scripts, backups). + +## Follow-ups + +- Overview: no exporter config binding (needs Telemetry 0.3.0 `ApplyConfiguration` in its `Program.cs`). +- OTLP logs: arrive with the Telemetry 0.3.0 builds (scadaproj#55); no collector change needed. +- A trace backend (Tempo/Jaeger) or a central collector, if a cross-box view is wanted. diff --git a/infra/otel-collector/wonder/Add-OtelDenyPath.ps1 b/infra/otel-collector/wonder/Add-OtelDenyPath.ps1 new file mode 100644 index 0000000..5ffc5e3 --- /dev/null +++ b/infra/otel-collector/wonder/Add-OtelDenyPath.ps1 @@ -0,0 +1,19 @@ +# scadaproj#54, QA only: append E:\ApiInstall\OtelCollector to the recorded OtOpcUaScriptWorker -DenyPaths (lmxopcua#891 procedure). +param([switch]$Apply) +$ErrorActionPreference = 'Stop' +$inst = 'E:\psrem-stage\otopcua-890-f75ee7d0\install\Install-ScriptWorkerIsolation.ps1' +$recorded = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths) +$deny = @($recorded | Where-Object { $_ -ne 'E:\ApiInstall\OtelCollector' }) + 'E:\ApiInstall\OtelCollector' +"recorded $($recorded.Count) -> $($deny.Count) paths" +function EnvSig { $e = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost' -Name Environment).Environment) + $b = [Text.Encoding]::UTF8.GetBytes($e -join "`n"); "$($e.Count) entries, block " + (-join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | % { $_.ToString('x2') })) } +"OtOpcUaHost env before: $(EnvSig)" +$common = @{ InstallRoot = 'E:\ApiInstall\OtOpcUa\app'; HostServiceAccount = 'LocalSystem'; ConfigureHostService = $true; DenyPaths = $deny } +if ($Apply) { + & $inst @common 2>&1 | Out-String -Width 300 + "--- re-apply (expect all unchanged)" + & $inst @common 2>&1 | Out-String -Width 300 +} else { & $inst @common -WhatIf 2>&1 | Out-String -Width 300 } +"OtOpcUaHost env after: $(EnvSig)" +"recorded now: " + (@((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths) -join '; ') +(icacls 'E:\ApiInstall\OtelCollector') | Select-String 'OtOpcUaScriptWorker' | % { "acl $_" } diff --git a/infra/otel-collector/wonder/Install-OtelCollector.ps1 b/infra/otel-collector/wonder/Install-OtelCollector.ps1 new file mode 100644 index 0000000..4ad1b99 --- /dev/null +++ b/infra/otel-collector/wonder/Install-OtelCollector.ps1 @@ -0,0 +1,116 @@ +<# +.SYNOPSIS + Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54). + +.DESCRIPTION + Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\} + Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit. + ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit + ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and + Modify on data\ and logs\. + Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service + wrapper, so otelcol runs in console mode). + Every listener in config.yaml binds 127.0.0.1. The script checks that after start. + +.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt. +.PARAMETER Tier DEV or QA. +.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\. +.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed. (kept). +#> +param( + [Parameter(Mandatory)] [string]$StageDir, + [ValidateSet('DEV','QA')] [string]$Tier, + [string]$NssmPath, + [switch]$Uninstall +) +$ErrorActionPreference = 'Stop' +$svcName = 'ZbOtelCollector' +$root = 'E:\ApiInstall\OtelCollector' +$acct = "NT SERVICE\$svcName" +$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } + +if ($Uninstall) { + $s = Get-Service $svcName -ErrorAction SilentlyContinue + if ($s) { + if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) } + & sc.exe delete $svcName | Out-Null + Say "service $svcName deleted" + } + if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" } + return +} + +if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' } +if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" } +if (Test-Path $root) { throw "$root already exists" } + +# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git). +$sums = @{} +Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } | + ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() } +foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') { + $h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash + if ($h -ne $sums[$f]) { throw "hash mismatch $f" } + Say "hash ok $f $($h.Substring(0,12))" +} + +# 2. Layout +foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null } +$tmp = Join-Path $StageDir 'x' +if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force } +Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp +$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash +if ($h -ne $exeSha) { throw "exe hash mismatch $h" } +Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe') +Remove-Item $tmp -Recurse -Force +Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe') +Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml') +Say "layout ok, exe sha $($exeSha.Substring(0,12))" + +# 3. Service +$nssm = Join-Path $root 'bin\nssm.exe' +& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null +$set = @( + @('AppParameters', "--config=$root\config\config.yaml"), + @('AppDirectory', $root), + @('DisplayName', 'ZB OpenTelemetry Collector'), + @('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'), + @('Start', 'SERVICE_AUTO_START'), + @('AppStdout', "$root\logs\collector.log"), + @('AppStderr', "$root\logs\collector.log"), + @('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'), + @('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000')) +foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } } +Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1') +& sc.exe config $svcName obj= $acct | Out-Null +if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" } +Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)" + +# 4. ACL +& icacls $root /inheritance:d | Out-Null +& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null +& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null +foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null } +(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() } + +# 5. Start + checks +Start-Service $svcName +$ok = $false +for ($i = 0; $i -lt 30; $i++) { + Start-Sleep 2 + try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {} +} +Say "service $((Get-Service $svcName).Status); health_check ok=$ok" +if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' } +$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId +$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe' +$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId +$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" } +$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' } +if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" } +$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces' +Say "OTLP/HTTP empty POST -> $($r.StatusCode)" +$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false +Say "collector.log error lines: $(@($err).Count)" +Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) } diff --git a/infra/otel-collector/wonder/README.md b/infra/otel-collector/wonder/README.md new file mode 100644 index 0000000..13deb38 --- /dev/null +++ b/infra/otel-collector/wonder/README.md @@ -0,0 +1,89 @@ +# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54) + +One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 / +vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on +`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged. + +Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md). + +## What is installed + +| Item | Value | +|---|---| +| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. | +| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` | +| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s | +| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` | +| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM | +| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` | + +## Endpoints (all 127.0.0.1, none reachable off-box) + +| Port | What | +|---|---| +| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** | +| 4318 | OTLP HTTP receiver | +| 8889 | Prometheus exporter: everything received, labelled `job=/`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` | +| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) | +| 13133 | `health_check` | + +**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and +no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac, +a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter +on these collectors (OTLP out with TLS and auth) rather than opening the receivers. + +## Backend and retention + +- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export + request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at + about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and + 231 GB (vq04). +- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update. +- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files. + At `info` level the collector writes a few lines at start and then only errors, so this stays small. +- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do + not export logs. When 0.3.0 ships, logs start flowing with no collector change. +- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline. + +## App settings (service environment) + +| App | Service | Where the env lives | Entries added | +|---|---|---|---| +| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` | +| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` | +| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` | +| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) | +| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change | + +These key names are unchanged in Telemetry 0.3.0 (`:Telemetry` section), so the entries carry over to that upgrade. +Prometheus `/metrics` on every app stays served (OTLP is additive). + +## Scripts + +All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only). + +| Script | Does | +|---|---| +| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.` | +| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts | +| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` | + +```powershell +# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe) +& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe +# switch apps (Service|Kind|Prefix|Port), one at a time in this order +& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup- -Apps ` + 'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', ` + 'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130' +& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content +``` + +On QA, order the Akka pairs standby-first across the two boxes (see the rollout record). + +## Rollback + +- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir -Apps ''` removes the two entries and + restarts. Alternatively, `reg import \-pre-otlp-.reg` and restart. On QA, restart Akka-pair nodes standby-first. +- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then + run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation + installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`. diff --git a/infra/otel-collector/wonder/Switch-AppsToOtlp.ps1 b/infra/otel-collector/wonder/Switch-AppsToOtlp.ps1 new file mode 100644 index 0000000..f25b332 --- /dev/null +++ b/infra/otel-collector/wonder/Switch-AppsToOtlp.ps1 @@ -0,0 +1,108 @@ +<# +.SYNOPSIS + Adds the OTLP exporter settings to app services' environments and restarts them one at a time (scadaproj#54). + +.DESCRIPTION + For each -Apps entry, in order: + 1. reads the service environment (service-key 'Environment' for the .NET-hosted services, NSSM + Parameters\AppEnvironmentExtra for the NSSM ones) and exports the service key (.reg) to -BackupDir; + 2. writes Telemetry__Exporter=Otlp and Telemetry__OtlpEndpoint=http://127.0.0.1:4317 + (replacing any existing entries of those two names), as REG_MULTI_SZ, never via nssm/install scripts; + 3. reads it back: every pre-existing entry must be hash-identical and the count must be +2 (or +0 when the + entries were already present); + 4. stops the service (waits for Stopped and for its process to exit), starts it, waits for /health/ready 200. + If 3 or 4 fails, the original environment (held in memory) is written back, the service is restarted and the + script stops with exit 1. Only entry NAMES and SHA-256 prefixes are printed, never values. + + -Rollback writes back the environment from the .reg backups instead (removes the two entries) and restarts. + +.PARAMETER Apps Entries 'Service|Kind|Prefix|Port', Kind = Env or Nssm, Prefix e.g. 'OtOpcUa__' or '' (HistorianGateway). +#> +param( + [Parameter(Mandatory)] [string[]]$Apps, + [Parameter(Mandatory)] [string]$BackupDir, + [string]$Endpoint = 'http://127.0.0.1:4317', + [switch]$Rollback +) +$ErrorActionPreference = 'Stop' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } +function Sha([string]$s) { $b = [Text.Encoding]::UTF8.GetBytes($s); -join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | ForEach-Object { $_.ToString('x2') }) } + +if (-not (Test-Path $BackupDir)) { + New-Item -ItemType Directory -Path $BackupDir | Out-Null + $acl = New-Object Security.AccessControl.DirectorySecurity + $acl.SetAccessRuleProtection($true, $false) + foreach ($id in 'NT AUTHORITY\SYSTEM','BUILTIN\Administrators') { + $acl.AddAccessRule((New-Object Security.AccessControl.FileSystemAccessRule($id, 'FullControl', 'ContainerInherit,ObjectInherit', 'None', 'Allow'))) + } + Set-Acl $BackupDir $acl +} + +function Get-EnvLoc($svc, $kind) { + if ($kind -eq 'Nssm') { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc\Parameters"; Name = 'AppEnvironmentExtra' } } + else { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"; Name = 'Environment' } } +} +function Read-Env($loc) { @((Get-ItemProperty $loc.Path -Name $loc.Name -ErrorAction SilentlyContinue).($loc.Name) | Where-Object { $_ }) } +function Write-Env($loc, [string[]]$lines) { Set-ItemProperty $loc.Path -Name $loc.Name -Type MultiString -Value $lines } + +function Test-Ready($port) { + foreach ($h in 'localhost', '127.0.0.1', $env:COMPUTERNAME) { + try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 "http://${h}:$port/health/ready"; if ($r.StatusCode -eq 200) { return $true } } catch {} + } + $false +} +function Restart-App($svc, $port) { + $t0 = Get-Date + $procId = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId + Stop-Service $svc -NoWait -Force + $deadline = (Get-Date).AddSeconds(180) + while ((Get-Service $svc).Status -ne 'Stopped') { if ((Get-Date) -gt $deadline) { throw "$svc did not stop" }; Start-Sleep 1 } + while ($procId -and (Get-Process -Id $procId -ErrorAction SilentlyContinue)) { if ((Get-Date) -gt $deadline) { throw "$svc pid $procId still alive" }; Start-Sleep 1 } + $tStop = Get-Date + Start-Service $svc + $deadline = (Get-Date).AddSeconds(240) + while (-not (Test-Ready $port)) { if ((Get-Date) -gt $deadline) { throw "$svc not ready on :$port" }; Start-Sleep 2 } + $newPid = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId + Say ("{0} restarted: stop {1:N0} s, stop->ready {2:N0} s, pid {3} -> {4}" -f $svc, ($tStop - $t0).TotalSeconds, ((Get-Date) - $tStop).TotalSeconds, $procId, $newPid) +} + +$stamp = Get-Date -Format yyyyMMddTHHmmss +foreach ($entry in $Apps) { + $svc, $kind, $prefix, $port = $entry -split '\|' + $loc = Get-EnvLoc $svc $kind + $kExp = "${prefix}Telemetry__Exporter"; $kEnd = "${prefix}Telemetry__OtlpEndpoint" + $old = Read-Env $loc + if ($old.Count -eq 0) { throw "$svc has no environment at $($loc.Path)\$($loc.Name)" } + $oldHashes = $old | ForEach-Object { Sha $_ } + $pep = $old | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) } + Say "$svc env before: $($old.Count) entries, block $(Sha ($old -join "`n")), pepper [$($pep -join ',')]" + + if ($Rollback) { + $new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + } else { + $reg = Join-Path $BackupDir "$svc-pre-otlp-$stamp.reg" + & reg.exe export ($loc.Path -replace '^HKLM:\\','HKLM\' -replace '\\Parameters$','') $reg /y | Out-Null + if ($LASTEXITCODE -ne 0) { throw "reg export failed for $svc" } + Set-Content (Join-Path $BackupDir "$svc-env-names-pre-$stamp.txt") ($old | ForEach-Object { ($_ -split '=',2)[0] }) + $new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + @("$kExp=Otlp", "$kEnd=$Endpoint") + } + $kept = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + try { + Write-Env $loc $new + $back = Read-Env $loc + $backHashes = @($back | ForEach-Object { Sha $_ }) + foreach ($k in $kept) { if ($backHashes -notcontains (Sha $k)) { throw "$svc readback lost entry $(($k -split '=',2)[0])" } } + if ($back.Count -ne $new.Count) { throw "$svc readback count $($back.Count) != $($new.Count)" } + $pep2 = $back | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) } + if (($pep -join ',') -ne ($pep2 -join ',')) { throw "$svc pepper changed" } + $added = $back | Where-Object { $_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=") } + Say "$svc env after: $($back.Count) entries ($($kept.Count) pre-existing hash-identical), pepper [$($pep2 -join ',')], telemetry: $($added -join ' ')" + Restart-App $svc $port + } catch { + Say "FAIL $svc : $($_.Exception.Message) -- restoring the original environment" + Write-Env $loc $old + try { Restart-App $svc $port } catch { Say "ROLLBACK RESTART FAILED $svc : $($_.Exception.Message)" } + exit 1 + } +} +Say 'all done' diff --git a/infra/otel-collector/wonder/Test-OtelCollector.ps1 b/infra/otel-collector/wonder/Test-OtelCollector.ps1 new file mode 100644 index 0000000..913fc6b --- /dev/null +++ b/infra/otel-collector/wonder/Test-OtelCollector.ps1 @@ -0,0 +1,65 @@ +<# +.SYNOPSIS + Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent. +.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window. +.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP). +#> +param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20) +$ErrorActionPreference = 'Continue' +$root = 'E:\ApiInstall\OtelCollector' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } +$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'" +Say "service $($s.State) $($s.StartMode) $($s.StartName)" +try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" } +$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe' +if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" } + " collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) } +Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime } +" E: free GB " + [int]((Get-PSDrive E).Free/1GB) + +$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000 +function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue } +foreach ($sig in 'metrics','traces','logs') { + $f = "$root\data\$sig.jsonl" + if (-not (Test-Path $f)) { Say "${sig}: no file"; continue } + $agg = @{} + foreach ($line in (Get-Content $f -Tail $Tail)) { + try { $o = $line | ConvertFrom-Json } catch { continue } + $rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs } + foreach ($r in $rs) { + $key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role') + if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } } + $a = $agg[$key] + if ($sig -eq 'metrics') { + foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1 + foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } } + } elseif ($sig -eq 'traces') { + foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } } + } else { + foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } } + } + } + } + Say "$sig (last $Tail export requests):" + foreach ($k in ($agg.Keys | Sort-Object)) { + $a = $agg[$k] + $newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' } + $fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' } + " $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh" + if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') } + } +} +try { + $p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n" + $jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name + Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" } +} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" } +try { + $q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n" + Say "collector self-metrics:" + $q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') } +} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" } +foreach ($port in $AppPorts) { + $c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} } + " app :$port /metrics $c" +} diff --git a/infra/otel-collector/wonder/config.yaml b/infra/otel-collector/wonder/config.yaml new file mode 100644 index 0000000..3382867 --- /dev/null +++ b/infra/otel-collector/wonder/config.yaml @@ -0,0 +1,80 @@ +# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54). +# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA), +# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector. +# +# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317, +# and nothing is reachable from the network. That is why the receivers have no auth and no TLS. +# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md). +# +# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as +# deployment.environment.name, so files copied off two tiers can still be told apart. +# +# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older +# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB. + +extensions: + health_check: + endpoint: 127.0.0.1:13133 + +receivers: + otlp: + protocols: + grpc: + endpoint: 127.0.0.1:4317 + http: + endpoint: 127.0.0.1:4318 + +processors: + memory_limiter: + check_interval: 5s + limit_mib: 256 + spike_limit_mib: 64 + resource/tier: + attributes: + - key: deployment.environment.name + value: ${env:ZB_TIER} + action: upsert + batch: {} + +exporters: + file/metrics: + path: E:/ApiInstall/OtelCollector/data/metrics.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + file/traces: + path: E:/ApiInstall/OtelCollector/data/traces.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + file/logs: + path: E:/ApiInstall/OtelCollector/data/logs.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + prometheus: + endpoint: 127.0.0.1:8889 + resource_constant_labels: + included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name] + metric_expiration: 15m + +service: + extensions: [health_check] + telemetry: + logs: + level: info + metrics: + level: basic + readers: + - pull: + exporter: + prometheus: + host: 127.0.0.1 + port: 8888 + pipelines: + metrics: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/metrics, prometheus] + traces: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/traces] + logs: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/logs]