feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)

ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:10:15 -05:00
parent 85654d7b43
commit 71b8595742
12 changed files with 649 additions and 8 deletions
+1 -1
View File
@@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See
| Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
| UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
| Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | | Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). Open: wonder-tier collector #54. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). **Wonder tiers on OTLP since 2026-11-12 (#54, closed):** one loopback-only `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, `NT SERVICE\ZbOtelCollector`, `E:\ApiInstall\OtelCollector`, file + Prometheus `127.0.0.1:8889` exporters, ≤ ~1.8 GB per box) on vd03/vq03/vq04. OtOpcUa, ScadaBridge Central + Site, mxgw and HG export to `http://127.0.0.1:4317` via service env (Overview has no exporter binding). The deployed builds send metrics + traces; logs follow when 0.3.0 is deployed. Record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/). Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
@@ -163,7 +163,7 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates `AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
(`logs.jsonl` stayed empty). (`logs.jsonl` stayed empty).
- **ScadaBridge#133:** OTLP from the Central role is not yet proven. - **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers. - **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below) ## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
@@ -211,7 +211,7 @@ scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack)
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
wonder tier has a collector yet (#54), so no deployed box can trip it. wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on - **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
`ExportLogs=false` case with a metrics positive control, and a Prometheus case. `ExportLogs=false` case with a metrics positive control, and a Prometheus case.
+7 -3
View File
@@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the > listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously > "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there > contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain. > (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12.
> >
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector` > **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived > on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
@@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against - ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central → `zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still - ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a
Prometheus-only. → #54 per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1,
file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview**
has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds
are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry - ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP `0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the (OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the
+12
View File
@@ -401,6 +401,8 @@ nodes, verified Healthy at setup):
| HistorianGateway | Up | Up (since 2026-08-14) | | HistorianGateway | Up | Up (since 2026-08-14) |
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) | | Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) |
> **Active node as of 2026-11-12 22:00 EST** (after the scadaproj#54 rolling restart, standby-first): OtOpcUa **vq04** Primary (250) / vq03 240; ScadaBridge central **and** site **active on vq03**. The table above is the 2026-08-13 capture.
**Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge **Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge
`2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window, `2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window,
rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because
@@ -488,6 +490,16 @@ Neither install is a scripted in-repo flow — the operational procedures live i
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in `inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
those runbooks, or not probed in §7.1, as unverified. those runbooks, or not probed in §7.1, as unverified.
**OpenTelemetry collector on every wonder box (2026-11-12, scadaproj#54):** service `ZbOtelCollector`
(otelcol-contrib 0.162.0, NSSM, virtual account `NT SERVICE\ZbOtelCollector`) at `E:\ApiInstall\OtelCollector\` on
vd03, vq03 and vq04. All of its ports are **loopback only**: OTLP `127.0.0.1:4317`/`4318`, Prometheus exporter `:8889`,
self-metrics `:8888`, health `:13133`, so nothing new is reachable off-box. Data is in `data\{metrics,traces,logs}.jsonl`
(rotated, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway have
`<App>__Telemetry__Exporter=Otlp` + `__OtlpEndpoint=http://127.0.0.1:4317` in their service environment (HG: root `Telemetry__*`);
Overview is unchanged. On QA, `E:\ApiInstall\OtelCollector` is the 12th `-DenyPaths` entry of the OtOpcUa worker isolation.
The rolling restart left **OtOpcUa Primary on vq04** and **ScadaBridge Central + Site active on vq03**. Env backups:
`E:\psrem-stage\otel-54\backup-<COMPUTER>\`. Runbook + record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/).
**Auth posture differs between the two, and QA is the correct one:** **Auth posture differs between the two, and QA is the correct one:**
| | VD03 (DEV) | VQ03/VQ04 (QA) | | | VD03 (DEV) | VQ03/VQ04 (QA) |
+3 -2
View File
@@ -6,8 +6,9 @@ shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-sha
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md). [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA > Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA
> vq03/vq04) has a collector yet — that is scadaproj#54. > vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54):
> [`wonder/`](wonder/).
## Endpoints ## Endpoints
@@ -0,0 +1,147 @@
# 2026-11-12: OTel collector on the wonder tiers, apps switched to OTLP (scadaproj#54)
**Status: EXECUTED 2026-11-12, 21:38–21:56 EST.** DEV wonder-app-vd03 and QA wonder-app-vq03 + vq04 each run the
`ZbOtelCollector` service. Every app that has an exporter setting now exports OTLP to the collector on its own box.
No binary, appsettings file or Telemetry pin was changed. The change consists of a new service plus two service-environment entries per app.
Runbook and scripts: [`README.md`](README.md).
Channel: the `winrm` skill, used only for this deployment's on-box steps (preflight, install, env edits, restarts,
verification), with the user's request of 2026-11-12 ("resolve the remaining otel issues"). Files were staged over SMB to
`E:\psrem-stage\otel-54\`. Health, ScadaBridge site health, OPC UA ServiceLevel and Overview were checked from the Mac
over HTTP / opc.tcp.
## Design (owner-approved defaults)
| Decision | Choice | Why / trade-off |
|---|---|---|
| Topology | One collector **per box**, apps → `http://127.0.0.1:4317` | No new network path and nothing new reachable on the corporate network. On QA each node keeps its own copy, so the two pair nodes' data sits in two places. A tier-wide view needs a backend that the collectors export *to* |
| Receiver security | Loopback-only bind on all five ports, no auth or TLS | Only local processes can send. Opening the receivers to the network would require auth + TLS first |
| Backend | `file` exporter (OTLP-JSON, rotated) + `prometheus` exporter on `127.0.0.1:8889` | Durable local record plus a scrapeable view, with nothing new to operate. **No Tempo or Jaeger yet.** That is the future option: add an `otlp` exporter to the traces pipeline |
| Retention | Each signal file rotates at 100 MB, keeps 5 backups, max 7 days → ≤ ~600 MB per signal, ≤ ~1.8 GB total | E: had 180 / 216 / 231 GB free (vd03 / vq03 / vq04), so the cap is under 1 %. **At the observed rate (§ Disk) the size cap binds, not the 7 days:** about 1.5–2.8 days of metrics and 16 h – 1.5 days of traces |
| Version | `otelcol-contrib` 0.162.0 `windows_amd64` | Same release as the dev collector, so `config.yaml` is validated by the same binary. The tarball SHA-256 `b371829181e8…9e5c3d` matches the release's `.sha256` asset (this release publishes per-asset `.sha256` files and no combined `checksums.txt`) |
| Service shape | NSSM (a copy of the box's `nssm.exe` in `bin\`), Automatic, restart on exit after 5 s, stdout/stderr → `logs\collector.log` rotated at 10 MB | Same shape as the other apps. `NO_WINDOWS_SERVICE=1` keeps otelcol in console mode under NSSM |
| Account | Virtual account `NT SERVICE\ZbOtelCollector` | Least privilege that works: it binds unprivileged loopback ports and writes only `data\` + `logs\`. LocalSystem was not needed |
| Install location | `E:\ApiInstall\OtelCollector\{bin,config,data,logs}` | Next to the apps. The root DACL is protected, `BUILTIN\Users` removed, SYSTEM + Administrators full, service account RX / Modify on `data\` + `logs\` |
| Worker deny (lmxopcua#891) | vd03: the inherited `E:\ApiInstall` deny, kept as an explicit copy when inheritance was cut. QA: the isolation installer re-run with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` (11 → 12) | The collector holds no secrets: its config has no credentials and the receivers have no auth. Telemetry files can carry URL paths and host names, which is why `Users` was removed and the deny was added anyway. The procedure asks that every new `E:\ApiInstall` folder join the list |
## Per-app change
Two entries were added to each service's environment: `<Prefix>Telemetry__Exporter=Otlp` and
`<Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317`. The key names were read from each **deployed** commit's source:
| App (deployed build) | Service | Env store | Prefix | Entries vd03 | Entries vq03 | Entries vq04 |
|---|---|---|---|---|---|---|
| HistorianGateway (vd03 `ea9f21c`, QA `fa9cf31`) | `HistorianGateway` | NSSM `AppEnvironmentExtra` | *(root `Telemetry:`)* | 7 → 9 | 32 → 34 | 32 → 34 |
| OtOpcUa (`fea7de0a`) | `OtOpcUaHost` | service-key `Environment` | `OtOpcUa__` | 26 → 28 | 46 → 48 | 45 → 47 |
| ScadaBridge Central (vd03 `424db155`, QA as deployed) | `ScadaBridge-Central` | service-key `Environment` | `ScadaBridge__` | 10 → 12 | 13 → 15 | 13 → 15 |
| ScadaBridge Site | `ScadaBridge-Site` | service-key `Environment` | `ScadaBridge__` | 4 → 6 | 6 → 8 | 6 → 8 |
| MxAccessGateway (vd03 `76a71bc8`, QA `4f931e4`) | `MxAccessGw` | NSSM `AppEnvironmentExtra` | `MxGateway__` | 7 → 9 | 31 → 33 | 31 → 33 |
| Overview | `ZbOverview` | — | — | not changed | not changed | not changed |
- **Overview cannot be switched by config.** Its `AddZbTelemetry` call sets `ServiceName` and `Meters` only and binds no
exporter key, so it stays Prometheus-only. This needs a code change (Telemetry 0.3.0 `ApplyConfiguration`), which is out of scope here.
- Every pre-existing entry was hash-identical on readback. Peppers were compared by SHA-256 prefix only, never printed:
mxgw vd03 `0d01abcc1fe5` (env block `3f283e9a1d57` before, matching the #229 record), QA `88188ab11b50`; OtOpcUa vd03
`a26449ba4e13`, QA `d61fda284aa4`; HistorianGateway vd03 `8f8410157077`, QA `f779032d1cac`; ScadaBridge inbound vd03
`2fc0e4846d17`, QA `c358a1ef70c4`. All were unchanged after the edit.
- The values were written straight to the registry as REG_MULTI_SZ. `nssm set`, `install-service.bat` and the apps'
upgrade scripts were not used, so the vd03 mxgw env-only pepper stayed in place.
- These keys are the same in Telemetry 0.3.0 (`<App>:Telemetry` section, scadaproj#55), so they carry over. **OTLP logs will not
flow until the 0.3.0 builds are deployed**; `logs.jsonl` is empty until then.
## Timeline (EST)
| Time | Box | Step |
|---|---|---|
| 21:30 | all | Read-only preflight. Collector ports 4317/4318/8888/8889/13133 free on all three boxes, no existing service or folder |
| 21:38–21:39 | vd03 | Collector installed and healthy (process started in about 15 s) |
| 21:39:33–21:40:43 | vd03 | HG (5 s stop→ready), OtOpcUa (8 s), SB Central (13 s), SB Site (9 s), mxgw (5 s), one at a time |
| 21:44–21:46 | vq04, vq03 | Collectors installed and healthy |
| 21:46–21:47 | vq04, vq03 | Isolation installer `-WhatIf`, apply (2 `changed`: `DenyPathRecord` + `SensitiveDirDeny E:\ApiInstall\OtelCollector`), re-apply all `unchanged`; OtOpcUaHost env block hash unchanged |
| 21:48:06–21:49:05 | vq04 | HG (5 s), mxgw (6 s), OtOpcUa **Secondary** (stop 32 s, 10 s to ready) |
| 21:49:17–21:50:20 | vq03 | HG (5 s), mxgw (6 s), SB Central **standby** (6 s), SB Site **standby** (3 s), OtOpcUa **Primary** (failover to vq04) |
| 21:50–21:53:38 | — | Waited for the QA site's MxGateway tags to recover (see below) before touching the active ScadaBridge nodes |
| 21:53:59–21:55:03 | vq04 | SB Central **active** (6 s, failover to vq03), SB Site **active** (stop 41 s, failover to vq03) |
| 21:56:05 | — | QA site back to 2196/2196 Good on vq03 |
The QA pairs were restarted standby-first, one node at a time, as the 2026-10-22 ScadaBridge QA record does for a
config-only change and as the OtOpcUa QA record's Secondary restart + failover drill shows for `fea7de0a`. No pair was
ever fully down.
## Verification
### Apps (from the Mac)
- `/health/ready` **200** for OtOpcUa `:9000`, SB Central `:8085`, SB Site `:8084`, mxgw `:5130`, HG `:5220` and Overview
`:5320` on all three boxes.
- `/metrics` **200** on every app port (checked on the box): Prometheus is still served, since OTLP is additive.
- **OtOpcUa ServiceLevel (opc.tcp i=2267):** vd03 **250**. QA before was vq03 250 / vq04 240. After: **vq04 250 (Primary,
`/health/active` 200) / vq03 240** (503). Roles swapped by the rolling restart, with no failback, as recorded for `fea7de0a`.
- **ScadaBridge active nodes:** QA Central and Site are now **active on vq03** (were vq04). Both pairs report 2 nodes online.
- **ScadaBridge → MxGateway (site health report):**
- vd03: `Connected`, 810/810 resolved, good 398 / bad 412, **identical to the pre-change report** (the Bad tags are
pre-existing idle receivers).
- QA: 2196/2196 Good before and after. Two dips: (1) the vq03 gateway restart at 21:49:34 took the site (then Primary on
vq04) to 0 resolved until **21:53:38, about 4 min**; (2) the site failover at 21:55 recovered in about 1 min (21:56:05). The
first dip is longer than the ~1.5 min seen on 2026-09-10 and is worth knowing for future QA gateway restarts. It recovered unaided.
- **mxgw:** exactly one gateway process per box after the restart.
- **Overview:** QA **12/12 Up** on both vq03 and vq04 dashboards, DEV **6/6 Up**.
### Collector (on the box, `Test-OtelCollector.ps1`)
vd03, about 3 min after the last restart:
| Signal | What arrived (`service.name`, all `host.name=WONDER-APP-VD03`, `deployment.environment.name=DEV`) |
|---|---|
| Traces | `historian-gateway` (historian.Connect/Probe, HistoryService/*), `mxgateway` (OpenSession/Invoke), `otopcua` (`otopcua.opcua.address_space_rebuild`, HistorianStatus/Probe), `scadabridge` Central (`site.id=central node.role=Central`: CentralControlService heartbeat/report/reconcile, `/management`), `scadabridge` Site (`site.id=site-a node.role=Site`: MxAccessGateway Invoke/OpenSession client spans) |
| Metrics | Prometheus `:8889` series per job: `ZB.MOM.WW/historian-gateway` 289, `mxgateway` 376, `otopcua` 304, `scadabridge` 680 |
| Collector counters | accepted 1372 metric points / 3488 spans, **refused 0**; file + prometheus exporters sent all of it |
| Logs | none (expected until Telemetry 0.3.0, scadaproj#55) |
**ScadaBridge Central over OTLP is proven on vd03**: Central-role spans and metrics arrived with `node.role=Central`. That is
ScadaBridge#133's question. This record does not close that issue; it is left to its owner.
QA (22:07–22:08, files pulled over SMB and summarised on the Mac, plus the on-box check):
| Box | Every process present with metrics + traces (`deployment.environment.name=QA`) | `:8889` series per job | Accepted / refused |
|---|---|---|---|
| vq03 | `historian-gateway`, `mxgateway`, `otopcua`, `scadabridge` Central (`site.id=central`), `scadabridge` Site (`site.id=site-a`: MxAccessGateway Invoke 2216, LocalDbSync spans) | HG 306, mxgw 644, OtOpcUa 396, SB 771 | 9098 points, 15942 spans / **0** |
| vq04 | the same five; OtOpcUa sends `otopcua.script.*` (runner on) | HG 248, mxgw 235, OtOpcUa 472, SB 438 | 7081 points, 4941 spans / **0** |
Collector working set 115–126 MB. `Test-OtelCollector.ps1`'s per-line `ConvertFrom-Json` is too slow on PS 5.1 for QA-sized
export lines (it did not finish in 10 min), so its default `-Tail` is now 20. Run it with `-Tail 1` for the quick
service/listener/Prometheus/counter check, and pull `data\*.jsonl` over SMB for the content check.
## Disk
E: free after install: vd03 179 GB, vq03 215 GB, vq04 230 GB. The collector binary is about 370 MB.
Observed write rate in the first 20–30 minutes:
| Box | metrics.jsonl | traces.jsonl | → metrics retained | → traces retained |
|---|---|---|---|---|
| vd03 (28 min) | 4.9 MB (~250 MB/day) | 8.0 MB (~410 MB/day) | ~2.4 days | ~1.5 days |
| vq03 (21 min) | 3.1 MB (~210 MB/day) | 12.6 MB (~860 MB/day) | ~2.8 days | ~16 h |
| vq04 (22 min) | 2.4 MB (~160 MB/day) | 3.5 MB (~230 MB/day) | ~3.7 days | ~2.6 days |
The 600 MB per-signal cap therefore holds about 16 h to 3.7 days, not 7 days. Disk stays bounded at ~1.8 GB per box. The trace
volume is mostly the ScadaBridge Site's MxAccessGateway `Invoke` client spans (the active site) and HistorianGateway's
four `Get*InterfaceVersion` handshake spans per historian connect. Raise `max_backups` if a longer window is wanted; there is room.
## Backups and rollback
- Registry exports per service, taken before each edit: `E:\psrem-stage\otel-54\backup-<COMPUTER>\<Service>-pre-otlp-<ts>.reg`,
plus `<Service>-env-names-pre-<ts>.txt` (names only). The folder ACL is protected, SYSTEM + Administrators only.
vq04 has two stamps, one for step 1 (HG/mxgw/OtOpcUa) and one for step 3 (ScadaBridge).
- **App rollback:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<entry>'` removes the two entries and restarts,
or `reg import` the `.reg` and restart. QA Akka pairs go standby-first.
- **Collector rollback:** `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall` stops and deletes the
service and renames the folder to `OtelCollector.removed.<ts>`. On QA, re-run the isolation installer without
`E:\ApiInstall\OtelCollector` in `-DenyPaths`. With the collector gone, the apps keep running; only OTLP export fails, silently.
- Staging `E:\psrem-stage\otel-54\` stays on each box (zip, config, scripts, backups).
## Follow-ups
- Overview: no exporter config binding (needs Telemetry 0.3.0 `ApplyConfiguration` in its `Program.cs`).
- OTLP logs: arrive with the Telemetry 0.3.0 builds (scadaproj#55); no collector change needed.
- A trace backend (Tempo/Jaeger) or a central collector, if a cross-box view is wanted.
@@ -0,0 +1,19 @@
# scadaproj#54, QA only: append E:\ApiInstall\OtelCollector to the recorded OtOpcUaScriptWorker -DenyPaths (lmxopcua#891 procedure).
param([switch]$Apply)
$ErrorActionPreference = 'Stop'
$inst = 'E:\psrem-stage\otopcua-890-f75ee7d0\install\Install-ScriptWorkerIsolation.ps1'
$recorded = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths)
$deny = @($recorded | Where-Object { $_ -ne 'E:\ApiInstall\OtelCollector' }) + 'E:\ApiInstall\OtelCollector'
"recorded $($recorded.Count) -> $($deny.Count) paths"
function EnvSig { $e = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost' -Name Environment).Environment)
$b = [Text.Encoding]::UTF8.GetBytes($e -join "`n"); "$($e.Count) entries, block " + (-join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | % { $_.ToString('x2') })) }
"OtOpcUaHost env before: $(EnvSig)"
$common = @{ InstallRoot = 'E:\ApiInstall\OtOpcUa\app'; HostServiceAccount = 'LocalSystem'; ConfigureHostService = $true; DenyPaths = $deny }
if ($Apply) {
& $inst @common 2>&1 | Out-String -Width 300
"--- re-apply (expect all unchanged)"
& $inst @common 2>&1 | Out-String -Width 300
} else { & $inst @common -WhatIf 2>&1 | Out-String -Width 300 }
"OtOpcUaHost env after: $(EnvSig)"
"recorded now: " + (@((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths) -join '; ')
(icacls 'E:\ApiInstall\OtelCollector') | Select-String 'OtOpcUaScriptWorker' | % { "acl $_" }
@@ -0,0 +1,116 @@
<#
.SYNOPSIS
Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54).
.DESCRIPTION
Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\}
Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit.
ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit
ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and
Modify on data\ and logs\.
Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service
wrapper, so otelcol runs in console mode).
Every listener in config.yaml binds 127.0.0.1. The script checks that after start.
.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt.
.PARAMETER Tier DEV or QA.
.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\.
.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed.<ts> (kept).
#>
param(
[Parameter(Mandatory)] [string]$StageDir,
[ValidateSet('DEV','QA')] [string]$Tier,
[string]$NssmPath,
[switch]$Uninstall
)
$ErrorActionPreference = 'Stop'
$svcName = 'ZbOtelCollector'
$root = 'E:\ApiInstall\OtelCollector'
$acct = "NT SERVICE\$svcName"
$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
if ($Uninstall) {
$s = Get-Service $svcName -ErrorAction SilentlyContinue
if ($s) {
if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) }
& sc.exe delete $svcName | Out-Null
Say "service $svcName deleted"
}
if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" }
return
}
if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' }
if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" }
if (Test-Path $root) { throw "$root already exists" }
# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git).
$sums = @{}
Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } |
ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() }
foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') {
$h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash
if ($h -ne $sums[$f]) { throw "hash mismatch $f" }
Say "hash ok $f $($h.Substring(0,12))"
}
# 2. Layout
foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null }
$tmp = Join-Path $StageDir 'x'
if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force }
Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp
$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash
if ($h -ne $exeSha) { throw "exe hash mismatch $h" }
Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe')
Remove-Item $tmp -Recurse -Force
Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe')
Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml')
Say "layout ok, exe sha $($exeSha.Substring(0,12))"
# 3. Service
$nssm = Join-Path $root 'bin\nssm.exe'
& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null
$set = @(
@('AppParameters', "--config=$root\config\config.yaml"),
@('AppDirectory', $root),
@('DisplayName', 'ZB OpenTelemetry Collector'),
@('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'),
@('Start', 'SERVICE_AUTO_START'),
@('AppStdout', "$root\logs\collector.log"),
@('AppStderr', "$root\logs\collector.log"),
@('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'),
@('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000'))
foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } }
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1')
& sc.exe config $svcName obj= $acct | Out-Null
if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" }
Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)"
# 4. ACL
& icacls $root /inheritance:d | Out-Null
& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null
& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null
foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null }
(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() }
# 5. Start + checks
Start-Service $svcName
$ok = $false
for ($i = 0; $i -lt 30; $i++) {
Start-Sleep 2
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {}
}
Say "service $((Get-Service $svcName).Status); health_check ok=$ok"
if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' }
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe'
$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId
$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' }
if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" }
$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces'
Say "OTLP/HTTP empty POST -> $($r.StatusCode)"
$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false
Say "collector.log error lines: $(@($err).Count)"
Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) }
+89
View File
@@ -0,0 +1,89 @@
# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 /
vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged.
Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md).
## What is installed
| Item | Value |
|---|---|
| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. |
| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` |
| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` |
| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM |
| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` |
## Endpoints (all 127.0.0.1, none reachable off-box)
| Port | What |
|---|---|
| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** |
| 4318 | OTLP HTTP receiver |
| 8889 | Prometheus exporter: everything received, labelled `job=<namespace>/<service>`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` |
| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) |
| 13133 | `health_check` |
**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
## Backend and retention
- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export
request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at
about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and
231 GB (vq04).
- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files.
At `info` level the collector writes a few lines at start and then only errors, so this stays small.
- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do
not export logs. When 0.3.0 ships, logs start flowing with no collector change.
- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline.
## App settings (service environment)
| App | Service | Where the env lives | Entries added |
|---|---|---|---|
| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` |
| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` |
| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) |
| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
These key names are unchanged in Telemetry 0.3.0 (`<App>:Telemetry` section), so the entries carry over to that upgrade.
Prometheus `/metrics` on every app stays served (OTLP is additive).
## Scripts
All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only).
| Script | Does |
|---|---|
| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.<ts>` |
| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts |
| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` |
```powershell
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
```
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
## Rollback
- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'` removes the two entries and
restarts. Alternatively, `reg import <dir>\<Service>-pre-otlp-<ts>.reg` and restart. On QA, restart Akka-pair nodes standby-first.
- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation
installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`.
@@ -0,0 +1,108 @@
<#
.SYNOPSIS
Adds the OTLP exporter settings to app services' environments and restarts them one at a time (scadaproj#54).
.DESCRIPTION
For each -Apps entry, in order:
1. reads the service environment (service-key 'Environment' for the .NET-hosted services, NSSM
Parameters\AppEnvironmentExtra for the NSSM ones) and exports the service key (.reg) to -BackupDir;
2. writes <Prefix>Telemetry__Exporter=Otlp and <Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317
(replacing any existing entries of those two names), as REG_MULTI_SZ, never via nssm/install scripts;
3. reads it back: every pre-existing entry must be hash-identical and the count must be +2 (or +0 when the
entries were already present);
4. stops the service (waits for Stopped and for its process to exit), starts it, waits for /health/ready 200.
If 3 or 4 fails, the original environment (held in memory) is written back, the service is restarted and the
script stops with exit 1. Only entry NAMES and SHA-256 prefixes are printed, never values.
-Rollback writes back the environment from the .reg backups instead (removes the two entries) and restarts.
.PARAMETER Apps Entries 'Service|Kind|Prefix|Port', Kind = Env or Nssm, Prefix e.g. 'OtOpcUa__' or '' (HistorianGateway).
#>
param(
[Parameter(Mandatory)] [string[]]$Apps,
[Parameter(Mandatory)] [string]$BackupDir,
[string]$Endpoint = 'http://127.0.0.1:4317',
[switch]$Rollback
)
$ErrorActionPreference = 'Stop'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
function Sha([string]$s) { $b = [Text.Encoding]::UTF8.GetBytes($s); -join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | ForEach-Object { $_.ToString('x2') }) }
if (-not (Test-Path $BackupDir)) {
New-Item -ItemType Directory -Path $BackupDir | Out-Null
$acl = New-Object Security.AccessControl.DirectorySecurity
$acl.SetAccessRuleProtection($true, $false)
foreach ($id in 'NT AUTHORITY\SYSTEM','BUILTIN\Administrators') {
$acl.AddAccessRule((New-Object Security.AccessControl.FileSystemAccessRule($id, 'FullControl', 'ContainerInherit,ObjectInherit', 'None', 'Allow')))
}
Set-Acl $BackupDir $acl
}
function Get-EnvLoc($svc, $kind) {
if ($kind -eq 'Nssm') { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc\Parameters"; Name = 'AppEnvironmentExtra' } }
else { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"; Name = 'Environment' } }
}
function Read-Env($loc) { @((Get-ItemProperty $loc.Path -Name $loc.Name -ErrorAction SilentlyContinue).($loc.Name) | Where-Object { $_ }) }
function Write-Env($loc, [string[]]$lines) { Set-ItemProperty $loc.Path -Name $loc.Name -Type MultiString -Value $lines }
function Test-Ready($port) {
foreach ($h in 'localhost', '127.0.0.1', $env:COMPUTERNAME) {
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 "http://${h}:$port/health/ready"; if ($r.StatusCode -eq 200) { return $true } } catch {}
}
$false
}
function Restart-App($svc, $port) {
$t0 = Get-Date
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
Stop-Service $svc -NoWait -Force
$deadline = (Get-Date).AddSeconds(180)
while ((Get-Service $svc).Status -ne 'Stopped') { if ((Get-Date) -gt $deadline) { throw "$svc did not stop" }; Start-Sleep 1 }
while ($procId -and (Get-Process -Id $procId -ErrorAction SilentlyContinue)) { if ((Get-Date) -gt $deadline) { throw "$svc pid $procId still alive" }; Start-Sleep 1 }
$tStop = Get-Date
Start-Service $svc
$deadline = (Get-Date).AddSeconds(240)
while (-not (Test-Ready $port)) { if ((Get-Date) -gt $deadline) { throw "$svc not ready on :$port" }; Start-Sleep 2 }
$newPid = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
Say ("{0} restarted: stop {1:N0} s, stop->ready {2:N0} s, pid {3} -> {4}" -f $svc, ($tStop - $t0).TotalSeconds, ((Get-Date) - $tStop).TotalSeconds, $procId, $newPid)
}
$stamp = Get-Date -Format yyyyMMddTHHmmss
foreach ($entry in $Apps) {
$svc, $kind, $prefix, $port = $entry -split '\|'
$loc = Get-EnvLoc $svc $kind
$kExp = "${prefix}Telemetry__Exporter"; $kEnd = "${prefix}Telemetry__OtlpEndpoint"
$old = Read-Env $loc
if ($old.Count -eq 0) { throw "$svc has no environment at $($loc.Path)\$($loc.Name)" }
$oldHashes = $old | ForEach-Object { Sha $_ }
$pep = $old | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
Say "$svc env before: $($old.Count) entries, block $(Sha ($old -join "`n")), pepper [$($pep -join ',')]"
if ($Rollback) {
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
} else {
$reg = Join-Path $BackupDir "$svc-pre-otlp-$stamp.reg"
& reg.exe export ($loc.Path -replace '^HKLM:\\','HKLM\' -replace '\\Parameters$','') $reg /y | Out-Null
if ($LASTEXITCODE -ne 0) { throw "reg export failed for $svc" }
Set-Content (Join-Path $BackupDir "$svc-env-names-pre-$stamp.txt") ($old | ForEach-Object { ($_ -split '=',2)[0] })
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + @("$kExp=Otlp", "$kEnd=$Endpoint")
}
$kept = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
try {
Write-Env $loc $new
$back = Read-Env $loc
$backHashes = @($back | ForEach-Object { Sha $_ })
foreach ($k in $kept) { if ($backHashes -notcontains (Sha $k)) { throw "$svc readback lost entry $(($k -split '=',2)[0])" } }
if ($back.Count -ne $new.Count) { throw "$svc readback count $($back.Count) != $($new.Count)" }
$pep2 = $back | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
if (($pep -join ',') -ne ($pep2 -join ',')) { throw "$svc pepper changed" }
$added = $back | Where-Object { $_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=") }
Say "$svc env after: $($back.Count) entries ($($kept.Count) pre-existing hash-identical), pepper [$($pep2 -join ',')], telemetry: $($added -join ' ')"
Restart-App $svc $port
} catch {
Say "FAIL $svc : $($_.Exception.Message) -- restoring the original environment"
Write-Env $loc $old
try { Restart-App $svc $port } catch { Say "ROLLBACK RESTART FAILED $svc : $($_.Exception.Message)" }
exit 1
}
}
Say 'all done'
@@ -0,0 +1,65 @@
<#
.SYNOPSIS
Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent.
.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window.
.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP).
#>
param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20)
$ErrorActionPreference = 'Continue'
$root = 'E:\ApiInstall\OtelCollector'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'"
Say "service $($s.State) $($s.StartMode) $($s.StartName)"
try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" }
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe'
if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
" collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) }
Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime }
" E: free GB " + [int]((Get-PSDrive E).Free/1GB)
$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000
function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue }
foreach ($sig in 'metrics','traces','logs') {
$f = "$root\data\$sig.jsonl"
if (-not (Test-Path $f)) { Say "${sig}: no file"; continue }
$agg = @{}
foreach ($line in (Get-Content $f -Tail $Tail)) {
try { $o = $line | ConvertFrom-Json } catch { continue }
$rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs }
foreach ($r in $rs) {
$key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role')
if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } }
$a = $agg[$key]
if ($sig -eq 'metrics') {
foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1
foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } }
} elseif ($sig -eq 'traces') {
foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } }
} else {
foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } }
}
}
}
Say "$sig (last $Tail export requests):"
foreach ($k in ($agg.Keys | Sort-Object)) {
$a = $agg[$k]
$newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' }
$fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' }
" $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh"
if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') }
}
}
try {
$p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n"
$jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name
Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" }
} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" }
try {
$q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n"
Say "collector self-metrics:"
$q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') }
} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" }
foreach ($port in $AppPorts) {
$c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} }
" app :$port /metrics $c"
}
+80
View File
@@ -0,0 +1,80 @@
# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54).
# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA),
# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector.
#
# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317,
# and nothing is reachable from the network. That is why the receivers have no auth and no TLS.
# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md).
#
# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as
# deployment.environment.name, so files copied off two tiers can still be told apart.
#
# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older
# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB.
extensions:
health_check:
endpoint: 127.0.0.1:13133
receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:4317
http:
endpoint: 127.0.0.1:4318
processors:
memory_limiter:
check_interval: 5s
limit_mib: 256
spike_limit_mib: 64
resource/tier:
attributes:
- key: deployment.environment.name
value: ${env:ZB_TIER}
action: upsert
batch: {}
exporters:
file/metrics:
path: E:/ApiInstall/OtelCollector/data/metrics.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
file/traces:
path: E:/ApiInstall/OtelCollector/data/traces.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
file/logs:
path: E:/ApiInstall/OtelCollector/data/logs.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
prometheus:
endpoint: 127.0.0.1:8889
resource_constant_labels:
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name]
metric_expiration: 15m
service:
extensions: [health_check]
telemetry:
logs:
level: info
metrics:
level: basic
readers:
- pull:
exporter:
prometheus:
host: 127.0.0.1
port: 8888
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/metrics, prometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/traces]
logs:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/logs]