feat(infra): per-box OTel collector on the wonder tiers; apps switched to OTLP (#54)
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries each; rolling restarts, standby-first on the QA pairs. Overview has no exporter binding and is unchanged. Config, install/switch/verify scripts, runbook and the dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS, index row and deployment.md updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See
|
|||||||
| Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
|
| Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
|
||||||
| UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
|
| UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
|
||||||
| Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
|
| Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
|
||||||
| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). Open: wonder-tier collector #54. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
|
| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). **Wonder tiers on OTLP since 2026-11-12 (#54, closed):** one loopback-only `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, `NT SERVICE\ZbOtelCollector`, `E:\ApiInstall\OtelCollector`, file + Prometheus `127.0.0.1:8889` exporters, ≤ ~1.8 GB per box) on vd03/vq03/vq04. OtOpcUa, ScadaBridge Central + Site, mxgw and HG export to `http://127.0.0.1:4317` via service env (Overview has no exporter binding). The deployed builds send metrics + traces; logs follow when 0.3.0 is deployed. Record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/). Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
|
||||||
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
||||||
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
||||||
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
|
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
|
||||||
|
|||||||
@@ -163,7 +163,7 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew
|
|||||||
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
|
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
|
||||||
(`logs.jsonl` stayed empty).
|
(`logs.jsonl` stayed empty).
|
||||||
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
|
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
|
||||||
- **scadaproj#54:** collector on the wonder tiers.
|
- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
|
||||||
|
|
||||||
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
|
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
|
||||||
|
|
||||||
@@ -211,7 +211,7 @@ scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack)
|
|||||||
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
|
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
|
||||||
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
|
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
|
||||||
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
|
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
|
||||||
wonder tier has a collector yet (#54), so no deployed box can trip it.
|
wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
|
||||||
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
|
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
|
||||||
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
|
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
|
||||||
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
|
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
|
|||||||
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
|
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
|
||||||
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
|
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
|
||||||
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
|
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
|
||||||
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
|
> (further ScadaBridge instruments → ScadaBridge#81) remains; the collector on the wonder tiers (#54) is DONE 2026-11-12.
|
||||||
>
|
>
|
||||||
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
|
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
|
||||||
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
|
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
|
||||||
@@ -273,8 +273,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
|
|||||||
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
|
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
|
||||||
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
|
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
|
||||||
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
|
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
|
||||||
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
|
- ~~**No collector on the wonder tiers** (DEV vd03, QA vq03/vq04)~~ (→ #54). **DONE 2026-11-12:** a
|
||||||
Prometheus-only. → #54
|
per-box `ZbOtelCollector` (otelcol-contrib 0.162.0, NSSM, virtual account, all listeners on 127.0.0.1,
|
||||||
|
file + Prometheus exporters, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
|
||||||
|
HistorianGateway export OTLP there on all three boxes; Prometheus `/metrics` is unchanged. Remaining: **Overview**
|
||||||
|
has no exporter config binding (it needs a code change), and OTLP logs arrive only once the Telemetry 0.3.0 builds
|
||||||
|
are deployed. Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
|
||||||
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
|
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
|
||||||
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
|
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
|
||||||
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the
|
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the
|
||||||
|
|||||||
@@ -401,6 +401,8 @@ nodes, verified Healthy at setup):
|
|||||||
| HistorianGateway | Up | Up (since 2026-08-14) |
|
| HistorianGateway | Up | Up (since 2026-08-14) |
|
||||||
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) |
|
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 12-instance registry) |
|
||||||
|
|
||||||
|
> **Active node as of 2026-11-12 22:00 EST** (after the scadaproj#54 rolling restart, standby-first): OtOpcUa **vq04** Primary (250) / vq03 240; ScadaBridge central **and** site **active on vq03**. The table above is the 2026-08-13 capture.
|
||||||
|
|
||||||
**Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge
|
**Deployed builds on QA (2026-09-10 round, all via the `winrm` skill, both nodes):** ScadaBridge
|
||||||
`2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window,
|
`2b9a7c3c` (all four nodes stopped together, one shared-DB migration 53→55, 2 min 35 s window,
|
||||||
rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because
|
rollback = `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 because
|
||||||
@@ -488,6 +490,16 @@ Neither install is a scripted in-repo flow — the operational procedures live i
|
|||||||
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
|
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
|
||||||
those runbooks, or not probed in §7.1, as unverified.
|
those runbooks, or not probed in §7.1, as unverified.
|
||||||
|
|
||||||
|
**OpenTelemetry collector on every wonder box (2026-11-12, scadaproj#54):** service `ZbOtelCollector`
|
||||||
|
(otelcol-contrib 0.162.0, NSSM, virtual account `NT SERVICE\ZbOtelCollector`) at `E:\ApiInstall\OtelCollector\` on
|
||||||
|
vd03, vq03 and vq04. All of its ports are **loopback only**: OTLP `127.0.0.1:4317`/`4318`, Prometheus exporter `:8889`,
|
||||||
|
self-metrics `:8888`, health `:13133`, so nothing new is reachable off-box. Data is in `data\{metrics,traces,logs}.jsonl`
|
||||||
|
(rotated, ≤ ~1.8 GB per box). OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and HistorianGateway have
|
||||||
|
`<App>__Telemetry__Exporter=Otlp` + `__OtlpEndpoint=http://127.0.0.1:4317` in their service environment (HG: root `Telemetry__*`);
|
||||||
|
Overview is unchanged. On QA, `E:\ApiInstall\OtelCollector` is the 12th `-DenyPaths` entry of the OtOpcUa worker isolation.
|
||||||
|
The rolling restart left **OtOpcUa Primary on vq04** and **ScadaBridge Central + Site active on vq03**. Env backups:
|
||||||
|
`E:\psrem-stage\otel-54\backup-<COMPUTER>\`. Runbook + record: [`infra/otel-collector/wonder/`](infra/otel-collector/wonder/).
|
||||||
|
|
||||||
**Auth posture differs between the two, and QA is the correct one:**
|
**Auth posture differs between the two, and QA is the correct one:**
|
||||||
|
|
||||||
| | VD03 (DEV) | VQ03/VQ04 (QA) |
|
| | VD03 (DEV) | VQ03/VQ04 (QA) |
|
||||||
|
|||||||
@@ -6,8 +6,9 @@ shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-sha
|
|||||||
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
|
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
|
||||||
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
|
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
|
||||||
|
|
||||||
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA
|
> Scope: **dev/test only**. The receivers have no auth and no TLS. The wonder tiers (DEV vd03, QA
|
||||||
> vq03/vq04) has a collector yet — that is scadaproj#54.
|
> vq03/vq04) have their own per-box, loopback-only collector since 2026-11-12 (scadaproj#54):
|
||||||
|
> [`wonder/`](wonder/).
|
||||||
|
|
||||||
## Endpoints
|
## Endpoints
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,147 @@
|
|||||||
|
# 2026-11-12: OTel collector on the wonder tiers, apps switched to OTLP (scadaproj#54)
|
||||||
|
|
||||||
|
**Status: EXECUTED 2026-11-12, 21:38–21:56 EST.** DEV wonder-app-vd03 and QA wonder-app-vq03 + vq04 each run the
|
||||||
|
`ZbOtelCollector` service. Every app that has an exporter setting now exports OTLP to the collector on its own box.
|
||||||
|
No binary, appsettings file or Telemetry pin was changed. The change consists of a new service plus two service-environment entries per app.
|
||||||
|
Runbook and scripts: [`README.md`](README.md).
|
||||||
|
|
||||||
|
Channel: the `winrm` skill, used only for this deployment's on-box steps (preflight, install, env edits, restarts,
|
||||||
|
verification), with the user's request of 2026-11-12 ("resolve the remaining otel issues"). Files were staged over SMB to
|
||||||
|
`E:\psrem-stage\otel-54\`. Health, ScadaBridge site health, OPC UA ServiceLevel and Overview were checked from the Mac
|
||||||
|
over HTTP / opc.tcp.
|
||||||
|
|
||||||
|
## Design (owner-approved defaults)
|
||||||
|
|
||||||
|
| Decision | Choice | Why / trade-off |
|
||||||
|
|---|---|---|
|
||||||
|
| Topology | One collector **per box**, apps → `http://127.0.0.1:4317` | No new network path and nothing new reachable on the corporate network. On QA each node keeps its own copy, so the two pair nodes' data sits in two places. A tier-wide view needs a backend that the collectors export *to* |
|
||||||
|
| Receiver security | Loopback-only bind on all five ports, no auth or TLS | Only local processes can send. Opening the receivers to the network would require auth + TLS first |
|
||||||
|
| Backend | `file` exporter (OTLP-JSON, rotated) + `prometheus` exporter on `127.0.0.1:8889` | Durable local record plus a scrapeable view, with nothing new to operate. **No Tempo or Jaeger yet.** That is the future option: add an `otlp` exporter to the traces pipeline |
|
||||||
|
| Retention | Each signal file rotates at 100 MB, keeps 5 backups, max 7 days → ≤ ~600 MB per signal, ≤ ~1.8 GB total | E: had 180 / 216 / 231 GB free (vd03 / vq03 / vq04), so the cap is under 1 %. **At the observed rate (§ Disk) the size cap binds, not the 7 days:** about 1.5–2.8 days of metrics and 16 h – 1.5 days of traces |
|
||||||
|
| Version | `otelcol-contrib` 0.162.0 `windows_amd64` | Same release as the dev collector, so `config.yaml` is validated by the same binary. The tarball SHA-256 `b371829181e8…9e5c3d` matches the release's `.sha256` asset (this release publishes per-asset `.sha256` files and no combined `checksums.txt`) |
|
||||||
|
| Service shape | NSSM (a copy of the box's `nssm.exe` in `bin\`), Automatic, restart on exit after 5 s, stdout/stderr → `logs\collector.log` rotated at 10 MB | Same shape as the other apps. `NO_WINDOWS_SERVICE=1` keeps otelcol in console mode under NSSM |
|
||||||
|
| Account | Virtual account `NT SERVICE\ZbOtelCollector` | Least privilege that works: it binds unprivileged loopback ports and writes only `data\` + `logs\`. LocalSystem was not needed |
|
||||||
|
| Install location | `E:\ApiInstall\OtelCollector\{bin,config,data,logs}` | Next to the apps. The root DACL is protected, `BUILTIN\Users` removed, SYSTEM + Administrators full, service account RX / Modify on `data\` + `logs\` |
|
||||||
|
| Worker deny (lmxopcua#891) | vd03: the inherited `E:\ApiInstall` deny, kept as an explicit copy when inheritance was cut. QA: the isolation installer re-run with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` (11 → 12) | The collector holds no secrets: its config has no credentials and the receivers have no auth. Telemetry files can carry URL paths and host names, which is why `Users` was removed and the deny was added anyway. The procedure asks that every new `E:\ApiInstall` folder join the list |
|
||||||
|
|
||||||
|
## Per-app change
|
||||||
|
|
||||||
|
Two entries were added to each service's environment: `<Prefix>Telemetry__Exporter=Otlp` and
|
||||||
|
`<Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317`. The key names were read from each **deployed** commit's source:
|
||||||
|
|
||||||
|
| App (deployed build) | Service | Env store | Prefix | Entries vd03 | Entries vq03 | Entries vq04 |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| HistorianGateway (vd03 `ea9f21c`, QA `fa9cf31`) | `HistorianGateway` | NSSM `AppEnvironmentExtra` | *(root `Telemetry:`)* | 7 → 9 | 32 → 34 | 32 → 34 |
|
||||||
|
| OtOpcUa (`fea7de0a`) | `OtOpcUaHost` | service-key `Environment` | `OtOpcUa__` | 26 → 28 | 46 → 48 | 45 → 47 |
|
||||||
|
| ScadaBridge Central (vd03 `424db155`, QA as deployed) | `ScadaBridge-Central` | service-key `Environment` | `ScadaBridge__` | 10 → 12 | 13 → 15 | 13 → 15 |
|
||||||
|
| ScadaBridge Site | `ScadaBridge-Site` | service-key `Environment` | `ScadaBridge__` | 4 → 6 | 6 → 8 | 6 → 8 |
|
||||||
|
| MxAccessGateway (vd03 `76a71bc8`, QA `4f931e4`) | `MxAccessGw` | NSSM `AppEnvironmentExtra` | `MxGateway__` | 7 → 9 | 31 → 33 | 31 → 33 |
|
||||||
|
| Overview | `ZbOverview` | — | — | not changed | not changed | not changed |
|
||||||
|
|
||||||
|
- **Overview cannot be switched by config.** Its `AddZbTelemetry` call sets `ServiceName` and `Meters` only and binds no
|
||||||
|
exporter key, so it stays Prometheus-only. This needs a code change (Telemetry 0.3.0 `ApplyConfiguration`), which is out of scope here.
|
||||||
|
- Every pre-existing entry was hash-identical on readback. Peppers were compared by SHA-256 prefix only, never printed:
|
||||||
|
mxgw vd03 `0d01abcc1fe5` (env block `3f283e9a1d57` before, matching the #229 record), QA `88188ab11b50`; OtOpcUa vd03
|
||||||
|
`a26449ba4e13`, QA `d61fda284aa4`; HistorianGateway vd03 `8f8410157077`, QA `f779032d1cac`; ScadaBridge inbound vd03
|
||||||
|
`2fc0e4846d17`, QA `c358a1ef70c4`. All were unchanged after the edit.
|
||||||
|
- The values were written straight to the registry as REG_MULTI_SZ. `nssm set`, `install-service.bat` and the apps'
|
||||||
|
upgrade scripts were not used, so the vd03 mxgw env-only pepper stayed in place.
|
||||||
|
- These keys are the same in Telemetry 0.3.0 (`<App>:Telemetry` section, scadaproj#55), so they carry over. **OTLP logs will not
|
||||||
|
flow until the 0.3.0 builds are deployed**; `logs.jsonl` is empty until then.
|
||||||
|
|
||||||
|
## Timeline (EST)
|
||||||
|
|
||||||
|
| Time | Box | Step |
|
||||||
|
|---|---|---|
|
||||||
|
| 21:30 | all | Read-only preflight. Collector ports 4317/4318/8888/8889/13133 free on all three boxes, no existing service or folder |
|
||||||
|
| 21:38–21:39 | vd03 | Collector installed and healthy (process started in about 15 s) |
|
||||||
|
| 21:39:33–21:40:43 | vd03 | HG (5 s stop→ready), OtOpcUa (8 s), SB Central (13 s), SB Site (9 s), mxgw (5 s), one at a time |
|
||||||
|
| 21:44–21:46 | vq04, vq03 | Collectors installed and healthy |
|
||||||
|
| 21:46–21:47 | vq04, vq03 | Isolation installer `-WhatIf`, apply (2 `changed`: `DenyPathRecord` + `SensitiveDirDeny E:\ApiInstall\OtelCollector`), re-apply all `unchanged`; OtOpcUaHost env block hash unchanged |
|
||||||
|
| 21:48:06–21:49:05 | vq04 | HG (5 s), mxgw (6 s), OtOpcUa **Secondary** (stop 32 s, 10 s to ready) |
|
||||||
|
| 21:49:17–21:50:20 | vq03 | HG (5 s), mxgw (6 s), SB Central **standby** (6 s), SB Site **standby** (3 s), OtOpcUa **Primary** (failover to vq04) |
|
||||||
|
| 21:50–21:53:38 | — | Waited for the QA site's MxGateway tags to recover (see below) before touching the active ScadaBridge nodes |
|
||||||
|
| 21:53:59–21:55:03 | vq04 | SB Central **active** (6 s, failover to vq03), SB Site **active** (stop 41 s, failover to vq03) |
|
||||||
|
| 21:56:05 | — | QA site back to 2196/2196 Good on vq03 |
|
||||||
|
|
||||||
|
The QA pairs were restarted standby-first, one node at a time, as the 2026-10-22 ScadaBridge QA record does for a
|
||||||
|
config-only change and as the OtOpcUa QA record's Secondary restart + failover drill shows for `fea7de0a`. No pair was
|
||||||
|
ever fully down.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
### Apps (from the Mac)
|
||||||
|
|
||||||
|
- `/health/ready` **200** for OtOpcUa `:9000`, SB Central `:8085`, SB Site `:8084`, mxgw `:5130`, HG `:5220` and Overview
|
||||||
|
`:5320` on all three boxes.
|
||||||
|
- `/metrics` **200** on every app port (checked on the box): Prometheus is still served, since OTLP is additive.
|
||||||
|
- **OtOpcUa ServiceLevel (opc.tcp i=2267):** vd03 **250**. QA before was vq03 250 / vq04 240. After: **vq04 250 (Primary,
|
||||||
|
`/health/active` 200) / vq03 240** (503). Roles swapped by the rolling restart, with no failback, as recorded for `fea7de0a`.
|
||||||
|
- **ScadaBridge active nodes:** QA Central and Site are now **active on vq03** (were vq04). Both pairs report 2 nodes online.
|
||||||
|
- **ScadaBridge → MxGateway (site health report):**
|
||||||
|
- vd03: `Connected`, 810/810 resolved, good 398 / bad 412, **identical to the pre-change report** (the Bad tags are
|
||||||
|
pre-existing idle receivers).
|
||||||
|
- QA: 2196/2196 Good before and after. Two dips: (1) the vq03 gateway restart at 21:49:34 took the site (then Primary on
|
||||||
|
vq04) to 0 resolved until **21:53:38, about 4 min**; (2) the site failover at 21:55 recovered in about 1 min (21:56:05). The
|
||||||
|
first dip is longer than the ~1.5 min seen on 2026-09-10 and is worth knowing for future QA gateway restarts. It recovered unaided.
|
||||||
|
- **mxgw:** exactly one gateway process per box after the restart.
|
||||||
|
- **Overview:** QA **12/12 Up** on both vq03 and vq04 dashboards, DEV **6/6 Up**.
|
||||||
|
|
||||||
|
### Collector (on the box, `Test-OtelCollector.ps1`)
|
||||||
|
|
||||||
|
vd03, about 3 min after the last restart:
|
||||||
|
|
||||||
|
| Signal | What arrived (`service.name`, all `host.name=WONDER-APP-VD03`, `deployment.environment.name=DEV`) |
|
||||||
|
|---|---|
|
||||||
|
| Traces | `historian-gateway` (historian.Connect/Probe, HistoryService/*), `mxgateway` (OpenSession/Invoke), `otopcua` (`otopcua.opcua.address_space_rebuild`, HistorianStatus/Probe), `scadabridge` Central (`site.id=central node.role=Central`: CentralControlService heartbeat/report/reconcile, `/management`), `scadabridge` Site (`site.id=site-a node.role=Site`: MxAccessGateway Invoke/OpenSession client spans) |
|
||||||
|
| Metrics | Prometheus `:8889` series per job: `ZB.MOM.WW/historian-gateway` 289, `mxgateway` 376, `otopcua` 304, `scadabridge` 680 |
|
||||||
|
| Collector counters | accepted 1372 metric points / 3488 spans, **refused 0**; file + prometheus exporters sent all of it |
|
||||||
|
| Logs | none (expected until Telemetry 0.3.0, scadaproj#55) |
|
||||||
|
|
||||||
|
**ScadaBridge Central over OTLP is proven on vd03**: Central-role spans and metrics arrived with `node.role=Central`. That is
|
||||||
|
ScadaBridge#133's question. This record does not close that issue; it is left to its owner.
|
||||||
|
|
||||||
|
QA (22:07–22:08, files pulled over SMB and summarised on the Mac, plus the on-box check):
|
||||||
|
|
||||||
|
| Box | Every process present with metrics + traces (`deployment.environment.name=QA`) | `:8889` series per job | Accepted / refused |
|
||||||
|
|---|---|---|---|
|
||||||
|
| vq03 | `historian-gateway`, `mxgateway`, `otopcua`, `scadabridge` Central (`site.id=central`), `scadabridge` Site (`site.id=site-a`: MxAccessGateway Invoke 2216, LocalDbSync spans) | HG 306, mxgw 644, OtOpcUa 396, SB 771 | 9098 points, 15942 spans / **0** |
|
||||||
|
| vq04 | the same five; OtOpcUa sends `otopcua.script.*` (runner on) | HG 248, mxgw 235, OtOpcUa 472, SB 438 | 7081 points, 4941 spans / **0** |
|
||||||
|
|
||||||
|
Collector working set 115–126 MB. `Test-OtelCollector.ps1`'s per-line `ConvertFrom-Json` is too slow on PS 5.1 for QA-sized
|
||||||
|
export lines (it did not finish in 10 min), so its default `-Tail` is now 20. Run it with `-Tail 1` for the quick
|
||||||
|
service/listener/Prometheus/counter check, and pull `data\*.jsonl` over SMB for the content check.
|
||||||
|
|
||||||
|
## Disk
|
||||||
|
|
||||||
|
E: free after install: vd03 179 GB, vq03 215 GB, vq04 230 GB. The collector binary is about 370 MB.
|
||||||
|
|
||||||
|
Observed write rate in the first 20–30 minutes:
|
||||||
|
|
||||||
|
| Box | metrics.jsonl | traces.jsonl | → metrics retained | → traces retained |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| vd03 (28 min) | 4.9 MB (~250 MB/day) | 8.0 MB (~410 MB/day) | ~2.4 days | ~1.5 days |
|
||||||
|
| vq03 (21 min) | 3.1 MB (~210 MB/day) | 12.6 MB (~860 MB/day) | ~2.8 days | ~16 h |
|
||||||
|
| vq04 (22 min) | 2.4 MB (~160 MB/day) | 3.5 MB (~230 MB/day) | ~3.7 days | ~2.6 days |
|
||||||
|
|
||||||
|
The 600 MB per-signal cap therefore holds about 16 h to 3.7 days, not 7 days. Disk stays bounded at ~1.8 GB per box. The trace
|
||||||
|
volume is mostly the ScadaBridge Site's MxAccessGateway `Invoke` client spans (the active site) and HistorianGateway's
|
||||||
|
four `Get*InterfaceVersion` handshake spans per historian connect. Raise `max_backups` if a longer window is wanted; there is room.
|
||||||
|
|
||||||
|
## Backups and rollback
|
||||||
|
|
||||||
|
- Registry exports per service, taken before each edit: `E:\psrem-stage\otel-54\backup-<COMPUTER>\<Service>-pre-otlp-<ts>.reg`,
|
||||||
|
plus `<Service>-env-names-pre-<ts>.txt` (names only). The folder ACL is protected, SYSTEM + Administrators only.
|
||||||
|
vq04 has two stamps, one for step 1 (HG/mxgw/OtOpcUa) and one for step 3 (ScadaBridge).
|
||||||
|
- **App rollback:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<entry>'` removes the two entries and restarts,
|
||||||
|
or `reg import` the `.reg` and restart. QA Akka pairs go standby-first.
|
||||||
|
- **Collector rollback:** `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall` stops and deletes the
|
||||||
|
service and renames the folder to `OtelCollector.removed.<ts>`. On QA, re-run the isolation installer without
|
||||||
|
`E:\ApiInstall\OtelCollector` in `-DenyPaths`. With the collector gone, the apps keep running; only OTLP export fails, silently.
|
||||||
|
- Staging `E:\psrem-stage\otel-54\` stays on each box (zip, config, scripts, backups).
|
||||||
|
|
||||||
|
## Follow-ups
|
||||||
|
|
||||||
|
- Overview: no exporter config binding (needs Telemetry 0.3.0 `ApplyConfiguration` in its `Program.cs`).
|
||||||
|
- OTLP logs: arrive with the Telemetry 0.3.0 builds (scadaproj#55); no collector change needed.
|
||||||
|
- A trace backend (Tempo/Jaeger) or a central collector, if a cross-box view is wanted.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# scadaproj#54, QA only: append E:\ApiInstall\OtelCollector to the recorded OtOpcUaScriptWorker -DenyPaths (lmxopcua#891 procedure).
|
||||||
|
param([switch]$Apply)
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$inst = 'E:\psrem-stage\otopcua-890-f75ee7d0\install\Install-ScriptWorkerIsolation.ps1'
|
||||||
|
$recorded = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths)
|
||||||
|
$deny = @($recorded | Where-Object { $_ -ne 'E:\ApiInstall\OtelCollector' }) + 'E:\ApiInstall\OtelCollector'
|
||||||
|
"recorded $($recorded.Count) -> $($deny.Count) paths"
|
||||||
|
function EnvSig { $e = @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost' -Name Environment).Environment)
|
||||||
|
$b = [Text.Encoding]::UTF8.GetBytes($e -join "`n"); "$($e.Count) entries, block " + (-join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | % { $_.ToString('x2') })) }
|
||||||
|
"OtOpcUaHost env before: $(EnvSig)"
|
||||||
|
$common = @{ InstallRoot = 'E:\ApiInstall\OtOpcUa\app'; HostServiceAccount = 'LocalSystem'; ConfigureHostService = $true; DenyPaths = $deny }
|
||||||
|
if ($Apply) {
|
||||||
|
& $inst @common 2>&1 | Out-String -Width 300
|
||||||
|
"--- re-apply (expect all unchanged)"
|
||||||
|
& $inst @common 2>&1 | Out-String -Width 300
|
||||||
|
} else { & $inst @common -WhatIf 2>&1 | Out-String -Width 300 }
|
||||||
|
"OtOpcUaHost env after: $(EnvSig)"
|
||||||
|
"recorded now: " + (@((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters' -Name DenyPaths).DenyPaths) -join '; ')
|
||||||
|
(icacls 'E:\ApiInstall\OtelCollector') | Select-String 'OtOpcUaScriptWorker' | % { "acl $_" }
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54).
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\}
|
||||||
|
Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit.
|
||||||
|
ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit
|
||||||
|
ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and
|
||||||
|
Modify on data\ and logs\.
|
||||||
|
Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service
|
||||||
|
wrapper, so otelcol runs in console mode).
|
||||||
|
Every listener in config.yaml binds 127.0.0.1. The script checks that after start.
|
||||||
|
|
||||||
|
.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt.
|
||||||
|
.PARAMETER Tier DEV or QA.
|
||||||
|
.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\.
|
||||||
|
.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed.<ts> (kept).
|
||||||
|
#>
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory)] [string]$StageDir,
|
||||||
|
[ValidateSet('DEV','QA')] [string]$Tier,
|
||||||
|
[string]$NssmPath,
|
||||||
|
[switch]$Uninstall
|
||||||
|
)
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$svcName = 'ZbOtelCollector'
|
||||||
|
$root = 'E:\ApiInstall\OtelCollector'
|
||||||
|
$acct = "NT SERVICE\$svcName"
|
||||||
|
$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E'
|
||||||
|
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||||
|
|
||||||
|
if ($Uninstall) {
|
||||||
|
$s = Get-Service $svcName -ErrorAction SilentlyContinue
|
||||||
|
if ($s) {
|
||||||
|
if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) }
|
||||||
|
& sc.exe delete $svcName | Out-Null
|
||||||
|
Say "service $svcName deleted"
|
||||||
|
}
|
||||||
|
if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" }
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' }
|
||||||
|
if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" }
|
||||||
|
if (Test-Path $root) { throw "$root already exists" }
|
||||||
|
|
||||||
|
# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git).
|
||||||
|
$sums = @{}
|
||||||
|
Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } |
|
||||||
|
ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() }
|
||||||
|
foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') {
|
||||||
|
$h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash
|
||||||
|
if ($h -ne $sums[$f]) { throw "hash mismatch $f" }
|
||||||
|
Say "hash ok $f $($h.Substring(0,12))"
|
||||||
|
}
|
||||||
|
|
||||||
|
# 2. Layout
|
||||||
|
foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null }
|
||||||
|
$tmp = Join-Path $StageDir 'x'
|
||||||
|
if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force }
|
||||||
|
Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp
|
||||||
|
$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash
|
||||||
|
if ($h -ne $exeSha) { throw "exe hash mismatch $h" }
|
||||||
|
Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe')
|
||||||
|
Remove-Item $tmp -Recurse -Force
|
||||||
|
Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe')
|
||||||
|
Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml')
|
||||||
|
Say "layout ok, exe sha $($exeSha.Substring(0,12))"
|
||||||
|
|
||||||
|
# 3. Service
|
||||||
|
$nssm = Join-Path $root 'bin\nssm.exe'
|
||||||
|
& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null
|
||||||
|
$set = @(
|
||||||
|
@('AppParameters', "--config=$root\config\config.yaml"),
|
||||||
|
@('AppDirectory', $root),
|
||||||
|
@('DisplayName', 'ZB OpenTelemetry Collector'),
|
||||||
|
@('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'),
|
||||||
|
@('Start', 'SERVICE_AUTO_START'),
|
||||||
|
@('AppStdout', "$root\logs\collector.log"),
|
||||||
|
@('AppStderr', "$root\logs\collector.log"),
|
||||||
|
@('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'),
|
||||||
|
@('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000'))
|
||||||
|
foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } }
|
||||||
|
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1')
|
||||||
|
& sc.exe config $svcName obj= $acct | Out-Null
|
||||||
|
if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" }
|
||||||
|
Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)"
|
||||||
|
|
||||||
|
# 4. ACL
|
||||||
|
& icacls $root /inheritance:d | Out-Null
|
||||||
|
& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null
|
||||||
|
& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null
|
||||||
|
foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null }
|
||||||
|
(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() }
|
||||||
|
|
||||||
|
# 5. Start + checks
|
||||||
|
Start-Service $svcName
|
||||||
|
$ok = $false
|
||||||
|
for ($i = 0; $i -lt 30; $i++) {
|
||||||
|
Start-Sleep 2
|
||||||
|
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {}
|
||||||
|
}
|
||||||
|
Say "service $((Get-Service $svcName).Status); health_check ok=$ok"
|
||||||
|
if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' }
|
||||||
|
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId
|
||||||
|
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||||
|
$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId
|
||||||
|
$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||||
|
$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' }
|
||||||
|
if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" }
|
||||||
|
$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces'
|
||||||
|
Say "OTLP/HTTP empty POST -> $($r.StatusCode)"
|
||||||
|
$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false
|
||||||
|
Say "collector.log error lines: $(@($err).Count)"
|
||||||
|
Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) }
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# ZbOtelCollector: per-box OTel collector on the wonder tiers (scadaproj#54)
|
||||||
|
|
||||||
|
One OpenTelemetry Collector (contrib) per wonder box: **wonder-app-vd03** (DEV) and **wonder-app-vq03 /
|
||||||
|
vq04** (QA). The family apps on each box export OTLP to the collector on the same box. The shared dev collector on
|
||||||
|
`10.100.0.35` ([`../README.md`](../README.md)) is a separate thing and is unchanged.
|
||||||
|
|
||||||
|
Design note and rollout record: [`2026-11-12-wonder-otel-collector.md`](2026-11-12-wonder-otel-collector.md).
|
||||||
|
|
||||||
|
## What is installed
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Binary | `otelcol-contrib` **0.162.0** `windows_amd64` (the same release as the dev collector). The tarball's SHA-256 `b3718291…9e5c3d` matches the release's `.sha256` asset. The exe SHA-256 is `f8f69bd6…e436e`. It was repacked on the Mac as a zip (`97a675ee…a167`) because PS 5.1 `Expand-Archive` can't read `.tar.gz`. |
|
||||||
|
| Folder | `E:\ApiInstall\OtelCollector\` containing `bin\otelcol-contrib.exe`, `bin\nssm.exe` (a copy of the box's NSSM), `config\config.yaml` ([`config.yaml`](config.yaml)), `data\` and `logs\` |
|
||||||
|
| Service | **`ZbOtelCollector`** (NSSM), display name "ZB OpenTelemetry Collector", Automatic, restarts on exit after 5 s |
|
||||||
|
| Account | Virtual account **`NT SERVICE\ZbOtelCollector`**, not LocalSystem. It needs only to bind loopback ports and write `data\` + `logs\` |
|
||||||
|
| Service env | `ZB_TIER=DEV` or `QA`, stamped on every signal as `deployment.environment.name`. `NO_WINDOWS_SERVICE=1` makes otelcol run in console mode under NSSM |
|
||||||
|
| ACL | The root is protected: the inherited ACEs were copied and `BUILTIN\Users` was removed. SYSTEM and Administrators have full control. The service account has RX on the tree and Modify on `data\` and `logs\`. The `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891) is present: on vd03 it is the explicit copy of the inherited `E:\ApiInstall` deny, and on QA it was added by re-running the isolation installer with `E:\ApiInstall\OtelCollector` appended to `-DenyPaths` |
|
||||||
|
|
||||||
|
## Endpoints (all 127.0.0.1, none reachable off-box)
|
||||||
|
|
||||||
|
| Port | What |
|
||||||
|
|---|---|
|
||||||
|
| 4317 | OTLP gRPC receiver. **Apps point here: `http://127.0.0.1:4317`** |
|
||||||
|
| 4318 | OTLP HTTP receiver |
|
||||||
|
| 8889 | Prometheus exporter: everything received, labelled `job=<namespace>/<service>`, plus `site_id`, `node_role`, `host_name` and `deployment_environment_name` |
|
||||||
|
| 8888 | The collector's own metrics (`otelcol_receiver_accepted_*`, `otelcol_exporter_send_failed_*`) |
|
||||||
|
| 13133 | `health_check` |
|
||||||
|
|
||||||
|
**Trade-off: loopback-only receivers.** Nothing on the network can send to the collector, so the receivers need no auth and
|
||||||
|
no TLS. The cost is that there is no tier-wide view: each box keeps its own files, and anything off-box (the Mac,
|
||||||
|
a future Prometheus or Tempo) cannot scrape `:8889` or query it. When a central backend is wanted, put it behind an exporter
|
||||||
|
on these collectors (OTLP out with TLS and auth) rather than opening the receivers.
|
||||||
|
|
||||||
|
## Backend and retention
|
||||||
|
|
||||||
|
- `file/metrics`, `file/traces` and `file/logs` write `data\{metrics,traces,logs}.jsonl` as OTLP-JSON, one export
|
||||||
|
request per line. Rotation is set to `max_megabytes: 100`, `max_backups: 5` and `max_days: 7`. Each signal is capped at
|
||||||
|
about 600 MB and all three at about **1.8 GB**. At the rate seen on day one that holds **16 h to 3.7 days**, depending on the signal and box (size binds before `max_days`; see the rollout record § Disk). Free space on E: at install was 180 GB (vd03), 216 GB (vq03) and
|
||||||
|
231 GB (vq04).
|
||||||
|
- The Prometheus exporter on 127.0.0.1:8889 holds no history; series expire 15 min after their last update.
|
||||||
|
- `logs\collector.log` is NSSM's stdout/stderr capture, rotated online at 10 MB. NSSM never deletes rotated files.
|
||||||
|
At `info` level the collector writes a few lines at start and then only errors, so this stays small.
|
||||||
|
- `logs.jsonl` stays empty until the apps run Telemetry **0.3.0** (scadaproj#55, OTLP logs). The builds deployed today do
|
||||||
|
not export logs. When 0.3.0 ships, logs start flowing with no collector change.
|
||||||
|
- No Tempo or Jaeger. A trace backend is a future option: add an `otlp` exporter to the `traces` pipeline.
|
||||||
|
|
||||||
|
## App settings (service environment)
|
||||||
|
|
||||||
|
| App | Service | Where the env lives | Entries added |
|
||||||
|
|---|---|---|---|
|
||||||
|
| OtOpcUa | `OtOpcUaHost` | service key `Environment` | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://127.0.0.1:4317` |
|
||||||
|
| ScadaBridge | `ScadaBridge-Central`, `ScadaBridge-Site` | service key `Environment` | `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
|
||||||
|
| MxAccessGateway | `MxAccessGw` | NSSM `Parameters\AppEnvironmentExtra` | `MxGateway__Telemetry__Exporter`, `MxGateway__Telemetry__OtlpEndpoint` |
|
||||||
|
| HistorianGateway | `HistorianGateway` | NSSM `Parameters\AppEnvironmentExtra` | `Telemetry__Exporter`, `Telemetry__OtlpEndpoint` (root-level section) |
|
||||||
|
| Overview | `ZbOverview` | not changed | Its `AddZbTelemetry` call binds no exporter key on the deployed build, so it stays Prometheus-only. This needs a code change |
|
||||||
|
|
||||||
|
These key names are unchanged in Telemetry 0.3.0 (`<App>:Telemetry` section), so the entries carry over to that upgrade.
|
||||||
|
Prometheus `/metrics` on every app stays served (OTLP is additive).
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
All are run on the box from the stage folder `E:\psrem-stage\otel-54\` through the `winrm` skill (deployment scope only).
|
||||||
|
|
||||||
|
| Script | Does |
|
||||||
|
|---|---|
|
||||||
|
| [`Install-OtelCollector.ps1`](Install-OtelCollector.ps1) | Checks hashes, lays out the folder, creates the service + virtual account, sets ACLs, starts it. It then checks health, that every listener is loopback, and that an empty OTLP POST returns 200. `-Uninstall` stops and deletes the service and renames the folder to `OtelCollector.removed.<ts>` |
|
||||||
|
| [`Switch-AppsToOtlp.ps1`](Switch-AppsToOtlp.ps1) | Per app, in order: `.reg` export into a protected backup folder (SYSTEM + Administrators only). Then it adds the two entries as REG_MULTI_SZ, never through `nssm set` or `install-service.bat`. On readback, every existing entry must be hash-identical and the pepper hash unchanged. Then stop (wait for the process to exit), start and wait for `/health/ready` 200. On failure it restores the in-memory original and restarts. `-Rollback` removes the two entries and restarts |
|
||||||
|
| [`Test-OtelCollector.ps1`](Test-OtelCollector.ps1) | Read-only: service, health, listeners, what each `service.name` sent (newest timestamp), Prometheus series per job, collector accepted/failed counters, and each app's `/metrics` |
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# install (vd03 example; QA uses -Tier QA -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe)
|
||||||
|
& E:\psrem-stage\otel-54\Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Tier DEV -NssmPath E:\ApiInstall\MxGateway\nssm.exe
|
||||||
|
# switch apps (Service|Kind|Prefix|Port), one at a time in this order
|
||||||
|
& E:\psrem-stage\otel-54\Switch-AppsToOtlp.ps1 -BackupDir E:\psrem-stage\otel-54\backup-<COMPUTER> -Apps `
|
||||||
|
'HistorianGateway|Nssm||5220','OtOpcUaHost|Env|OtOpcUa__|9000','ScadaBridge-Central|Env|ScadaBridge__|8085', `
|
||||||
|
'ScadaBridge-Site|Env|ScadaBridge__|8084','MxAccessGw|Nssm|MxGateway__|5130'
|
||||||
|
& E:\psrem-stage\otel-54\Test-OtelCollector.ps1 -Tail 1 -AppPorts 9000,8085,8084,5130,5220 # -Tail >20 is slow on PS 5.1; pull data\*.jsonl over SMB for content
|
||||||
|
```
|
||||||
|
|
||||||
|
On QA, order the Akka pairs standby-first across the two boxes (see the rollout record).
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
- **One app:** `Switch-AppsToOtlp.ps1 -Rollback -BackupDir <dir> -Apps '<that app entry>'` removes the two entries and
|
||||||
|
restarts. Alternatively, `reg import <dir>\<Service>-pre-otlp-<ts>.reg` and restart. On QA, restart Akka-pair nodes standby-first.
|
||||||
|
- **Collector:** roll the apps back first, or leave them: an app whose collector is gone only loses OTLP export. Then
|
||||||
|
run `Install-OtelCollector.ps1 -StageDir E:\psrem-stage\otel-54 -Uninstall`. On QA, also re-run the isolation
|
||||||
|
installer with `E:\ApiInstall\OtelCollector` removed from `-DenyPaths`.
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Adds the OTLP exporter settings to app services' environments and restarts them one at a time (scadaproj#54).
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
For each -Apps entry, in order:
|
||||||
|
1. reads the service environment (service-key 'Environment' for the .NET-hosted services, NSSM
|
||||||
|
Parameters\AppEnvironmentExtra for the NSSM ones) and exports the service key (.reg) to -BackupDir;
|
||||||
|
2. writes <Prefix>Telemetry__Exporter=Otlp and <Prefix>Telemetry__OtlpEndpoint=http://127.0.0.1:4317
|
||||||
|
(replacing any existing entries of those two names), as REG_MULTI_SZ, never via nssm/install scripts;
|
||||||
|
3. reads it back: every pre-existing entry must be hash-identical and the count must be +2 (or +0 when the
|
||||||
|
entries were already present);
|
||||||
|
4. stops the service (waits for Stopped and for its process to exit), starts it, waits for /health/ready 200.
|
||||||
|
If 3 or 4 fails, the original environment (held in memory) is written back, the service is restarted and the
|
||||||
|
script stops with exit 1. Only entry NAMES and SHA-256 prefixes are printed, never values.
|
||||||
|
|
||||||
|
-Rollback writes back the environment from the .reg backups instead (removes the two entries) and restarts.
|
||||||
|
|
||||||
|
.PARAMETER Apps Entries 'Service|Kind|Prefix|Port', Kind = Env or Nssm, Prefix e.g. 'OtOpcUa__' or '' (HistorianGateway).
|
||||||
|
#>
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory)] [string[]]$Apps,
|
||||||
|
[Parameter(Mandatory)] [string]$BackupDir,
|
||||||
|
[string]$Endpoint = 'http://127.0.0.1:4317',
|
||||||
|
[switch]$Rollback
|
||||||
|
)
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||||
|
function Sha([string]$s) { $b = [Text.Encoding]::UTF8.GetBytes($s); -join ([Security.Cryptography.SHA256]::Create().ComputeHash($b)[0..5] | ForEach-Object { $_.ToString('x2') }) }
|
||||||
|
|
||||||
|
if (-not (Test-Path $BackupDir)) {
|
||||||
|
New-Item -ItemType Directory -Path $BackupDir | Out-Null
|
||||||
|
$acl = New-Object Security.AccessControl.DirectorySecurity
|
||||||
|
$acl.SetAccessRuleProtection($true, $false)
|
||||||
|
foreach ($id in 'NT AUTHORITY\SYSTEM','BUILTIN\Administrators') {
|
||||||
|
$acl.AddAccessRule((New-Object Security.AccessControl.FileSystemAccessRule($id, 'FullControl', 'ContainerInherit,ObjectInherit', 'None', 'Allow')))
|
||||||
|
}
|
||||||
|
Set-Acl $BackupDir $acl
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-EnvLoc($svc, $kind) {
|
||||||
|
if ($kind -eq 'Nssm') { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc\Parameters"; Name = 'AppEnvironmentExtra' } }
|
||||||
|
else { @{ Path = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"; Name = 'Environment' } }
|
||||||
|
}
|
||||||
|
function Read-Env($loc) { @((Get-ItemProperty $loc.Path -Name $loc.Name -ErrorAction SilentlyContinue).($loc.Name) | Where-Object { $_ }) }
|
||||||
|
function Write-Env($loc, [string[]]$lines) { Set-ItemProperty $loc.Path -Name $loc.Name -Type MultiString -Value $lines }
|
||||||
|
|
||||||
|
function Test-Ready($port) {
|
||||||
|
foreach ($h in 'localhost', '127.0.0.1', $env:COMPUTERNAME) {
|
||||||
|
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 "http://${h}:$port/health/ready"; if ($r.StatusCode -eq 200) { return $true } } catch {}
|
||||||
|
}
|
||||||
|
$false
|
||||||
|
}
|
||||||
|
function Restart-App($svc, $port) {
|
||||||
|
$t0 = Get-Date
|
||||||
|
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
|
||||||
|
Stop-Service $svc -NoWait -Force
|
||||||
|
$deadline = (Get-Date).AddSeconds(180)
|
||||||
|
while ((Get-Service $svc).Status -ne 'Stopped') { if ((Get-Date) -gt $deadline) { throw "$svc did not stop" }; Start-Sleep 1 }
|
||||||
|
while ($procId -and (Get-Process -Id $procId -ErrorAction SilentlyContinue)) { if ((Get-Date) -gt $deadline) { throw "$svc pid $procId still alive" }; Start-Sleep 1 }
|
||||||
|
$tStop = Get-Date
|
||||||
|
Start-Service $svc
|
||||||
|
$deadline = (Get-Date).AddSeconds(240)
|
||||||
|
while (-not (Test-Ready $port)) { if ((Get-Date) -gt $deadline) { throw "$svc not ready on :$port" }; Start-Sleep 2 }
|
||||||
|
$newPid = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
|
||||||
|
Say ("{0} restarted: stop {1:N0} s, stop->ready {2:N0} s, pid {3} -> {4}" -f $svc, ($tStop - $t0).TotalSeconds, ((Get-Date) - $tStop).TotalSeconds, $procId, $newPid)
|
||||||
|
}
|
||||||
|
|
||||||
|
$stamp = Get-Date -Format yyyyMMddTHHmmss
|
||||||
|
foreach ($entry in $Apps) {
|
||||||
|
$svc, $kind, $prefix, $port = $entry -split '\|'
|
||||||
|
$loc = Get-EnvLoc $svc $kind
|
||||||
|
$kExp = "${prefix}Telemetry__Exporter"; $kEnd = "${prefix}Telemetry__OtlpEndpoint"
|
||||||
|
$old = Read-Env $loc
|
||||||
|
if ($old.Count -eq 0) { throw "$svc has no environment at $($loc.Path)\$($loc.Name)" }
|
||||||
|
$oldHashes = $old | ForEach-Object { Sha $_ }
|
||||||
|
$pep = $old | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
|
||||||
|
Say "$svc env before: $($old.Count) entries, block $(Sha ($old -join "`n")), pepper [$($pep -join ',')]"
|
||||||
|
|
||||||
|
if ($Rollback) {
|
||||||
|
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
|
||||||
|
} else {
|
||||||
|
$reg = Join-Path $BackupDir "$svc-pre-otlp-$stamp.reg"
|
||||||
|
& reg.exe export ($loc.Path -replace '^HKLM:\\','HKLM\' -replace '\\Parameters$','') $reg /y | Out-Null
|
||||||
|
if ($LASTEXITCODE -ne 0) { throw "reg export failed for $svc" }
|
||||||
|
Set-Content (Join-Path $BackupDir "$svc-env-names-pre-$stamp.txt") ($old | ForEach-Object { ($_ -split '=',2)[0] })
|
||||||
|
$new = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) }) + @("$kExp=Otlp", "$kEnd=$Endpoint")
|
||||||
|
}
|
||||||
|
$kept = @($old | Where-Object { -not ($_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=")) })
|
||||||
|
try {
|
||||||
|
Write-Env $loc $new
|
||||||
|
$back = Read-Env $loc
|
||||||
|
$backHashes = @($back | ForEach-Object { Sha $_ })
|
||||||
|
foreach ($k in $kept) { if ($backHashes -notcontains (Sha $k)) { throw "$svc readback lost entry $(($k -split '=',2)[0])" } }
|
||||||
|
if ($back.Count -ne $new.Count) { throw "$svc readback count $($back.Count) != $($new.Count)" }
|
||||||
|
$pep2 = $back | Where-Object { $_ -match '^[^=]*Pepper=' } | ForEach-Object { ($_ -split '=',2)[0] + ':' + (Sha (($_ -split '=',2)[1])) }
|
||||||
|
if (($pep -join ',') -ne ($pep2 -join ',')) { throw "$svc pepper changed" }
|
||||||
|
$added = $back | Where-Object { $_.StartsWith("$kExp=") -or $_.StartsWith("$kEnd=") }
|
||||||
|
Say "$svc env after: $($back.Count) entries ($($kept.Count) pre-existing hash-identical), pepper [$($pep2 -join ',')], telemetry: $($added -join ' ')"
|
||||||
|
Restart-App $svc $port
|
||||||
|
} catch {
|
||||||
|
Say "FAIL $svc : $($_.Exception.Message) -- restoring the original environment"
|
||||||
|
Write-Env $loc $old
|
||||||
|
try { Restart-App $svc $port } catch { Say "ROLLBACK RESTART FAILED $svc : $($_.Exception.Message)" }
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Say 'all done'
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent.
|
||||||
|
.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window.
|
||||||
|
.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP).
|
||||||
|
#>
|
||||||
|
param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20)
|
||||||
|
$ErrorActionPreference = 'Continue'
|
||||||
|
$root = 'E:\ApiInstall\OtelCollector'
|
||||||
|
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||||
|
$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'"
|
||||||
|
Say "service $($s.State) $($s.StartMode) $($s.StartName)"
|
||||||
|
try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" }
|
||||||
|
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||||
|
if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||||
|
" collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) }
|
||||||
|
Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime }
|
||||||
|
" E: free GB " + [int]((Get-PSDrive E).Free/1GB)
|
||||||
|
|
||||||
|
$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000
|
||||||
|
function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue }
|
||||||
|
foreach ($sig in 'metrics','traces','logs') {
|
||||||
|
$f = "$root\data\$sig.jsonl"
|
||||||
|
if (-not (Test-Path $f)) { Say "${sig}: no file"; continue }
|
||||||
|
$agg = @{}
|
||||||
|
foreach ($line in (Get-Content $f -Tail $Tail)) {
|
||||||
|
try { $o = $line | ConvertFrom-Json } catch { continue }
|
||||||
|
$rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs }
|
||||||
|
foreach ($r in $rs) {
|
||||||
|
$key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role')
|
||||||
|
if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } }
|
||||||
|
$a = $agg[$key]
|
||||||
|
if ($sig -eq 'metrics') {
|
||||||
|
foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1
|
||||||
|
foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } }
|
||||||
|
} elseif ($sig -eq 'traces') {
|
||||||
|
foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } }
|
||||||
|
} else {
|
||||||
|
foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Say "$sig (last $Tail export requests):"
|
||||||
|
foreach ($k in ($agg.Keys | Sort-Object)) {
|
||||||
|
$a = $agg[$k]
|
||||||
|
$newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' }
|
||||||
|
$fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' }
|
||||||
|
" $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh"
|
||||||
|
if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n"
|
||||||
|
$jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name
|
||||||
|
Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" }
|
||||||
|
} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" }
|
||||||
|
try {
|
||||||
|
$q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n"
|
||||||
|
Say "collector self-metrics:"
|
||||||
|
$q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') }
|
||||||
|
} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" }
|
||||||
|
foreach ($port in $AppPorts) {
|
||||||
|
$c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} }
|
||||||
|
" app :$port /metrics $c"
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54).
|
||||||
|
# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA),
|
||||||
|
# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector.
|
||||||
|
#
|
||||||
|
# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317,
|
||||||
|
# and nothing is reachable from the network. That is why the receivers have no auth and no TLS.
|
||||||
|
# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md).
|
||||||
|
#
|
||||||
|
# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as
|
||||||
|
# deployment.environment.name, so files copied off two tiers can still be told apart.
|
||||||
|
#
|
||||||
|
# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older
|
||||||
|
# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB.
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
health_check:
|
||||||
|
endpoint: 127.0.0.1:13133
|
||||||
|
|
||||||
|
receivers:
|
||||||
|
otlp:
|
||||||
|
protocols:
|
||||||
|
grpc:
|
||||||
|
endpoint: 127.0.0.1:4317
|
||||||
|
http:
|
||||||
|
endpoint: 127.0.0.1:4318
|
||||||
|
|
||||||
|
processors:
|
||||||
|
memory_limiter:
|
||||||
|
check_interval: 5s
|
||||||
|
limit_mib: 256
|
||||||
|
spike_limit_mib: 64
|
||||||
|
resource/tier:
|
||||||
|
attributes:
|
||||||
|
- key: deployment.environment.name
|
||||||
|
value: ${env:ZB_TIER}
|
||||||
|
action: upsert
|
||||||
|
batch: {}
|
||||||
|
|
||||||
|
exporters:
|
||||||
|
file/metrics:
|
||||||
|
path: E:/ApiInstall/OtelCollector/data/metrics.jsonl
|
||||||
|
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||||
|
file/traces:
|
||||||
|
path: E:/ApiInstall/OtelCollector/data/traces.jsonl
|
||||||
|
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||||
|
file/logs:
|
||||||
|
path: E:/ApiInstall/OtelCollector/data/logs.jsonl
|
||||||
|
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||||
|
prometheus:
|
||||||
|
endpoint: 127.0.0.1:8889
|
||||||
|
resource_constant_labels:
|
||||||
|
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name]
|
||||||
|
metric_expiration: 15m
|
||||||
|
|
||||||
|
service:
|
||||||
|
extensions: [health_check]
|
||||||
|
telemetry:
|
||||||
|
logs:
|
||||||
|
level: info
|
||||||
|
metrics:
|
||||||
|
level: basic
|
||||||
|
readers:
|
||||||
|
- pull:
|
||||||
|
exporter:
|
||||||
|
prometheus:
|
||||||
|
host: 127.0.0.1
|
||||||
|
port: 8888
|
||||||
|
pipelines:
|
||||||
|
metrics:
|
||||||
|
receivers: [otlp]
|
||||||
|
processors: [memory_limiter, resource/tier, batch]
|
||||||
|
exporters: [file/metrics, prometheus]
|
||||||
|
traces:
|
||||||
|
receivers: [otlp]
|
||||||
|
processors: [memory_limiter, resource/tier, batch]
|
||||||
|
exporters: [file/traces]
|
||||||
|
logs:
|
||||||
|
receivers: [otlp]
|
||||||
|
processors: [memory_limiter, resource/tier, batch]
|
||||||
|
exporters: [file/logs]
|
||||||
Reference in New Issue
Block a user