feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)

- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 —
  OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889,
  health :13133; README runbook + otelsum.py summariser. Deployed.
- components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces
  received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway;
  OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996).
- GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups
  #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133
  (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 20:48:36 -05:00
parent f98ba08910
commit e1a2df83f7
12 changed files with 409 additions and 7 deletions
+5 -1
View File
@@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See
| Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
| UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
| Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | | Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
| Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. **OTLP proven end to end 2026-11-12 (#13, closed):** shared dev collector `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)); metrics + traces from all four apps (ScadaBridge Site role; Central → ScadaBridge#133); OtOpcUa Galaxy driver telemetry was missing from the allowlists → fixed lmxopcua#892 (`master` `db86b996`, not deployed); no lib change, so `0.2.1` stays unpublished. Open: wonder-tier collector #54; `service.version`/`site.id`/`node.role` + logs-over-OTLP #55. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
@@ -249,6 +249,10 @@ msbuild Database\Database.sqlproj /p:Configuration=Release # SSDT → bin\Rel
> single `zb-shared-glauth` container on the Linux fixture host **`10.100.0.35:3893`** > single `zb-shared-glauth` container on the Linux fixture host **`10.100.0.35:3893`**
> (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy runbook: > (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy runbook:
> [`scadaproj/infra/glauth/`](infra/glauth/) (`config.toml` + `docker-compose.yml` + `README.md`). > [`scadaproj/infra/glauth/`](infra/glauth/) (`config.toml` + `docker-compose.yml` + `README.md`).
>
> **Shared OTel collector (dev):** `zb-otel-collector` on the same host — OTLP gRPC `:4317` / HTTP `:4318`,
> Prometheus re-export `:8889`, health `:13133`. Point an app at it with `<App>:Telemetry:Exporter=Otlp` +
> `:OtlpEndpoint=http://10.100.0.35:4317`. Runbook: [`scadaproj/infra/otel-collector/`](infra/otel-collector/).
## Backlog / open work (family standard, 2026-11-06) ## Backlog / open work (family standard, 2026-11-06)
+2 -2
View File
@@ -2,7 +2,7 @@
Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application. Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #13 (OTel collector / OTLP end to end). The history below is a record, not a status tracker; file new follow-ups as issues. **Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); no library change was needed, so `0.2.1` stays unpublished. Open: #54 (wonder-tier collector), #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP). The history below is a record, not a status tracker; file new follow-ups as issues.
The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation. The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation.
@@ -88,7 +88,7 @@ Built at **0.1.0**, published to the Gitea NuGet feed, and **adopted across all
2026-06-01** (branch `feat/adopt-zb-telemetry` per repo, behaviour-preserving). MxAccessGateway's 2026-06-01** (branch `feat/adopt-zb-telemetry` per repo, behaviour-preserving). MxAccessGateway's
MEL→Serilog migration and metrics export both landed in this pass (not beforehand, despite the MEL→Serilog migration and metrics export both landed in this pass (not beforehand, despite the
earlier claim). Deferred follow-ons (MxGateway `ms`→`s` + Meter rename, ScadaBridge app instruments earlier claim). Deferred follow-ons (MxGateway `ms`→`s` + Meter rename, ScadaBridge app instruments
+ Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook, #13 OTLP collector): + Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook — closed; #13 OTLP collector — closed 2026-11-12):
- `~/Desktop/scadaproj/components/observability/GAPS.md` — adoption status + deferred follow-ons - `~/Desktop/scadaproj/components/observability/GAPS.md` — adoption status + deferred follow-ons
@@ -0,0 +1,173 @@
# OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
First run of the family's OTLP export path against a real collector. Before this, every app had
the `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from
it, and OtOpcUa's spans had no exporter anywhere.
**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site
role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished.
## Collector
`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`,
OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file`
exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The
four ports were free on the host before deploy, and no other container was touched.
```
$ curl -s http://10.100.0.35:13133/
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
200
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
True
```
## How each app was run
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and
`OTEL_METRIC_EXPORT_INTERVAL=10000`.
| App | Build | Telemetry pin | How |
|---|---|---|---|
| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. |
| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. |
## What arrived (collector `otelsum.py` output, trimmed)
Standard instrumentation scopes that every app sent are listed once at the end.
### OtOpcUa — metrics + traces
```
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
```
```json
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
```
OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend
with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply`
(no parent — the context does not cross the Akka actor hop); the two join on
`otopcua.correlation_id`.
### HistorianGateway — metrics + traces
```
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
```
`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException`
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
```
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
```
### ScadaBridge (Site) — metrics + traces
```
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
```
The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own,
so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133.
### MxAccessGateway — metrics + traces
```
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
```
Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …)
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not
reachable from off-box (windev firewall), so the traffic was generated on-box.
### Standard instrumentation (all four)
`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`),
`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`,
`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`;
plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard.
## Defect found and fixed
**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters`
and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the
LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`,
`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on
ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*`
counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP.
Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in
the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it.
`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not
driven: that would need a Galaxy driver deployment against a gateway.
The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(`
in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb),
HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway
(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter.
## Checked, no defect
- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317`
works for all four. No checked-in config carries a scheme-less endpoint.
- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span)
(the overlay is additive, as documented).
- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name`
present on all four.
## Gaps filed (not fixed here)
- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent
on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
(`logs.jsonl` stayed empty).
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers.
## Telemetry 0.2.1: not published
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
built-but-unpublished until a pin bump wants it.
+15 -4
View File
@@ -22,7 +22,13 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the > listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously > "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there > contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
> (deploying an OTel collector → #13; further ScadaBridge instruments → ScadaBridge#81) remain. > (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
>
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
> from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway. One defect fixed: OtOpcUa's
> Galaxy driver meter/source were missing from the allowlists (lmxopcua#892). Evidence:
> [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
## Divergence vs spec ## Divergence vs spec
@@ -256,7 +262,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
**Still open (not code — operational/future):** **Still open (not code — operational/future):**
- **OTLP is opt-in but unexercised** (→ #13) until an OTel collector endpoint is deployed and the - ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
`<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` config is set. The wiring is in place; only a `zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
collector is missing. ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
Prometheus-only. → #54
- **Resource identity + logs-over-OTLP gaps** found by #13: `service.version` absent on all four apps;
`site.id`/`node.role` absent on OtOpcUa, MxAccessGateway, HistorianGateway; no app passes the exporter
settings to `AddZbSerilog`, so logs never travel over OTLP. → #55
- **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81 - **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81
@@ -1,5 +1,11 @@
# Observability — current state: MxAccessGateway # Observability — current state: MxAccessGateway
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. Since then the gateway
> exports its metrics via `AddZbTelemetry`; the meter is `ZB.MOM.WW.MxGateway` (seconds), and logging is Serilog.
> OTLP export of **metrics and traces** is proven end to end (scratch copy of the deployed windev build; the
> `mxgateway.*` gauges and ASP.NET Core spans arrived). The gateway still has no `ActivitySource` of its own.
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**); Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**);
solution `src/MxGateway.sln`. Telemetry code is concentrated in solution `src/MxGateway.sln`. Telemetry code is concentrated in
`src/ZB.MOM.WW.MxGateway.Server/Metrics/` (instruments) and `src/ZB.MOM.WW.MxGateway.Server/Metrics/` (instruments) and
@@ -1,5 +1,13 @@
# Observability — current state: OtOpcUa # Observability — current state: OtOpcUa
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls
> `AddZbTelemetry` (Resource + OTLP opt-in), and OTLP export of **metrics and traces** is proven end to end
> against the dev collector (`otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`, `otopcua.deploy.*`
> metrics arrived). That ends the "tracing no-op" below once `OtOpcUa:Telemetry:Exporter=Otlp` is set. The
> Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in
> lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role`
> (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`. Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side
bootstrap) and `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/` (instruments + enricher). bootstrap) and `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/` (instruments + enricher).
@@ -1,5 +1,11 @@
# Observability — current state: ScadaBridge # Observability — current state: ScadaBridge
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. ScadaBridge now
> has the `ZB.MOM.WW.ScadaBridge` meter and `AddZbSerilog`. OTLP export of **metrics and traces** is proven
> end to end for a **Site** node (6 `scadabridge.*` instruments, `localdb.oplog.depth`, gRPC-client spans,
> with `site.id`/`node.role` on the Resource); the Central role is not yet proven (ScadaBridge#133).
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution
`ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref `ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref
(metrics/traces) and a substantive Serilog setup (logs). All paths relative to repo root. (metrics/traces) and a substantive Serilog setup (logs). All paths relative to repo root.
+1
View File
@@ -0,0 +1 @@
data/
+69
View File
@@ -0,0 +1,69 @@
# Shared dev OpenTelemetry collector (`zb-otel-collector`)
One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib)
that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the
shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the
OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run:
[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md).
> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA
> vq03/vq04) has a collector yet — that is scadaproj#54.
## Endpoints
| Port | What | Use |
|---|---|---|
| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. |
| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). |
| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=<service.namespace>/<service.name>`, `instance=<service.instance.id>` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. |
| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` |
Reachable from the Mac and from windev (`10.100.0.48`) — both verified.
## Pointing an app at it
OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on.
| App | Config keys (env form) |
|---|---|
| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less
`host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there.
- The SDK's default metric export interval is 60 s. For a quick check set
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called
without the exporter settings (scadaproj#55).
## Reading what arrived
```bash
ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua # one service
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
```
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc.
## Deploy / update
```bash
# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
```
- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or
`sudo chown 10001:10001 data`).
- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check
and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this
release; the config uses its replacement `resource_constant_labels`.
- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no
network or volume with any other container. Tear down with `docker compose down` there.
+65
View File
@@ -0,0 +1,65 @@
# zb-otel-collector — dev/test collector for the ZB.MOM.WW family (scadaproj#13).
# Receives OTLP (gRPC 4317 + HTTP 4318) from OtOpcUa, ScadaBridge, MxAccessGateway and
# HistorianGateway and proves receipt three ways:
# * debug exporter -> `docker logs zb-otel-collector` (one summary line per batch)
# * file exporters -> /data/{metrics,traces,logs}.jsonl (OTLP-JSON, rotated)
# * prometheus exporter-> :8889/metrics, every received metric re-exposed with
# job/instance + site_id/node_role/host_name labels (resource_constant_labels)
# No auth on the receivers: dev network only. Do not expose this config to a production tier.
extensions:
health_check:
endpoint: 0.0.0.0:13133
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 5s
limit_mib: 512
spike_limit_mib: 128
batch: {}
exporters:
debug:
verbosity: basic
file/metrics:
path: /data/metrics.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
file/traces:
path: /data/traces.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
file/logs:
path: /data/logs.jsonl
rotation: { max_megabytes: 50, max_backups: 3 }
prometheus:
endpoint: 0.0.0.0:8889
# service.name/namespace become job/instance automatically; also lift the family identity triple.
resource_constant_labels:
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name]
metric_expiration: 15m
service:
extensions: [health_check]
telemetry:
logs:
level: info
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/metrics, prometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/traces]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, file/logs]
+31
View File
@@ -0,0 +1,31 @@
# Shared dev OpenTelemetry collector for the ZB.MOM.WW family (scadaproj#13).
# Deploy on the shared Docker host 10.100.0.35:
# mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data, copy docker-compose.yml,
# config.yaml and otelsum.py into ~/zb-otel-collector/ (README "Deploy / update")
# cd ~/zb-otel-collector && docker compose up -d
# Verify: curl -s http://10.100.0.35:13133/ -> {"status":"Server available",...}
# curl -s http://10.100.0.35:8889/metrics -> received metrics, re-exposed for Prometheus
# tail -f ~/zb-otel-collector/data/traces.jsonl (also metrics.jsonl / logs.jsonl)
name: zb-otel-collector
services:
otel-collector:
# Pinned exact release (published 2026-10-05). Bump deliberately; never :latest.
image: otel/opentelemetry-collector-contrib:0.162.0
container_name: zb-otel-collector
restart: unless-stopped
# The image runs as uid 10001; the bind-mounted data dir must be writable by it
# (README: `sudo chown 10001:10001 data` or `chmod 777 data`).
command: ["--config=/etc/otelcol-contrib/config.yaml"]
ports:
- "4317:4317" # OTLP gRPC (family default: <App>:Telemetry:OtlpEndpoint=http://10.100.0.35:4317)
- "4318:4318" # OTLP HTTP/protobuf
- "8889:8889" # Prometheus exporter: everything received over OTLP, re-exposed for scraping
- "13133:13133" # health_check extension
volumes:
- ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
- ./data:/data
logging:
driver: json-file
options:
max-size: "20m"
max-file: "3"
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env python3
# Summarise what zb-otel-collector received: per service.name, the Resource attributes, the
# metric names per instrumentation scope, and span counts per (scope, name).
# Usage on the collector host: cd ~/zb-otel-collector/data && python3 ../otelsum.py [service.name]
import json,sys,collections
svc_filter=sys.argv[1] if len(sys.argv)>1 else None
def attrs(r): return {a["key"]:list(a["value"].values())[0] if a["value"] else None for a in r.get("attributes",[])}
res={}; mets=collections.defaultdict(set); spans=collections.Counter()
for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")):
try: lines=open(f)
except FileNotFoundError: continue
for l in lines:
d=json.loads(l)
for r in d.get(key,[]):
a=attrs(r["resource"]); s=a.get("service.name")
if svc_filter and s!=svc_filter: continue
res[s]={k:v for k,v in a.items() if not k.startswith("telemetry.sdk")} | {"_sdk":a.get("telemetry.sdk.version")}
if key=="resourceMetrics":
for sm in r["scopeMetrics"]:
for m in sm["metrics"]: mets[(s,sm["scope"]["name"])].add(m["name"])
elif key=="resourceSpans":
for ss in r["scopeSpans"]:
for sp in ss["spans"]: spans[(s,ss["scope"]["name"],sp["name"])]+=1
else:
for sl in r["scopeLogs"]: spans[(s,"<logs>",sl["scope"].get("name",""))]+=len(sl["logRecords"])
for s,a in res.items(): print("RESOURCE",s,json.dumps(a))
for (s,sc),names in sorted(mets.items()): print("METRICS",s,sc,len(names),sorted(names))
for k,v in sorted(spans.items()): print("SPANS",v,k)