docs(observability): Telemetry 0.3.0 published, adopted by all four apps, logs over OTLP proven (#55)

Evidence doc gains the #55 section (root cause of no logs, 0.3.0 contents + SHA256,
per-app commits, collector output per app: service.version, site.id, node.role and
log records on OtOpcUa, HistorianGateway, ScadaBridge Site and MxAccessGateway).
GAPS, current-state, SPEC, the Telemetry CLAUDE.md and the index row updated;
infra/otel-collector gains otelidentity.py (Resource per signal + log samples).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 21:28:50 -05:00
parent 2011c8d6b9
commit f05ce80269
10 changed files with 247 additions and 27 deletions
+1 -1
View File
@@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See
| Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
| UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
| Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
| Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. **OTLP proven end to end 2026-11-12 (#13, closed):** shared dev collector `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)); metrics + traces from all four apps (ScadaBridge Site role; Central → ScadaBridge#133); OtOpcUa Galaxy driver telemetry was missing from the allowlists → fixed lmxopcua#892 (`master` `db86b996`, not deployed); no lib change, so `0.2.1` stays unpublished. Open: wonder-tier collector #54; `service.version`/`site.id`/`node.role` + logs-over-OTLP #55. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). Open: wonder-tier collector #54. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
+32 -14
View File
@@ -2,7 +2,7 @@
Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); no library change was needed, so `0.2.1` stays unpublished. Open: #54 (wonder-tier collector), #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP). The history below is a record, not a status tracker; file new follow-ups as issues.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP) is **closed** by `0.3.0` (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues.
The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation.
@@ -14,8 +14,8 @@ The library normalizes the three-project observability surface: a shared OpenTel
| Package | Responsibilities | Key Dependencies |
|---|---|---|
| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry` — wires OTel SDK (metrics + tracing), populates shared Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`, `host.name`), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. `app.MapZbMetrics()` — mounts `/metrics`. `ZbTelemetryOptions` — the single options object shared by both packages. | `Microsoft.AspNetCore.App` (framework ref), `OpenTelemetry.*` stack |
| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog` — shared two-stage Serilog bootstrap: `ReadFrom.Configuration`-driven sinks, `MinimumLevel.Is(Information)` default (config-overridable), `SiteId`/`NodeRole`/`NodeHostname` enrichers from `ZbTelemetryOptions`, `TraceContextEnricher` (writes `trace_id`/`span_id` from `Activity.Current`), `ILogRedactor` seam (per-project sensitive-field redaction via `RedactionEnricher`). **0.2.0:** optional `ZbSerilogOptions` overload — `MinimumLevelGovernor` (app key outranks `Serilog:MinimumLevel`) + `ConfigureLogger` (final filters/enrichers/sinks). Does NOT freeze `Log.Logger` — safe for multi-host/test scenarios. | `ZB.MOM.WW.Telemetry`, `Serilog.*` stack |
| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry` — wires OTel SDK (metrics + tracing), populates shared Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`, `host.name`), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. `app.MapZbMetrics()` — mounts `/metrics`. `ZbTelemetryOptions` — the single options object shared by both packages. **0.3.0:** `service.version` defaults from the entry assembly's informational version; `ResourceAttributes`; `ApplyConfiguration(<App>:Telemetry section)`; publishes its options in DI for the Serilog log sink. | `Microsoft.AspNetCore.App` (framework ref), `OpenTelemetry.*` stack |
| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog` — shared two-stage Serilog bootstrap: `ReadFrom.Configuration`-driven sinks, `MinimumLevel.Is(Information)` default (config-overridable), `SiteId`/`NodeRole`/`NodeHostname` enrichers from `ZbTelemetryOptions`, `TraceContextEnricher` (writes `trace_id`/`span_id` from `Activity.Current`), `ILogRedactor` seam (per-project sensitive-field redaction via `RedactionEnricher`). **0.2.0:** optional `ZbSerilogOptions` overload — `MinimumLevelGovernor` (app key outranks `Serilog:MinimumLevel`) + `ConfigureLogger` (final filters/enrichers/sinks). **0.3.0:** the OTLP log sink follows the `AddZbTelemetry` registration in the same container (`ExportLogs=false` opts out). Does NOT freeze `Log.Logger` — safe for multi-host/test scenarios. | `ZB.MOM.WW.Telemetry`, `Serilog.*` stack |
---
@@ -23,14 +23,13 @@ The library normalizes the three-project observability surface: a shared OpenTel
| Consumer | `ZB.MOM.WW.Telemetry` (core) | `ZB.MOM.WW.Telemetry.Serilog` |
|---|:---:|:---:|
| **OtOpcUa** | ✅ adopted | ✅ adopted (`AddZbSerilog`) |
| **MxAccessGateway** | ✅ adopted (`GatewayMetrics` exported) | ✅ adopted (MEL→Serilog migrated in this pass) |
| **ScadaBridge** | ✅ adopted (both roots) | ✅ `AddZbSerilog` **0.2.0** (2026-11-12, ScadaBridge#94, `main` `2d2e9f35`) — `MinimumLevelGovernor` + `ConfigureLogger` (standby filter); run on a pre-host builder so the static `Log.Logger` stays the one process logger |
| **HistorianGateway** | ✅ adopted | ✅ adopted (`AddZbSerilog`) |
| **OtOpcUa** | ✅ **0.3.0** (`master` `412da27c`) — `site.id` = Cluster id, `node.role` = plain roles | ✅ **0.3.0** (`AddZbSerilog`; logs follow `AddZbTelemetry`) |
| **MxAccessGateway** | ✅ **0.3.0** (`main` `d9d9ff48`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
| **ScadaBridge** | ✅ **0.3.0** (`main` `a633005f`, both roots) | ✅ **0.3.0** `AddZbSerilog` with `MinimumLevelGovernor` + `ConfigureLogger` on a pre-host builder; the exporter is passed explicitly via `ScadaBridgeTelemetryConfig` |
| **HistorianGateway** | ✅ **0.3.0** (`main` `ae1dbcc`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
All three adopted on 2026-06-01 (branch `feat/adopt-zb-telemetry` per repo); HistorianGateway adopted since. OtOpcUa, MxAccessGateway and HistorianGateway pin both packages at `0.1.0` on their origin default branches
(verified 2026-11-09; their `AddZbSerilog(o => …)` call sites are source-compatible with 0.2.0 — the new overload is
additive). ScadaBridge pins **`0.2.0`** since 2026-11-12. See `components/observability/GAPS.md` for the full result.
All four pin both packages at **`0.3.0`** on their origin default branches since 2026-11-12 (scadaproj#55; not
deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See `components/observability/GAPS.md` for the full result.
---
@@ -54,9 +53,9 @@ All test assemblies run offline:
| Assembly | Tests |
|---|---|
| `ZB.MOM.WW.Telemetry.Tests` | 12 |
| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 30 |
| **Total** | **42** |
| `ZB.MOM.WW.Telemetry.Tests` | 38 |
| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 39 |
| **Total** | **77** |
`GeneratePackageOnBuild` is off — pack explicitly with the command above.
@@ -64,7 +63,26 @@ All test assemblies run offline:
## Status
**`0.2.1` (2026-11-09, scadaproj#48) — built, NOT published.** P3 review fixes: `RedactionEnricher` is now registered
**`0.3.0` (2026-11-12, scadaproj#55) — PUBLISHED** (both packages, packed from the tree of scadaproj `main` `334c82c` (committed as `df9a328` before rebasing onto main); 38 + 39
tests green in Release). SHA256 (feed download = local pack): `ZB.MOM.WW.Telemetry.0.3.0.nupkg`
`a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2`, `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg`
`04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c`. Adopted by all four apps the same day.
- `service.version`: `ServiceVersion = null` (default) now resolves to the entry assembly's
`AssemblyInformationalVersion` (else its assembly version); `""` omits; any value overrides. **Behaviour
change** from 0.2.x, where null omitted the attribute.
- `ResourceAttributes` (extra Resource attributes on all signals; first-class keys reserved, rejected at startup).
- `ApplyConfiguration(IConfiguration section)` — `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`,
`NodeRole`, `ExportLogs`, `ResourceAttributes:<key>`; absent keys keep code defaults.
- **Logs over OTLP**: `AddZbTelemetry` registers its options in DI. The `AddZbSerilog` OTLP log sink
follows them when its own options do not select OTLP. Root cause of "no app ever exported a log": the
sink keyed only on the `AddZbSerilog` options, which every app filled with the service name alone.
`ExportLogs = false` keeps logs local.
- Includes the never-published **0.2.1** fixes below, including the stricter OTLP endpoint check. That check
was shipped deliberately: no checked-in config sets an endpoint, it only runs when `Exporter=Otlp`, and no
wonder tier has a collector (#54).
- Feed versions: `0.1.0`, `0.2.0`, `0.3.0` (`0.2.1` was never published).
**`0.2.1` (2026-11-09, scadaproj#48) — never published; shipped inside `0.3.0`.** P3 review fixes: `RedactionEnricher` is now registered
AFTER `ConfigureLogger`, so enrichers added in the hook are redacted (009); dictionary keys that print identically are
all kept (later ones get a `#n` suffix in the redactor's view; the original key is restored on rebuild) (010); the OTLP
endpoint must be `http`/`https` with a host (`collector:4317` is now rejected) and the pre-built-options
@@ -165,9 +165,143 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers.
## Telemetry 0.2.1: not published
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
built-but-unpublished until a pin bump wants it.
---
# Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day)
Closes the three #55 gaps found above: no `service.version` anywhere, no `site.id`/`node.role` on OtOpcUa,
MxAccessGateway and HistorianGateway, and no logs over OTLP from any app.
## Root cause of "no logs"
`ZbSerilogConfig.ApplyOpenTelemetryExport` adds the `WriteTo.OpenTelemetry` sink only when the options
passed **to `AddZbSerilog`** say `Exporter=Otlp`. Every app filled those options with the service name alone
and set the exporter only on `AddZbTelemetry`, a separate options object the Serilog package never saw. The
sink condition could therefore never be true.
## Library: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.0 — published
scadaproj `main` `b34049b` + `334c82c`. Feed SHA256 (feed download = local pack):
| Package | SHA256 |
|---|---|
| `ZB.MOM.WW.Telemetry.0.3.0.nupkg` | `a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2` |
| `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` | `04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c` |
- **`service.version`**: `ServiceVersion = null` (the default) now means the entry assembly's
`AssemblyInformationalVersion` (the git SHA the apps stamp), falling back to its assembly version. Set a
value to override it; `""` omits the attribute.
- **`ResourceAttributes`**: extra Resource attributes on all three signals. The first-class keys
(`service.*`, `host.name`, `site.id`, `node.role`) are reserved and rejected at startup.
- **`ApplyConfiguration(section)`**: reads `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, `NodeRole`,
`ExportLogs`, `ResourceAttributes:<key>` from an app's own `<App>:Telemetry` section. Absent keys keep the
code defaults, so the four hand-written exporter parsers are gone.
- **Logs follow `AddZbTelemetry`**: `AddZbTelemetry` registers its options in DI. `AddZbSerilog` resolves them
inside its `AddSerilog` callback (so call order does not matter), and when its own options do not select
OTLP, the OTLP log sink uses that registration's endpoint and Resource. `ExportLogs = false` keeps logs
local. Levels and the configuration-driven sinks are unchanged.
- **0.2.1 shipped in the same release.** It holds the P3 review fixes: redaction after `ConfigureLogger`,
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
wonder tier has a collector yet (#54), so no deployed box can trip it.
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
Mutation checks, each caught by the named tests:
- Drop the DI lookup → the e2e log test fails.
- Drop the version default → the pipeline-Resource and e2e tests fail.
- Drop `ResourceAttributes` → 2 attribute tests and the e2e test fail.
- Ignore `ExportLogs` → the decision test and the e2e test fail.
- Disable the reserved-key check → 7 theory cases fail.
- Drop `ApplyConfiguration`'s attribute or `SiteId` read → its test fails.
## Apps (all pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Identity wiring |
|---|---|---|---|
| OtOpcUa | `master` `412da27c` | 0.1.0 → 0.3.0 | `site.id` = the application Cluster id (the `cluster-<ClusterId>` role suffix; omitted on a legacy node); `node.role` = the plain roles, sorted (`admin,driver`); Program passes its `EffectiveRoles`. |
| MxAccessGateway | `main` `d9d9ff48` | 0.1.0 → 0.3.0 (csproj) | `node.role=standalone` (one gateway per box); `site.id` from `MxGateway:Telemetry:SiteId` (there is no node identity in its config, so it is omitted unless set). |
| HistorianGateway | `main` `ae1dbcc` | 0.1.0 → 0.3.0 (csproj) | Same as mxgw, under the root-level `Telemetry:` section. |
| ScadaBridge | `main` `a633005f` | 0.2.0 → 0.3.0 | Unchanged `site.id`/`node.role`. The new `ScadaBridgeTelemetryConfig` fills both `AddZbTelemetry` and the pre-host `AddZbSerilog` logger. The logger is built on its own container with no `AddZbTelemetry` to follow, so the exporter is passed to it explicitly. |
Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452
(6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity
tests (OtOpcUa `ObservabilityResourceIdentityTests`, HG `GatewayTracingTests`, ScadaBridge
`ScadaBridgeTelemetryConfigTests`).
## What arrived (collector, [`otelidentity.py`](../../infra/otel-collector/otelidentity.py), one process each)
All runs: `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`, `OTEL_METRIC_EXPORT_INTERVAL=10000`.
**OtOpcUa** — local Release host, fused `admin,driver,cluster-OTEL55` single node. Its own Akka mesh on
`localhost:14053`. Throwaway ConfigDb `OtOpcUaOtel55` on the docker-dev SQL: EF-migrated, one
`ServerCluster` `OTEL55` + one `ClusterNode`, gRPC mesh/telemetry keys of its own. The DB was **dropped
afterwards**. Activity: one `POST /api/deployments` (202) plus health/metrics GETs.
```
RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12}
LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"}
LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"}
```
The `otopcua.deploy.apply` and `otopcua.opcua.address_space_rebuild` spans arrived again.
**HistorianGateway** — local Release host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores,
`Telemetry__SiteId=otel-probe-55`. Activity: `HistorianStatus/Probe` + health GETs.
```
RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"}
```
**ScadaBridge (Site)** — local Release **Site** host, own site id `otel-probe-55`, own 1-node mesh (`localhost:18082`),
scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable.
```
RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"}
```
The first line, `Starting ScadaBridge host …`, is written to the static logger before any web host exists, so
the explicit exporter path on the pre-host logger works. **Central not run:** the rig was in use by the
ScadaBridge#132 failover drills (`sb132-p` / `sb132-probe` running, centrals recreated 3–6 minutes earlier), so
ScadaBridge#133 stays open.
**MxAccessGateway** — Release `main` `54aa61a0`, published framework-dependent win-x64 from the Mac. (It was
pushed as `d9d9ff48` after rebasing over one docs-only commit.) Run as a scratch copy at `C:\otel55\mxgw` on
windev, on `:15120`/`:15130`, under `Staging` + `DisableLogin`, with alarms off, scratch auth/secrets/snapshot
stores and a throwaway pepper. `Worker:ExecutablePath` pointed at the non-existent
`C:\otel55\OtelProbeNoWorker.exe`. It was launched through `Win32_Process.Create`, because a process started
from the ssh session dies with it. `MxGateway__Telemetry__SiteId=otel-probe-55`.
```
RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12}
LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."}
LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"}
```
The live `MxAccessGw` service stayed `Running` with `:5130/health/ready` = 200 throughout. Its gateway PID
(9148) and worker PIDs (`14700,22108`) were unchanged before and after. The scratch copy and `C:\otel55` were
deleted.
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
backend. The Resource is byte-identical across the three signals of each process.
+12 -3
View File
@@ -29,6 +29,14 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway. One defect fixed: OtOpcUa's
> Galaxy driver meter/source were missing from the allowlists (lmxopcua#892). Evidence:
> [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
>
> **✅ RESOURCE IDENTITY + LOGS OVER OTLP — 2026-11-12 (#55, closed).** Telemetry(.Serilog) **`0.3.0`**
> (published; ships the 0.2.1 review fixes): `service.version` defaults from the entry assembly's
> informational version (the git SHA), `ResourceAttributes` + `ApplyConfiguration(<App>:Telemetry)`, and the
> Serilog OTLP log sink follows `AddZbTelemetry`'s exporter. All four apps pin `0.3.0` and report
> `service.version`, `site.id` and `node.role` (OtOpcUa: Cluster id / plain roles; mxgw + HG: `standalone`
> + configurable `SiteId`; ScadaBridge unchanged) and send logs — proven at the collector for OtOpcUa,
> HistorianGateway, ScadaBridge (Site) and MxAccessGateway. Central role still → ScadaBridge#133.
## Divergence vs spec
@@ -267,7 +275,8 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
Prometheus-only. → #54
- **Resource identity + logs-over-OTLP gaps** found by #13: `service.version` absent on all four apps;
`site.id`/`node.role` absent on OtOpcUa, MxAccessGateway, HistorianGateway; no app passes the exporter
settings to `AddZbSerilog`, so logs never travel over OTLP. → #55
- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry
`0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP
(OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the
#55 section of [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81
@@ -5,6 +5,12 @@
> OTLP export of **metrics and traces** is proven end to end (scratch copy of the deployed windev build; the
> `mxgateway.*` gauges and ASP.NET Core spans arrived). The gateway still has no `ActivitySource` of its own.
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
>
> **2026-11-12 update (scadaproj#55):** closed on `main` `d9d9ff48` (Telemetry `0.3.0`). The Resource now
> carries `service.version` (entry-assembly informational version, e.g. `0.1.2+54aa61a0`), `node.role=standalone`
> (one gateway per box; `MxGateway:Telemetry:NodeRole` overrides) and `site.id` from
> `MxGateway:Telemetry:SiteId` (no natural value in the gateway's config, so it is omitted unless set).
> `MxGateway:Telemetry:Exporter=Otlp` now also sends the Serilog logs (proven: 21 log records).
Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**);
solution `src/MxGateway.sln`. Telemetry code is concentrated in
@@ -7,6 +7,13 @@
> Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in
> lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role`
> (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
>
> **2026-11-12 update (scadaproj#55):** closed on `master` `412da27c` (Telemetry `0.3.0`). The Resource now
> carries `service.version` (entry-assembly informational version, e.g. `1.0.0+412da27c…`), `site.id` = the
> node's application Cluster id (suffix of its `cluster-<ClusterId>` role; omitted on a legacy node) and
> `node.role` = its plain roles (`admin`, `driver`, `admin,driver`). `OtOpcUa:Telemetry:SiteId` / `:NodeRole` /
> `:ServiceVersion` / `:ResourceAttributes:*` override them. `OtOpcUa:Telemetry:Exporter=Otlp` now also sends
> the Serilog logs to the collector (proven: 73 log records with the same Resource).
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side
@@ -5,6 +5,12 @@
> end to end for a **Site** node (6 `scadabridge.*` instruments, `localdb.oplog.depth`, gRPC-client spans,
> with `site.id`/`node.role` on the Resource); the Central role is not yet proven (ScadaBridge#133).
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
>
> **2026-11-12 update (scadaproj#55):** closed on `main` `a633005f` (Telemetry `0.3.0`). `service.version` now
> on the Resource (entry-assembly informational version); `site.id`/`node.role` unchanged.
> `ScadaBridgeTelemetryConfig` fills both the `AddZbTelemetry` registration and the pre-host `AddZbSerilog`
> logger, so `ScadaBridge:Telemetry:Exporter=Otlp` also sends logs (proven on a Site node: 78 log records,
> including the pre-host startup line). Central still → ScadaBridge#133.
Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution
`ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref
+3 -2
View File
@@ -67,10 +67,11 @@ The OTel `Resource` attached to all three signals is built from `ZbTelemetryOpti
|---|---|---|
| `service.name` | `ServiceName` | Required. Lower-case short identifier (`otopcua`, `mxgateway`, `scadabridge`) |
| `service.namespace` | `ServiceNamespace` | Default `"ZB.MOM.WW"` — constant across the fleet |
| `service.version` | `ServiceVersion` | Optional; recommend populating from `AssemblyInformationalVersion` |
| `service.version` | `ServiceVersion` | Defaults (Telemetry 0.3.0+) to the entry assembly's `AssemblyInformationalVersion` — the git SHA every family app stamps; set a value to override, `""` to omit |
| `service.instance.id` | _(auto)_ | Always populated from `ZbResource.InstanceId` — a deterministic `MachineName:ProcessId`. The OTel SDK's random-GUID default is disabled so every signal from the same process carries an identical, restart-stable instance id for cross-signal correlation |
| `site.id` | `SiteId` | Optional; identifies the physical/logical site |
| `node.role` | `NodeRole` | Optional; e.g. `"central"`, `"site"`, `"hub"` |
| `node.role` | `NodeRole` | Optional; e.g. `"central"`, `"site"`, `"hub"`, `"standalone"` |
| _(any other)_ | `ResourceAttributes` | Optional extra attributes (0.3.0+); the keys above are reserved |
| `host.name` | _(auto)_ | Always populated from `Environment.MachineName` |
The same `SiteId` and `NodeRole` values are passed to the Serilog enrichers (§4) so a
+11 -6
View File
@@ -31,12 +31,14 @@ OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is
| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` |
| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) |
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less
`host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there.
- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.3.0` (what the apps pin since
2026-11-12) rejects a scheme-less `host:port` at startup when `Exporter=Otlp`.
- The SDK's default metric export interval is 60 s. For a quick check set
`OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process.
- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called
without the exporter settings (scadaproj#55).
- What travels: **metrics, traces and logs** (Telemetry `0.3.0`+, scadaproj#55): with `Exporter=Otlp` the
Serilog logs go to the same endpoint with the same Resource. `<prefix>:Telemetry:ExportLogs=false` keeps them
local. Every app's Resource carries `service.version` (git SHA), `site.id` and `node.role`; mxgw and
HistorianGateway report `node.role=standalone` and take `site.id` from `<prefix>:Telemetry:SiteId`.
## Reading what arrived
@@ -45,18 +47,21 @@ ssh dohertj2@10.100.0.35
docker logs --tail 20 zb-otel-collector # one debug line per received batch
cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names
python3 ../otelsum.py otopcua # one service
python3 ../otelidentity.py otopcua :12345 # Resource per signal + log samples, one process (instance-id suffix)
curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head
```
`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc.
(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc;
[`otelidentity.py`](otelidentity.py) prints the Resource each signal arrived with, the log count by severity
and sample log records, filtered to one process (scadaproj#55).
## Deploy / update
```bash
# from this directory on the Mac
ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data'
scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/
scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/
ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate'
```
+34
View File
@@ -0,0 +1,34 @@
# Usage (collector host, in ~/zb-otel-collector/data): python3 ../otelidentity.py <service.name> [instance-id-substring, e.g. :12345]
# Prints the Resource each signal (metrics/traces/logs) arrived with, counts, and sample log records (scadaproj#55).
import json,sys,collections
svc=sys.argv[1]; inst=sys.argv[2] if len(sys.argv)>2 else None
def attrs(lst): return {a["key"]:(list(a["value"].values())[0] if a.get("value") else None) for a in lst or []}
KEEP=("service.name","service.version","service.instance.id","site.id","node.role","host.name")
out={}
counts=collections.Counter(); samples=[]; sev=collections.Counter()
for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")):
try: lines=open(f)
except FileNotFoundError: continue
for l in lines:
d=json.loads(l)
for r in d.get(key,[]):
a=attrs(r["resource"].get("attributes"))
if a.get("service.name")!=svc: continue
if inst and inst not in str(a.get("service.instance.id")): continue
extra={k:v for k,v in a.items() if k not in KEEP and not k.startswith(("telemetry.sdk","process.","os."))}
out[key]={k:a.get(k) for k in KEEP}|({"extra":extra} if extra else {})
if key=="resourceMetrics":
counts["metric points"]+=sum(1 for sm in r["scopeMetrics"] for m in sm["metrics"])
elif key=="resourceSpans":
counts["spans"]+=sum(len(ss["spans"]) for ss in r["scopeSpans"])
else:
for sl in r["scopeLogs"]:
for lr in sl["logRecords"]:
counts["log records"]+=1; sev[lr.get("severityText","?")]+=1
la=attrs(lr.get("attributes"))
if len(samples)<4 or (lr.get("traceId") and not any(s[2] for s in samples)):
body=lr.get("body",{}).get("stringValue","")
samples.append((lr.get("severityText"),body[:110],lr.get("traceId",""),la.get("SourceContext") or la.get("message_template.text","")[:40]))
for k,v in out.items(): print("RESOURCE",k,json.dumps(v))
print("COUNTS",dict(counts),"SEVERITY",dict(sev))
for s in samples[:5]: print("LOG",json.dumps({"severity":s[0],"body":s[1],"traceId":s[2],"src":s[3]}))