[Telemetry] Resource identity gaps and OTLP logs never enabled (found exercising OTLP, #13) #55

Closed
opened 2026-11-12 20:45:12 -05:00 by dohertj2 · 1 comment
Owner

Found by #13 reading what actually reached the collector (components/observability/2026-11-12-otlp-end-to-end-evidence.md):

  1. service.version is absent from every app's Resource (OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). SPEC §1 recommends it from AssemblyInformationalVersion; every app already stamps the git SHA there. Options: each app sets o.ServiceVersion, or the lib defaults it from the entry assembly when null (a behaviour change → needs a Telemetry release + pin bumps).
  2. site.id / node.role absent for OtOpcUa, MxAccessGateway, HistorianGateway — only ScadaBridge sets them (site.id=otel-probe, node.role=Site arrived). OtOpcUa has a natural value for both (the cluster-<ClusterId> role and admin/driver roles). Without them two OtOpcUa nodes are distinguishable only by service.instance.id (= MachineName:PID).
  3. No app exports logs over OTLP. ZB.MOM.WW.Telemetry.Serilog adds a WriteTo.OpenTelemetry sink only when the Serilog options carry Exporter=Otlp + OtlpEndpoint, but every app calls AddZbSerilog(o => o.ServiceName = ...) with the service name only, so <App>:Telemetry:Exporter=Otlp switches metrics + traces but never logs (logs.jsonl stayed empty for all four). Wire the same exporter/endpoint into the AddZbSerilog options, or document logs-over-OTLP as out of scope.

Per-repo issues once a direction is chosen.

Found by #13 reading what actually reached the collector (`components/observability/2026-11-12-otlp-end-to-end-evidence.md`): 1. **`service.version` is absent from every app's Resource** (OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). SPEC §1 recommends it from `AssemblyInformationalVersion`; every app already stamps the git SHA there. Options: each app sets `o.ServiceVersion`, or the lib defaults it from the entry assembly when null (a behaviour change → needs a Telemetry release + pin bumps). 2. **`site.id` / `node.role` absent for OtOpcUa, MxAccessGateway, HistorianGateway** — only ScadaBridge sets them (`site.id=otel-probe`, `node.role=Site` arrived). OtOpcUa has a natural value for both (the `cluster-<ClusterId>` role and admin/driver roles). Without them two OtOpcUa nodes are distinguishable only by `service.instance.id` (= MachineName:PID). 3. **No app exports logs over OTLP.** `ZB.MOM.WW.Telemetry.Serilog` adds a `WriteTo.OpenTelemetry` sink only when the *Serilog* options carry `Exporter=Otlp` + `OtlpEndpoint`, but every app calls `AddZbSerilog(o => o.ServiceName = ...)` with the service name only, so `<App>:Telemetry:Exporter=Otlp` switches metrics + traces but never logs (`logs.jsonl` stayed empty for all four). Wire the same exporter/endpoint into the AddZbSerilog options, or document logs-over-OTLP as out of scope. Per-repo issues once a direction is chosen.
dohertj2 added the type:followuppriority:p3cross-repo labels 2026-11-12 20:45:12 -05:00
Author
Owner

Fixed in the library and adopted in all four apps. Telemetry(.Serilog) 0.3.0 is published.

Root cause of "no logs": the Serilog OTLP sink only looked at the options passed to AddZbSerilog. Every app filled those options with the service name alone, and set the exporter only on AddZbTelemetry, a separate options object the Serilog package never saw.

Library, 0.3.0 (scadaproj main cfc7168 + 2011c8d; the 0.2.1 review fixes ship in the same release):

  • service.version: a null ServiceVersion (the default) now resolves to the entry assembly's AssemblyInformationalVersion, which is the git SHA. "" omits it; any other value overrides it.
  • ResourceAttributes: extra Resource attributes on every signal. First-class keys are reserved and rejected at startup.
  • ApplyConfiguration(<App>:Telemetry): reads Exporter, OtlpEndpoint, ServiceVersion, SiteId, NodeRole, ExportLogs and ResourceAttributes:*.
  • Logs: AddZbTelemetry now registers its options in DI, and the AddZbSerilog OTLP log sink follows them, so Exporter=Otlp sends logs to the same endpoint with the same Resource. ExportLogs=false opts out. Levels and sinks are unchanged.
  • Stricter endpoint check is included (a scheme-less host:port now fails when Exporter=Otlp). This is safe to ship: no checked-in config sets an endpoint, and no wonder tier has a collector (#54).
  • Tests 42 → 77 in Release, including a loopback OTLP/gRPC receiver. Mutation checks were run on every new behaviour.
  • Feed SHA256 (feed download = local pack):
    • ZB.MOM.WW.Telemetry.0.3.0.nupkg a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2
    • ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg 04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c

Apps (pushed to their default branches, not deployed):

App Commit site.id / node.role Tests
OtOpcUa master 412da27c Cluster id (cluster-<id> role) / plain roles (admin,driver) Host 288/288, Host.Integration 325/327 (2 skipped)
MxAccessGateway main d9d9ff48 MxGateway:Telemetry:SiteId / standalone 1446/1452 (6 skipped)
HistorianGateway main ae1dbcc Telemetry:SiteId / standalone 1414 + 206 + 186 passed
ScadaBridge main a633005f unchanged; the exporter is passed to the pre-host logger via ScadaBridgeTelemetryConfig Host 699/699

At the collector (one throwaway process each; the Resource is identical on metrics, spans and logs):

otopcua           service.version=1.0.0+412da27c…  site.id=OTEL55         node.role=admin,driver  logs=73
historian-gateway service.version=1.0.0+ae1dbccc…  site.id=otel-probe-55  node.role=standalone    logs=29
scadabridge       service.version=1.0.0+a633005f…  site.id=otel-probe-55  node.role=Site          logs=78 (incl. the pre-host startup line)
mxgateway         service.version=0.1.2+54aa61a0   site.id=otel-probe-55  node.role=standalone    logs=21

Log records written inside a request carry their traceId.

Clean-up: the OtOpcUa throwaway ConfigDb was dropped. The windev scratch gateway was removed; the live MxAccessGw stayed Running with :5130/health/ready 200 and its worker PIDs unchanged.

Not done: ScadaBridge Central. The rig was busy with the #132 failover drills, so ScadaBridge#133 stays open.

Full record: components/observability/2026-11-12-otlp-end-to-end-evidence.md (new #55 section), plus GAPS, current-state, SPEC, ZB.MOM.WW.Telemetry/CLAUDE.md and the index row.

Fixed in the library and adopted in all four apps. **Telemetry(.Serilog) `0.3.0` is published.** **Root cause of "no logs":** the Serilog OTLP sink only looked at the options passed *to `AddZbSerilog`*. Every app filled those options with the service name alone, and set the exporter only on `AddZbTelemetry`, a separate options object the Serilog package never saw. **Library, 0.3.0** (scadaproj `main` `cfc7168` + `2011c8d`; the 0.2.1 review fixes ship in the same release): - `service.version`: a null `ServiceVersion` (the default) now resolves to the entry assembly's `AssemblyInformationalVersion`, which is the git SHA. `""` omits it; any other value overrides it. - `ResourceAttributes`: extra Resource attributes on every signal. First-class keys are reserved and rejected at startup. - `ApplyConfiguration(<App>:Telemetry)`: reads `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, `NodeRole`, `ExportLogs` and `ResourceAttributes:*`. - Logs: `AddZbTelemetry` now registers its options in DI, and the `AddZbSerilog` OTLP log sink follows them, so `Exporter=Otlp` sends logs to the same endpoint with the same Resource. `ExportLogs=false` opts out. Levels and sinks are unchanged. - Stricter endpoint check is included (a scheme-less `host:port` now fails when `Exporter=Otlp`). This is safe to ship: no checked-in config sets an endpoint, and no wonder tier has a collector (#54). - Tests 42 → 77 in Release, including a loopback OTLP/gRPC receiver. Mutation checks were run on every new behaviour. - Feed SHA256 (feed download = local pack): - `ZB.MOM.WW.Telemetry.0.3.0.nupkg` `a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2` - `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` `04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c` **Apps** (pushed to their default branches, not deployed): | App | Commit | `site.id` / `node.role` | Tests | |---|---|---|---| | OtOpcUa | `master` `412da27c` | Cluster id (`cluster-<id>` role) / plain roles (`admin,driver`) | Host 288/288, Host.Integration 325/327 (2 skipped) | | MxAccessGateway | `main` `d9d9ff48` | `MxGateway:Telemetry:SiteId` / `standalone` | 1446/1452 (6 skipped) | | HistorianGateway | `main` `ae1dbcc` | `Telemetry:SiteId` / `standalone` | 1414 + 206 + 186 passed | | ScadaBridge | `main` `a633005f` | unchanged; the exporter is passed to the pre-host logger via `ScadaBridgeTelemetryConfig` | Host 699/699 | **At the collector** (one throwaway process each; the Resource is identical on metrics, spans and logs): ``` otopcua service.version=1.0.0+412da27c… site.id=OTEL55 node.role=admin,driver logs=73 historian-gateway service.version=1.0.0+ae1dbccc… site.id=otel-probe-55 node.role=standalone logs=29 scadabridge service.version=1.0.0+a633005f… site.id=otel-probe-55 node.role=Site logs=78 (incl. the pre-host startup line) mxgateway service.version=0.1.2+54aa61a0 site.id=otel-probe-55 node.role=standalone logs=21 ``` Log records written inside a request carry their `traceId`. **Clean-up:** the OtOpcUa throwaway ConfigDb was dropped. The windev scratch gateway was removed; the live `MxAccessGw` stayed Running with `:5130/health/ready` 200 and its worker PIDs unchanged. **Not done:** ScadaBridge Central. The rig was busy with the #132 failover drills, so ScadaBridge#133 stays open. Full record: `components/observability/2026-11-12-otlp-end-to-end-evidence.md` (new #55 section), plus GAPS, current-state, SPEC, `ZB.MOM.WW.Telemetry/CLAUDE.md` and the index row.
Sign in to join this conversation.