diff --git a/CLAUDE.md b/CLAUDE.md index 346d873..d5caac7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | -| Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. **OTLP proven end to end 2026-11-12 (#13, closed):** shared dev collector `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)); metrics + traces from all four apps (ScadaBridge Site role; Central → ScadaBridge#133); OtOpcUa Galaxy driver telemetry was missing from the allowlists → fixed lmxopcua#892 (`master` `db86b996`, not deployed); no lib change, so `0.2.1` stays unpublished. Open: wonder-tier collector #54; `service.version`/`site.id`/`node.role` + logs-over-OTLP #55. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | +| Observability (metrics / traces / logs) | **Adopted** — **`0.3.0` published 2026-11-12** (feed `0.1.0`, `0.2.0`, `0.3.0`; `0.2.1` never published, shipped inside 0.3.0). **All 4 apps pin `0.3.0`** on their default branches (#55: OtOpcUa `412da27c`, ScadaBridge `a633005f`, mxgw `d9d9ff48`, HistorianGateway `ae1dbcc`; not deployed) and call `AddZbTelemetry` + `AddZbSerilog`. 0.3.0: `service.version` defaults to the entry assembly's informational version (git SHA); `ResourceAttributes` + `ApplyConfiguration(:Telemetry)`; the Serilog OTLP log sink follows `AddZbTelemetry`'s exporter, so `Exporter=Otlp` now sends **logs** too; stricter endpoint check (scheme-less `host:port` fails when `Exporter=Otlp`). Resource identity: OtOpcUa `site.id`=Cluster id / `node.role`=plain roles; mxgw + HG `node.role=standalone`, `site.id` from config; ScadaBridge unchanged. **OTLP proven end to end 2026-11-12** (#13 + #55, both closed) against `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)): metrics, traces and logs with the full Resource from all four apps (ScadaBridge Site role; Central → ScadaBridge#133). Open: wonder-tier collector #54. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | diff --git a/ZB.MOM.WW.Telemetry/CLAUDE.md b/ZB.MOM.WW.Telemetry/CLAUDE.md index 4d055db..f309d68 100644 --- a/ZB.MOM.WW.Telemetry/CLAUDE.md +++ b/ZB.MOM.WW.Telemetry/CLAUDE.md @@ -2,7 +2,7 @@ Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application. -**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); no library change was needed, so `0.2.1` stays unpublished. Open: #54 (wonder-tier collector), #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP). The history below is a record, not a status tracker; file new follow-ups as issues. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP) is **closed** by `0.3.0` (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues. The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation. @@ -14,8 +14,8 @@ The library normalizes the three-project observability surface: a shared OpenTel | Package | Responsibilities | Key Dependencies | |---|---|---| -| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry` — wires OTel SDK (metrics + tracing), populates shared Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`, `host.name`), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. `app.MapZbMetrics()` — mounts `/metrics`. `ZbTelemetryOptions` — the single options object shared by both packages. | `Microsoft.AspNetCore.App` (framework ref), `OpenTelemetry.*` stack | -| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog` — shared two-stage Serilog bootstrap: `ReadFrom.Configuration`-driven sinks, `MinimumLevel.Is(Information)` default (config-overridable), `SiteId`/`NodeRole`/`NodeHostname` enrichers from `ZbTelemetryOptions`, `TraceContextEnricher` (writes `trace_id`/`span_id` from `Activity.Current`), `ILogRedactor` seam (per-project sensitive-field redaction via `RedactionEnricher`). **0.2.0:** optional `ZbSerilogOptions` overload — `MinimumLevelGovernor` (app key outranks `Serilog:MinimumLevel`) + `ConfigureLogger` (final filters/enrichers/sinks). Does NOT freeze `Log.Logger` — safe for multi-host/test scenarios. | `ZB.MOM.WW.Telemetry`, `Serilog.*` stack | +| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry` — wires OTel SDK (metrics + tracing), populates shared Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`, `host.name`), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. `app.MapZbMetrics()` — mounts `/metrics`. `ZbTelemetryOptions` — the single options object shared by both packages. **0.3.0:** `service.version` defaults from the entry assembly's informational version; `ResourceAttributes`; `ApplyConfiguration(:Telemetry section)`; publishes its options in DI for the Serilog log sink. | `Microsoft.AspNetCore.App` (framework ref), `OpenTelemetry.*` stack | +| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog` — shared two-stage Serilog bootstrap: `ReadFrom.Configuration`-driven sinks, `MinimumLevel.Is(Information)` default (config-overridable), `SiteId`/`NodeRole`/`NodeHostname` enrichers from `ZbTelemetryOptions`, `TraceContextEnricher` (writes `trace_id`/`span_id` from `Activity.Current`), `ILogRedactor` seam (per-project sensitive-field redaction via `RedactionEnricher`). **0.2.0:** optional `ZbSerilogOptions` overload — `MinimumLevelGovernor` (app key outranks `Serilog:MinimumLevel`) + `ConfigureLogger` (final filters/enrichers/sinks). **0.3.0:** the OTLP log sink follows the `AddZbTelemetry` registration in the same container (`ExportLogs=false` opts out). Does NOT freeze `Log.Logger` — safe for multi-host/test scenarios. | `ZB.MOM.WW.Telemetry`, `Serilog.*` stack | --- @@ -23,14 +23,13 @@ The library normalizes the three-project observability surface: a shared OpenTel | Consumer | `ZB.MOM.WW.Telemetry` (core) | `ZB.MOM.WW.Telemetry.Serilog` | |---|:---:|:---:| -| **OtOpcUa** | ✅ adopted | ✅ adopted (`AddZbSerilog`) | -| **MxAccessGateway** | ✅ adopted (`GatewayMetrics` exported) | ✅ adopted (MEL→Serilog migrated in this pass) | -| **ScadaBridge** | ✅ adopted (both roots) | ✅ `AddZbSerilog` **0.2.0** (2026-11-12, ScadaBridge#94, `main` `2d2e9f35`) — `MinimumLevelGovernor` + `ConfigureLogger` (standby filter); run on a pre-host builder so the static `Log.Logger` stays the one process logger | -| **HistorianGateway** | ✅ adopted | ✅ adopted (`AddZbSerilog`) | +| **OtOpcUa** | ✅ **0.3.0** (`master` `412da27c`) — `site.id` = Cluster id, `node.role` = plain roles | ✅ **0.3.0** (`AddZbSerilog`; logs follow `AddZbTelemetry`) | +| **MxAccessGateway** | ✅ **0.3.0** (`main` `d9d9ff48`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) | +| **ScadaBridge** | ✅ **0.3.0** (`main` `a633005f`, both roots) | ✅ **0.3.0** `AddZbSerilog` with `MinimumLevelGovernor` + `ConfigureLogger` on a pre-host builder; the exporter is passed explicitly via `ScadaBridgeTelemetryConfig` | +| **HistorianGateway** | ✅ **0.3.0** (`main` `ae1dbcc`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) | -All three adopted on 2026-06-01 (branch `feat/adopt-zb-telemetry` per repo); HistorianGateway adopted since. OtOpcUa, MxAccessGateway and HistorianGateway pin both packages at `0.1.0` on their origin default branches -(verified 2026-11-09; their `AddZbSerilog(o => …)` call sites are source-compatible with 0.2.0 — the new overload is -additive). ScadaBridge pins **`0.2.0`** since 2026-11-12. See `components/observability/GAPS.md` for the full result. +All four pin both packages at **`0.3.0`** on their origin default branches since 2026-11-12 (scadaproj#55; not +deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See `components/observability/GAPS.md` for the full result. --- @@ -54,9 +53,9 @@ All test assemblies run offline: | Assembly | Tests | |---|---| -| `ZB.MOM.WW.Telemetry.Tests` | 12 | -| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 30 | -| **Total** | **42** | +| `ZB.MOM.WW.Telemetry.Tests` | 38 | +| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 39 | +| **Total** | **77** | `GeneratePackageOnBuild` is off — pack explicitly with the command above. @@ -64,7 +63,26 @@ All test assemblies run offline: ## Status -**`0.2.1` (2026-11-09, scadaproj#48) — built, NOT published.** P3 review fixes: `RedactionEnricher` is now registered +**`0.3.0` (2026-11-12, scadaproj#55) — PUBLISHED** (both packages, packed from the tree of scadaproj `main` `334c82c` (committed as `df9a328` before rebasing onto main); 38 + 39 +tests green in Release). SHA256 (feed download = local pack): `ZB.MOM.WW.Telemetry.0.3.0.nupkg` +`a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2`, `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` +`04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c`. Adopted by all four apps the same day. +- `service.version`: `ServiceVersion = null` (default) now resolves to the entry assembly's + `AssemblyInformationalVersion` (else its assembly version); `""` omits; any value overrides. **Behaviour + change** from 0.2.x, where null omitted the attribute. +- `ResourceAttributes` (extra Resource attributes on all signals; first-class keys reserved, rejected at startup). +- `ApplyConfiguration(IConfiguration section)` — `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, + `NodeRole`, `ExportLogs`, `ResourceAttributes:`; absent keys keep code defaults. +- **Logs over OTLP**: `AddZbTelemetry` registers its options in DI. The `AddZbSerilog` OTLP log sink + follows them when its own options do not select OTLP. Root cause of "no app ever exported a log": the + sink keyed only on the `AddZbSerilog` options, which every app filled with the service name alone. + `ExportLogs = false` keeps logs local. +- Includes the never-published **0.2.1** fixes below, including the stricter OTLP endpoint check. That check + was shipped deliberately: no checked-in config sets an endpoint, it only runs when `Exporter=Otlp`, and no + wonder tier has a collector (#54). +- Feed versions: `0.1.0`, `0.2.0`, `0.3.0` (`0.2.1` was never published). + +**`0.2.1` (2026-11-09, scadaproj#48) — never published; shipped inside `0.3.0`.** P3 review fixes: `RedactionEnricher` is now registered AFTER `ConfigureLogger`, so enrichers added in the hook are redacted (009); dictionary keys that print identically are all kept (later ones get a `#n` suffix in the redactor's view; the original key is restored on rebuild) (010); the OTLP endpoint must be `http`/`https` with a host (`collector:4317` is now rejected) and the pre-built-options diff --git a/components/observability/2026-11-12-otlp-end-to-end-evidence.md b/components/observability/2026-11-12-otlp-end-to-end-evidence.md index 99c8082..4d81cd7 100644 --- a/components/observability/2026-11-12-otlp-end-to-end-evidence.md +++ b/components/observability/2026-11-12-otlp-end-to-end-evidence.md @@ -165,9 +165,143 @@ HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGatew - **ScadaBridge#133:** OTLP from the Central role is not yet proven. - **scadaproj#54:** collector on the wonder tiers. -## Telemetry 0.2.1: not published +## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below) No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays built-but-unpublished until a pin bump wants it. + +--- + +# Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day) + +Closes the three #55 gaps found above: no `service.version` anywhere, no `site.id`/`node.role` on OtOpcUa, +MxAccessGateway and HistorianGateway, and no logs over OTLP from any app. + +## Root cause of "no logs" + +`ZbSerilogConfig.ApplyOpenTelemetryExport` adds the `WriteTo.OpenTelemetry` sink only when the options +passed **to `AddZbSerilog`** say `Exporter=Otlp`. Every app filled those options with the service name alone +and set the exporter only on `AddZbTelemetry`, a separate options object the Serilog package never saw. The +sink condition could therefore never be true. + +## Library: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.0 — published + +scadaproj `main` `b34049b` + `334c82c`. Feed SHA256 (feed download = local pack): + +| Package | SHA256 | +|---|---| +| `ZB.MOM.WW.Telemetry.0.3.0.nupkg` | `a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2` | +| `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` | `04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c` | + +- **`service.version`**: `ServiceVersion = null` (the default) now means the entry assembly's + `AssemblyInformationalVersion` (the git SHA the apps stamp), falling back to its assembly version. Set a + value to override it; `""` omits the attribute. +- **`ResourceAttributes`**: extra Resource attributes on all three signals. The first-class keys + (`service.*`, `host.name`, `site.id`, `node.role`) are reserved and rejected at startup. +- **`ApplyConfiguration(section)`**: reads `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, `NodeRole`, + `ExportLogs`, `ResourceAttributes:` from an app's own `:Telemetry` section. Absent keys keep the + code defaults, so the four hand-written exporter parsers are gone. +- **Logs follow `AddZbTelemetry`**: `AddZbTelemetry` registers its options in DI. `AddZbSerilog` resolves them + inside its `AddSerilog` callback (so call order does not matter), and when its own options do not select + OTLP, the OTLP log sink uses that registration's endpoint and Resource. `ExportLogs = false` keeps logs + local. Levels and the configuration-driven sinks are unchanged. +- **0.2.1 shipped in the same release.** It holds the P3 review fixes: redaction after `ConfigureLogger`, + dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at + startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in + config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no + wonder tier has a collector yet (#54), so no deployed box can trip it. +- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on + Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an + `ExportLogs=false` case with a metrics positive control, and a Prometheus case. + + Mutation checks, each caught by the named tests: + - Drop the DI lookup → the e2e log test fails. + - Drop the version default → the pipeline-Resource and e2e tests fail. + - Drop `ResourceAttributes` → 2 attribute tests and the e2e test fail. + - Ignore `ExportLogs` → the decision test and the e2e test fail. + - Disable the reserved-key check → 7 theory cases fail. + - Drop `ApplyConfiguration`'s attribute or `SiteId` read → its test fails. + +## Apps (all pushed to their default branch; no wonder-box deploys) + +| App | Commit | Pin | Identity wiring | +|---|---|---|---| +| OtOpcUa | `master` `412da27c` | 0.1.0 → 0.3.0 | `site.id` = the application Cluster id (the `cluster-` role suffix; omitted on a legacy node); `node.role` = the plain roles, sorted (`admin,driver`); Program passes its `EffectiveRoles`. | +| MxAccessGateway | `main` `d9d9ff48` | 0.1.0 → 0.3.0 (csproj) | `node.role=standalone` (one gateway per box); `site.id` from `MxGateway:Telemetry:SiteId` (there is no node identity in its config, so it is omitted unless set). | +| HistorianGateway | `main` `ae1dbcc` | 0.1.0 → 0.3.0 (csproj) | Same as mxgw, under the root-level `Telemetry:` section. | +| ScadaBridge | `main` `a633005f` | 0.2.0 → 0.3.0 | Unchanged `site.id`/`node.role`. The new `ScadaBridgeTelemetryConfig` fills both `AddZbTelemetry` and the pre-host `AddZbSerilog` logger. The logger is built on its own container with no `AddZbTelemetry` to follow, so the exporter is passed to it explicitly. | + +Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452 +(6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity +tests (OtOpcUa `ObservabilityResourceIdentityTests`, HG `GatewayTracingTests`, ScadaBridge +`ScadaBridgeTelemetryConfigTests`). + +## What arrived (collector, [`otelidentity.py`](../../infra/otel-collector/otelidentity.py), one process each) + +All runs: `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`, `OTEL_METRIC_EXPORT_INTERVAL=10000`. + +**OtOpcUa** — local Release host, fused `admin,driver,cluster-OTEL55` single node. Its own Akka mesh on +`localhost:14053`. Throwaway ConfigDb `OtOpcUaOtel55` on the docker-dev SQL: EF-migrated, one +`ServerCluster` `OTEL55` + one `ClusterNode`, gRPC mesh/telemetry keys of its own. The DB was **dropped +afterwards**. Activity: one `POST /api/deployments` (202) plus health/metrics GETs. + +``` +RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"} +RESOURCE resourceSpans {... identical ...} +RESOURCE resourceLogs {... identical ...} +COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12} +LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"} +LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"} +``` +The `otopcua.deploy.apply` and `otopcua.opcua.address_space_rebuild` spans arrived again. + +**HistorianGateway** — local Release host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores, +`Telemetry__SiteId=otel-probe-55`. Activity: `HistorianStatus/Probe` + health GETs. + +``` +RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"} +RESOURCE resourceSpans {... identical ...} +RESOURCE resourceLogs {... identical ...} +COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2} +LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"} +``` + +**ScadaBridge (Site)** — local Release **Site** host, own site id `otel-probe-55`, own 1-node mesh (`localhost:18082`), +scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. + +``` +RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"} +RESOURCE resourceSpans {... identical ...} +RESOURCE resourceLogs {... identical ...} +COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2} +LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"} +LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"} +``` +The first line, `Starting ScadaBridge host …`, is written to the static logger before any web host exists, so +the explicit exporter path on the pre-host logger works. **Central not run:** the rig was in use by the +ScadaBridge#132 failover drills (`sb132-p` / `sb132-probe` running, centrals recreated 3–6 minutes earlier), so +ScadaBridge#133 stays open. + +**MxAccessGateway** — Release `main` `54aa61a0`, published framework-dependent win-x64 from the Mac. (It was +pushed as `d9d9ff48` after rebasing over one docs-only commit.) Run as a scratch copy at `C:\otel55\mxgw` on +windev, on `:15120`/`:15130`, under `Staging` + `DisableLogin`, with alarms off, scratch auth/secrets/snapshot +stores and a throwaway pepper. `Worker:ExecutablePath` pointed at the non-existent +`C:\otel55\OtelProbeNoWorker.exe`. It was launched through `Win32_Process.Create`, because a process started +from the ssh session dies with it. `MxGateway__Telemetry__SiteId=otel-probe-55`. + +``` +RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"} +RESOURCE resourceSpans {... identical ...} +RESOURCE resourceLogs {... identical ...} +COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12} +LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."} +LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"} +``` +The live `MxAccessGw` service stayed `Running` with `:5130/health/ready` = 200 throughout. Its gateway PID +(9148) and worker PIDs (`14700,22108`) were unchanged before and after. The scratch copy and `C:\otel55` were +deleted. + +On every app, log records carry `traceId` when written inside a request, so logs join their spans in the +backend. The Resource is byte-identical across the three signals of each process. diff --git a/components/observability/GAPS.md b/components/observability/GAPS.md index 059cd1f..6f49012 100644 --- a/components/observability/GAPS.md +++ b/components/observability/GAPS.md @@ -29,6 +29,14 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p > from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway. One defect fixed: OtOpcUa's > Galaxy driver meter/source were missing from the allowlists (lmxopcua#892). Evidence: > [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). +> +> **✅ RESOURCE IDENTITY + LOGS OVER OTLP — 2026-11-12 (#55, closed).** Telemetry(.Serilog) **`0.3.0`** +> (published; ships the 0.2.1 review fixes): `service.version` defaults from the entry assembly's +> informational version (the git SHA), `ResourceAttributes` + `ApplyConfiguration(:Telemetry)`, and the +> Serilog OTLP log sink follows `AddZbTelemetry`'s exporter. All four apps pin `0.3.0` and report +> `service.version`, `site.id` and `node.role` (OtOpcUa: Cluster id / plain roles; mxgw + HG: `standalone` +> + configurable `SiteId`; ScadaBridge unchanged) and send logs — proven at the collector for OtOpcUa, +> HistorianGateway, ScadaBridge (Site) and MxAccessGateway. Central role still → ScadaBridge#133. ## Divergence vs spec @@ -267,7 +275,8 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). - **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still Prometheus-only. → #54 -- **Resource identity + logs-over-OTLP gaps** found by #13: `service.version` absent on all four apps; - `site.id`/`node.role` absent on OtOpcUa, MxAccessGateway, HistorianGateway; no app passes the exporter - settings to `AddZbSerilog`, so logs never travel over OTLP. → #55 +- ~~**Resource identity + logs-over-OTLP gaps** found by #13~~ — **DONE 2026-11-12 (#55)**: Telemetry + `0.3.0`; all four apps report `service.version` / `site.id` / `node.role` and export logs over OTLP + (OtOpcUa `412da27c`, ScadaBridge `a633005f`, MxAccessGateway `d9d9ff48`, HistorianGateway `ae1dbcc`). See the + #55 section of [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). - **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81 diff --git a/components/observability/current-state/mxaccessgw/CURRENT-STATE.md b/components/observability/current-state/mxaccessgw/CURRENT-STATE.md index afc88b9..b4ee96f 100644 --- a/components/observability/current-state/mxaccessgw/CURRENT-STATE.md +++ b/components/observability/current-state/mxaccessgw/CURRENT-STATE.md @@ -5,6 +5,12 @@ > OTLP export of **metrics and traces** is proven end to end (scratch copy of the deployed windev build; the > `mxgateway.*` gauges and ASP.NET Core spans arrived). The gateway still has no `ActivitySource` of its own. > Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). +> +> **2026-11-12 update (scadaproj#55):** closed on `main` `d9d9ff48` (Telemetry `0.3.0`). The Resource now +> carries `service.version` (entry-assembly informational version, e.g. `0.1.2+54aa61a0`), `node.role=standalone` +> (one gateway per box; `MxGateway:Telemetry:NodeRole` overrides) and `site.id` from +> `MxGateway:Telemetry:SiteId` (no natural value in the gateway's config, so it is omitted unless set). +> `MxGateway:Telemetry:Exporter=Otlp` now also sends the Serilog logs (proven: 21 log records). Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**); solution `src/MxGateway.sln`. Telemetry code is concentrated in diff --git a/components/observability/current-state/otopcua/CURRENT-STATE.md b/components/observability/current-state/otopcua/CURRENT-STATE.md index b15cf4e..bfe180d 100644 --- a/components/observability/current-state/otopcua/CURRENT-STATE.md +++ b/components/observability/current-state/otopcua/CURRENT-STATE.md @@ -7,6 +7,13 @@ > Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in > lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role` > (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). +> +> **2026-11-12 update (scadaproj#55):** closed on `master` `412da27c` (Telemetry `0.3.0`). The Resource now +> carries `service.version` (entry-assembly informational version, e.g. `1.0.0+412da27c…`), `site.id` = the +> node's application Cluster id (suffix of its `cluster-` role; omitted on a legacy node) and +> `node.role` = its plain roles (`admin`, `driver`, `admin,driver`). `OtOpcUa:Telemetry:SiteId` / `:NodeRole` / +> `:ServiceVersion` / `:ResourceAttributes:*` override them. `OtOpcUa:Telemetry:Exporter=Otlp` now also sends +> the Serilog logs to the collector (proven: 73 log records with the same Resource). Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`. Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side diff --git a/components/observability/current-state/scadabridge/CURRENT-STATE.md b/components/observability/current-state/scadabridge/CURRENT-STATE.md index ef77e43..081979f 100644 --- a/components/observability/current-state/scadabridge/CURRENT-STATE.md +++ b/components/observability/current-state/scadabridge/CURRENT-STATE.md @@ -5,6 +5,12 @@ > end to end for a **Site** node (6 `scadabridge.*` instruments, `localdb.oplog.depth`, gRPC-client spans, > with `site.id`/`node.role` on the Resource); the Central role is not yet proven (ScadaBridge#133). > Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). +> +> **2026-11-12 update (scadaproj#55):** closed on `main` `a633005f` (Telemetry `0.3.0`). `service.version` now +> on the Resource (entry-assembly informational version); `site.id`/`node.role` unchanged. +> `ScadaBridgeTelemetryConfig` fills both the `AddZbTelemetry` registration and the pre-host `AddZbSerilog` +> logger, so `ScadaBridge:Telemetry:Exporter=Otlp` also sends logs (proven on a Site node: 78 log records, +> including the pre-host startup line). Central still → ScadaBridge#133. Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution `ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref diff --git a/components/observability/spec/SPEC.md b/components/observability/spec/SPEC.md index b54e12f..3f7f3c7 100644 --- a/components/observability/spec/SPEC.md +++ b/components/observability/spec/SPEC.md @@ -67,10 +67,11 @@ The OTel `Resource` attached to all three signals is built from `ZbTelemetryOpti |---|---|---| | `service.name` | `ServiceName` | Required. Lower-case short identifier (`otopcua`, `mxgateway`, `scadabridge`) | | `service.namespace` | `ServiceNamespace` | Default `"ZB.MOM.WW"` — constant across the fleet | -| `service.version` | `ServiceVersion` | Optional; recommend populating from `AssemblyInformationalVersion` | +| `service.version` | `ServiceVersion` | Defaults (Telemetry 0.3.0+) to the entry assembly's `AssemblyInformationalVersion` — the git SHA every family app stamps; set a value to override, `""` to omit | | `service.instance.id` | _(auto)_ | Always populated from `ZbResource.InstanceId` — a deterministic `MachineName:ProcessId`. The OTel SDK's random-GUID default is disabled so every signal from the same process carries an identical, restart-stable instance id for cross-signal correlation | | `site.id` | `SiteId` | Optional; identifies the physical/logical site | -| `node.role` | `NodeRole` | Optional; e.g. `"central"`, `"site"`, `"hub"` | +| `node.role` | `NodeRole` | Optional; e.g. `"central"`, `"site"`, `"hub"`, `"standalone"` | +| _(any other)_ | `ResourceAttributes` | Optional extra attributes (0.3.0+); the keys above are reserved | | `host.name` | _(auto)_ | Always populated from `Environment.MachineName` | The same `SiteId` and `NodeRole` values are passed to the Serilog enrichers (§4) so a diff --git a/infra/otel-collector/README.md b/infra/otel-collector/README.md index 388a2c2..5e44e0e 100644 --- a/infra/otel-collector/README.md +++ b/infra/otel-collector/README.md @@ -31,12 +31,14 @@ OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is | MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | | HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) | -- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less - `host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there. +- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.3.0` (what the apps pin since + 2026-11-12) rejects a scheme-less `host:port` at startup when `Exporter=Otlp`. - The SDK's default metric export interval is 60 s. For a quick check set `OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process. -- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called - without the exporter settings (scadaproj#55). +- What travels: **metrics, traces and logs** (Telemetry `0.3.0`+, scadaproj#55): with `Exporter=Otlp` the + Serilog logs go to the same endpoint with the same Resource. `:Telemetry:ExportLogs=false` keeps them + local. Every app's Resource carries `service.version` (git SHA), `site.id` and `node.role`; mxgw and + HistorianGateway report `node.role=standalone` and take `site.id` from `:Telemetry:SiteId`. ## Reading what arrived @@ -45,18 +47,21 @@ ssh dohertj2@10.100.0.35 docker logs --tail 20 zb-otel-collector # one debug line per received batch cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names python3 ../otelsum.py otopcua # one service +python3 ../otelidentity.py otopcua :12345 # Resource per signal + log samples, one process (instance-id suffix) curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head ``` `data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB -(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc. +(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc; +[`otelidentity.py`](otelidentity.py) prints the Resource each signal arrived with, the log count by severity +and sample log records, filtered to one process (scadaproj#55). ## Deploy / update ```bash # from this directory on the Mac ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data' -scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/ +scp docker-compose.yml config.yaml otelsum.py otelidentity.py dohertj2@10.100.0.35:zb-otel-collector/ ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate' ``` diff --git a/infra/otel-collector/otelidentity.py b/infra/otel-collector/otelidentity.py new file mode 100644 index 0000000..8b637be --- /dev/null +++ b/infra/otel-collector/otelidentity.py @@ -0,0 +1,34 @@ +# Usage (collector host, in ~/zb-otel-collector/data): python3 ../otelidentity.py [instance-id-substring, e.g. :12345] +# Prints the Resource each signal (metrics/traces/logs) arrived with, counts, and sample log records (scadaproj#55). +import json,sys,collections +svc=sys.argv[1]; inst=sys.argv[2] if len(sys.argv)>2 else None +def attrs(lst): return {a["key"]:(list(a["value"].values())[0] if a.get("value") else None) for a in lst or []} +KEEP=("service.name","service.version","service.instance.id","site.id","node.role","host.name") +out={} +counts=collections.Counter(); samples=[]; sev=collections.Counter() +for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")): + try: lines=open(f) + except FileNotFoundError: continue + for l in lines: + d=json.loads(l) + for r in d.get(key,[]): + a=attrs(r["resource"].get("attributes")) + if a.get("service.name")!=svc: continue + if inst and inst not in str(a.get("service.instance.id")): continue + extra={k:v for k,v in a.items() if k not in KEEP and not k.startswith(("telemetry.sdk","process.","os."))} + out[key]={k:a.get(k) for k in KEEP}|({"extra":extra} if extra else {}) + if key=="resourceMetrics": + counts["metric points"]+=sum(1 for sm in r["scopeMetrics"] for m in sm["metrics"]) + elif key=="resourceSpans": + counts["spans"]+=sum(len(ss["spans"]) for ss in r["scopeSpans"]) + else: + for sl in r["scopeLogs"]: + for lr in sl["logRecords"]: + counts["log records"]+=1; sev[lr.get("severityText","?")]+=1 + la=attrs(lr.get("attributes")) + if len(samples)<4 or (lr.get("traceId") and not any(s[2] for s in samples)): + body=lr.get("body",{}).get("stringValue","") + samples.append((lr.get("severityText"),body[:110],lr.get("traceId",""),la.get("SourceContext") or la.get("message_template.text","")[:40])) +for k,v in out.items(): print("RESOURCE",k,json.dumps(v)) +print("COUNTS",dict(counts),"SEVERITY",dict(sev)) +for s in samples[:5]: print("LOG",json.dumps({"severity":s[0],"body":s[1],"traceId":s[2],"src":s[3]}))