# Design — Health & Observability normalization components + shared libraries

Date: 2026-06-01
Status: **Approved design** (brainstorm output). Implementation plans follow separately
(one per library) via the writing-plans workflow.

This design adds the next two entries to the [component-normalization](../../components/README.md)
program, following the exact arc already used for **Auth** (`ZB.MOM.WW.Auth`) and **UI-Theme**
(`ZB.MOM.WW.Theme`): normalize the concern in `components/`, then build the shared library in this
repo. The two concerns are the top-ranked candidates in [`upcoming.md`](../../upcoming.md) (Health #1,
Observability #2 — the "operability cluster").

## Scope decisions (locked during brainstorm)

1. **Deliverable depth** — normalization docs **+ build both shared libraries** (.NET 10, tested,
   `dotnet pack`). *Not* a docs-only pass.
2. **Structure** — two separate components → two separate libraries (one component = one library,
   per house precedent): `components/health/` → `ZB.MOM.WW.Health`; `components/observability/` →
   `ZB.MOM.WW.Telemetry`. A future `ZB.MOM.WW.Hosting` aggregator can bundle both behind one call.
3. **Telemetry reach** — all three OpenTelemetry signals (metrics + traces + logs), including a shared
   Serilog bootstrap, enrichers, and trace↔log correlation.
4. **Sister-repo touch** — exactly one: migrate **MxAccessGateway** off `Microsoft.Extensions.Logging`
   onto the shared Serilog bootstrap. No other app adoption — wiring Health/Telemetry into the three
   apps stays a future `GAPS.md` item, identical to where Auth and UI-Theme sit today.
5. **Packaging** — dependency-split packages (mirrors Auth's 4-package split and the
   `AspNetCore.HealthChecks.*` ecosystem). Heavy probes live in opt-in satellites so MxGateway never
   transitively pulls Akka or EF.
6. **Current-state docs** — full code-verified depth with `file:line` refs, per
   `components/README.md`'s mandate (matching auth's current-state docs).

## The unifying hinge

A single identity triple — `service.name` / `site.id` / `node.role` (+ host) — populates **both** the
OpenTelemetry `Resource` **and** the Serilog enrichers. A metric, a span, and a log line from the same
node therefore carry identical dimensions and join up in a backend. This symmetry is the reason
Health and Telemetry are designed together even though they ship as separate libraries.

## Repo layout

```
scadaproj/
├─ components/
│   ├─ health/                       NEW normalization component (docs)
│   │   ├─ README.md
│   │   ├─ spec/SPEC.md
│   │   ├─ shared-contract/ZB.MOM.WW.Health.md
│   │   ├─ current-state/{otopcua,mxaccessgw,scadabridge}/CURRENT-STATE.md
│   │   └─ GAPS.md
│   └─ observability/                NEW normalization component (docs)
│       ├─ README.md
│       ├─ spec/SPEC.md
│       ├─ spec/METRIC-CONVENTIONS.md   (mirrors auth CANONICAL-ROLES / theme DESIGN-TOKENS)
│       ├─ shared-contract/ZB.MOM.WW.Telemetry.md
│       ├─ current-state/{otopcua,mxaccessgw,scadabridge}/CURRENT-STATE.md
│       └─ GAPS.md
├─ ZB.MOM.WW.Health/                 NEW built library (nested git repo, .NET 10) → 3 nupkgs @ 0.1.0
├─ ZB.MOM.WW.Telemetry/              NEW built library (nested git repo, .NET 10) → 2 nupkgs @ 0.1.0
└─ docs/plans/
    ├─ 2026-06-01-zb-mom-ww-health-shared-library.md       (impl plan — from writing-plans)
    └─ 2026-06-01-zb-mom-ww-telemetry-shared-library.md     (impl plan — from writing-plans)
```

Index updates (same discipline as the prior two components): add both rows to
`components/README.md`, the `CLAUDE.md` Component-normalization table, and check off Health +
Observability in `upcoming.md`.

## Code-verified current state (2026-06-01 scan)

### Health
| | OtOpcUa | ScadaBridge | MxGateway |
|---|---|---|---|
| Endpoints | `/health/ready`, `/health/active`, `/healthz` | `/health/ready`, `/health/active` (no `/healthz`) | `/health/live` only (custom `GatewayHealthReply`) |
| Probes | Database, AkkaCluster, AdminRoleLeader | Database, AkkaCluster, ActiveNode | **none** (`AddHealthChecks()` called but unused) |
| Tagging | tags on the check | named + predicate, `HealthChecks.UI.Client` JSON | — |
| Extra | — | `IActiveNodeGate` route gate + `HealthMonitoring/` domain pipeline | net48 x86 worker has no endpoint |

Both descend from the same "ScadaLink three-tier pattern" (OtOpcUa's `HealthEndpoints.cs:13` says
so) but the Akka/leader probe logic and the DB probe technique already differ. MxGateway is **not**
Akka-based — a hard dependency-hygiene constraint.

Key refs: OtOpcUa `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/{HealthEndpoints,DatabaseHealthCheck,AkkaClusterHealthCheck,AdminRoleLeaderHealthCheck}.cs`;
ScadaBridge `src/ZB.MOM.WW.ScadaBridge.Host/Health/{DatabaseHealthCheck,AkkaClusterHealthCheck,ActiveNodeHealthCheck,ActiveNodeGate}.cs` + `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/`;
MxGateway `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:61,139–145`.

### Telemetry
| | OtOpcUa | MxGateway | ScadaBridge |
|---|---|---|---|
| OTel SDK | full (`WithMetrics`+`WithTracing`) | **none** (hand-rolled `System.Diagnostics.Metrics`, no export) | **none** (`OpenTelemetry.Api` is a dangling CVE-patch ref) |
| Exporter | Prometheus `/metrics` | in-memory snapshot only (`GetSnapshot()`) | — |
| Meter | `ZB.MOM.WW.OtOpcUa` | `MxGateway.Server` (13 ctr / 3 hist `ms` / 4 gauge) | — |
| Tracing | ActivitySource (2 spans) | none | none |
| Resource / `service.name` | **none anywhere** | none | none |

Nobody sets a resource/`service.name` — the fleet is indistinguishable in a collector. Durations
split `s` (OtOpcUa, OTel-correct) vs `ms` (MxGateway).

Key refs: OtOpcUa `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs` +
`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`;
MxGateway `src/ZB.MOM.WW.MxGateway.Server/Metrics/GatewayMetrics.cs`;
ScadaBridge `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj:31` (Api only, zero usage).

### Logging
Serilog in OtOpcUa (`Program.cs:49`) + ScadaBridge (`LoggerConfigurationFactory.cs:28–126`,
enrichers `SiteId`/`NodeRole`/`NodeHostname`); MEL in MxGateway (`appsettings.json`, correlation via
`GatewayLogScope`/`BeginScope` middleware + `GatewayLogRedactor`). ScadaBridge's enricher set is the
cleanest and its property names match the Resource attributes Telemetry needs. Nobody enriches logs
with `trace_id`/`span_id`.

## Library design — `ZB.MOM.WW.Health` (3 packages)

**① `ZB.MOM.WW.Health`** (core; deps: `Microsoft.Extensions.Diagnostics.HealthChecks` + ASP.NET Core abstractions)
- Tier convention: canonical tags `ready` / `active` / `live`; `app.MapZbHealth()` maps all three —
  `/health/ready` (tag `ready` → can this node serve?), `/health/active` (tag `active` → is this the
  leader/active node?), `/healthz` (predicate `_ => false` → bare process liveness).
- Canonical JSON response writer (lifts ScadaBridge's `HealthChecks.UI.Client` style to the default).
- `IActiveNodeGate` seam (generalized from ScadaBridge's `ActiveNodeGate`) + `MapZbHealth` integration.
- `GrpcDependencyHealthCheck` — "is my downstream gRPC dependency reachable" (MxGateway → worker;
  OtOpcUa → gateway channel).

**② `ZB.MOM.WW.Health.Akka`** (dep: Akka.Cluster)
- `AkkaClusterHealthCheck` with a configurable status policy. Default = ScadaBridge's
  (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded, else Unhealthy); OtOpcUa's (self-`Up`-among-
  members → Healthy/Degraded) ships as a preset.
- `ActiveNodeHealthCheck` with an optional role filter — role-less default gives ScadaBridge's
  `ActiveNode` (Up && leader); passing a role gives OtOpcUa's `AdminRoleLeader` behavior.

**③ `ZB.MOM.WW.Health.EntityFrameworkCore`** (dep: EF Core)
- `DatabaseHealthCheck<TContext>` — default probe `CanConnectAsync()` (ScadaBridge), optional
  probe-query delegate for OtOpcUa's "query `Deployments`" style.

**Stays per-project:** which probes each app registers; orchestrator/Traefik wiring; ScadaBridge's
`HealthMonitoring/` domain aggregation pipeline (distributed domain health, not an ASP.NET probe).

**Consumer matrix:** MxGateway → core only (+ gRPC-dep probe; no Akka/EF); OtOpcUa & ScadaBridge → all three.

## Library design — `ZB.MOM.WW.Telemetry` (2 packages)

**① `ZB.MOM.WW.Telemetry`** (OTel metrics + traces; deps: OpenTelemetry SDK + hosting/exporter)
- `builder.AddZbTelemetry(options)` — the missing front door:
  ```csharp
  builder.AddZbTelemetry(o => {
      o.ServiceName = "mxgateway";        // → Resource service.name
      o.ServiceNamespace = "ZB.MOM.WW";   // constant across the fleet
      o.SiteId = cfg.SiteId;              // → resource attr site.id
      o.NodeRole = cfg.NodeRole;          // → resource attr node.role
      o.Meters = ["MxGateway.Server"];    // app's own Meter name(s)
      o.ActivitySources = [...];          // app's own ActivitySource name(s)
      o.Exporter = Prometheus;            // default; OTLP opt-in
  });
  app.MapZbMetrics();                     // Prometheus /metrics
  ```
- Shared `Resource`: `service.name` + `service.namespace` + `service.version` + `site.id` +
  `node.role` + `host.name`. **The headline fix** — nobody sets this today.
- Standard instrumentation everyone should have (only OtOpcUa has it now): ASP.NET Core, gRPC client,
  HttpClient, runtime + process meters.
- Exporter: Prometheus `/metrics` default; **OTLP opt-in** via options (path to a real collector).
- App instruments stay per-project. MxGateway's hand-rolled `GatewayMetrics` keeps its 13/3/4
  instruments but its `Meter` is registered through `AddZbTelemetry` so it finally **exports** instead
  of dying in an in-memory snapshot.

**② `ZB.MOM.WW.Telemetry.Serilog`** (logs signal + Serilog convergence; deps: Serilog + the core package)
- `AddZbSerilog()` — shared two-stage bootstrap generalizing ScadaBridge's `LoggerConfigurationFactory`
  (`ReadFrom.Configuration` for sinks + explicit `MinimumLevel.Is` override).
- Shared enrichers `SiteId` / `NodeRole` / `NodeHostname`, **bound from the same options object as the
  OTel Resource** so logs and metrics carry identical dimensions.
- **NEW `TraceContextEnricher`** — stamps `trace_id`/`span_id` from `Activity.Current` onto every log
  event (makes a log line clickable from a trace; nobody has this today).
- OTel log export — logs flow through the OTel pipeline with the same Resource (all three signals
  correlated in a backend).
- `ILogRedactor` seam — generalized from MxGateway's `GatewayLogRedactor` (the only app with real
  secret redaction). The seam is shared; the policy (which fields/commands) stays per-project.

**Convergence the spec pins down:** Meter name = `<app>` namespace; instrument name =
`<app>.<subsystem>.<event>`; duration unit = **seconds** (OTel semconv) — flags MxGateway's `ms`
histograms as a convergence item.

### The one adoption — MxGateway MEL → Serilog
Replace `WebApplicationBuilder` default logging with `AddZbSerilog()`; re-express the
`GatewayLogScope`/`BeginScope` correlation middleware as a Serilog `LogContext.PushProperty` scope;
move `GatewayLogRedactor` behind the shared `ILogRedactor` seam. The net48 x86 worker's `IWorkerLogger`
(stderr key=value) stays bespoke — out of process and out of scope.

## Normalization component docs

Both trees follow `components/README.md`'s six-part layout (matching auth + ui-theme). Each `spec`
opens with a Section 0 stating normalized vs. left-per-project explicitly. `observability/` adds one
reference doc — `spec/METRIC-CONVENTIONS.md` — mirroring auth's `CANONICAL-ROLES.md` / theme's
`DESIGN-TOKENS.md`. Three `current-state/<project>/CURRENT-STATE.md` per component at full
code-verified depth, each ending in an Adoption plan. `GAPS.md` turns deltas into a prioritized
backlog (MxGateway "no probes" + "MEL→Serilog" are top entries; `ms`→`s` and the missing Resource are
convergence items). Both register at status **Draft** (`Draft → Reviewed → Adopting → Converged`).

## Testing & verification

Every package ships tests (mirrors auth's 172 / theme's 32; `dotnet test` from each library root):
- **Health** — `WebApplicationFactory` tests for the three tiers + JSON shape; `IActiveNodeGate`
  gates a route (200 active / 503 standby); `GrpcDependencyHealthCheck` on a stub channel.
- **Health.Akka** — table-driven status-policy + role-filter unit tests over faked cluster state.
- **Health.EntityFrameworkCore** — `DatabaseHealthCheck<T>` against SQLite in-memory (healthy / broken
  context / custom probe delegate).
- **Telemetry** — Resource carries every options attribute; in-memory exporter sees a registered app
  Meter's instrument; `MapZbMetrics` serves Prometheus text.
- **Telemetry.Serilog** — in-memory/TestCorrelator sink asserts enricher properties present;
  `TraceContextEnricher` stamps `trace_id`/`span_id` under an active `Activity` and omits cleanly
  otherwise; `ILogRedactor` scrubs a policy-marked secret.
- **MxGateway migration** — existing `MxGateway.Tests` (fake worker) still green + correlation scope
  still emits + secrets still redacted.

Verification gates (evidence, not assertions): each library `dotnet test` green + `dotnet pack`
produces nupkgs @ 0.1.0; MxGateway `dotnet build src/MxGateway.sln` + `dotnet test` green.

## Build order

```
1. components/health/ + components/observability/ docs   (spec first — drives the APIs)
2. ZB.MOM.WW.Health        (3 pkgs)                       ─┐ parallelizable
3. ZB.MOM.WW.Telemetry     (core: metrics+traces)         ─┘
4. ZB.MOM.WW.Telemetry.Serilog   (needs #3)
5. MxGateway MEL→Serilog migration  (needs #4)            ← the one sister-repo touch
6. Index/registry updates + GAPS cross-check
```

## Implementation tasks (native task IDs)

- #7  Build `ZB.MOM.WW.Health` library (3 packages)
- #8  Build `ZB.MOM.WW.Telemetry` library (2 packages)
- #9  Migrate MxGateway logging MEL → shared Serilog (sister-repo) — blocked by #8
- #10 Author `components/health/` normalization docs
- #11 Author `components/observability/` normalization docs

Dependency: #9 blocked by #8 (needs `ZB.MOM.WW.Telemetry.Serilog`). Docs (#10/#11) precede the
libraries logically (spec drives API) but can be drafted in parallel from the captured current-state.