# Code Review — Telemetry
| Field | Value |
|-------|-------|
| Library | `ZB.MOM.WW.Telemetry/` |
| Packages | `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Telemetry.Serilog` |
| Component spec | `components/observability/spec/SPEC.md` |
| Shared contract | `components/observability/shared-contract/ZB.MOM.WW.Telemetry.md` |
| Status | Reviewed |
| Last reviewed | 2026-11-09 (re-review; baseline 2026-06-01 at `5f75cd4`) |
| Reviewer | Claude (2026-11-09 re-review, scadaproj#15; baseline: Claude automated) |
| Commit reviewed | `91fb2b4` |
| Open findings | 0 |
## Summary
**2026-11-09 re-review (`91fb2b4`, scadaproj#15).** Re-reviewed at `91fb2b4` (2026-11-09) against the 2026-06-01 baseline `5f75cd4`. Source is at `0.2.0`,
which is built but not published; the feed still carries `0.1.0`. Changes since the baseline:
- nested redaction (`05cc62a`): `RedactionEnricher` projects `StructureValue`/`SequenceValue`/
`DictionaryValue` into mutable views, honours removals, and writes back only what changed;
- one canonical Resource-attribute map (`ZbResource.BuildAttributes`) shared by the OTel SDK and the
Serilog OTLP sink;
- an eager OTLP options validator;
- the 0.2.0 `ZbSerilogOptions` hooks (`MinimumLevelGovernor`, `ConfigureLogger`).
The baseline redaction fixes (Telemetry-001/002/003) hold: removal and nested masking work, and are
tested. The Resource map is genuinely single-sourced. The DI extensions have container-building tests
(`AddZbTelemetryTests`, `MultiHostTests`, `MinimumLevelGovernanceTests`). All 42 tests pass (Serilog 30,
core 12).
There are three new Low findings, each verified with a scratchpad probe:
- **Telemetry-009:** enrichers added through the new `ConfigureLogger` hook run after
`RedactionEnricher`, so properties they add are never redacted.
- **Telemetry-010:** the redaction projection keys logged dictionaries by `ToString()`. Distinct keys
that stringify the same, such as `DateTime` keys differing only by milliseconds, collapse, and an
entry is silently dropped from the log even when the redactor changes nothing.
- **Telemetry-011:** the Telemetry-006 fail-fast validator accepts a scheme-less `host:port` endpoint
as an "absolute URI", and the pre-built-options `AddZbTelemetry(IServiceCollection, ZbTelemetryOptions)`
overload skips validation entirely.
**2026-06-01 baseline (`5f75cd4`).** The library is small, focused, and well-structured: two packages with a clean Serilog/OTel
boundary (the `Serilog.*` stack appears only in the `.Serilog` package; the core package is
pure OTel + ASP.NET Core framework reference), correct argument validation, deliberate
`sealed` types, thorough XML docs, and a deliberate no-process-global-state design for
`AddZbSerilog` that is well covered by `MultiHostTests`. The identity triple, Resource
omission rules, exporter wiring (Prometheus always-on, OTLP additive), and trace/log
correlation all match the spec's intent and are exercised by the 19 tests.
The most material problems are in the redaction seam — the one component the review brief
flags as security-critical. `RedactionEnricher` honours only *replacement* of scalar
properties: it silently ignores the redactor **removing** a key (a documented capability of
`ILogRedactor`), and it cannot see inside destructured/structured property values, so a
secret logged as a field of `{@Object}` is never scrubbed. Both let secrets reach sinks
despite a conforming redactor. Secondary themes: a spec drift around an undocumented
`service.instance.id` Resource attribute, two hand-maintained Resource-attribute builders
that can drift apart, and a stale doc-comment on `MapZbMetrics`. Tests are solid for the
happy paths but have no coverage for redactor removal or structured-value redaction.
## Checklist coverage
2026-11-09 re-review at `91fb2b4`:
| # | Category | Examined | Notes |
|---|----------|----------|-------|
| 1 | Correctness & logic bugs | ☑ | Nested projection/rebuild, removal reconciliation and change detection (`ValueEquals`) are correct, except that dictionary keys collide on `ToString()` → Telemetry-010. Governor ordering (after `ReadFrom.Configuration`) works as documented. |
| 2 | Public API surface & compatibility | ☑ | Additive: `ZbResource.BuildAttributes` (public), the `ZbSerilogOptions` overload, and `InternalsVisibleTo` for the Serilog package. The options-object `AddZbTelemetry` overload skips the new validation → Telemetry-011. |
| 3 | Concurrency & thread safety | ☑ | Enricher state is a `Lazy` with `ExecutionAndPublication`, and projection is per event. No issues found. |
| 4 | Error handling & resilience | ☑ | The validator gives named `ArgumentException`s, but its absolute-URI test is too permissive and is bypassed by one overload → Telemetry-011. |
| 5 | Security & secret handling | ☑ | Redaction reach is now top-level plus nested, with removal honoured. Properties added by `ConfigureLogger` enrichers bypass redaction → Telemetry-009. `Exception` text remains outside the redactor seam, by design and unchanged. |
| 6 | Performance & resource management | ☑ | The per-event projection allocates only when a redactor is registered and the event has properties. Untouched properties are not reallocated. No issues found. |
| 7 | Spec & shared-contract adherence | ☑ | Resource attributes (`service.*`, `host.name`, `site.id`, `node.role`) are identical across all three signals via `BuildAttributes`. No issues found. |
| 8 | Packaging, dependencies & project layout | ☑ | Version 0.2.0; repository URLs corrected to `scadaproj`; the Serilog stack is still confined to `.Serilog`. No issues found. |
| 9 | Testing coverage | ☑ | 42 tests pass, including removal and nested redaction and governor precedence. Untested: enricher ordering against `ConfigureLogger` (Telemetry-009), colliding dictionary keys (Telemetry-010), a scheme-less OTLP endpoint (Telemetry-011). |
| 10 | Documentation & XML docs | ☑ | The `ILogRedactor`/README redaction-reach text is accurate. `ConfigureLogger` docs advertise "extra enrichers" without saying they run after redaction (Telemetry-009). |
2026-06-01 baseline checklist (5f75cd4)
| # | Category | Examined | Notes |
|---|----------|----------|-------|
| 1 | Correctness & logic bugs | ☑ | Redactor "remove" path is a no-op (Telemetry-001); structured values opaque to redactor (Telemetry-002). |
| 2 | Public API surface & compatibility | ☑ | Surface minimal, `sealed`, nullable-correct. `ZbResource.InstanceId` is an added public member not in the contract (Telemetry-004). |
| 3 | Concurrency & thread safety | ☑ | No issues found. Enrichers stateless; `Lazy` uses `ExecutionAndPublication`; `Activity.Current` is async-local. |
| 4 | Error handling & resilience | ☑ | Guard clauses present. `new Uri(OtlpEndpoint)` can throw late on malformed input (Telemetry-006). |
| 5 | Security & secret handling | ☑ | Redaction gaps (Telemetry-001/002) are security-relevant — secrets can survive a conforming redactor. |
| 6 | Performance & resource management | ☑ | Per-event dictionary snapshot when a redactor is registered (Telemetry-007); acceptable but noted. |
| 7 | Spec & shared-contract adherence | ☑ | Undocumented `service.instance.id` attribute (Telemetry-004); two Resource builders that can drift (Telemetry-005). |
| 8 | Packaging, dependencies & project layout | ☑ | No issues found. Serilog stack confined to `.Serilog`; central versions; correct net10.0; framework ref justified. |
| 9 | Testing coverage | ☑ | No tests for redactor removal or structured-value redaction (Telemetry-003). |
| 10 | Documentation & XML docs | ☑ | `MapZbMetrics` doc-comment is stale: claims "only valid when Exporter = Prometheus" (Telemetry-008). |
## Findings
### Telemetry-001 — `RedactionEnricher` ignores property removal, leaving secrets in the event
| | |
|--|--|
| Severity | High |
| Category | Security & secret handling |
| Status | Resolved |
| Location | `ZB.MOM.WW.Telemetry/src/ZB.MOM.WW.Telemetry.Serilog/RedactionEnricher.cs:49-67` |
**Description**
`ILogRedactor.Redact` is documented to let a project "remove or replace any sensitive
values" (`ILogRedactor.cs:13` and the XML doc on the interface method: *"remove or replace"*;
the shared contract repeats *"Remove or replace any sensitive values"*). `RedactionEnricher`
builds a `snapshot` dictionary, hands it to the redactor, then writes back only via
`AddOrUpdateProperty` for entries that remain in the snapshot and `HasChanged`:
```csharp
foreach (var entry in snapshot)
{
if (HasChanged(logEvent, entry.Key, entry.Value))
logEvent.AddOrUpdateProperty(propertyFactory.CreateProperty(entry.Key, entry.Value));
}
```
If a redactor *removes* a key from the dictionary (`properties.Remove("apiKey")`) — the most
natural way to implement "must not leave the process" — that key simply no longer appears in
the write-back loop, so the original property is **never removed from `logEvent`**. The
secret reaches every sink unredacted, even though the redactor did exactly what its contract
permits. This defeats the seam's stated operational guarantee ("secrets never leave the
process in log events") for any removal-style redactor.
**Recommendation**
After calling the redactor, reconcile deletions: for each property key present on the
original `logEvent` but absent from the returned `snapshot`, call
`logEvent.RemovePropertyIfPresent(key)`. (Capture the original key set before mutation, then
diff.) Add a test asserting a removing redactor scrubs the property (see Telemetry-003).
**Resolution**
Resolved in `544a6dd`, 2026-06-01 — `RedactionEnricher` now captures the original property
key set and calls `RemovePropertyIfPresent` for any key the redactor dropped from the snapshot,
so a removing redactor scrubs the property; covered by a new removing-redactor test.
### Telemetry-002 — Redactor cannot inspect or scrub destructured/structured property values
| | |
|--|--|
| Severity | Medium |
| Category | Security & secret handling |
| Status | Resolved |
| Location | `ZB.MOM.WW.Telemetry/src/ZB.MOM.WW.Telemetry.Serilog/RedactionEnricher.cs:49-55` |
**Description**
The snapshot only unwraps `ScalarValue`; every other `LogEventPropertyValue`
(`StructureValue` from `{@Object}`, `SequenceValue`, `DictionaryValue`) is passed to the
redactor as the raw Serilog wrapper object:
```csharp
snapshot[property.Key] = property.Value is ScalarValue scalar ? scalar.Value : property.Value;
```
A project redactor written against the seam (`IDictionary` of "values")
therefore sees an opaque `StructureValue` for a destructured payload — it cannot read or
mask a secret *field inside* a logged object (e.g. `logger.Information("{@Command}", cmd)`
where `cmd.ApiKey` is sensitive). MxGateway's reference redactor specifically guards
"which command payloads must not leave the process" (per `ILogRedactor` XML doc and the
contract), which is precisely the destructured-object case. The seam silently cannot meet
that requirement; the redactor only works for top-level scalar properties.
**Recommendation**
Document the seam's actual reach (scalar top-level properties only) on `ILogRedactor` and in
the shared contract, and/or recursively project `StructureValue`/`SequenceValue`/
`DictionaryValue` into the snapshot and rebuild them on write-back so nested fields are
redactable. At minimum, make the limitation explicit so consumers do not assume nested
payloads are scrubbed when they are not.
**Resolution**
Resolved in `544a6dd`, 2026-06-01 (documented the scalar-only limitation), then **superseded by
`05cc62a`, 2026-06-01 — nested redaction implemented.** `RedactionEnricher` now projects each
structured value into a mutable nested view the redactor descends into recursively
(`StructureValue` → `IDictionary`, `SequenceValue` → `IList