OTLP export not yet proven for the Central role (scadaproj#13 proved Site only) #133

New Issue

2026-11-12T20:45:23-05:00

dohertj2 commented

2026-11-12 20:45:23 -05:00

Fixed in dd4f1c2a (main, rebased on e36cd605). Owner decision, approved 2026-11-12: take the recommendation.

What changed

One direction. AlarmConditionState.Severity is higher = more urgent for every kind. New Commons.Types.Alarms.AlarmPriorityBands.ToSeverity is OtOpcUa's GalaxyAlarmPriority.Map (lmxopcua#887, b61f3e05) value for value: 1–250 → 1000, 251–500 → 750, 501–750 → 500, 751–999 → 250, with ≤ 0 → 1000 and > 999 → 250. It is used by MxGatewayAlarmMapper (the gateway severity is the AVEVA priority) and by AlarmConditionStateFactory.ForComputed (PriorityLevel). The OPC UA path is unchanged. OtOpcUa's history-only "not an alarm" row (AVEVA severity 0 → 1) has no live counterpart, so it is not ported.
Raw priority kept. AlarmConditionState.Priority (new, trailing int?), AlarmStateChanged.Priority and ScriptAlarm.Priority (new, trailing) all carry exactly the pre-#129 Severity: the AVEVA priority clamped to 0–1000, PriorityLevel, or Part 9 Severity for OPC UA. There is no proto change. AlarmStateUpdate.priority keeps its meaning, and central derives Severity by kind (SeverityFromPriority), so the decode is right for un-upgraded sites too. The Debug View snapshot decode restores Condition.Priority. NativeAlarmActor upgrades pre-#129 native_alarm_state rows on rehydrate.
MES 900–999 band. It keys off Priority, never Severity. A test pins Priority == pre-#129 Severity for every input from int.MinValue to int.MaxValue, so the band selects identical alarms. The rig scripts docker/seed-mes-alarm-demo.sh and seed-mes-test-machine.sh now use a.Priority for the filters and the MES Severity field, so their output is unchanged. Deployed scripts must make the same switch: deploy gate #131.
Consumers.
- Changed: AlarmActor raise event-log severity is now banded severity ≥ 700 (was raw priority ≥ 700); SiteStreamGrpcClient.ConvertToDomainEvent; SiteCommandDtoMapper.FromProto; ScriptRuntimeContext.ToScriptAlarm; NativeAlarmActor (Emit/rehydrate).
- Now correct with no code change: NativeAlarmActor.ErrorSeverityThreshold, AlarmSummaryService worst = max, AlarmSummary.razor >= filter and descending sort, AlarmStateBadges.
- Deliberately left alone: CLI debug snapshot Priority column (raw, labelled), AlarmContext.Priority (raw), PriorityLevel authoring in UI/CLI/Transport. Notifications, Inbound API host code and Flow Debugger have no severity sort or filter.
Docs: Component-Commons, -DataConnectionLayer, -SiteRuntime, -Communication, -CentralUI, -InboundAPI, HighLevelReqs, docs/plans/2026-06-30-mes-alarm-status-api.md, CLAUDE.md, CLI README.

Tests (Release, rebased tree): Commons 977/977, DataConnectionLayer 549/549, Communication 834/834, SiteRuntime 1055/1055, CentralUI (bUnit) 1545/1545, IntegrationTests SiteAlarmStream 2/2. One unrelated DCL flake (TagResolutionCounts_ConnectionLevelFailureThenRecovery…) failed once and passed on its own and in every later full run.

New coverage: the band table at every edge and clamp, monotonicity, MxGateway live and snapshot, the MES-band invariance, the stream decode per kind, Debug snapshot Priority, legacy-row rehydrate, the computed and native event-log thresholds, Alarm Summary worst/sort.
Mutation check: reverting ToSeverity to the old clamp turned 62 tests red across all five projects, bUnit included. Keying Priority off the new Severity turned the MES-band test (plus 3 others) red. Both were restored.

Consumer impact: see the previous comment. No deploys were made.

**Fixed in `dd4f1c2a`** (main, rebased on `e36cd605`). Owner decision, approved 2026-11-12: take the recommendation. **What changed** 1. **One direction.** `AlarmConditionState.Severity` is higher = more urgent for every kind. New `Commons.Types.Alarms.AlarmPriorityBands.ToSeverity` is OtOpcUa's `GalaxyAlarmPriority.Map` (lmxopcua#887, `b61f3e05`) value for value: 1–250 → 1000, 251–500 → 750, 501–750 → 500, 751–999 → 250, with ≤ 0 → 1000 and > 999 → 250. It is used by `MxGatewayAlarmMapper` (the gateway `severity` is the AVEVA priority) and by `AlarmConditionStateFactory.ForComputed` (`PriorityLevel`). The OPC UA path is unchanged. OtOpcUa's history-only "not an alarm" row (AVEVA severity 0 → 1) has no live counterpart, so it is not ported. 2. **Raw priority kept.** `AlarmConditionState.Priority` (new, trailing `int?`), `AlarmStateChanged.Priority` and `ScriptAlarm.Priority` (new, trailing) all carry exactly the pre-#129 Severity: the AVEVA priority clamped to 0–1000, `PriorityLevel`, or Part 9 Severity for OPC UA. There is no proto change. `AlarmStateUpdate.priority` keeps its meaning, and central derives Severity by kind (`SeverityFromPriority`), so the decode is right for un-upgraded sites too. The Debug View snapshot decode restores `Condition.Priority`. `NativeAlarmActor` upgrades pre-#129 `native_alarm_state` rows on rehydrate. 3. **MES 900–999 band.** It keys off `Priority`, never `Severity`. A test pins `Priority` == pre-#129 `Severity` for every input from `int.MinValue` to `int.MaxValue`, so the band selects identical alarms. The rig scripts `docker/seed-mes-alarm-demo.sh` and `seed-mes-test-machine.sh` now use `a.Priority` for the filters and the MES `Severity` field, so their output is unchanged. **Deployed scripts must make the same switch: deploy gate #131.** 4. **Consumers.** - Changed: `AlarmActor` raise event-log severity is now banded severity ≥ 700 (was raw priority ≥ 700); `SiteStreamGrpcClient.ConvertToDomainEvent`; `SiteCommandDtoMapper.FromProto`; `ScriptRuntimeContext.ToScriptAlarm`; `NativeAlarmActor` (Emit/rehydrate). - Now correct with no code change: `NativeAlarmActor.ErrorSeverityThreshold`, `AlarmSummaryService` worst = max, `AlarmSummary.razor` `>=` filter and descending sort, `AlarmStateBadges`. - Deliberately left alone: CLI `debug snapshot` Priority column (raw, labelled), `AlarmContext.Priority` (raw), `PriorityLevel` authoring in UI/CLI/Transport. Notifications, Inbound API host code and Flow Debugger have no severity sort or filter. 5. **Docs:** Component-Commons, -DataConnectionLayer, -SiteRuntime, -Communication, -CentralUI, -InboundAPI, HighLevelReqs, `docs/plans/2026-06-30-mes-alarm-status-api.md`, CLAUDE.md, CLI README. **Tests** (Release, rebased tree): Commons 977/977, DataConnectionLayer 549/549, Communication 834/834, SiteRuntime 1055/1055, CentralUI (bUnit) 1545/1545, IntegrationTests `SiteAlarmStream` 2/2. One unrelated DCL flake (`TagResolutionCounts_ConnectionLevelFailureThenRecovery…`) failed once and passed on its own and in every later full run. - New coverage: the band table at every edge and clamp, monotonicity, MxGateway live and snapshot, the MES-band invariance, the stream decode per kind, Debug snapshot `Priority`, legacy-row rehydrate, the computed and native event-log thresholds, Alarm Summary worst/sort. - **Mutation check:** reverting `ToSeverity` to the old clamp turned 62 tests red across all five projects, bUnit included. Keying `Priority` off the new Severity turned the MES-band test (plus 3 others) red. Both were restored. Consumer impact: see the previous comment. No deploys were made.

dohertj2 added the type:followup type:test priority:p3 labels 2026-11-12 20:45:23 -05:00

dohertj2 commented

2026-11-12 21:30:22 -05:00

Update from scadaproj#55 (2026-11-12): ScadaBridge main a633005f pins Telemetry 0.3.0. With ScadaBridge:Telemetry:Exporter=Otlp, logs now go to the collector as well, alongside metrics and traces. The Resource also carries service.version. This is proven on a throwaway Site node (78 log records with site.id/node.role/service.version).

Central still not run. The docker rig was busy with the #132 failover drills (sb132-p/sb132-probe were running, and the centrals had been recreated minutes earlier). The "Do" above still applies; when it is run, also check for log records with node.role=Central, e.g. python3 ../otelidentity.py scadabridge on the collector host.

Update from scadaproj#55 (2026-11-12): ScadaBridge `main` `a633005f` pins Telemetry `0.3.0`. With `ScadaBridge:Telemetry:Exporter=Otlp`, **logs** now go to the collector as well, alongside metrics and traces. The Resource also carries `service.version`. This is proven on a throwaway Site node (78 log records with `site.id`/`node.role`/`service.version`). **Central still not run.** The docker rig was busy with the #132 failover drills (`sb132-p`/`sb132-probe` were running, and the centrals had been recreated minutes earlier). The "Do" above still applies; when it is run, also check for log records with `node.role=Central`, e.g. `python3 ../otelidentity.py scadabridge` on the collector host.

dohertj2 commented

2026-11-12 21:50:16 -05:00

Done: Central role over OTLP is proven on the docker rig (2026-11-12). Evidence: scadaproj components/observability/2026-11-12-otlp-end-to-end-evidence.md, section "ScadaBridge Central role over OTLP" (scadaproj main d93ca7b).

Run: the rig image predated Telemetry 0.3.0 (its Telemetry dll was 0.2.0), so it was rebuilt with docker/deploy.sh from a clean main at 482ac665. A temporary compose override set ScadaBridge__Telemetry__Exporter=Otlp + OtlpEndpoint=http://10.100.0.35:4317 on central-a and central-b, and both were recreated together. Activity: 9 rejected inbound calls (401), 3 authorized POST /api/SimpleAlarmStatus (200) with a temporary scoped key (deleted afterwards), one deploy instance --id 1492 (Success), and CLI/UI/health traffic.

Arrived at zb-otel-collector. The Resource is identical across metrics, traces and logs: service.name=scadabridge, service.namespace=ZB.MOM.WW, service.version=1.0.0, node.role=Central, site.id=central, service.instance.id/host.name = container id.

node	metric points	spans	log records
central-a (active)	463	321	3305 (3046 Info / 259 Warn)
central-b (standby)	421	77	331 (311 Info / 20 Warn)

Central instruments (central-a, cumulative): scadabridge.deployments.applied = 1; scadabridge.inbound_api.requests {SimpleAlarmStatus} = 3, {<unauthorized>} = 9. These match the activity exactly. Also central.transport.import_sessions.open and the shared gauges. The standby exports only the gauges. Spans include the CentralControlService gRPC server calls (Heartbeat/SubmitNotification/IngestAuditEvents/ReportSiteHealth), POST /management, POST /api/{methodName}, the deployment config fetch, and the gRPC client spans SiteCommandService/ExecuteRoute/ExecuteLifecycle. The first log record is the pre-host Starting ScadaBridge host as Central on scadabridge-central-a.

Allowlist: ObservedMeters covers every meter in the repo (ZB.MOM.WW.ScadaBridge + LocalDb), and there is no app ActivitySource. No defect, no code change.

Restored: both centrals were recreated together on the default config (0 telemetry env vars). All 8 nodes /health/ready 200, one active per pair, central memberCount 2 / unreachableCount 0, Traefik /health/active 200. The new central processes send nothing to the collector (negative control). The rig stays on the rebuilt 482ac665 image.

Follow-ups filed:

#134: rig images stamp service.version without the +<sha>, because .dockerignore excludes .git/. Rig-only.
scadaproj#56: the Serilog OTLP sink's own LogsService/Export calls are traced as gRPC client spans in all four apps. It is a library fix in ZB.MOM.WW.Telemetry.

Closing.

**Done: Central role over OTLP is proven on the docker rig (2026-11-12).** Evidence: scadaproj `components/observability/2026-11-12-otlp-end-to-end-evidence.md`, section "ScadaBridge Central role over OTLP" (scadaproj `main` `d93ca7b`). **Run:** the rig image predated Telemetry 0.3.0 (its Telemetry dll was `0.2.0`), so it was rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665`. A temporary compose override set `ScadaBridge__Telemetry__Exporter=Otlp` + `OtlpEndpoint=http://10.100.0.35:4317` on central-a **and** central-b, and both were recreated together. Activity: 9 rejected inbound calls (401), 3 authorized `POST /api/SimpleAlarmStatus` (200) with a temporary scoped key (deleted afterwards), one `deploy instance --id 1492` (Success), and CLI/UI/health traffic. **Arrived at `zb-otel-collector`**. The Resource is identical across metrics, traces and logs: `service.name=scadabridge`, `service.namespace=ZB.MOM.WW`, `service.version=1.0.0`, **`node.role=Central`**, **`site.id=central`**, `service.instance.id`/`host.name` = container id. | node | metric points | spans | log records | |---|---|---|---| | central-a (active) | 463 | 321 | 3305 (3046 Info / 259 Warn) | | central-b (standby) | 421 | 77 | 331 (311 Info / 20 Warn) | Central instruments (central-a, cumulative): `scadabridge.deployments.applied` = **1**; `scadabridge.inbound_api.requests` {SimpleAlarmStatus} = **3**, {`<unauthorized>`} = **9**. These match the activity exactly. Also `central.transport.import_sessions.open` and the shared gauges. The standby exports only the gauges. Spans include the CentralControlService gRPC server calls (Heartbeat/SubmitNotification/IngestAuditEvents/ReportSiteHealth), `POST /management`, `POST /api/{methodName}`, the deployment config fetch, and the gRPC client spans `SiteCommandService/ExecuteRoute`/`ExecuteLifecycle`. The first log record is the pre-host `Starting ScadaBridge host as Central on scadabridge-central-a`. **Allowlist:** `ObservedMeters` covers every meter in the repo (`ZB.MOM.WW.ScadaBridge` + LocalDb), and there is no app `ActivitySource`. **No defect, no code change.** **Restored:** both centrals were recreated together on the default config (0 telemetry env vars). All 8 nodes `/health/ready` 200, one active per pair, central `memberCount 2 / unreachableCount 0`, Traefik `/health/active` 200. The new central processes send nothing to the collector (negative control). The rig stays on the rebuilt `482ac665` image. **Follow-ups filed:** - #134: rig images stamp `service.version` without the `+<sha>`, because `.dockerignore` excludes `.git/`. Rig-only. - scadaproj#56: the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in all four apps. It is a library fix in `ZB.MOM.WW.Telemetry`. Closing.

dohertj2 closed this issue

2026-11-12 21:50:17 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#133