merge: observability (register rows 33, 54, 56, 63, 66) — LocalDb backlog KPI history + sustained-threshold badge, honest stream-subscriber gauge rename + real data-connection gauges, central metrics port on the rig, ASPNETCORE_ENVIRONMENT on site containers, import-session rejection counter
This commit is contained in:
Joseph Doherty
@@ -295,7 +295,7 @@ public record AlarmStateChanged(
|
||||
|
||||
### Observability
|
||||
|
||||
`Observability/ScadaBridgeTelemetry` defines the singleton `Meter` named `ZB.MOM.WW.ScadaBridge` and the application-wide instrument definitions. Components call the static emit helpers (`RecordDeploymentApplied`, `SiteConnectionOpened`, etc.) rather than creating their own meters. Instruments are no-ops until an OTel listener attaches, so uninstrumented hosts pay no overhead.
|
||||
`Observability/ScadaBridgeTelemetry` defines the singleton `Meter` named `ZB.MOM.WW.ScadaBridge` and the application-wide instrument definitions. Components call the static emit helpers (`RecordDeploymentApplied`, `SiteStreamSubscriberOpened`, etc.) rather than creating their own meters. Instruments are no-ops until an OTel listener attaches, so uninstrumented hosts pay no overhead.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -52,6 +52,9 @@ Without this, an abandoned wizard or a plain CLI preview held one of the eight `
|
||||
**Session-cap observability (register row 53).** That leak was only ever inferable from rotating test failures, because nothing counted open sessions and nothing logged a rejection. Two signals now close the gap:
|
||||
|
||||
- **Gauge `scadabridge.central.transport.import_sessions.open`** — open (unexpired) sessions on this node, on the `ZB.MOM.WW.ScadaBridge` meter, which is already in `SiteServiceRegistration.ObservedMeters` (the `ZbTelemetryOptions.Meters` allowlist; an unlisted meter exports nothing, silently). The provider is registered by `BundleSessionEvictionService.StartAsync` and identity-checked-cleared on `StopAsync`, mirroring the StoreAndForward queue-depth gauge. `IBundleSessionStore.OpenSessionCount` deliberately excludes expired-but-unswept entries — counting the raw dictionary would report the node as pinned at the cap for up to a sweep interval after a burst had actually drained, which is the exact false positive this instrument exists to rule out. Transport is registered only in the host's Central branch, so the gauge reports 0 on a site node.
|
||||
- **Counter `scadabridge.central.transport.import_sessions.rejected`** (register row 66) — bundle-import sessions turned away because the cap was exhausted, on the same meter (so likewise no allowlist change). Row 53 shipped the gauge and a Warning log, but a log line cannot be alerted on; this is the series to alert on. It is incremented **only at the post-eviction recheck** in `BundleSessionStore.Open`: the cap is tested twice, and a store full of merely *expired* sessions reclaims and admits the caller, so counting the first test would fire the signal routinely on any node whose sweep has not yet run. Re-`Open`ing an already-open session id skips the cap block entirely and is likewise never counted.
|
||||
|
||||
Read the two instruments **together** — the pair is what discriminates the failure modes. Rejections climbing while the gauge oscillates *below* the cap is genuine concurrency: raise `MaxConcurrentImportSessions`. Rejections climbing while the gauge sits *pinned at* the cap is a session leak (the register row 43 signature); raising the cap only defers it, so find the leak instead.
|
||||
- **Warning log on rejection** — `BundleSessionStore.Open` logs the open count, the cap, and the TTL when it rejects. It fires only on a real cap hit: a store full of *expired* sessions is reclaimed first and logs nothing. A count that stays pinned at the cap across repeated warnings is a leak; an occasional warning is genuine concurrency, and the fix there is to raise `ScadaBridge:Transport:MaxConcurrentImportSessions` (each slot pins up to `MaxBundleSizeMb` of decrypted plaintext, so the cap is a memory budget).
|
||||
|
||||
### Conflict resolution and `BundleImportId` correlation
|
||||
|
||||
@@ -255,6 +255,32 @@ LocalDbReplicationConnected staying true with the backlog returning to ~0 is the
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
|
||||
row 33):
|
||||
|
||||
- **The "Replication backlog" badge** on the site tile of the Health dashboard. It is deliberately
|
||||
a *sustained* signal, not a bare threshold: the badge appears only once the backlog has stayed
|
||||
above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
|
||||
purge burst described above drains in seconds to a couple of minutes, so it never reaches the
|
||||
dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
|
||||
would fire once a day on every replicated site and be ignored within a week. The dwell is
|
||||
validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
|
||||
single sample and would collapse back into a bare threshold.
|
||||
- **The "LocalDb Replication Backlog" trend chart** in the dashboard's per-site Site Health Trends
|
||||
panel, backed by the KPI history store (sampled every minute, retained 90 days raw / 365 days
|
||||
hourly). This is what answers the question a point-in-time reading cannot: *did that spike
|
||||
drain?* A past purge burst appears as a spike that returns to ~0; a real fault appears as a
|
||||
level that never comes back down.
|
||||
|
||||
**Both surfaces are silent on an unreplicated site, by design.** `LocalDbOplogBacklog` and
|
||||
`LocalDbReplicationConnected` are nullable, and null means "no reading", never zero. The KPI
|
||||
sampler therefore writes **no sample at all** rather than a fabricated 0, so site-b/site-c show an
|
||||
empty chart rather than a flat zero line, and they never accumulate the dwell anchor the badge
|
||||
keys off. A gap in those series means "not replicated / not reporting" — which is the truth. Note
|
||||
that `false` and null differ for the connected flag: a configured link that is DOWN samples as 0,
|
||||
while no link at all samples as nothing.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
|
||||
@@ -108,7 +108,7 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
|
||||
|
||||
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel:
|
||||
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
|
||||
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
|
||||
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteStreamSubscriberOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
|
||||
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s.
|
||||
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
|
||||
|
||||
@@ -34,7 +34,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported |
|
||||
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
|
||||
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
|
||||
@@ -90,7 +90,7 @@ The Audit Log spans both sites (hot-path append + telemetry forward) and central
|
||||
|
||||
- **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
|
||||
- **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The per-site tile surfaces a warning badge when its `SiteAuditBacklog` crosses the configurable threshold.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).
|
||||
|
||||
These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs.
|
||||
|
||||
@@ -101,6 +101,75 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
|
||||
- Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
|
||||
- Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.
|
||||
|
||||
## Sustained LocalDb Replication Backlog Badge
|
||||
|
||||
The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has
|
||||
stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under
|
||||
`ScadaBridge:HealthMonitoring` and validated at startup.
|
||||
|
||||
- **Sustained, not instantaneous — and that is the design, not a refinement.** The daily
|
||||
`site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so
|
||||
the backlog jumps by the deleted batch size and drains as the peer acks it. A bare
|
||||
`backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is
|
||||
defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor`
|
||||
is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single
|
||||
sample and would collapse back into the bare threshold.
|
||||
- **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`,
|
||||
the instant the backlog first rose above the threshold. Reports at or below the threshold clear
|
||||
it; reports above it leave the original instant intact (a sustained window, not a sliding one).
|
||||
- **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged*
|
||||
rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is
|
||||
evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports
|
||||
null forever, so it never accumulates an anchor and can never be badged.
|
||||
- Offline sites are not badged; the Offline badge is the signal that matters there.
|
||||
|
||||
## KPI History Sampling of Replication State
|
||||
|
||||
`SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1)
|
||||
into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site
|
||||
Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the
|
||||
signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing
|
||||
distinguishing an expected purge burst from a stalled peer.
|
||||
|
||||
Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a
|
||||
coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a
|
||||
pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once
|
||||
a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site,
|
||||
14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not
|
||||
reporting". Note `false` and `null` are different states for the connected flag — a configured link
|
||||
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
|
||||
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).
|
||||
|
||||
## Prometheus Metrics (site node)
|
||||
|
||||
Distinct from the health *report* above: these are scraped from the site node's `/metrics`
|
||||
endpoint (`MetricsPort`, default 8084) rather than sent to central.
|
||||
|
||||
| Series | Meaning |
|
||||
| --- | --- |
|
||||
| `scadabridge_site_data_connections_connected` | Data connections on this node in the `Connected` state |
|
||||
| `scadabridge_site_data_connections_total` | Data connections known to this node, in any state |
|
||||
| `scadabridge_site_stream_subscribers_active` | Open central→site gRPC subscriber streams |
|
||||
|
||||
- The two `data_connections` gauges (register row 54) are the **only exported data-connection
|
||||
signal**; before them the health report's `DataConnectionStatuses` was visible in the UI and
|
||||
the CLI but never as a metric. `connected < total` is the alert condition — neither number
|
||||
alone distinguishes "a device is down" from "nothing is configured". They are fed directly by
|
||||
the Data Connection Layer at each connection state transition, deliberately **not** derived
|
||||
from the health-report path, so the metric survives a dead report pipeline (the exact failure
|
||||
the *metrics stale* flag exists to catch).
|
||||
- Both read **0 on the standby node** — data connections are opened by the deployment path on
|
||||
the active node only. A standby reporting 0 is healthy; alert on the active node, or on
|
||||
`max()` across the pair.
|
||||
- `scadabridge_site_stream_subscribers_active` was renamed from `scadabridge_site_connection_up`
|
||||
by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide
|
||||
alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name
|
||||
invited reading it as data-connection health, and did cause exactly that misdiagnosis once.
|
||||
- All three ride the `ZB.MOM.WW.ScadaBridge` meter, already in the `SiteServiceRegistration.ObservedMeters`
|
||||
allowlist. `ZbTelemetryOptions.Meters` is an **allowlist**: an unlisted meter exports nothing,
|
||||
with no error and no missing-metric signal.
|
||||
|
||||
## No Alerting
|
||||
|
||||
- Health monitoring is **display-only** for now — no automated notifications or alerts are triggered by health status changes.
|
||||
@@ -122,4 +191,4 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
|
||||
|
||||
- **Central UI**: Health Monitoring Dashboard displays aggregated metrics.
|
||||
- **Communication Layer**: Health reports flow as periodic messages.
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`. Of these, **only four are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, and `deadLetters` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The remaining eight are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
|
||||
@@ -143,6 +143,17 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: *
|
||||
- `disabledInstances`
|
||||
- `auditBacklogPending`
|
||||
- `eventLogWriteFailures`
|
||||
- `localDbOplogBacklog` — **conditionally emitted** (register row 33)
|
||||
- `localDbReplicationConnected` — **conditionally emitted** (register row 33), 0/1
|
||||
|
||||
The last two are written **only when the site actually reports a reading**. Both underlying health-report
|
||||
fields are nullable and null means "no reading" (replication not configured, or a failed poll), never
|
||||
zero — so unlike `auditBacklogPending` above, they are not coerced with `?? 0`. A site therefore emits
|
||||
**12 metrics when unreplicated and 14 when replicated**, and a gap in these two series truthfully means
|
||||
"not replicated / not reporting" rather than a flat zero line that would render an unreplicated pair, or
|
||||
one that cannot read its own oplog, as perfectly healthy. Both fold as **Gauge** in
|
||||
`KpiMetricAggregationCatalog` (they are instantaneous readings; summing an oplog depth or a 0/1 flag per
|
||||
hour would be meaningless). `localDbOplogBacklog` is charted on the Health dashboard's per-site trend panel.
|
||||
|
||||
## Query + UI
|
||||
|
||||
|
||||
@@ -134,7 +134,7 @@ flowchart TD
|
||||
class T muted
|
||||
```
|
||||
|
||||
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory. **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing.
|
||||
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory. **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing. **A log line is not alertable, so register row 66 adds the counter `scadabridge.central.transport.import_sessions.rejected`** on the same meter, incremented at the same single genuine-rejection point (the post-eviction recheck — never on the reclaim-then-succeed path, and never when re-opening an already-open session id). Alert on the counter; interpret it against the gauge — rejections rising while the gauge stays below the cap means genuine concurrency (raise the cap), whereas rejections rising while the gauge is pinned at the cap means a leaked session (raising the cap only defers it).
|
||||
|
||||
**Session release is explicit, not TTL-only (register row 43).** `ApplyAsync` releases the session it applied (zero-then-remove, T-007), and `IBundleImporter.Abandon(sessionId)` does the same for a session that will never be applied. Every abandonment path calls it: the Central UI wizard on new upload / Back-to-Upload / `ResetSessionState` / lockout reset / `Dispose` (navigation away or circuit teardown), the CLI `transport preview` handler (a preview never applies), and the CLI `transport import` handler for the aborts that precede apply (blocker rows, unresolved mappings). `Abandon` is idempotent — double release, an unknown id, and an already-expired id are all silent no-ops — and since session ids are random GUIDs handed only to their opener, abandoning the id you hold can never evict another operator's in-progress import. Relying on the TTL alone was an availability *and* secret-lifetime defect: eight abandoned wizards exhausted `MaxConcurrentImportSessions` and locked out every import on the active node for 30 minutes, each still holding its decrypted bundle content.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user