Joseph Doherty
325 lines
23 KiB
Markdown
325 lines
23 KiB
Markdown
# Live Telemetry Transport (v2)
|
|
|
|
## Overview
|
|
|
|
The AdminUI's live observability panels (`/alerts`, `/script-log`, `/hosts` driver table, driver
|
|
resilience status) are fed by four node→central event channels. Historically all four rode
|
|
DistributedPubSub (DPS) on the shared Akka mesh — the same gossip ring that carries redundancy
|
|
state and the command-plane topics. Per-cluster mesh **Phase 5** adds a second transport for these
|
|
four channels only: one gRPC server-streaming contract, selected per node/central pair by
|
|
`Telemetry:Mode` / `TelemetryDial:Mode` (`Dps` default | `Grpc`).
|
|
|
|
This is the same motivation as [Phase 2's `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)
|
|
and [Phase 3's `ConfigSource`/`ConfigServe`](Configuration.md#configsource--configserve-config-fetch-and-cache):
|
|
DPS only works when central and the node share a gossip ring, and [Phase 6](plans/2026-07-21-per-cluster-mesh-design.md)
|
|
splits the fleet into one Akka mesh per application `Cluster`. Once that split lands, a driver node
|
|
in a site's mesh is no longer a cluster member of central's mesh, so DPS can no longer reach it —
|
|
telemetry needs its own out-of-band transport, same as commands (Phase 2) and config bytes (Phase 3).
|
|
|
|
## Direction: the node hosts, central dials
|
|
|
|
**Load-bearing and easy to get backwards:** each **driver** node hosts the telemetry gRPC server
|
|
(a dedicated Kestrel h2c listener); **central (admin role) is the client** and dials in. This
|
|
mirrors the inversion ScadaBridge already uses for the same problem (its `SiteStreamService`).
|
|
|
|
```
|
|
driver node central (admin)
|
|
┌─────────────────────┐ ┌──────────────────────────┐
|
|
│ 4 publish seams │ │ TelemetryDialSupervisor │
|
|
│ -> node-local hub │ Subscribe │ - discovers ClusterNode │
|
|
│ -> DPS (unchanged) │◄─────────────│ rows (Host+GrpcPort) │
|
|
│ │ (gRPC │ - one reconnecting │
|
|
│ TelemetryStreamGrpc- │ stream) │ dialer per node │
|
|
│ Service (h2c server) │─────────────►│ - feeds the SAME sinks │
|
|
│ Telemetry:GrpcListen- │ TelemetryEvent │ the DPS bridges feed │
|
|
│ Port │ (oneof, 4 kinds)│ today │
|
|
└─────────────────────┘ └──────────────────────────┘
|
|
```
|
|
|
|
A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its
|
|
redundant pair peer, the same way central dials site nodes.
|
|
|
|
**This same driver-role listener also hosts the mesh gRPC command-plane ingress.** Whenever
|
|
`Telemetry:GrpcListenPort > 0`, the node maps both `TelemetryStreamGrpcService` (this stream) and
|
|
`NodeCommandService` (central→node commands under `MeshTransport:Mode=Grpc` — see
|
|
[`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
|
|
on the one port. They share it rather than each claiming its own, since both are dialled by the same
|
|
central admin nodes and a second cleartext h2c port would buy nothing but another firewall rule.
|
|
`NodeCommandService` is gated by its own `MeshCommandAuthInterceptor`/`GrpcPsk` bearer, independent
|
|
of this stream's `Telemetry:ApiKey`.
|
|
|
|
Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` +
|
|
`GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery
|
|
Phase 1's ack set and Phase 2's gRPC command dial already use (see
|
|
[`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)).
|
|
No shared gossip membership is required to find or dial a node.
|
|
|
|
## The dark switch
|
|
|
|
`Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` (central/dial side) each default to
|
|
`Dps` and can be independently set to `Grpc`. Both code paths are compiled into every binary —
|
|
flipping either flag is an appsettings/env change plus a restart, **not a redeploy or rebuild** —
|
|
the same discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
|
|
|
|
What actually changes per mode, precisely:
|
|
|
|
- **The node ALWAYS does both things, regardless of `Telemetry:Mode`.** Every one of the four
|
|
publish seams emits into the node-local `ITelemetryLocalHub` **and** publishes to DPS,
|
|
unconditionally. The node additionally **always** hosts the gRPC server whenever
|
|
`Telemetry:GrpcListenPort > 0` — hosting is not mode-gated at all. `Telemetry:Mode` only affects
|
|
central's own bookkeeping about which side it expects to be consulted (it is otherwise inert on
|
|
the node).
|
|
- **Only central's ingest source switches**, driven by `TelemetryDial:Mode`:
|
|
- `Dps` (default) — today's four DPS SignalR bridges (alert, script-log, driver-status,
|
|
resilience) subscribe and feed the AdminUI sinks. Unchanged behavior.
|
|
- `Grpc` — those four DPS bridges are **not spawned**. Instead the `TelemetryDialSupervisor`
|
|
actor dials every driver node's gRPC stream and feeds the **identical** in-process sinks
|
|
(`IInProcessBroadcaster<AlarmTransitionEvent>`, `IInProcessBroadcaster<ScriptLogEntry>`,
|
|
`IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`).
|
|
- **The AdminUI panels themselves are untouched.** `/alerts`, `/script-log`, and the `/hosts`
|
|
driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps.
|
|
|
|
Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be
|
|
flipped node-by-node and central-by-central with no coordination window where telemetry is lost —
|
|
the node is always serving, so central can switch to `Grpc` whenever it likes.
|
|
|
|
## The four migrated channels
|
|
|
|
| DPS topic (today) | Domain record | Central sink fed |
|
|
|---|---|---|
|
|
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`); `/alerts` page; live/disconnected pill |
|
|
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`); `/script-log` page; live pill |
|
|
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore`; `/hosts` driver table |
|
|
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` |
|
|
|
|
These four are carried as `oneof` event kinds on one `TelemetryStreamService.Subscribe` RPC — a
|
|
single stream per (central, driver-node) pair carries all four, rather than one stream per topic.
|
|
Proto field evolution is additive-only (never renumber/reuse a tag), locked by a contract test that
|
|
reflects over the `oneof` cases.
|
|
|
|
### `driver-health` is keyed per driver instance, not per Akka member
|
|
|
|
`DriverHealthChanged` carries no per-Akka-member identity — a snapshot names the driver instance it
|
|
describes, never the node that published it. `IDriverStatusSnapshotStore` therefore keys on the
|
|
driver instance, and the `/hosts` driver table renders **cluster-scoped** rows: one entry per
|
|
configured driver instance, grouped under its cluster, rather than one entry per (member, driver)
|
|
pair. A pair's two nodes running the same deployed driver contribute to the same row, with the most
|
|
recent snapshot winning.
|
|
|
|
That is the correct granularity for what the channel reports — a driver instance's transport health
|
|
is a property of the instance's configuration and its device, not of which member happens to host
|
|
it — but it does mean the panel cannot answer "which node saw this?". Adding that would mean
|
|
widening the wire record, not changing the page. The one sub-instance breakdown the channel *does*
|
|
carry is per-host connectivity, below.
|
|
|
|
### Driver diagnostics ride `driver-health` (Gitea #640)
|
|
|
|
A driver may implement the optional `IDriverDiagnostics` (Core.Abstractions): a pure in-memory
|
|
`IReadOnlyDictionary<string,string> GetDiagnostics()` of operator-facing facts that are not driver
|
|
*state* (S7's negotiated PDU size is the first; others opt in later). `DriverInstanceActor` pulls it on
|
|
every health publish (try/caught, unguarded by the invoker like `GetHostStatuses`), bounds it
|
|
(`DriverDiagnostics.Bound`: 16 entries, text truncated to 200 chars, ordinal key order), and carries it as
|
|
`DriverHealthChanged.Diagnostics` -> proto `DriverHealth.has_diagnostics = 13` / `map<string,string>
|
|
diagnostics = 14` (additive; old nodes never set them, old central ignores them) -> `HostsDriverRow.DiagnosticItems`,
|
|
rendered as a small key/value list under the driver name on `/hosts` (nothing when empty).
|
|
|
|
- **null ≠ empty**: null = driver lacks the capability; the presence flag preserves an empty bag.
|
|
- **The diagnostics digest is part of the publish fingerprint, and must stay there** - a diagnostic fact
|
|
(e.g. PDU size changing on a reconnect) moves no other component, so without it the dedup swallows the
|
|
publish. Pinned by `DriverInstanceActorDiagnosticsTests.A_diagnostics_change_republishes_despite_otherwise_unchanged_health`
|
|
(stable-health stub - the shared `StubDriver` returns `UtcNow` and would pass vacuously).
|
|
|
|
### Per-host connectivity rides `driver-health` (Gitea #521)
|
|
|
|
`DriverHealthChanged.HostStatuses` carries each driver's `IHostConnectivityProbe.GetHostStatuses()`
|
|
result, rendered as the **Hosts** column on `/hosts`. It exists to show the one thing the driver-level
|
|
state chip structurally cannot: a **multi-device driver stays aggregate-`Healthy` while one of its
|
|
devices is unreachable**. A FOCAS or AbLegacy instance owning several PLCs previously hid that
|
|
entirely.
|
|
|
|
Three things to know before touching it:
|
|
|
|
- **This deliberately does NOT go through SQL.** A `DriverHostStatus` table existed, with an entity
|
|
doc describing a publisher hosted service that upserted rows from each driver node. That publisher
|
|
was never written, and **per-cluster mesh Phase 4 made it unbuildable as described** — `Program.cs`
|
|
gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to
|
|
write rows with. The table was dropped (migration `DropDriverHostStatusTable`); it was empty on
|
|
every deployment. This channel needs no DB and already survives the mesh split.
|
|
- **null ≠ empty, on the wire too.** Null means "the driver has no probe"; an empty list means "it has
|
|
one that currently knows no hosts". proto3 cannot distinguish an absent repeated field from an empty
|
|
one, so `DriverHealth.has_host_statuses` carries that bit explicitly. Collapsing the two would render
|
|
every probe-less driver as one whose devices are all fine.
|
|
- **The host digest is part of the publish fingerprint, and must stay there.** `PublishHealthSnapshot`
|
|
dedups on that fingerprint, and on a single-host-down transition *every other component is
|
|
unchanged* — so without it the dedup swallows precisely the publish carrying the news. Same trap
|
|
that bit the rediscovery signal; pinned by
|
|
`DriverInstanceActorHostStatusTests.A_single_host_going_down_is_not_swallowed_by_the_unchanged_health_dedup`.
|
|
The digest is a flattened, host-name-ordered **string** for the converse reason: a tuple holding an
|
|
`IReadOnlyList` compares by reference and would re-publish on every 30 s heartbeat forever.
|
|
|
|
**`DriverInstanceResilienceStatus` was dropped (Gitea #524, migration
|
|
`DropDriverInstanceResilienceStatusTable`).** The table had no writer and no reader; the live data
|
|
reaches the AdminUI over the `driver-resilience-status` channel above, which is its only home.
|
|
|
|
## The three non-migrated channels (NOT migrated in Phase 5 — do not read this as "seven done")
|
|
|
|
The program sketch originally named seven observability topics for Phase 5. Three were scoped out,
|
|
each for a distinct, settled reason — and one of those three has since been deleted outright:
|
|
|
|
- **`redundancy-state`** — bidirectional, built directly from `Cluster.State`, and **pair-local**:
|
|
it drives ServiceLevel and the Primary data-plane gate, consumed in-process by
|
|
`OpcUaPublishActor`, `ScriptedAlarmHostActor`, `DriverHostActor`, and `HistorianAdapterActor`.
|
|
It stays on DPS in **both** `MeshTransport` modes today, and it is genuinely mesh-bound by
|
|
design — under Phase 6 each pair keeps sharing its own small mesh, so DPS keeps working for it
|
|
in-mesh. Central's *display* of each pair's redundancy state is a Phase 6 cross-mesh concern
|
|
(a future observability channel, once central no longer shares gossip with any site pair), not a
|
|
Phase 5 telemetry-panel migration.
|
|
- **`fleet-status`** — **DELETED, 2026-08-06, Gitea #542.** There is no `fleet-status` channel any
|
|
more, on DPS or anywhere else. It was central-internal rather than a node→central stream: the
|
|
admin singleton `FleetStatusBroadcaster` built it from the admin node's own cluster membership/
|
|
reachability/leader events and published to a SignalR hub that **no client anywhere ever
|
|
connected to** — `Fleet.razor` polls `NodeDeploymentState` in the Config DB and always ignored
|
|
the feed. Its second input, `DriverHostStatusHeartbeat`, had no production sender either. Phase 6
|
|
then made the central-only view wrong as well (central's gossip ring is its own pair). The whole
|
|
chain — broadcaster, singleton registration, DPS topic, SignalR bridge, `FleetStatusHub`, and the
|
|
`FleetStatusChanged`/`FleetNodeStatus` messages — was removed rather than rewritten cross-mesh.
|
|
Fleet visibility that does work today: `/fleet` over `NodeDeploymentState`, and the four Phase 5
|
|
telemetry channels above, which are per-node dials and therefore already mesh-independent.
|
|
- **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2
|
|
gRPC command transport (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a
|
|
command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.
|
|
|
|
See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state`
|
|
and the command transports fit together.
|
|
|
|
## Authentication — fail-closed from day one
|
|
|
|
A shared node bearer key gates the stream: `Telemetry:ApiKey` (node/serve side) must equal
|
|
`TelemetryDial:ApiKey` (central/dial side). `TelemetryStreamAuthInterceptor` enforces it — path-scoped
|
|
to `/telemetry.v1.TelemetryStreamService/`, comparing the `Authorization: Bearer` token with
|
|
`CryptographicOperations.FixedTimeEquals`, and rejecting with `PermissionDenied`. An **unset** key
|
|
rejects every call rather than allowing an open stream — the same fail-closed posture as
|
|
`ConfigServeAuthInterceptor` and `LocalDbSyncAuthInterceptor`.
|
|
|
|
This **supersedes** [the design doc's §6.3](plans/2026-07-21-per-cluster-mesh-design.md), which had
|
|
provisionally decided to "match ScadaBridge's unauthenticated posture for now" for inter-cluster
|
|
transports. ScadaBridge itself has since closed that gap with this identical
|
|
interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a
|
|
later hardening pass. See the design doc's superseded note for the full history.
|
|
|
|
**On a fused `admin,driver` node, `TelemetryStreamService` and `NodeCommandService` are `AllowAnonymous`
|
|
by design** — a *different* gate from the interceptor above. `TelemetryStreamAuthInterceptor` is the PSK
|
|
check; on an admin-role node the AdminUI cookie `FallbackPolicy` sits in front of it and would 401/redirect
|
|
a correctly-keyed call before the interceptor's verdict matters, because it runs in the authorization
|
|
middleware, strictly *before* the interceptor executes (archreview 03/S14). A driver-only node never
|
|
registers that middleware at all, so it is structurally unaffected either way. See
|
|
[`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).
|
|
|
|
## Upgrading an existing deployment: populate `ClusterNode.GrpcPort` first (#493)
|
|
|
|
`GrpcPort` is a **nullable column added by Phase 5**, and nothing backfills it onto rows that
|
|
already existed. On an upgraded deployment every `ClusterNode` row therefore carries `NULL`,
|
|
central finds **no dial targets**, and flipping `TelemetryDial:Mode` to `Grpc` connects to
|
|
nothing. Fresh installs are unaffected. `AkkaPort` did not have this problem only because it is
|
|
`NOT NULL` with a default of 4053.
|
|
|
|
Nothing fails loudly, which is what makes this worth stating up front — the code degrades
|
|
gracefully, once per node, throttled:
|
|
|
|
```
|
|
ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
|
|
(further skips of this node are silent)
|
|
```
|
|
|
|
Other symptoms: no `ESTABLISHED` sockets on the telemetry port (a `LISTEN` but nothing else in
|
|
`docker exec <node> cat /proc/net/tcp6`), and the `/alerts` / `/script-log` pill never turning
|
|
live in `Grpc` mode.
|
|
|
|
**Before flipping any node to `Grpc`**, set each driver node's `GrpcPort` to that node's own
|
|
`Telemetry:GrpcListenPort` — via the AdminUI node editor, or directly:
|
|
|
|
```sql
|
|
UPDATE dbo.ClusterNode SET GrpcPort = <that node's Telemetry:GrpcListenPort> WHERE NodeId = '<node>';
|
|
```
|
|
|
|
There is deliberately **no EF data migration** backfilling this. The correct value is that node's
|
|
own listener port, which is per-deployment configuration the database cannot derive; a migration
|
|
could only invent one, and a *wrong* dial target is worse than the `NULL` the dialer already skips
|
|
explicitly. The docker-dev seed does backfill (`GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'`
|
|
→ 4056) because there the port is fixed by `docker-compose.yml` and therefore actually knowable.
|
|
|
|
## Reconnect and reliability
|
|
|
|
Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per
|
|
driver node**:
|
|
|
|
- **Discovery** — enabled, non-maintenance `ClusterNode` rows are read on start and refreshed every
|
|
`TelemetryDial:ContactRefreshSeconds` (default 60), plus on an admin-side topology change. A row
|
|
with a null `GrpcPort` is skipped (that node exposes no telemetry endpoint — logged, not an
|
|
error). Dial targets are added/removed as the row set changes.
|
|
- **Reconnect backoff** — immediate first retry, then a fixed ~5 second backoff, indefinitely. The
|
|
dialer never gives up permanently: this is an observability channel, not the data plane, so a
|
|
persistently-unreachable node just means a persistently-stale panel, not a fleet fault.
|
|
- **Generation stamping** — each dialer carries a monotonically-increasing generation counter, so
|
|
a late error or late event from a superseded stream (e.g. one that raced a reconnect) is
|
|
recognized and ignored rather than corrupting the current stream's state.
|
|
- **Snapshot replay on reconnect** — the node-local hub keeps a last-value cache for
|
|
`driver-health` and `driver-resilience-status` (there is nothing meaningful to replay for
|
|
`alerts`/`script-logs`, which are append logs). On every new `Subscribe` — including a
|
|
reconnect — the hub drains the cached snapshots to the new stream before live deltas, so
|
|
`IDriverStatusSnapshotStore` / `IDriverResilienceStatusStore` re-prime immediately rather than
|
|
sitting stale until the next natural event. `alerts`/`script-logs` simply tolerate the gap — a
|
|
reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would.
|
|
The cache is evicted per instance (`ITelemetryLocalHub.Evict`) when `DriverHostActor.StopChild` removes a
|
|
driver from the plan (#609), so a decommissioned instance's last state is not replayed to new subscribers.
|
|
`StopChild` first releases the instance's resilience state (`IDriverCapabilityInvokerFactory.Release` →
|
|
pipeline-builder + `DriverResilienceStatusTracker.Retire`): `DriverResilienceStatusPublisherService`
|
|
re-emits the whole tracker every 5 s, so evicting only the hub was undone within one tick (review R1 M1).
|
|
⚠️ **Both releases HOLD the id, not just clear it.** `Context.Stop` is asynchronous, so the stopping child
|
|
can still finish a capability call it had in flight — observed: its `SubscribeBulk`, whose
|
|
`RecordCallStart`/`RecordSuccess`/`RecordCallComplete` ran after the release and re-created the tracker
|
|
entry, which the publisher then re-emitted every tick for the life of the process — and can still publish a
|
|
Health snapshot. So the tracker ignores every `Record*` for a retired id, and the hub drops every later
|
|
Health/Resilience item for an evicted id (neither cached nor fanned out), until the same id is spawned
|
|
again: `IDriverCapabilityInvokerFactory.Create` reinstates it in the tracker and `SpawnChild` calls
|
|
`ITelemetryLocalHub.Readmit`. Residual: a late write from the *old* child that lands after a re-add of the
|
|
same id is indistinguishable from the new child's and is kept — the same window a respawn always had.
|
|
- **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by
|
|
**aggregate** stream health across all dialers: connected when **at least one** node stream is
|
|
up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill
|
|
semantics — the pill has never meant "every node is live," only "the panel has a source." The
|
|
two store-backed panels (`driver-health`, `driver-resilience-status`) carry no fleet-wide
|
|
connection flag today, on gRPC or on DPS; per-row staleness is unchanged.
|
|
|
|
## Configuration reference
|
|
|
|
See [`docs/Configuration.md` § `Telemetry` / `TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport)
|
|
for the full keys table. In brief:
|
|
|
|
- **`Telemetry`** (node/serve side): `Mode` (`Dps`|`Grpc`, default `Dps`), `GrpcListenPort` (`0` =
|
|
disabled — the driver node's dedicated telemetry h2c port), `ApiKey` (the shared node key; supply
|
|
via `${secret:}`/env, never commit).
|
|
- **`TelemetryDial`** (central side): `Mode` (`Dps`|`Grpc`), `ApiKey` (must equal the nodes'
|
|
`Telemetry:ApiKey`), `ContactRefreshSeconds` (default `60`), `CallTimeoutSeconds` (default `30`).
|
|
|
|
A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0` (nothing to serve
|
|
otherwise), and `Grpc` mode on either side requires a non-empty `ApiKey` — both are enforced by
|
|
`ValidateOnStart` validators, fail-fast at boot.
|
|
|
|
## Relationship to the rest of the mesh program
|
|
|
|
- **Same dark-switch discipline as Phases 2 and 3.** `MeshTransport:Mode`, `ConfigSource:Mode`, and
|
|
`Telemetry:Mode`/`TelemetryDial:Mode` are all independently-flippable, restart-only config
|
|
changes, and in every case the "new" side is always wired so the flip needs no coordinated
|
|
redeploy.
|
|
- **Why the inversion matters for Phase 6.** Every other cross-boundary transport in this program
|
|
(`MeshTransport`, `ConfigSource`/`ConfigServe`) already has central as the addressable, discoverable
|
|
side and the node as the caller or the callee-by-address. Telemetry is the one channel that is
|
|
naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out
|
|
to each node by its `ClusterNode`-recorded address — is what lets telemetry survive the mesh split
|
|
cleanly: central never needs cluster membership with a site's mesh to keep watching it.
|
|
- **Status:** merged to `master` (`35552a96`); the live gate (flip the docker-dev rig to `Grpc`,
|
|
confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site
|
|
node) PASSED 2026-07-23. See `docs/plans/2026-07-22-per-cluster-mesh-program.md` § Phase 5 and
|
|
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan.
|