Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (accept, or add a UseTls option) #545

New Issue

2026-08-06T07:14:14-04:00

dohertj2 commented

2026-08-06 07:14:14 -04:00

The two mesh planes introduced by the ClusterClient→gRPC migration are h2c (cleartext HTTP/2) only, and their authentication is a per-cluster pre-shared key carried in an authorization: Bearer header. So the PSK crosses the network in cleartext on every call, and — unlike the three planes that precede it — the surface it opens is a write surface.

What is exposed.

Node side: mesh_transport.v1.NodeCommandService/Execute on each driver node's telemetry listener (:4056), gated by MeshCommandAuthInterceptor against MeshTransport:GrpcPsk. Anyone who can read one call off the wire can afterwards restart drivers, acknowledge alarms and dispatch deployments to every node in that cluster.
Central side: mesh_transport.v1.CentralCommandService/ReportApply on the ConfigServe listener (:4055), gated by CentralCommandAuthInterceptor against the per-cluster key from IMeshPskProvider. Capturing that key lets a caller forge deployment acks.

Both dial via GrpcChannel.ForAddress("http://…") (MeshGrpcCommandDialer, GrpcApplyAckSender) and are configured with http:// endpoints (MeshTransport:CentralGrpcEndpoints, ClusterNode.Host+GrpcPort).

This is precedent-consistent, not a regression. The three existing cross-mesh gRPC planes — telemetry stream (Phase 5), config-serve/artifact fetch (Phase 3) and LocalDb pair sync — are all h2c with a shared-bearer interceptor, on the same trusted-site-link assumption. Per-cluster keys already bound the blast radius to one cluster (a compromised SITE-A key cannot drive SITE-B), which is strictly better than the fleet-wide-key alternative that was rejected in the design.

Decision needed (it is deliberately not being taken inside the migration program, which is a like-for-like transport swap):

Accept the risk and record it — the mesh link is assumed private (VPN / dedicated WAN), the same assumption every sibling plane already makes. Document it once, for all five planes, rather than per plane.
Add a UseTls / scheme option to MeshTransport (and, for consistency, to the sibling planes) so an operator on an untrusted link can move both mesh planes to https:// with a server certificate. That is a per-plane options + validator + rig-config change, and it should be done for all five planes together or not at all — a mixed posture where two planes are TLS and three are not is harder to reason about than either uniform answer.

Note that the honest scope of a TLS flip is wider than the two new planes: doing only the mesh planes would leave the same key material class (Telemetry:ApiKey, ConfigServe:ApiKey, LocalDb sync key) still in cleartext on the same links.

Filed from the ClusterClient→gRPC Phase 1A live gate (docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md § Issues filed by this program; raised in the 1A.2 Opus code review as a non-blocking follow-up).

The two mesh planes introduced by the ClusterClient→gRPC migration are **h2c (cleartext HTTP/2) only**, and their authentication is a per-cluster pre-shared key carried in an `authorization: Bearer` header. So the PSK crosses the network in cleartext on every call, and — unlike the three planes that precede it — the surface it opens is a **write** surface. **What is exposed.** - Node side: `mesh_transport.v1.NodeCommandService/Execute` on each driver node's telemetry listener (`:4056`), gated by `MeshCommandAuthInterceptor` against `MeshTransport:GrpcPsk`. Anyone who can read one call off the wire can afterwards restart drivers, acknowledge alarms and dispatch deployments to every node in that cluster. - Central side: `mesh_transport.v1.CentralCommandService/ReportApply` on the ConfigServe listener (`:4055`), gated by `CentralCommandAuthInterceptor` against the per-cluster key from `IMeshPskProvider`. Capturing that key lets a caller forge deployment acks. Both dial via `GrpcChannel.ForAddress("http://…")` (`MeshGrpcCommandDialer`, `GrpcApplyAckSender`) and are configured with `http://` endpoints (`MeshTransport:CentralGrpcEndpoints`, `ClusterNode.Host`+`GrpcPort`). **This is precedent-consistent, not a regression.** The three existing cross-mesh gRPC planes — telemetry stream (Phase 5), config-serve/artifact fetch (Phase 3) and LocalDb pair sync — are all h2c with a shared-bearer interceptor, on the same trusted-site-link assumption. Per-cluster keys already bound the blast radius to one cluster (a compromised SITE-A key cannot drive SITE-B), which is strictly better than the fleet-wide-key alternative that was rejected in the design. **Decision needed** (it is deliberately not being taken inside the migration program, which is a like-for-like transport swap): 1. **Accept the risk** and record it — the mesh link is assumed private (VPN / dedicated WAN), the same assumption every sibling plane already makes. Document it once, for all five planes, rather than per plane. 2. **Add a `UseTls` / scheme option** to `MeshTransport` (and, for consistency, to the sibling planes) so an operator on an untrusted link can move both mesh planes to `https://` with a server certificate. That is a per-plane options + validator + rig-config change, and it should be done for all five planes together or not at all — a mixed posture where two planes are TLS and three are not is harder to reason about than either uniform answer. Note that the honest scope of a TLS flip is wider than the two new planes: doing only the mesh planes would leave the same key material class (`Telemetry:ApiKey`, `ConfigServe:ApiKey`, LocalDb sync key) still in cleartext on the same links. Filed from the ClusterClient→gRPC Phase 1A live gate (`docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md` § Issues filed by this program; raised in the 1A.2 Opus code review as a non-blocking follow-up).

dohertj2 referenced this issue from a commit

2026-08-06 19:36:00 -04:00

docs(security): #545 record the accepted h2c posture for the five internal gRPC planes

dohertj2 referenced this issue from a commit

2026-08-06 19:36:00 -04:00

Merge fix/issues-543-544-545: whitespace-key fail-closed backport (#543), de-flaked deadline test (#544), h2c accepted-posture record (#545)

dohertj2 commented

2026-08-06 19:37:00 -04:00

Resolved by recording the accepted posture — option 1, merged to master @ 1df84703 and pushed (commit 17a822b7, docs-only).

Decision: accept the risk and record it once, for all five planes. New subsection in docs/security.md under Transport Security: "Internal gRPC planes (h2c) — accepted cleartext posture", containing the five-plane table (service, listener config key, auth key, interceptor), an honest read/write characterization (telemetry + config-serve are read-planes; LocalDb sync writes the node's boot cache; the two mesh planes are command/write surfaces — a three-way split, sharper than this issue's two-way framing), the trusted-link assumption, the per-cluster blast-radius bound (OT-MESH-PSK-<clusterId> via ${secret:}; dev keys enumerated), and the recorded alternative: a UseTls/scheme option must cover all five planes together or not at all. docs/Configuration.md's MeshTransport section cross-links it.

A dedicated docs-accuracy review verified every claim against source: proto service names, the Program.cs listener-sharing claims (NodeCommandService on the Telemetry:GrpcListenPort listener, CentralCommandService on ConfigServe:GrpcListenPort), options-class key names, MeshPskProvider.SecretNamePrefix, compose values phrased as rig values (not fixed ports), and GrpcChannel.ForAddress("http://…") in all five dialers. The issue's note that the honest TLS scope is wider than the two new planes is now the doc's own framing.

Deliberately not done here: no code change, no UseTls option. If an untrusted link ever carries these planes, reopen as the five-plane TLS program the doc describes.

Resolved by recording the accepted posture — **option 1**, merged to `master` @ `1df84703` and pushed (commit `17a822b7`, docs-only). **Decision: accept the risk and record it once, for all five planes.** New subsection in `docs/security.md` under *Transport Security*: **"Internal gRPC planes (h2c) — accepted cleartext posture"**, containing the five-plane table (service, listener config key, auth key, interceptor), an honest read/write characterization (telemetry + config-serve are read-planes; **LocalDb sync writes the node's boot cache**; the two mesh planes are command/write surfaces — a three-way split, sharper than this issue's two-way framing), the trusted-link assumption, the per-cluster blast-radius bound (`OT-MESH-PSK-<clusterId>` via `${secret:}`; dev keys enumerated), and the recorded alternative: a UseTls/scheme option must cover **all five planes together or not at all**. `docs/Configuration.md`'s MeshTransport section cross-links it. A dedicated docs-accuracy review verified every claim against source: proto service names, the `Program.cs` listener-sharing claims (NodeCommandService on the `Telemetry:GrpcListenPort` listener, CentralCommandService on `ConfigServe:GrpcListenPort`), options-class key names, `MeshPskProvider.SecretNamePrefix`, compose values phrased as rig values (not fixed ports), and `GrpcChannel.ForAddress("http://…")` in all five dialers. The issue's note that the honest TLS scope is wider than the two new planes is now the doc's own framing. Deliberately not done here: no code change, no UseTls option. If an untrusted link ever carries these planes, reopen as the five-plane TLS program the doc describes.

dohertj2 closed this issue

2026-08-06 19:37:05 -04:00

dohertj2 referenced this issue

2026-08-06 19:37:24 -04:00

docs/security.md summary still claims NodeAcl enforcement is "never wired" and reads carry no authorization — stale since #520 stages 5–6 #547

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#545