docs(plans): TLS production profile design for Traefik, Kestrel, gRPC and Akka remoting (ScadaBridge#64)
Per-hop current state with code evidence (browser/Traefik/Kestrel, CLI, central<->site gRPC + PSK, LocalDb sync, Akka remoting, OPC UA / MxGateway), the recommended mode per hop (Kestrel HTTPS + ALPN, gRPC over TLS with validation and the PSK retained, Akka DotNetty SSL with a cert per node and mutual auth), certificate issuance/storage/rotation, rig and wonder impact, a staged rollout, rollback, a test plan and 13 open owner decisions. Findings: the central ASPNETCORE_URLS re-bind drops the scheme (an https URL binds plaintext); Akka.NET has no secure-cookie, so mTLS is the only remoting authentication. Component-ClusterInfrastructure documents the new RemotingTls option. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,536 @@
|
|||||||
|
# TLS production profile: Traefik, Kestrel, gRPC and Akka remoting (design, 2026-11-09)
|
||||||
|
|
||||||
|
**Status: DESIGN, plus opt-in groundwork that is OFF by default (ScadaBridge#64, register row 128).**
|
||||||
|
No deployed configuration changes. No environment is switched to TLS by this document or by the code
|
||||||
|
that ships with it. The revisit trigger in the register is unchanged: the first deployment onto a
|
||||||
|
network that is not wholly inside the plant security boundary.
|
||||||
|
|
||||||
|
Groundwork in the same change (section 9): an opt-in Akka.Remote TLS block
|
||||||
|
(`ScadaBridge:Cluster:RemotingTls`), a fix that lets the control-plane PSK credentials work on an
|
||||||
|
`https://` gRPC endpoint, and a fix to the bootstrap guard's seed parser for `akka.ssl.tcp`. With
|
||||||
|
the defaults, the Akka HOCON document is byte-identical to before (12 rendered documents compared
|
||||||
|
by SHA). Kestrel configuration is not touched.
|
||||||
|
|
||||||
|
Every "current state" claim below cites the code it was read from. Anything about the wonder
|
||||||
|
environment that the code cannot show is listed as an **open owner decision** in section 11, not
|
||||||
|
assumed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Hops in scope
|
||||||
|
|
||||||
|
| # | Hop | Transport today | Listener / client |
|
||||||
|
|---|---|---|---|
|
||||||
|
| H1 | Browser to Central UI (and Inbound API callers) | HTTP/1.1, plaintext | Traefik `web` :80 to Kestrel :5000 (rig); Kestrel :8085 directly (wonder) |
|
||||||
|
| H2 | CLI to `/management` | HTTP, plaintext; Basic or bearer token | same Kestrel HTTP surface as H1 |
|
||||||
|
| H3 | Central and site gRPC control plane | h2c (HTTP/2 without TLS) + per-site PSK | site :8083, central :8083 |
|
||||||
|
| H3b | LocalDb pair sync (site node to site node) | h2c + bearer API key | shares the site :8083 listener |
|
||||||
|
| H4 | Akka remoting within a pair | DotNetty TCP, plaintext, no authentication | central :8081, site :8082 |
|
||||||
|
| H5 | Site to external OPC UA server | OPC UA SecureChannel; `None` by default | DCL `RealOpcUaClient` |
|
||||||
|
| H5b | Site to MxGateway | gRPC; h2c by default, TLS when configured | DCL `RealMxGatewayClient` |
|
||||||
|
| H6 | Standby central to active central (Inbound API relay) | HTTP, plaintext | `StandbyForwardPeerEndpoints` |
|
||||||
|
| H7 | Prometheus / Overview to `/health/*`, `/metrics` | HTTP/1.1, plaintext | site :8084, central HTTP surface |
|
||||||
|
|
||||||
|
H6 and H7 follow the same Kestrel decision as H1. SQL Server and LDAP connections are out of scope
|
||||||
|
here. They are noted in section 10.
|
||||||
|
|
||||||
|
## 2. Current state, with evidence
|
||||||
|
|
||||||
|
### H1: Browser to Traefik to Central UI
|
||||||
|
|
||||||
|
- **Rig Traefik is HTTP only.** `docker/traefik/traefik.yml` declares one entrypoint,
|
||||||
|
`web: address: ":80"`, and `api: insecure: true`. `docker/traefik/dynamic.yml` routes
|
||||||
|
`PathPrefix(/)` to `http://scadabridge-central-a:5000` and `http://scadabridge-central-b:5000`,
|
||||||
|
health-checked on `/health/active`. Host port `9000:80` (`docker/docker-compose.yml`).
|
||||||
|
- **Traefik exists only on the docker rigs.** The wonder deployment records show centrals served
|
||||||
|
directly by Kestrel on `:8085`, with no load balancer in front
|
||||||
|
(`docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md`;
|
||||||
|
`scadaproj/deployment.md` section 7.1, port 8085 "ScadaBridge Central: CentralUI + Management +
|
||||||
|
Inbound API"). QA users address `vq03:8085` or `vq04:8085` by name.
|
||||||
|
- **Kestrel central HTTP surface.** `Program.cs` (Central branch) reads `ASPNETCORE_URLS`, falling
|
||||||
|
back to `http://+:5000`, and re-binds each port with `options.ListenAnyIP(httpPort, ...)` and
|
||||||
|
`HttpProtocols.Http1AndHttp2`. It does this because the explicit gRPC listener switches Kestrel
|
||||||
|
into explicit-endpoints mode.
|
||||||
|
- **Finding F1: the scheme in `ASPNETCORE_URLS` is silently dropped.** `Program.ParseHttpBindPorts`
|
||||||
|
keeps only the port (its own remarks: "only the port matters"), and the re-bind never calls
|
||||||
|
`UseHttps`. So `ASPNETCORE_URLS=https://+:443` today binds **plaintext** HTTP on 443, with no
|
||||||
|
startup error. Any TLS profile must close this. Either honour the scheme (bind TLS with a configured
|
||||||
|
certificate) or refuse to boot on an `https` URL that has no certificate.
|
||||||
|
- **Session cookie.** `SecurityOptions.RequireHttpsCookie` defaults to `true`, which sets the
|
||||||
|
cookie's `Secure` flag (`src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs`). Both rig centrals
|
||||||
|
set it to `false` (`docker/central-node-*/appsettings.Central.json`). When it is false the code logs
|
||||||
|
a startup warning and uses `SameAsRequest` (`Security/ServiceCollectionExtensions.cs`). The value on
|
||||||
|
the wonder boxes is not recorded in the repo; see section 11.
|
||||||
|
- **Forwarded headers.** `ForwardedHeadersSetup` honours `X-Forwarded-For | X-Forwarded-Proto` from
|
||||||
|
configured proxies only, and is disabled by default. The rig enables it for `172.16.0.0/12`. This
|
||||||
|
matters for TLS that terminates at a proxy: with `X-Forwarded-Proto: https` trusted, the app sees
|
||||||
|
the request as HTTPS, so the session cookie is `Secure` even under `SameAsRequest`.
|
||||||
|
|
||||||
|
### H2: CLI to management
|
||||||
|
|
||||||
|
- `ManagementHttpClient` takes any absolute `http` or `https` URL
|
||||||
|
(`CLI/Commands/CommandHelpers.cs`: "Expected an absolute http/https URL"). It exchanges
|
||||||
|
credentials for a bearer token at `POST /auth/token`, falling back to HTTP Basic
|
||||||
|
(`CLI/ManagementHttpClient.cs` header remarks). Over `http` both credential forms travel in
|
||||||
|
cleartext.
|
||||||
|
- It uses a plain `SocketsHttpHandler { ConnectTimeout, AllowAutoRedirect = false }`, with no
|
||||||
|
certificate-validation override anywhere in the CLI. So an `https` URL already gets OS-trust-store
|
||||||
|
validation. **No CLI code change is needed for TLS.** Operators change the URL, and the box must
|
||||||
|
trust the issuing CA.
|
||||||
|
|
||||||
|
### H3: Central and site gRPC control plane
|
||||||
|
|
||||||
|
- **Site listener.** `SiteKestrelSetup.Configure` binds `ListenAnyIP(grpcPort)` with
|
||||||
|
`HttpProtocols.Http2` only (h2c), plus `ListenAnyIP(metricsPort)` with `Http1AndHttp2`. Neither
|
||||||
|
calls `UseHttps`.
|
||||||
|
- **Central listener.** `Program.cs` binds `ListenAnyIP(centralGrpcPort)` with `Http2` only (h2c).
|
||||||
|
The comment states: "gRPC does NOT go through Traefik (HTTP/1 only); sites reach this port by
|
||||||
|
container name".
|
||||||
|
- **Endpoint validation already allows https.** `GrpcEndpointRules` accepts "an absolute http:// or
|
||||||
|
https:// URL" (`Commons/Validation/GrpcEndpointRules.cs`). The site addresses are DB rows
|
||||||
|
(`docker/seed-sites.sh` sets `--grpc-node-a-address "http://scadabridge-site-a-a:8083"`). The
|
||||||
|
site-to-central list is `ScadaBridge:Communication:CentralGrpcEndpoints` in each
|
||||||
|
`appsettings.Site.json`.
|
||||||
|
- **Client builders.** These build channels:
|
||||||
|
- `SitePairChannelProvider.BuildChannel`: central to site commands.
|
||||||
|
- `SiteStreamGrpcClient`: central to site streams.
|
||||||
|
- `CentralChannelProvider`: site to central control.
|
||||||
|
- `GrpcPull{AuditEvents,SiteCalls,TraceEvents}Client`: central to site audit pulls, in AuditLog.
|
||||||
|
|
||||||
|
All of them build a `SocketsHttpHandler` with keepalive, then call `.WithMessageLimits()` and
|
||||||
|
`.WithSiteCredentials(...)`.
|
||||||
|
- **PSK.** `ControlPlaneCredentials.WithSiteCredentials` attaches `Authorization: Bearer <psk>` and
|
||||||
|
`x-scadabridge-site` as call credentials (`Communication/Grpc/ISitePskProvider.cs`). On the server,
|
||||||
|
`ControlPlaneAuthInterceptor` (site) and `CentralControlAuthInterceptor` (central) are fail-closed.
|
||||||
|
Central resolves one key per site from `ScadaBridge:Communication:SitePsks` or the secret
|
||||||
|
`SB-GRPC-PSK-<siteId>` (`Host/SitePskProvider.cs`). Gitea #96 (`3a31dd2d`) added in-place rotation:
|
||||||
|
`SitePskRotationInvalidator`, plus `PskRejectionRefreshHandler`, which drops the cached key on a
|
||||||
|
`PermissionDenied`/`Unauthenticated` response.
|
||||||
|
- **Finding F2: the PSK composition could not run over TLS. Fixed for 4 of 7 builders in this
|
||||||
|
change.** The credentials were always composed over `ChannelCredentials.Insecure`, with
|
||||||
|
`UnsafeUseInsecureChannelCallCredentials = true`. Grpc.Net.Client 2.76 refuses that on an `https`
|
||||||
|
address, with the error "Channel is configured with insecure channel credentials and can't use a
|
||||||
|
HttpClient with a 'https' scheme". So every https endpoint threw at channel construction. In
|
||||||
|
`SitePairChannelProvider` that throw is caught and the address logged as malformed, so a TLS site
|
||||||
|
would have been **silently undialable**. The class comment claimed TLS "requires no change to this
|
||||||
|
code"; it was wrong. The new test
|
||||||
|
`ControlPlaneTlsEndToEndTests.Https_channel_with_insecure_composition_throws_at_construction` pins
|
||||||
|
the failure. Section 9 describes the fix. **Still open:** the three AuditLog pull clients. Each
|
||||||
|
needs a one-argument change (`WithSiteCredentials(_pskProvider, siteId, endpoint)`). They were left
|
||||||
|
untouched on purpose because AuditLog is being changed concurrently for #42. Until they get the
|
||||||
|
argument, audit pulls from an `https` site fail at channel build.
|
||||||
|
|
||||||
|
### H3b: LocalDb pair sync
|
||||||
|
|
||||||
|
- Sync rides the site's :8083 listener. The dialling half's `LocalDb:Replication:PeerAddress` is
|
||||||
|
`http://scadabridge-site-a-b:8083` (`docker/site-a-node-a/appsettings.Site.json`).
|
||||||
|
- The LocalDb library (`ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.DialPeer`) builds
|
||||||
|
`GrpcChannel.ForAddress(PeerAddress, …)` with **no channel credentials**, and sends its key as an
|
||||||
|
`Authorization` metadata header. An `https` PeerAddress therefore works unchanged, using
|
||||||
|
OS-trust-store validation. But **a site's :8083 cannot be TLS for central and h2c for its peer**:
|
||||||
|
the pair's PeerAddress must flip in the same step as the listener.
|
||||||
|
|
||||||
|
### H4: Akka remoting within a pair
|
||||||
|
|
||||||
|
- `AkkaHostedService.BuildHocon` emits `akka.remote.dot-netty.tcp { hostname, port }` and a transport
|
||||||
|
failure detector. It emits **no SSL keys**, so Akka's reference default `enable-ssl = false`
|
||||||
|
applies. Ports are central 8081 and site 8082; the rig publishes them to the host (`9011:8081`,
|
||||||
|
`9021:8082`, and so on).
|
||||||
|
- **No authentication.** Akka.Remote 1.5.62 (`Directory.Packages.props`) has no shared-secret or
|
||||||
|
"secure cookie" mechanism; the strings `secure-cookie` and `require-cookie` do not exist in
|
||||||
|
`Akka.Remote.dll`. `untrusted-mode` exists but is off, and it only filters system messages and
|
||||||
|
actor selections. Anyone who can reach :8081/:8082 can join the cluster or message its actors.
|
||||||
|
**Mutual TLS is the only authentication Akka.Remote offers.** The issue text says "Akka
|
||||||
|
`enable-ssl` + secure cookie"; the cookie half does not exist in Akka.NET.
|
||||||
|
- What Akka.Remote 1.5.62 supports (read from its embedded reference config and strings):
|
||||||
|
- `enable-ssl`
|
||||||
|
- `ssl.certificate.{path,password,flags}`, or `{use-thumbprint-over-file, thumbprint, store-name,
|
||||||
|
store-location}`
|
||||||
|
- `ssl.suppress-validation`
|
||||||
|
- `ssl.require-mutual-authentication`, **default true**
|
||||||
|
- `ssl.validate-certificate-hostname`, default false
|
||||||
|
- a programmatic `DotNettySslSetup` with a `CustomValidator`, plus helpers for chain, issuer,
|
||||||
|
subject, hostname and pinned-certificate validation
|
||||||
|
|
||||||
|
At startup it refuses a certificate without an accessible private key.
|
||||||
|
- **The address scheme changes under TLS.** With SSL on, node addresses are `akka.ssl.tcp://…`. The
|
||||||
|
new test `HoconRemotingTlsTests.Enabled_document_boots_a_tls_transport_with_akka_ssl_tcp_addresses`
|
||||||
|
checks this by booting a real `ActorSystem` on the emitted document. Effects on code that parses
|
||||||
|
seeds:
|
||||||
|
- `StartupValidator.SeedNodeIsSelf` and `SeedNodePort/Host` use `@` and the last `:`, so they do
|
||||||
|
not depend on the scheme. They still work.
|
||||||
|
- `ClusterBootstrapGuard.SeedPattern` was `^akka(?:\.[a-z0-9]+)?://`, a single dot segment.
|
||||||
|
**Finding F3:** it did not match `akka.ssl.tcp`, so with TLS on the guard would have parsed no
|
||||||
|
seeds and silently never engaged. Fixed in this change.
|
||||||
|
- `ClusterBootstrapCoordinator` uses Akka's own `Address.Parse`, which handles the TLS scheme.
|
||||||
|
- Overview's `LeaderResolver.HostOf` (scadaproj) parses after `@`, so it does not depend on the
|
||||||
|
scheme either.
|
||||||
|
|
||||||
|
### H5: Site to OPC UA and MxGateway
|
||||||
|
|
||||||
|
- **OPC UA** uses OPC UA SecureChannel, not TLS. `OpcUaEndpointConfig.SecurityMode` defaults to
|
||||||
|
`None`. The values are `None`, `Sign` and `SignAndEncrypt`. `AutoAcceptUntrustedCerts` defaults to
|
||||||
|
false, and when on it logs a man-in-the-middle warning (`RealOpcUaClient`). The application PKI
|
||||||
|
stores come from `ScadaBridge:OpcUa:{TrustedIssuer,TrustedPeer,RejectedCertificate}StorePath`.
|
||||||
|
`OpcUaGlobalOptions` says empty paths "fall back to a default under Path.GetTempPath()". A temp
|
||||||
|
path is the wrong home for a production trust list, so the production profile should set these
|
||||||
|
explicitly. The rig simulators run `--unsecuretransport --autoaccept` (`deployment.md` section 3.3).
|
||||||
|
- **MxGateway.** Each connection carries `Endpoint` (default `http://localhost:5000`), `UseTls`,
|
||||||
|
`CaFile` and `ServerName` (`Commons/Types/DataConnections/MxGatewayEndpointConfig.cs`). These are
|
||||||
|
passed straight to `MxGatewayClientOptions` (`RealMxGatewayClient.ConnectAsync`). TLS already works
|
||||||
|
per connection, with a private-CA file. **No ScadaBridge code change is needed.** Enabling it
|
||||||
|
depends on the gateway serving TLS (its `…Http__Url=https://…`; `deployment.md` mxgw section).
|
||||||
|
|
||||||
|
### H6 and H7: relay and health
|
||||||
|
|
||||||
|
- `InboundApiOptions.StandbyForwardPeerEndpoints` accepts `http` or `https` and is used through a
|
||||||
|
named `HttpClient` (`InboundAPI/ServiceCollectionExtensions.cs`) with default validation. It flips
|
||||||
|
when the central surface flips.
|
||||||
|
- The site metrics and health listener (:8084) is HTTP/1.1 by design, because Prometheus and the
|
||||||
|
Overview dashboard poll it. It carries no credentials (`/health/*` is anonymous). Making it TLS is
|
||||||
|
optional, and section 11 lists it as an owner decision.
|
||||||
|
|
||||||
|
## 3. Threat summary (why each hop matters)
|
||||||
|
|
||||||
|
| Hop | Credential or data exposed in cleartext today | Integrity or authentication gap |
|
||||||
|
|---|---|---|
|
||||||
|
| H1 | session cookie (when `RequireHttpsCookie=false`), LDAP password at login, Inbound API `sbk_` keys | none at the transport layer |
|
||||||
|
| H2 | operator LDAP password (Basic) or bearer token | none at the transport layer |
|
||||||
|
| H3 | per-site PSK (bearer), deployments incl. encrypted secret artifacts, audit rows | the PSK authenticates the caller, but it is replayable by anyone on the path |
|
||||||
|
| H3b | LocalDb API key, all replicated site rows | as H3 |
|
||||||
|
| H4 | every cluster message | **no authentication at all**: a reachable host can join |
|
||||||
|
| H5 | OPC UA values; MxGateway API key + writes | per connection; secure modes exist but are off by default |
|
||||||
|
|
||||||
|
The PSK work (#96 and its predecessors) limits who may *call* the control plane. It does nothing for
|
||||||
|
confidentiality, and a key captured on the wire is valid until it is rotated. H4 is the weakest
|
||||||
|
link: unauthenticated cluster membership.
|
||||||
|
|
||||||
|
## 4. Recommended TLS mode per hop
|
||||||
|
|
||||||
|
### 4.1 Kestrel: HTTPS with ALPN (H1, H2, H6, H7 and the gRPC listeners)
|
||||||
|
|
||||||
|
- Add a node-level certificate option, `ScadaBridge:Node:Tls`, with the same shape as
|
||||||
|
`RemotingTls`: `CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName`
|
||||||
|
+ `StoreLocation`. Apply it with `listenOptions.UseHttps(cert)` to every listener the node binds.
|
||||||
|
- **Central HTTP surface:** `Http1AndHttp2` on one TLS port. ALPN negotiates h2 for gRPC-capable
|
||||||
|
clients and HTTP/1.1 for browsers. This is the family pattern: HistorianGateway runs one
|
||||||
|
`Kestrel:Endpoints:Https` with `Http1AndHttp2`.
|
||||||
|
- **gRPC ports (central :8083, site :8083):** `Http2` over TLS. Clients negotiate h2 by ALPN, and
|
||||||
|
no h2c remains.
|
||||||
|
- **Site :8084:** optional (section 11).
|
||||||
|
- Close **F1**: `ParseHttpBindPorts` must return the scheme as well as the port, and an `https` entry
|
||||||
|
must bind TLS. With no certificate configured it must **refuse to boot**, so it never downgrades
|
||||||
|
silently.
|
||||||
|
- Keep TLS configuration in code, as the existing listeners are, not in `Kestrel:Endpoints`. The
|
||||||
|
explicit-endpoints comment in `Program.cs` already documents one surprise from mixing the two
|
||||||
|
sources. A config-section endpoint added on top of code-bound endpoints is a second surprise
|
||||||
|
waiting to happen. It is unverified whether Kestrel unions the two sources or replaces one with
|
||||||
|
the other.
|
||||||
|
- HSTS (`UseHsts`) only after an environment is confirmed HTTPS-only. Section 11 has the decision.
|
||||||
|
|
||||||
|
### 4.2 Traefik (rig): TLS entrypoint, re-encrypted backend
|
||||||
|
|
||||||
|
- Add a `websecure` entrypoint on :443 (host `9443`), with a file-provider `tls.certificates` entry
|
||||||
|
that holds the lab certificate. Redirect `web` to `websecure`.
|
||||||
|
- Backend: in the production profile, Traefik dials `https://scadabridge-central-{a,b}:5000`, with a
|
||||||
|
`serversTransport` whose `rootCAs` is the lab CA. TLS runs end to end and the same Kestrel TLS path
|
||||||
|
is exercised on the rig. Plaintext on the internal docker network is an acceptable *rig-only*
|
||||||
|
shortcut, but it would leave F1 and §4.1 untested on the rig.
|
||||||
|
- Keep `ForwardedHeaders` trusting Traefik. Set `RequireHttpsCookie` back to its default, `true`, on
|
||||||
|
every TLS-fronted node.
|
||||||
|
- Turn off `api.insecure` (the dashboard) or bind it to loopback in any non-rig use.
|
||||||
|
- gRPC stays off Traefik, as it is today. Nodes dial each other directly.
|
||||||
|
|
||||||
|
### 4.3 gRPC: TLS with certificate validation; PSK retained
|
||||||
|
|
||||||
|
- Endpoints become `https://host:8083`: the site rows `GrpcNodeAAddress`/`GrpcNodeBAddress`,
|
||||||
|
`CentralGrpcEndpoints`, and `LocalDb:Replication:PeerAddress`.
|
||||||
|
- Clients validate the server chain and hostname with `SocketsHttpHandler` defaults. No
|
||||||
|
`RemoteCertificateValidationCallback` override and no "accept any" switch is to be added.
|
||||||
|
**Hostnames in endpoints must be SANs on the server certificate.**
|
||||||
|
- **Keep the PSK.** TLS gives confidentiality and server authentication. The PSK stays the per-site
|
||||||
|
*client* identity, which central's interceptor keys per-site authorization on. With TLS the PSK
|
||||||
|
stops being replayable from the wire. `WithSiteCredentials` now composes it over
|
||||||
|
`ChannelCredentials.SecureSsl` for https (section 9).
|
||||||
|
- Client-certificate mTLS on gRPC is **not** recommended for the first step. It would duplicate the
|
||||||
|
PSK's job and add a client-certificate distribution path to every site. Section 11 lists it as an
|
||||||
|
owner option.
|
||||||
|
|
||||||
|
### 4.4 Akka.Remote: DotNetty SSL, one certificate per node, mutual authentication
|
||||||
|
|
||||||
|
- Set `ScadaBridge:Cluster:RemotingTls:Enabled = true` with this node's own certificate.
|
||||||
|
`RequireMutualAuthentication = true` (the default). Chain validation is on and cannot be turned off
|
||||||
|
through ScadaBridge configuration. The `suppress-validation` key is always emitted as `false`.
|
||||||
|
- All seeds use `akka.ssl.tcp://scadabridge@host:port`. `ClusterOptionsValidator` refuses a
|
||||||
|
TLS-enabled config that still has an `akka.tcp` seed. Without that check, the pair would retry
|
||||||
|
joins forever with no startup error.
|
||||||
|
- `ValidateCertificateHostname = false` at first, which is Akka's default; the trust anchor is a
|
||||||
|
dedicated issuing CA. Turn it on once every `NodeHostname`/seed host is confirmed as a SAN on the
|
||||||
|
peer's certificate.
|
||||||
|
- If the CA is a broad corporate CA, chain trust alone admits *any* host holding a certificate from
|
||||||
|
that CA. In that case add issuer or subject pinning through `DotNettySslSetup.CustomValidator`. That
|
||||||
|
is a code change in `GetOrCreateActorSystem`: `ActorSystem.Create` with a `BootstrapSetup`, which
|
||||||
|
section 9 does not include. Section 11 has the decision.
|
||||||
|
- Both nodes of a pair flip together. A TLS node and a plaintext node cannot talk to each other. This
|
||||||
|
matches the existing rule that a site pair stops and starts together (LocalDb Phase 2).
|
||||||
|
|
||||||
|
### 4.5 Site to OPC UA and MxGateway
|
||||||
|
|
||||||
|
- **OPC UA:** set `SecurityMode = SignAndEncrypt` per connection, wherever the server offers it
|
||||||
|
(OtOpcUa does). Set explicit `ScadaBridge:OpcUa:*StorePath` directories under the node's data
|
||||||
|
root, not the temp default. Trust the server certificates explicitly. `AutoAcceptUntrustedCerts`
|
||||||
|
stays false.
|
||||||
|
- **MxGateway:** use an `https://` endpoint with `UseTls = true`, `CaFile` (the issuing CA) and
|
||||||
|
`ServerName` matching the gateway certificate. This needs no code change, only the gateway serving
|
||||||
|
TLS.
|
||||||
|
|
||||||
|
## 5. Certificates: issuance, storage, rotation
|
||||||
|
|
||||||
|
### 5.1 One certificate per node
|
||||||
|
|
||||||
|
- **Subject Alternative Names:** the node's `NodeHostname` exactly as it appears in seeds and gRPC
|
||||||
|
endpoints (the FQDN on wonder, e.g. `wonder-app-vq03.zmr.zimmer.com`), plus the short name. On the
|
||||||
|
rig, the container name (`scadabridge-site-a-a`).
|
||||||
|
- **Extended Key Usage:** `serverAuth` **and** `clientAuth`. Akka mTLS makes every node both a TLS
|
||||||
|
server and a TLS client. Kestrel and gRPC need `serverAuth`.
|
||||||
|
- One certificate per node serves both Kestrel and Akka on that node. On the co-located wonder boxes,
|
||||||
|
the central and site services on one box can share the box certificate. Both services run on the
|
||||||
|
same host name, so the SANs are identical.
|
||||||
|
|
||||||
|
### 5.2 Issuance
|
||||||
|
|
||||||
|
- **wonder:** this needs a CA that the boxes trust. Whether a corporate PKI exists (for example AD
|
||||||
|
Certificate Services with auto-enrolment) and is available to these boxes is **not knowable from
|
||||||
|
the code**. It is open decision D1.
|
||||||
|
- **Rig:** use a lab CA generated by a new script, `docker/tls/gen-lab-ca.sh` (openssl, idempotent,
|
||||||
|
output gitignored). It issues one certificate per container. Both the CA and the leaves are
|
||||||
|
clearly marked lab-only, like the rig's other DEV-ONLY credentials.
|
||||||
|
- **Trust on Linux containers:** Akka's chain validation and `SocketsHttpHandler` both use the OS
|
||||||
|
trust store. So the rig image must install the lab CA. Mount it into
|
||||||
|
`/usr/local/share/ca-certificates/` and run `update-ca-certificates` at container start. Do not
|
||||||
|
bake it into the image, because the image is shared with non-rig uses.
|
||||||
|
|
||||||
|
### 5.3 Where certificates and keys live
|
||||||
|
|
||||||
|
| Environment | Certificate + private key | Secret material |
|
||||||
|
|---|---|---|
|
||||||
|
| wonder (Windows) | **`LocalMachine\My`, referenced by thumbprint** (`CertificateThumbprint`; `StoreLocation = local-machine`). No key file on disk. The private key ACL grants read to the service account. | none: the store holds the key |
|
||||||
|
| wonder, fallback | `.pfx` outside the content root (same rule as `Secrets:SqlitePath`: absolute, not under `app\`, which an upgrade replaces) | password as `${secret:SB-TLS-PFX-PASSWORD}`, resolved from the Secrets store by the pre-host expander |
|
||||||
|
| rig (Linux) | `.pfx` on a read-only bind mount (`/certs/<node>.pfx`) | DEV-ONLY password in compose env, like the mesh PSKs |
|
||||||
|
|
||||||
|
Storing the PFX bytes themselves in the Secrets store is **not recommended**. Akka's HOCON path wants
|
||||||
|
a file, and Kestrel would need a materialised copy anyway. The Secrets store is the right home for
|
||||||
|
the **password**.
|
||||||
|
|
||||||
|
Do not log the HOCON document. With a file certificate it contains the PFX password, which
|
||||||
|
`BuildRemotingTlsBlock` escapes but does not redact. `GetOrCreateActorSystem` logs only individual
|
||||||
|
fields today, and `akka.log-config-on-start` is off by default. Both must stay that way.
|
||||||
|
|
||||||
|
### 5.4 Rotation
|
||||||
|
|
||||||
|
- **Leaf rotation under the same CA:** the peer validates the chain, not a pinned leaf. A node can
|
||||||
|
therefore get a new leaf and restart on its own, without coordinating with its partner:
|
||||||
|
- Kestrel picks up the new certificate on restart. A `ServerCertificateSelector` could make that
|
||||||
|
hot later.
|
||||||
|
- Akka.Remote reads its certificate only at `ActorSystem` start, so it needs a restart.
|
||||||
|
- On a pair, restart one node at a time. A one-node restart is ordinary failover. A pair-together
|
||||||
|
stop is required only for the plaintext/TLS flip, not for a leaf rotation.
|
||||||
|
- **CA rotation:** first distribute a trust bundle with both the old and the new CA to every node.
|
||||||
|
Then rotate the leaves. Then remove the old CA.
|
||||||
|
- **Expiry monitoring:** add a startup log and a `/health/ready` degraded entry when the node
|
||||||
|
certificate expires in fewer than N days. That is follow-on work. Who owns renewal is decision D9.
|
||||||
|
|
||||||
|
## 6. Impact on the docker rig and the wonder boxes
|
||||||
|
|
||||||
|
### 6.1 Rig (`docker/`, `docker-env2/`)
|
||||||
|
|
||||||
|
- New: `docker/tls/` (CA script, gitignored output), Traefik `websecure` + certificates,
|
||||||
|
`RemotingTls` blocks and `akka.ssl.tcp` seeds in each node's appsettings, `https` site addresses in
|
||||||
|
`seed-sites.sh`, `https` `CentralGrpcEndpoints` and LocalDb PeerAddress, and CA trust in the
|
||||||
|
containers.
|
||||||
|
- Host ports: add 9443 for Traefik TLS. Existing 9000 stays open (redirect) until the profile is the
|
||||||
|
default.
|
||||||
|
- Scripts that probe `http://localhost:900x`, the CLI recipes and the Playwright suites need a
|
||||||
|
base-URL switch. Their certificate trust must come from the lab CA, never from "ignore errors".
|
||||||
|
- Rig posture, mirroring the LocalDb rollout: **site-a on TLS, site-b and site-c on h2c**, so both
|
||||||
|
states are proven side by side. Central must dial both: per-site endpoints make that natural.
|
||||||
|
|
||||||
|
### 6.2 wonder (DEV vd03, QA vq03/vq04)
|
||||||
|
|
||||||
|
- No Traefik. Kestrel terminates TLS itself on the central port (:8085 today; whether to move to 443
|
||||||
|
is decision D4).
|
||||||
|
- Certificates in `LocalMachine\My` (if PKI is available, D1/D2). Thumbprints go in
|
||||||
|
`appsettings.{Central,Site}.json`, which are preserved across binary swaps per the deployment
|
||||||
|
records.
|
||||||
|
- The site rows in the shared ConfigDb change to `https://wonder-app-vq0X…:8083` through the CLI
|
||||||
|
`site update`. That is a data change, reversible the same way.
|
||||||
|
- **Remote access caveat:** under the 2026-09-28 WinRM scope, certificate installation counts as a
|
||||||
|
deploy step and is allowed only as part of a deploy the user asked for.
|
||||||
|
- Overview: when the central surface goes TLS, its registry `BaseUrl`s become `https`. Its leader
|
||||||
|
matching does not depend on the scheme (section 2, H4).
|
||||||
|
|
||||||
|
## 7. Staged rollout
|
||||||
|
|
||||||
|
Each stage is independently reversible (section 8) and proven before the next one starts.
|
||||||
|
|
||||||
|
| Stage | What | Where | Gate |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 0 | **This change:** design + default-off groundwork (Akka TLS block, https-capable PSK channels, guard regex) | code only | unit + real-transport tests green; default HOCON byte-identical |
|
||||||
|
| 1 | Code: `ScadaBridge:Node:Tls` for Kestrel; close F1; the AuditLog pull clients take the endpoint (after #42 lands); optional cert-expiry health entry | code only, default off | unit + real-Kestrel TLS tests; default Kestrel binding unchanged |
|
||||||
|
| 2 | Rig lab CA + Traefik `websecure` (re-encrypting backend) + central Kestrel TLS | docker | browser + CLI over https; Secure cookie observed; plaintext :5000 refused once flipped |
|
||||||
|
| 3 | Rig site-a: gRPC https (both directions + LocalDb PeerAddress) and Akka TLS on the site-a pair | docker site-a only | section 10 rig checks; failover drill on site-a; site-b/c unchanged |
|
||||||
|
| 4 | Rig: central pair Akka TLS; then all sites | docker | full failover drill (`docker/failover-drill.sh`) |
|
||||||
|
| 5 | DEV vd03 (single node: Akka TLS is self-only; gRPC + Kestrel TLS) | wonder DEV | post-deploy https `/health/ready`, CLI, MES test machines |
|
||||||
|
| 6 | QA vq03/vq04 pair | wonder QA | pair stop/start together; failover drill in a window |
|
||||||
|
|
||||||
|
Migration detail for the central gRPC port: during stage 3, central must answer both TLS sites and
|
||||||
|
h2c sites on site-to-central calls. A single Kestrel port is either TLS or not. Stage 1 therefore
|
||||||
|
adds an **optional second central gRPC listener** (for example `CentralGrpcTlsPort`). Sites move to
|
||||||
|
it one at a time, and the h2c port is removed once no site uses it.
|
||||||
|
|
||||||
|
The central-to-site direction has no such problem, because each site address is its own row.
|
||||||
|
|
||||||
|
## 8. Rollback
|
||||||
|
|
||||||
|
Every switch is configuration, and every default is off:
|
||||||
|
|
||||||
|
- **Akka:** set `RemotingTls:Enabled=false` and revert the seeds to `akka.tcp`. Stop and start the
|
||||||
|
pair **together**.
|
||||||
|
- **gRPC:** revert the site rows to `http://` (`site update`), revert `CentralGrpcEndpoints` and
|
||||||
|
`PeerAddress`, then restart. The PSK is untouched throughout, so auth never lapses during rollback.
|
||||||
|
- **Kestrel and Traefik:** revert `ASPNETCORE_URLS` and the node `Tls` block, and revert the Traefik
|
||||||
|
files.
|
||||||
|
- **Cookie:** `RequireHttpsCookie=true` must stay **true** while the surface is TLS. Rolling the
|
||||||
|
surface back to HTTP is the only case where setting it false again is justified.
|
||||||
|
- **HSTS:** browsers keep enforcing HTTPS for the max-age after rollback. This is why HSTS is enabled
|
||||||
|
last, with a short max-age first (decision D10).
|
||||||
|
- **No schema change** is involved at any stage. On wonder, the previous `appsettings*.json` is
|
||||||
|
already preserved in `app.bak.<ts>` by the standard deploy recipe.
|
||||||
|
|
||||||
|
## 9. Groundwork shipped with this design (default OFF)
|
||||||
|
|
||||||
|
1. **`ScadaBridge:Cluster:RemotingTls`** (`ClusterRemotingTlsOptions` in ClusterInfrastructure).
|
||||||
|
- Fields: `Enabled`, `CertificatePath`/`CertificatePassword` or `CertificateThumbprint`/
|
||||||
|
`StoreName`/`StoreLocation`, `RequireMutualAuthentication` (default true),
|
||||||
|
`ValidateCertificateHostname` (default false).
|
||||||
|
- `AkkaHostedService.BuildRemotingTlsBlock` renders `enable-ssl` + `ssl { … }` inside
|
||||||
|
`dot-netty.tcp`, **only when `Enabled`**.
|
||||||
|
- `ClusterOptionsValidator` (TLS on only) requires:
|
||||||
|
- exactly one certificate source;
|
||||||
|
- a legal store location;
|
||||||
|
- `akka.ssl.tcp://` on every seed.
|
||||||
|
- **Byte-identity proof:** before the change, the rendered HOCON was captured for 12
|
||||||
|
configurations: `auto-down` / `leader-auto-down` / `keep-oldest`, bootstrap guard on and off,
|
||||||
|
Central and Site. After the change all 12 were re-rendered and diffed: identical. A permanent
|
||||||
|
test (`Disabled_block_with_fields_is_byte_identical_to_default`) pins it: a disabled block with
|
||||||
|
every field set renders exactly the default document.
|
||||||
|
2. **https-capable PSK channels.** `ControlPlaneCredentials.WithSiteCredentials(..., endpoint)`
|
||||||
|
composes over `ChannelCredentials.SecureSsl` for an `https` endpoint. For anything else, including
|
||||||
|
callers that do not pass the endpoint, it keeps the previous insecure composition with the
|
||||||
|
explicit opt-in, unchanged. `SitePairChannelProvider`, `SiteStreamGrpcClient` and
|
||||||
|
`CentralChannelProvider` pass their endpoint. The three AuditLog pull clients are deliberately
|
||||||
|
untouched (F2). With today's `http` endpoints, behaviour is unchanged.
|
||||||
|
3. **`ClusterBootstrapGuard.SeedPattern`** accepts any number of scheme segments, so `akka.ssl.tcp`
|
||||||
|
parses (F3).
|
||||||
|
4. **Not changed:** `Program.cs`, `SiteKestrelSetup.cs`, Traefik and every appsettings and compose
|
||||||
|
file. Kestrel binding is identical by construction.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
- `HoconRemotingTlsTests` (5): includes booting a real TLS `ActorSystem` from a generated
|
||||||
|
certificate. A mutation check (forcing `enable-ssl = false`) turns it red with `akka.tcp`.
|
||||||
|
- `ClusterRemotingTlsValidatorTests` (8).
|
||||||
|
- `ClusterBootstrapGuardTests` (+2 for `akka.ssl.tcp`).
|
||||||
|
- `ControlPlaneTlsEndToEndTests` (6). These use real Kestrel with `Http2` over TLS (ALPN), the real
|
||||||
|
`ControlPlaneAuthInterceptor`, and a pinned test certificate:
|
||||||
|
- the correct PSK is accepted over TLS;
|
||||||
|
- a wrong PSK is still `PermissionDenied` over TLS;
|
||||||
|
- an untrusted server certificate fails with an `AuthenticationException` cause;
|
||||||
|
- the pre-fix composition throws on https;
|
||||||
|
- the h2c composition is unchanged on http.
|
||||||
|
- `HttpsEndpointChannelTests` (9). These check that all three production builders accept `https`. A
|
||||||
|
mutation check (disabling the TLS branch) turns the three construction tests red.
|
||||||
|
|
||||||
|
## 10. Test plan for stages 1–6
|
||||||
|
|
||||||
|
**Unit and in-process:** extend the stage-0 pattern to the Kestrel `Tls` block:
|
||||||
|
- certificate loading from a file and from the store (the store path is Windows-only, so skip it
|
||||||
|
elsewhere);
|
||||||
|
- F1: an `https` URL with no certificate must refuse to boot;
|
||||||
|
- ALPN: a real Kestrel `Http1AndHttp2` TLS listener serves an HTTP/1.1 GET and an h2 gRPC call on
|
||||||
|
one port.
|
||||||
|
|
||||||
|
**Rig, per hop** (each with a negative control, so a check that "passes" is shown to be able to fail):
|
||||||
|
|
||||||
|
- **H1:** `curl --cacert lab-ca.pem https://localhost:9443/health/ready` returns 200. Login over
|
||||||
|
https sets the cookie `Secure` (read the `Set-Cookie`). `http://localhost:9000` redirects. Traefik
|
||||||
|
to backend is re-encrypted: a plaintext probe to `central-a:5000` is refused once flipped.
|
||||||
|
- **H2:** CLI against the https URL with the lab CA trusted succeeds. With the CA *not* trusted, it
|
||||||
|
fails with a certificate error and never falls back to http.
|
||||||
|
- **H3:**
|
||||||
|
- `openssl s_client -alpn h2 -connect site-a-a:8083` shows TLS 1.2 or later and `h2`.
|
||||||
|
- An h2c client against the TLS port fails.
|
||||||
|
- Central to site commands, streams, audit pulls and site to central heartbeats all flow for
|
||||||
|
site-a over TLS while site-b stays h2c.
|
||||||
|
- The PSK is still enforced: a wrong key over TLS gives `PermissionDenied`.
|
||||||
|
- The #96 rotation drill passes over TLS.
|
||||||
|
- **H3b:** LocalDb `LocalDbReplicationConnected` is true on the site-a pair over https.
|
||||||
|
`MaxBatchBytes` behaviour is unchanged.
|
||||||
|
- **H4:**
|
||||||
|
- The pair forms with `akka.ssl.tcp` addresses (`/health/ready` shows memberCount 2).
|
||||||
|
- **Negative:** a third container with a certificate from a *different* CA cannot join. It gets a
|
||||||
|
handshake failure in its log and is never Up.
|
||||||
|
- **Negative:** a container with *no* certificate cannot join, which shows mTLS is enforced.
|
||||||
|
- Expired-certificate node: refused.
|
||||||
|
- Failover drill timings within the existing `StableAfter` budget. A TLS handshake on reconnect
|
||||||
|
must not push detection past it.
|
||||||
|
- **H5:** one OPC UA connection on `SignAndEncrypt` with an explicitly trusted server certificate.
|
||||||
|
One MxGateway connection with `UseTls` and `CaFile`.
|
||||||
|
|
||||||
|
**Rotation drill:** replace one node's leaf certificate under the same CA, restart that node only,
|
||||||
|
and confirm the pair re-forms. Then do a CA roll with a two-CA trust bundle.
|
||||||
|
|
||||||
|
**wonder (stages 5–6):** run the standard deploy verification, but over https: `/health/ready` on
|
||||||
|
both centrals and sites, CLI `site list`, the inbound API probe recipe and the MES test machines.
|
||||||
|
Then run a QA failover in an agreed window.
|
||||||
|
|
||||||
|
## 11. Open decisions for the owner
|
||||||
|
|
||||||
|
Each decision has a recommendation, marked **(rec)**.
|
||||||
|
|
||||||
|
- **D1. PKI availability on the wonder network.** Is there a corporate CA, such as AD CS with
|
||||||
|
auto-enrolment, that will issue certificates to `wonder-app-*` with `serverAuth` + `clientAuth`?
|
||||||
|
The code cannot show this.
|
||||||
|
- **(rec)** If yes, use it. If no, use a ScadaBridge-dedicated private CA managed offline, with its
|
||||||
|
root installed into `LocalMachine\Root` on each box.
|
||||||
|
- **D2. Where the key lives on Windows.**
|
||||||
|
- **(rec)** `LocalMachine\My` by thumbprint. The private key ACL grants read to the service account.
|
||||||
|
- Alternative: a PFX outside the content root, with its password in the Secrets store.
|
||||||
|
- **D3. The ScadaBridge service account on the wonder boxes.** It decides the private key ACL. The
|
||||||
|
deployment records show Windows services but not the account.
|
||||||
|
- **D4. The production front door.**
|
||||||
|
- **(rec)** Kestrel terminates TLS directly, as wonder works today.
|
||||||
|
- Alternatives: Traefik, IIS/ARR, or an F5/hardware load balancer.
|
||||||
|
- Also decide whether to keep port :8085 or move to 443.
|
||||||
|
- **D5. Has the revisit trigger been met?** Are the current wonder networks wholly inside the plant
|
||||||
|
security boundary? If yes, this stays design-only and nothing is flipped.
|
||||||
|
- **D6. Client auth on gRPC.**
|
||||||
|
- **(rec)** PSK over TLS.
|
||||||
|
- Alternative: add client-certificate mTLS.
|
||||||
|
- **D7. Akka trust scope.**
|
||||||
|
- **(rec)** If the issuing CA is broad (corporate-wide), add issuer or subject pinning through
|
||||||
|
`DotNettySslSetup.CustomValidator`. That is a code change.
|
||||||
|
- If the CA is ScadaBridge-dedicated, chain trust is enough.
|
||||||
|
- Also decide when to turn on `ValidateCertificateHostname`.
|
||||||
|
- **D8. Site :8084 metrics and health over TLS.** Prometheus and Overview would need the CA. Today the
|
||||||
|
listener carries no credentials.
|
||||||
|
- **(rec)** Leave it HTTP for the first rollout.
|
||||||
|
- **D9. Certificate lifetime, renewal cadence, and who owns renewal and expiry alerts.**
|
||||||
|
- **D10. HSTS.** Once it is on, browsers refuse plain HTTP for the max-age.
|
||||||
|
- **(rec)** Enable it last, with a short max-age first.
|
||||||
|
- **D11. Rig Traefik backend.**
|
||||||
|
- **(rec)** Re-encrypt to Kestrel.
|
||||||
|
- Alternative: terminate TLS at Traefik with plaintext on the docker network.
|
||||||
|
- **D12. OPC UA and MxGateway.** Move OtOpcUa connections to `SignAndEncrypt`, and serve mxgw over
|
||||||
|
TLS. These depend on those services' own rollout.
|
||||||
|
- **D13. SQL Server and LDAP** (out of scope here). The rig uses `TrustServerCertificate=true` and
|
||||||
|
GLAuth with `Transport=None`. Production equivalents need their own decision.
|
||||||
@@ -244,6 +244,7 @@ Each node is configured with:
|
|||||||
- **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site).
|
- **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site).
|
||||||
- **gRPC port** (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport.
|
- **gRPC port** (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport.
|
||||||
- **Local storage paths**: SQLite database locations (site nodes only).
|
- **Local storage paths**: SQLite database locations (site nodes only).
|
||||||
|
- **Akka.Remote TLS** (opt-in, `ScadaBridge:Cluster:RemotingTls`, default **off**; ScadaBridge#64): when `Enabled`, `BuildHocon` renders `akka.remote.dot-netty.tcp.enable-ssl` with this node's own certificate (`CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName`/`StoreLocation`), mutual authentication on by default and chain validation always on (no `suppress-validation` switch). Node addresses become `akka.ssl.tcp://…`, so every seed must use that scheme — `ClusterOptionsValidator` refuses an `akka.tcp` seed with TLS on. Both nodes of a pair flip together. With TLS off the HOCON is byte-identical to before. Akka.Remote has no other authentication, so mTLS is the only way to stop an arbitrary reachable host joining the pair. Design and rollout: `docs/plans/2026-11-09-tls-production-profile-design.md`.
|
||||||
|
|
||||||
## Windows Service
|
## Windows Service
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user