# TLS production profile: Traefik, Kestrel, gRPC and Akka remoting (design, 2026-11-09) **Status: DESIGN, plus opt-in groundwork that is OFF by default (ScadaBridge#64, register row 128).** No deployed configuration changes. No environment is switched to TLS by this document or by the code that ships with it. The revisit trigger in the register is unchanged: the first deployment onto a network that is not wholly inside the plant security boundary. Groundwork in the same change (section 9): an opt-in Akka.Remote TLS block (`ScadaBridge:Cluster:RemotingTls`), a fix that lets the control-plane PSK credentials work on an `https://` gRPC endpoint, and a fix to the bootstrap guard's seed parser for `akka.ssl.tcp`. With the defaults, the Akka HOCON document is byte-identical to before (12 rendered documents compared by SHA). Kestrel configuration is not touched. Every "current state" claim below cites the code it was read from. Anything about the wonder environment that the code cannot show is listed as an **open owner decision** in section 11, not assumed. --- ## 1. Hops in scope | # | Hop | Transport today | Listener / client | |---|---|---|---| | H1 | Browser to Central UI (and Inbound API callers) | HTTP/1.1, plaintext | Traefik `web` :80 to Kestrel :5000 (rig); Kestrel :8085 directly (wonder) | | H2 | CLI to `/management` | HTTP, plaintext; Basic or bearer token | same Kestrel HTTP surface as H1 | | H3 | Central and site gRPC control plane | h2c (HTTP/2 without TLS) + per-site PSK | site :8083, central :8083 | | H3b | LocalDb pair sync (site node to site node) | h2c + bearer API key | shares the site :8083 listener | | H4 | Akka remoting within a pair | DotNetty TCP, plaintext, no authentication | central :8081, site :8082 | | H5 | Site to external OPC UA server | OPC UA SecureChannel; `None` by default | DCL `RealOpcUaClient` | | H5b | Site to MxGateway | gRPC; h2c by default, TLS when configured | DCL `RealMxGatewayClient` | | H6 | Standby central to active central (Inbound API relay) | HTTP, plaintext | `StandbyForwardPeerEndpoints` | | H7 | Prometheus / Overview to `/health/*`, `/metrics` | HTTP/1.1, plaintext | site :8084, central HTTP surface | H6 and H7 follow the same Kestrel decision as H1. SQL Server and LDAP connections are out of scope here. They are noted in section 10. ## 2. Current state, with evidence ### H1: Browser to Traefik to Central UI - **Rig Traefik is HTTP only.** `docker/traefik/traefik.yml` declares one entrypoint, `web: address: ":80"`, and `api: insecure: true`. `docker/traefik/dynamic.yml` routes `PathPrefix(/)` to `http://scadabridge-central-a:5000` and `http://scadabridge-central-b:5000`, health-checked on `/health/active`. Host port `9000:80` (`docker/docker-compose.yml`). - **Traefik exists only on the docker rigs.** The wonder deployment records show centrals served directly by Kestrel on `:8085`, with no load balancer in front (`docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md`; `scadaproj/deployment.md` section 7.1, port 8085 "ScadaBridge Central: CentralUI + Management + Inbound API"). QA users address `vq03:8085` or `vq04:8085` by name. - **Kestrel central HTTP surface.** `Program.cs` (Central branch) reads `ASPNETCORE_URLS`, falling back to `http://+:5000`, and re-binds each port with `options.ListenAnyIP(httpPort, ...)` and `HttpProtocols.Http1AndHttp2`. It does this because the explicit gRPC listener switches Kestrel into explicit-endpoints mode. - **Finding F1: the scheme in `ASPNETCORE_URLS` is silently dropped.** `Program.ParseHttpBindPorts` keeps only the port (its own remarks: "only the port matters"), and the re-bind never calls `UseHttps`. So `ASPNETCORE_URLS=https://+:443` today binds **plaintext** HTTP on 443, with no startup error. Any TLS profile must close this. Either honour the scheme (bind TLS with a configured certificate) or refuse to boot on an `https` URL that has no certificate. - **Session cookie.** `SecurityOptions.RequireHttpsCookie` defaults to `true`, which sets the cookie's `Secure` flag (`src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs`). Both rig centrals set it to `false` (`docker/central-node-*/appsettings.Central.json`). When it is false the code logs a startup warning and uses `SameAsRequest` (`Security/ServiceCollectionExtensions.cs`). The value on the wonder boxes is not recorded in the repo; see section 11. - **Forwarded headers.** `ForwardedHeadersSetup` honours `X-Forwarded-For | X-Forwarded-Proto` from configured proxies only, and is disabled by default. The rig enables it for `172.16.0.0/12`. This matters for TLS that terminates at a proxy: with `X-Forwarded-Proto: https` trusted, the app sees the request as HTTPS, so the session cookie is `Secure` even under `SameAsRequest`. ### H2: CLI to management - `ManagementHttpClient` takes any absolute `http` or `https` URL (`CLI/Commands/CommandHelpers.cs`: "Expected an absolute http/https URL"). It exchanges credentials for a bearer token at `POST /auth/token`, falling back to HTTP Basic (`CLI/ManagementHttpClient.cs` header remarks). Over `http` both credential forms travel in cleartext. - It uses a plain `SocketsHttpHandler { ConnectTimeout, AllowAutoRedirect = false }`, with no certificate-validation override anywhere in the CLI. So an `https` URL already gets OS-trust-store validation. **No CLI code change is needed for TLS.** Operators change the URL, and the box must trust the issuing CA. ### H3: Central and site gRPC control plane - **Site listener.** `SiteKestrelSetup.Configure` binds `ListenAnyIP(grpcPort)` with `HttpProtocols.Http2` only (h2c), plus `ListenAnyIP(metricsPort)` with `Http1AndHttp2`. Neither calls `UseHttps`. - **Central listener.** `Program.cs` binds `ListenAnyIP(centralGrpcPort)` with `Http2` only (h2c). The comment states: "gRPC does NOT go through Traefik (HTTP/1 only); sites reach this port by container name". - **Endpoint validation already allows https.** `GrpcEndpointRules` accepts "an absolute http:// or https:// URL" (`Commons/Validation/GrpcEndpointRules.cs`). The site addresses are DB rows (`docker/seed-sites.sh` sets `--grpc-node-a-address "http://scadabridge-site-a-a:8083"`). The site-to-central list is `ScadaBridge:Communication:CentralGrpcEndpoints` in each `appsettings.Site.json`. - **Client builders.** These build channels: - `SitePairChannelProvider.BuildChannel`: central to site commands. - `SiteStreamGrpcClient`: central to site streams. - `CentralChannelProvider`: site to central control. - `GrpcPull{AuditEvents,SiteCalls,TraceEvents}Client`: central to site audit pulls, in AuditLog. All of them build a `SocketsHttpHandler` with keepalive, then call `.WithMessageLimits()` and `.WithSiteCredentials(...)`. - **PSK.** `ControlPlaneCredentials.WithSiteCredentials` attaches `Authorization: Bearer ` and `x-scadabridge-site` as call credentials (`Communication/Grpc/ISitePskProvider.cs`). On the server, `ControlPlaneAuthInterceptor` (site) and `CentralControlAuthInterceptor` (central) are fail-closed. Central resolves one key per site from `ScadaBridge:Communication:SitePsks` or the secret `SB-GRPC-PSK-` (`Host/SitePskProvider.cs`). Gitea #96 (`3a31dd2d`) added in-place rotation: `SitePskRotationInvalidator`, plus `PskRejectionRefreshHandler`, which drops the cached key on a `PermissionDenied`/`Unauthenticated` response. - **Finding F2: the PSK composition could not run over TLS. Fixed for 4 of 7 builders in this change.** The credentials were always composed over `ChannelCredentials.Insecure`, with `UnsafeUseInsecureChannelCallCredentials = true`. Grpc.Net.Client 2.76 refuses that on an `https` address, with the error "Channel is configured with insecure channel credentials and can't use a HttpClient with a 'https' scheme". So every https endpoint threw at channel construction. In `SitePairChannelProvider` that throw is caught and the address logged as malformed, so a TLS site would have been **silently undialable**. The class comment claimed TLS "requires no change to this code"; it was wrong. The new test `ControlPlaneTlsEndToEndTests.Https_channel_with_insecure_composition_throws_at_construction` pins the failure. Section 9 describes the fix. **Still open:** the three AuditLog pull clients. Each needs a one-argument change (`WithSiteCredentials(_pskProvider, siteId, endpoint)`). They were left untouched on purpose because AuditLog is being changed concurrently for #42. Until they get the argument, audit pulls from an `https` site fail at channel build. ### H3b: LocalDb pair sync - Sync rides the site's :8083 listener. The dialling half's `LocalDb:Replication:PeerAddress` is `http://scadabridge-site-a-b:8083` (`docker/site-a-node-a/appsettings.Site.json`). - The LocalDb library (`ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.DialPeer`) builds `GrpcChannel.ForAddress(PeerAddress, …)` with **no channel credentials**, and sends its key as an `Authorization` metadata header. An `https` PeerAddress therefore works unchanged, using OS-trust-store validation. But **a site's :8083 cannot be TLS for central and h2c for its peer**: the pair's PeerAddress must flip in the same step as the listener. ### H4: Akka remoting within a pair - `AkkaHostedService.BuildHocon` emits `akka.remote.dot-netty.tcp { hostname, port }` and a transport failure detector. It emits **no SSL keys**, so Akka's reference default `enable-ssl = false` applies. Ports are central 8081 and site 8082; the rig publishes them to the host (`9011:8081`, `9021:8082`, and so on). - **No authentication.** Akka.Remote 1.5.62 (`Directory.Packages.props`) has no shared-secret or "secure cookie" mechanism; the strings `secure-cookie` and `require-cookie` do not exist in `Akka.Remote.dll`. `untrusted-mode` exists but is off, and it only filters system messages and actor selections. Anyone who can reach :8081/:8082 can join the cluster or message its actors. **Mutual TLS is the only authentication Akka.Remote offers.** The issue text says "Akka `enable-ssl` + secure cookie"; the cookie half does not exist in Akka.NET. - What Akka.Remote 1.5.62 supports (read from its embedded reference config and strings): - `enable-ssl` - `ssl.certificate.{path,password,flags}`, or `{use-thumbprint-over-file, thumbprint, store-name, store-location}` - `ssl.suppress-validation` - `ssl.require-mutual-authentication`, **default true** - `ssl.validate-certificate-hostname`, default false - a programmatic `DotNettySslSetup` with a `CustomValidator`, plus helpers for chain, issuer, subject, hostname and pinned-certificate validation At startup it refuses a certificate without an accessible private key. - **The address scheme changes under TLS.** With SSL on, node addresses are `akka.ssl.tcp://…`. The new test `HoconRemotingTlsTests.Enabled_document_boots_a_tls_transport_with_akka_ssl_tcp_addresses` checks this by booting a real `ActorSystem` on the emitted document. Effects on code that parses seeds: - `StartupValidator.SeedNodeIsSelf` and `SeedNodePort/Host` use `@` and the last `:`, so they do not depend on the scheme. They still work. - `ClusterBootstrapGuard.SeedPattern` was `^akka(?:\.[a-z0-9]+)?://`, a single dot segment. **Finding F3:** it did not match `akka.ssl.tcp`, so with TLS on the guard would have parsed no seeds and silently never engaged. Fixed in this change. - `ClusterBootstrapCoordinator` uses Akka's own `Address.Parse`, which handles the TLS scheme. - Overview's `LeaderResolver.HostOf` (scadaproj) parses after `@`, so it does not depend on the scheme either. ### H5: Site to OPC UA and MxGateway - **OPC UA** uses OPC UA SecureChannel, not TLS. `OpcUaEndpointConfig.SecurityMode` defaults to `None`. The values are `None`, `Sign` and `SignAndEncrypt`. `AutoAcceptUntrustedCerts` defaults to false, and when on it logs a man-in-the-middle warning (`RealOpcUaClient`). The application PKI stores come from `ScadaBridge:OpcUa:{TrustedIssuer,TrustedPeer,RejectedCertificate}StorePath`. `OpcUaGlobalOptions` says empty paths "fall back to a default under Path.GetTempPath()". A temp path is the wrong home for a production trust list, so the production profile should set these explicitly. The rig simulators run `--unsecuretransport --autoaccept` (`deployment.md` section 3.3). - **MxGateway.** Each connection carries `Endpoint` (default `http://localhost:5000`), `UseTls`, `CaFile` and `ServerName` (`Commons/Types/DataConnections/MxGatewayEndpointConfig.cs`). These are passed straight to `MxGatewayClientOptions` (`RealMxGatewayClient.ConnectAsync`). TLS already works per connection, with a private-CA file. **No ScadaBridge code change is needed.** Enabling it depends on the gateway serving TLS (its `…Http__Url=https://…`; `deployment.md` mxgw section). ### H6 and H7: relay and health - `InboundApiOptions.StandbyForwardPeerEndpoints` accepts `http` or `https` and is used through a named `HttpClient` (`InboundAPI/ServiceCollectionExtensions.cs`) with default validation. It flips when the central surface flips. - The site metrics and health listener (:8084) is HTTP/1.1 by design, because Prometheus and the Overview dashboard poll it. It carries no credentials (`/health/*` is anonymous). Making it TLS is optional, and section 11 lists it as an owner decision. ## 3. Threat summary (why each hop matters) | Hop | Credential or data exposed in cleartext today | Integrity or authentication gap | |---|---|---| | H1 | session cookie (when `RequireHttpsCookie=false`), LDAP password at login, Inbound API `sbk_` keys | none at the transport layer | | H2 | operator LDAP password (Basic) or bearer token | none at the transport layer | | H3 | per-site PSK (bearer), deployments incl. encrypted secret artifacts, audit rows | the PSK authenticates the caller, but it is replayable by anyone on the path | | H3b | LocalDb API key, all replicated site rows | as H3 | | H4 | every cluster message | **no authentication at all**: a reachable host can join | | H5 | OPC UA values; MxGateway API key + writes | per connection; secure modes exist but are off by default | The PSK work (#96 and its predecessors) limits who may *call* the control plane. It does nothing for confidentiality, and a key captured on the wire is valid until it is rotated. H4 is the weakest link: unauthenticated cluster membership. ## 4. Recommended TLS mode per hop ### 4.1 Kestrel: HTTPS with ALPN (H1, H2, H6, H7 and the gRPC listeners) - Add a node-level certificate option, `ScadaBridge:Node:Tls`, with the same shape as `RemotingTls`: `CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName` + `StoreLocation`. Apply it with `listenOptions.UseHttps(cert)` to every listener the node binds. - **Central HTTP surface:** `Http1AndHttp2` on one TLS port. ALPN negotiates h2 for gRPC-capable clients and HTTP/1.1 for browsers. This is the family pattern: HistorianGateway runs one `Kestrel:Endpoints:Https` with `Http1AndHttp2`. - **gRPC ports (central :8083, site :8083):** `Http2` over TLS. Clients negotiate h2 by ALPN, and no h2c remains. - **Site :8084:** optional (section 11). - Close **F1**: `ParseHttpBindPorts` must return the scheme as well as the port, and an `https` entry must bind TLS. With no certificate configured it must **refuse to boot**, so it never downgrades silently. - Keep TLS configuration in code, as the existing listeners are, not in `Kestrel:Endpoints`. The explicit-endpoints comment in `Program.cs` already documents one surprise from mixing the two sources. A config-section endpoint added on top of code-bound endpoints is a second surprise waiting to happen. It is unverified whether Kestrel unions the two sources or replaces one with the other. - HSTS (`UseHsts`) only after an environment is confirmed HTTPS-only. Section 11 has the decision. ### 4.2 Traefik (rig): TLS entrypoint, re-encrypted backend - Add a `websecure` entrypoint on :443 (host `9443`), with a file-provider `tls.certificates` entry that holds the lab certificate. Redirect `web` to `websecure`. - Backend: in the production profile, Traefik dials `https://scadabridge-central-{a,b}:5000`, with a `serversTransport` whose `rootCAs` is the lab CA. TLS runs end to end and the same Kestrel TLS path is exercised on the rig. Plaintext on the internal docker network is an acceptable *rig-only* shortcut, but it would leave F1 and §4.1 untested on the rig. - Keep `ForwardedHeaders` trusting Traefik. Set `RequireHttpsCookie` back to its default, `true`, on every TLS-fronted node. - Turn off `api.insecure` (the dashboard) or bind it to loopback in any non-rig use. - gRPC stays off Traefik, as it is today. Nodes dial each other directly. ### 4.3 gRPC: TLS with certificate validation; PSK retained - Endpoints become `https://host:8083`: the site rows `GrpcNodeAAddress`/`GrpcNodeBAddress`, `CentralGrpcEndpoints`, and `LocalDb:Replication:PeerAddress`. - Clients validate the server chain and hostname with `SocketsHttpHandler` defaults. No `RemoteCertificateValidationCallback` override and no "accept any" switch is to be added. **Hostnames in endpoints must be SANs on the server certificate.** - **Keep the PSK.** TLS gives confidentiality and server authentication. The PSK stays the per-site *client* identity, which central's interceptor keys per-site authorization on. With TLS the PSK stops being replayable from the wire. `WithSiteCredentials` now composes it over `ChannelCredentials.SecureSsl` for https (section 9). - Client-certificate mTLS on gRPC is **not** recommended for the first step. It would duplicate the PSK's job and add a client-certificate distribution path to every site. Section 11 lists it as an owner option. ### 4.4 Akka.Remote: DotNetty SSL, one certificate per node, mutual authentication - Set `ScadaBridge:Cluster:RemotingTls:Enabled = true` with this node's own certificate. `RequireMutualAuthentication = true` (the default). Chain validation is on and cannot be turned off through ScadaBridge configuration. The `suppress-validation` key is always emitted as `false`. - All seeds use `akka.ssl.tcp://scadabridge@host:port`. `ClusterOptionsValidator` refuses a TLS-enabled config that still has an `akka.tcp` seed. Without that check, the pair would retry joins forever with no startup error. - `ValidateCertificateHostname = false` at first, which is Akka's default; the trust anchor is a dedicated issuing CA. Turn it on once every `NodeHostname`/seed host is confirmed as a SAN on the peer's certificate. - If the CA is a broad corporate CA, chain trust alone admits *any* host holding a certificate from that CA. In that case add issuer or subject pinning through `DotNettySslSetup.CustomValidator`. That is a code change in `GetOrCreateActorSystem`: `ActorSystem.Create` with a `BootstrapSetup`, which section 9 does not include. Section 11 has the decision. - Both nodes of a pair flip together. A TLS node and a plaintext node cannot talk to each other. This matches the existing rule that a site pair stops and starts together (LocalDb Phase 2). ### 4.5 Site to OPC UA and MxGateway - **OPC UA:** set `SecurityMode = SignAndEncrypt` per connection, wherever the server offers it (OtOpcUa does). Set explicit `ScadaBridge:OpcUa:*StorePath` directories under the node's data root, not the temp default. Trust the server certificates explicitly. `AutoAcceptUntrustedCerts` stays false. - **MxGateway:** use an `https://` endpoint with `UseTls = true`, `CaFile` (the issuing CA) and `ServerName` matching the gateway certificate. This needs no code change, only the gateway serving TLS. ## 5. Certificates: issuance, storage, rotation ### 5.1 One certificate per node - **Subject Alternative Names:** the node's `NodeHostname` exactly as it appears in seeds and gRPC endpoints (the FQDN on wonder, e.g. `wonder-app-vq03.zmr.zimmer.com`), plus the short name. On the rig, the container name (`scadabridge-site-a-a`). - **Extended Key Usage:** `serverAuth` **and** `clientAuth`. Akka mTLS makes every node both a TLS server and a TLS client. Kestrel and gRPC need `serverAuth`. - One certificate per node serves both Kestrel and Akka on that node. On the co-located wonder boxes, the central and site services on one box can share the box certificate. Both services run on the same host name, so the SANs are identical. ### 5.2 Issuance - **wonder:** this needs a CA that the boxes trust. Whether a corporate PKI exists (for example AD Certificate Services with auto-enrolment) and is available to these boxes is **not knowable from the code**. It is open decision D1. - **Rig:** use a lab CA generated by a new script, `docker/tls/gen-lab-ca.sh` (openssl, idempotent, output gitignored). It issues one certificate per container. Both the CA and the leaves are clearly marked lab-only, like the rig's other DEV-ONLY credentials. - **Trust on Linux containers:** Akka's chain validation and `SocketsHttpHandler` both use the OS trust store. So the rig image must install the lab CA. Mount it into `/usr/local/share/ca-certificates/` and run `update-ca-certificates` at container start. Do not bake it into the image, because the image is shared with non-rig uses. ### 5.3 Where certificates and keys live | Environment | Certificate + private key | Secret material | |---|---|---| | wonder (Windows) | **`LocalMachine\My`, referenced by thumbprint** (`CertificateThumbprint`; `StoreLocation = local-machine`). No key file on disk. The private key ACL grants read to the service account. | none: the store holds the key | | wonder, fallback | `.pfx` outside the content root (same rule as `Secrets:SqlitePath`: absolute, not under `app\`, which an upgrade replaces) | password as `${secret:SB-TLS-PFX-PASSWORD}`, resolved from the Secrets store by the pre-host expander | | rig (Linux) | `.pfx` on a read-only bind mount (`/certs/.pfx`) | DEV-ONLY password in compose env, like the mesh PSKs | Storing the PFX bytes themselves in the Secrets store is **not recommended**. Akka's HOCON path wants a file, and Kestrel would need a materialised copy anyway. The Secrets store is the right home for the **password**. Do not log the HOCON document. With a file certificate it contains the PFX password, which `BuildRemotingTlsBlock` escapes but does not redact. `GetOrCreateActorSystem` logs only individual fields today, and `akka.log-config-on-start` is off by default. Both must stay that way. ### 5.4 Rotation - **Leaf rotation under the same CA:** the peer validates the chain, not a pinned leaf. A node can therefore get a new leaf and restart on its own, without coordinating with its partner: - Kestrel picks up the new certificate on restart. A `ServerCertificateSelector` could make that hot later. - Akka.Remote reads its certificate only at `ActorSystem` start, so it needs a restart. - On a pair, restart one node at a time. A one-node restart is ordinary failover. A pair-together stop is required only for the plaintext/TLS flip, not for a leaf rotation. - **CA rotation:** first distribute a trust bundle with both the old and the new CA to every node. Then rotate the leaves. Then remove the old CA. - **Expiry monitoring:** add a startup log and a `/health/ready` degraded entry when the node certificate expires in fewer than N days. That is follow-on work. Who owns renewal is decision D9. ## 6. Impact on the docker rig and the wonder boxes ### 6.1 Rig (`docker/`, `docker-env2/`) - New: `docker/tls/` (CA script, gitignored output), Traefik `websecure` + certificates, `RemotingTls` blocks and `akka.ssl.tcp` seeds in each node's appsettings, `https` site addresses in `seed-sites.sh`, `https` `CentralGrpcEndpoints` and LocalDb PeerAddress, and CA trust in the containers. - Host ports: add 9443 for Traefik TLS. Existing 9000 stays open (redirect) until the profile is the default. - Scripts that probe `http://localhost:900x`, the CLI recipes and the Playwright suites need a base-URL switch. Their certificate trust must come from the lab CA, never from "ignore errors". - Rig posture, mirroring the LocalDb rollout: **site-a on TLS, site-b and site-c on h2c**, so both states are proven side by side. Central must dial both: per-site endpoints make that natural. ### 6.2 wonder (DEV vd03, QA vq03/vq04) - No Traefik. Kestrel terminates TLS itself on the central port (:8085 today; whether to move to 443 is decision D4). - Certificates in `LocalMachine\My` (if PKI is available, D1/D2). Thumbprints go in `appsettings.{Central,Site}.json`, which are preserved across binary swaps per the deployment records. - The site rows in the shared ConfigDb change to `https://wonder-app-vq0X…:8083` through the CLI `site update`. That is a data change, reversible the same way. - **Remote access caveat:** under the 2026-09-28 WinRM scope, certificate installation counts as a deploy step and is allowed only as part of a deploy the user asked for. - Overview: when the central surface goes TLS, its registry `BaseUrl`s become `https`. Its leader matching does not depend on the scheme (section 2, H4). ## 7. Staged rollout Each stage is independently reversible (section 8) and proven before the next one starts. | Stage | What | Where | Gate | |---|---|---|---| | 0 | **This change:** design + default-off groundwork (Akka TLS block, https-capable PSK channels, guard regex) | code only | unit + real-transport tests green; default HOCON byte-identical | | 1 | Code: `ScadaBridge:Node:Tls` for Kestrel; close F1; the AuditLog pull clients take the endpoint (after #42 lands); optional cert-expiry health entry | code only, default off | unit + real-Kestrel TLS tests; default Kestrel binding unchanged | | 2 | Rig lab CA + Traefik `websecure` (re-encrypting backend) + central Kestrel TLS | docker | browser + CLI over https; Secure cookie observed; plaintext :5000 refused once flipped | | 3 | Rig site-a: gRPC https (both directions + LocalDb PeerAddress) and Akka TLS on the site-a pair | docker site-a only | section 10 rig checks; failover drill on site-a; site-b/c unchanged | | 4 | Rig: central pair Akka TLS; then all sites | docker | full failover drill (`docker/failover-drill.sh`) | | 5 | DEV vd03 (single node: Akka TLS is self-only; gRPC + Kestrel TLS) | wonder DEV | post-deploy https `/health/ready`, CLI, MES test machines | | 6 | QA vq03/vq04 pair | wonder QA | pair stop/start together; failover drill in a window | Migration detail for the central gRPC port: during stage 3, central must answer both TLS sites and h2c sites on site-to-central calls. A single Kestrel port is either TLS or not. Stage 1 therefore adds an **optional second central gRPC listener** (for example `CentralGrpcTlsPort`). Sites move to it one at a time, and the h2c port is removed once no site uses it. The central-to-site direction has no such problem, because each site address is its own row. ## 8. Rollback Every switch is configuration, and every default is off: - **Akka:** set `RemotingTls:Enabled=false` and revert the seeds to `akka.tcp`. Stop and start the pair **together**. - **gRPC:** revert the site rows to `http://` (`site update`), revert `CentralGrpcEndpoints` and `PeerAddress`, then restart. The PSK is untouched throughout, so auth never lapses during rollback. - **Kestrel and Traefik:** revert `ASPNETCORE_URLS` and the node `Tls` block, and revert the Traefik files. - **Cookie:** `RequireHttpsCookie=true` must stay **true** while the surface is TLS. Rolling the surface back to HTTP is the only case where setting it false again is justified. - **HSTS:** browsers keep enforcing HTTPS for the max-age after rollback. This is why HSTS is enabled last, with a short max-age first (decision D10). - **No schema change** is involved at any stage. On wonder, the previous `appsettings*.json` is already preserved in `app.bak.` by the standard deploy recipe. ## 9. Groundwork shipped with this design (default OFF) 1. **`ScadaBridge:Cluster:RemotingTls`** (`ClusterRemotingTlsOptions` in ClusterInfrastructure). - Fields: `Enabled`, `CertificatePath`/`CertificatePassword` or `CertificateThumbprint`/ `StoreName`/`StoreLocation`, `RequireMutualAuthentication` (default true), `ValidateCertificateHostname` (default false). - `AkkaHostedService.BuildRemotingTlsBlock` renders `enable-ssl` + `ssl { … }` inside `dot-netty.tcp`, **only when `Enabled`**. - `ClusterOptionsValidator` (TLS on only) requires: - exactly one certificate source; - a legal store location; - `akka.ssl.tcp://` on every seed. - **Byte-identity proof:** before the change, the rendered HOCON was captured for 12 configurations: `auto-down` / `leader-auto-down` / `keep-oldest`, bootstrap guard on and off, Central and Site. After the change all 12 were re-rendered and diffed: identical. A permanent test (`Disabled_block_with_fields_is_byte_identical_to_default`) pins it: a disabled block with every field set renders exactly the default document. 2. **https-capable PSK channels.** `ControlPlaneCredentials.WithSiteCredentials(..., endpoint)` composes over `ChannelCredentials.SecureSsl` for an `https` endpoint. For anything else, including callers that do not pass the endpoint, it keeps the previous insecure composition with the explicit opt-in, unchanged. `SitePairChannelProvider`, `SiteStreamGrpcClient` and `CentralChannelProvider` pass their endpoint. The three AuditLog pull clients are deliberately untouched (F2). With today's `http` endpoints, behaviour is unchanged. 3. **`ClusterBootstrapGuard.SeedPattern`** accepts any number of scheme segments, so `akka.ssl.tcp` parses (F3). 4. **Not changed:** `Program.cs`, `SiteKestrelSetup.cs`, Traefik and every appsettings and compose file. Kestrel binding is identical by construction. Tests: - `HoconRemotingTlsTests` (5): includes booting a real TLS `ActorSystem` from a generated certificate. A mutation check (forcing `enable-ssl = false`) turns it red with `akka.tcp`. - `ClusterRemotingTlsValidatorTests` (8). - `ClusterBootstrapGuardTests` (+2 for `akka.ssl.tcp`). - `ControlPlaneTlsEndToEndTests` (6). These use real Kestrel with `Http2` over TLS (ALPN), the real `ControlPlaneAuthInterceptor`, and a pinned test certificate: - the correct PSK is accepted over TLS; - a wrong PSK is still `PermissionDenied` over TLS; - an untrusted server certificate fails with an `AuthenticationException` cause; - the pre-fix composition throws on https; - the h2c composition is unchanged on http. - `HttpsEndpointChannelTests` (9). These check that all three production builders accept `https`. A mutation check (disabling the TLS branch) turns the three construction tests red. ## 10. Test plan for stages 1–6 **Unit and in-process:** extend the stage-0 pattern to the Kestrel `Tls` block: - certificate loading from a file and from the store (the store path is Windows-only, so skip it elsewhere); - F1: an `https` URL with no certificate must refuse to boot; - ALPN: a real Kestrel `Http1AndHttp2` TLS listener serves an HTTP/1.1 GET and an h2 gRPC call on one port. **Rig, per hop** (each with a negative control, so a check that "passes" is shown to be able to fail): - **H1:** `curl --cacert lab-ca.pem https://localhost:9443/health/ready` returns 200. Login over https sets the cookie `Secure` (read the `Set-Cookie`). `http://localhost:9000` redirects. Traefik to backend is re-encrypted: a plaintext probe to `central-a:5000` is refused once flipped. - **H2:** CLI against the https URL with the lab CA trusted succeeds. With the CA *not* trusted, it fails with a certificate error and never falls back to http. - **H3:** - `openssl s_client -alpn h2 -connect site-a-a:8083` shows TLS 1.2 or later and `h2`. - An h2c client against the TLS port fails. - Central to site commands, streams, audit pulls and site to central heartbeats all flow for site-a over TLS while site-b stays h2c. - The PSK is still enforced: a wrong key over TLS gives `PermissionDenied`. - The #96 rotation drill passes over TLS. - **H3b:** LocalDb `LocalDbReplicationConnected` is true on the site-a pair over https. `MaxBatchBytes` behaviour is unchanged. - **H4:** - The pair forms with `akka.ssl.tcp` addresses (`/health/ready` shows memberCount 2). - **Negative:** a third container with a certificate from a *different* CA cannot join. It gets a handshake failure in its log and is never Up. - **Negative:** a container with *no* certificate cannot join, which shows mTLS is enforced. - Expired-certificate node: refused. - Failover drill timings within the existing `StableAfter` budget. A TLS handshake on reconnect must not push detection past it. - **H5:** one OPC UA connection on `SignAndEncrypt` with an explicitly trusted server certificate. One MxGateway connection with `UseTls` and `CaFile`. **Rotation drill:** replace one node's leaf certificate under the same CA, restart that node only, and confirm the pair re-forms. Then do a CA roll with a two-CA trust bundle. **wonder (stages 5–6):** run the standard deploy verification, but over https: `/health/ready` on both centrals and sites, CLI `site list`, the inbound API probe recipe and the MES test machines. Then run a QA failover in an agreed window. ## 11. Open decisions for the owner Each decision has a recommendation, marked **(rec)**. - **D1. PKI availability on the wonder network.** Is there a corporate CA, such as AD CS with auto-enrolment, that will issue certificates to `wonder-app-*` with `serverAuth` + `clientAuth`? The code cannot show this. - **(rec)** If yes, use it. If no, use a ScadaBridge-dedicated private CA managed offline, with its root installed into `LocalMachine\Root` on each box. - **D2. Where the key lives on Windows.** - **(rec)** `LocalMachine\My` by thumbprint. The private key ACL grants read to the service account. - Alternative: a PFX outside the content root, with its password in the Secrets store. - **D3. The ScadaBridge service account on the wonder boxes.** It decides the private key ACL. The deployment records show Windows services but not the account. - **D4. The production front door.** - **(rec)** Kestrel terminates TLS directly, as wonder works today. - Alternatives: Traefik, IIS/ARR, or an F5/hardware load balancer. - Also decide whether to keep port :8085 or move to 443. - **D5. Has the revisit trigger been met?** Are the current wonder networks wholly inside the plant security boundary? If yes, this stays design-only and nothing is flipped. - **D6. Client auth on gRPC.** - **(rec)** PSK over TLS. - Alternative: add client-certificate mTLS. - **D7. Akka trust scope.** - **(rec)** If the issuing CA is broad (corporate-wide), add issuer or subject pinning through `DotNettySslSetup.CustomValidator`. That is a code change. - If the CA is ScadaBridge-dedicated, chain trust is enough. - Also decide when to turn on `ValidateCertificateHostname`. - **D8. Site :8084 metrics and health over TLS.** Prometheus and Overview would need the CA. Today the listener carries no credentials. - **(rec)** Leave it HTTP for the first rollout. - **D9. Certificate lifetime, renewal cadence, and who owns renewal and expiry alerts.** - **D10. HSTS.** Once it is on, browsers refuse plain HTTP for the max-age. - **(rec)** Enable it last, with a short max-age first. - **D11. Rig Traefik backend.** - **(rec)** Re-encrypt to Kestrel. - Alternative: terminate TLS at Traefik with plaintext on the docker network. - **D12. OPC UA and MxGateway.** Move OtOpcUa connections to `SignAndEncrypt`, and serve mxgw over TLS. These depend on those services' own rollout. - **D13. SQL Server and LDAP** (out of scope here). The rig uses `TrustServerCertificate=true` and GLAuth with `Transport=None`. Production equivalents need their own decision.