From 31cb2c79e3e17fddcd89607f30f9a5e5cc95c19b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 17:06:01 -0500 Subject: [PATCH] docs(plans): TLS production profile design for Traefik, Kestrel, gRPC and Akka remoting (ScadaBridge#64) Per-hop current state with code evidence (browser/Traefik/Kestrel, CLI, central<->site gRPC + PSK, LocalDb sync, Akka remoting, OPC UA / MxGateway), the recommended mode per hop (Kestrel HTTPS + ALPN, gRPC over TLS with validation and the PSK retained, Akka DotNetty SSL with a cert per node and mutual auth), certificate issuance/storage/rotation, rig and wonder impact, a staged rollout, rollback, a test plan and 13 open owner decisions. Findings: the central ASPNETCORE_URLS re-bind drops the scheme (an https URL binds plaintext); Akka.NET has no secure-cookie, so mTLS is the only remoting authentication. Component-ClusterInfrastructure documents the new RemotingTls option. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- ...026-11-09-tls-production-profile-design.md | 536 ++++++++++++++++++ .../Component-ClusterInfrastructure.md | 1 + 2 files changed, 537 insertions(+) create mode 100644 docs/plans/2026-11-09-tls-production-profile-design.md diff --git a/docs/plans/2026-11-09-tls-production-profile-design.md b/docs/plans/2026-11-09-tls-production-profile-design.md new file mode 100644 index 00000000..f310442a --- /dev/null +++ b/docs/plans/2026-11-09-tls-production-profile-design.md @@ -0,0 +1,536 @@ +# TLS production profile: Traefik, Kestrel, gRPC and Akka remoting (design, 2026-11-09) + +**Status: DESIGN, plus opt-in groundwork that is OFF by default (ScadaBridge#64, register row 128).** +No deployed configuration changes. No environment is switched to TLS by this document or by the code +that ships with it. The revisit trigger in the register is unchanged: the first deployment onto a +network that is not wholly inside the plant security boundary. + +Groundwork in the same change (section 9): an opt-in Akka.Remote TLS block +(`ScadaBridge:Cluster:RemotingTls`), a fix that lets the control-plane PSK credentials work on an +`https://` gRPC endpoint, and a fix to the bootstrap guard's seed parser for `akka.ssl.tcp`. With +the defaults, the Akka HOCON document is byte-identical to before (12 rendered documents compared +by SHA). Kestrel configuration is not touched. + +Every "current state" claim below cites the code it was read from. Anything about the wonder +environment that the code cannot show is listed as an **open owner decision** in section 11, not +assumed. + +--- + +## 1. Hops in scope + +| # | Hop | Transport today | Listener / client | +|---|---|---|---| +| H1 | Browser to Central UI (and Inbound API callers) | HTTP/1.1, plaintext | Traefik `web` :80 to Kestrel :5000 (rig); Kestrel :8085 directly (wonder) | +| H2 | CLI to `/management` | HTTP, plaintext; Basic or bearer token | same Kestrel HTTP surface as H1 | +| H3 | Central and site gRPC control plane | h2c (HTTP/2 without TLS) + per-site PSK | site :8083, central :8083 | +| H3b | LocalDb pair sync (site node to site node) | h2c + bearer API key | shares the site :8083 listener | +| H4 | Akka remoting within a pair | DotNetty TCP, plaintext, no authentication | central :8081, site :8082 | +| H5 | Site to external OPC UA server | OPC UA SecureChannel; `None` by default | DCL `RealOpcUaClient` | +| H5b | Site to MxGateway | gRPC; h2c by default, TLS when configured | DCL `RealMxGatewayClient` | +| H6 | Standby central to active central (Inbound API relay) | HTTP, plaintext | `StandbyForwardPeerEndpoints` | +| H7 | Prometheus / Overview to `/health/*`, `/metrics` | HTTP/1.1, plaintext | site :8084, central HTTP surface | + +H6 and H7 follow the same Kestrel decision as H1. SQL Server and LDAP connections are out of scope +here. They are noted in section 10. + +## 2. Current state, with evidence + +### H1: Browser to Traefik to Central UI + +- **Rig Traefik is HTTP only.** `docker/traefik/traefik.yml` declares one entrypoint, + `web: address: ":80"`, and `api: insecure: true`. `docker/traefik/dynamic.yml` routes + `PathPrefix(/)` to `http://scadabridge-central-a:5000` and `http://scadabridge-central-b:5000`, + health-checked on `/health/active`. Host port `9000:80` (`docker/docker-compose.yml`). +- **Traefik exists only on the docker rigs.** The wonder deployment records show centrals served + directly by Kestrel on `:8085`, with no load balancer in front + (`docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md`; + `scadaproj/deployment.md` section 7.1, port 8085 "ScadaBridge Central: CentralUI + Management + + Inbound API"). QA users address `vq03:8085` or `vq04:8085` by name. +- **Kestrel central HTTP surface.** `Program.cs` (Central branch) reads `ASPNETCORE_URLS`, falling + back to `http://+:5000`, and re-binds each port with `options.ListenAnyIP(httpPort, ...)` and + `HttpProtocols.Http1AndHttp2`. It does this because the explicit gRPC listener switches Kestrel + into explicit-endpoints mode. +- **Finding F1: the scheme in `ASPNETCORE_URLS` is silently dropped.** `Program.ParseHttpBindPorts` + keeps only the port (its own remarks: "only the port matters"), and the re-bind never calls + `UseHttps`. So `ASPNETCORE_URLS=https://+:443` today binds **plaintext** HTTP on 443, with no + startup error. Any TLS profile must close this. Either honour the scheme (bind TLS with a configured + certificate) or refuse to boot on an `https` URL that has no certificate. +- **Session cookie.** `SecurityOptions.RequireHttpsCookie` defaults to `true`, which sets the + cookie's `Secure` flag (`src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs`). Both rig centrals + set it to `false` (`docker/central-node-*/appsettings.Central.json`). When it is false the code logs + a startup warning and uses `SameAsRequest` (`Security/ServiceCollectionExtensions.cs`). The value on + the wonder boxes is not recorded in the repo; see section 11. +- **Forwarded headers.** `ForwardedHeadersSetup` honours `X-Forwarded-For | X-Forwarded-Proto` from + configured proxies only, and is disabled by default. The rig enables it for `172.16.0.0/12`. This + matters for TLS that terminates at a proxy: with `X-Forwarded-Proto: https` trusted, the app sees + the request as HTTPS, so the session cookie is `Secure` even under `SameAsRequest`. + +### H2: CLI to management + +- `ManagementHttpClient` takes any absolute `http` or `https` URL + (`CLI/Commands/CommandHelpers.cs`: "Expected an absolute http/https URL"). It exchanges + credentials for a bearer token at `POST /auth/token`, falling back to HTTP Basic + (`CLI/ManagementHttpClient.cs` header remarks). Over `http` both credential forms travel in + cleartext. +- It uses a plain `SocketsHttpHandler { ConnectTimeout, AllowAutoRedirect = false }`, with no + certificate-validation override anywhere in the CLI. So an `https` URL already gets OS-trust-store + validation. **No CLI code change is needed for TLS.** Operators change the URL, and the box must + trust the issuing CA. + +### H3: Central and site gRPC control plane + +- **Site listener.** `SiteKestrelSetup.Configure` binds `ListenAnyIP(grpcPort)` with + `HttpProtocols.Http2` only (h2c), plus `ListenAnyIP(metricsPort)` with `Http1AndHttp2`. Neither + calls `UseHttps`. +- **Central listener.** `Program.cs` binds `ListenAnyIP(centralGrpcPort)` with `Http2` only (h2c). + The comment states: "gRPC does NOT go through Traefik (HTTP/1 only); sites reach this port by + container name". +- **Endpoint validation already allows https.** `GrpcEndpointRules` accepts "an absolute http:// or + https:// URL" (`Commons/Validation/GrpcEndpointRules.cs`). The site addresses are DB rows + (`docker/seed-sites.sh` sets `--grpc-node-a-address "http://scadabridge-site-a-a:8083"`). The + site-to-central list is `ScadaBridge:Communication:CentralGrpcEndpoints` in each + `appsettings.Site.json`. +- **Client builders.** These build channels: + - `SitePairChannelProvider.BuildChannel`: central to site commands. + - `SiteStreamGrpcClient`: central to site streams. + - `CentralChannelProvider`: site to central control. + - `GrpcPull{AuditEvents,SiteCalls,TraceEvents}Client`: central to site audit pulls, in AuditLog. + + All of them build a `SocketsHttpHandler` with keepalive, then call `.WithMessageLimits()` and + `.WithSiteCredentials(...)`. +- **PSK.** `ControlPlaneCredentials.WithSiteCredentials` attaches `Authorization: Bearer ` and + `x-scadabridge-site` as call credentials (`Communication/Grpc/ISitePskProvider.cs`). On the server, + `ControlPlaneAuthInterceptor` (site) and `CentralControlAuthInterceptor` (central) are fail-closed. + Central resolves one key per site from `ScadaBridge:Communication:SitePsks` or the secret + `SB-GRPC-PSK-` (`Host/SitePskProvider.cs`). Gitea #96 (`3a31dd2d`) added in-place rotation: + `SitePskRotationInvalidator`, plus `PskRejectionRefreshHandler`, which drops the cached key on a + `PermissionDenied`/`Unauthenticated` response. +- **Finding F2: the PSK composition could not run over TLS. Fixed for 4 of 7 builders in this + change.** The credentials were always composed over `ChannelCredentials.Insecure`, with + `UnsafeUseInsecureChannelCallCredentials = true`. Grpc.Net.Client 2.76 refuses that on an `https` + address, with the error "Channel is configured with insecure channel credentials and can't use a + HttpClient with a 'https' scheme". So every https endpoint threw at channel construction. In + `SitePairChannelProvider` that throw is caught and the address logged as malformed, so a TLS site + would have been **silently undialable**. The class comment claimed TLS "requires no change to this + code"; it was wrong. The new test + `ControlPlaneTlsEndToEndTests.Https_channel_with_insecure_composition_throws_at_construction` pins + the failure. Section 9 describes the fix. **Still open:** the three AuditLog pull clients. Each + needs a one-argument change (`WithSiteCredentials(_pskProvider, siteId, endpoint)`). They were left + untouched on purpose because AuditLog is being changed concurrently for #42. Until they get the + argument, audit pulls from an `https` site fail at channel build. + +### H3b: LocalDb pair sync + +- Sync rides the site's :8083 listener. The dialling half's `LocalDb:Replication:PeerAddress` is + `http://scadabridge-site-a-b:8083` (`docker/site-a-node-a/appsettings.Site.json`). +- The LocalDb library (`ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.DialPeer`) builds + `GrpcChannel.ForAddress(PeerAddress, …)` with **no channel credentials**, and sends its key as an + `Authorization` metadata header. An `https` PeerAddress therefore works unchanged, using + OS-trust-store validation. But **a site's :8083 cannot be TLS for central and h2c for its peer**: + the pair's PeerAddress must flip in the same step as the listener. + +### H4: Akka remoting within a pair + +- `AkkaHostedService.BuildHocon` emits `akka.remote.dot-netty.tcp { hostname, port }` and a transport + failure detector. It emits **no SSL keys**, so Akka's reference default `enable-ssl = false` + applies. Ports are central 8081 and site 8082; the rig publishes them to the host (`9011:8081`, + `9021:8082`, and so on). +- **No authentication.** Akka.Remote 1.5.62 (`Directory.Packages.props`) has no shared-secret or + "secure cookie" mechanism; the strings `secure-cookie` and `require-cookie` do not exist in + `Akka.Remote.dll`. `untrusted-mode` exists but is off, and it only filters system messages and + actor selections. Anyone who can reach :8081/:8082 can join the cluster or message its actors. + **Mutual TLS is the only authentication Akka.Remote offers.** The issue text says "Akka + `enable-ssl` + secure cookie"; the cookie half does not exist in Akka.NET. +- What Akka.Remote 1.5.62 supports (read from its embedded reference config and strings): + - `enable-ssl` + - `ssl.certificate.{path,password,flags}`, or `{use-thumbprint-over-file, thumbprint, store-name, + store-location}` + - `ssl.suppress-validation` + - `ssl.require-mutual-authentication`, **default true** + - `ssl.validate-certificate-hostname`, default false + - a programmatic `DotNettySslSetup` with a `CustomValidator`, plus helpers for chain, issuer, + subject, hostname and pinned-certificate validation + + At startup it refuses a certificate without an accessible private key. +- **The address scheme changes under TLS.** With SSL on, node addresses are `akka.ssl.tcp://…`. The + new test `HoconRemotingTlsTests.Enabled_document_boots_a_tls_transport_with_akka_ssl_tcp_addresses` + checks this by booting a real `ActorSystem` on the emitted document. Effects on code that parses + seeds: + - `StartupValidator.SeedNodeIsSelf` and `SeedNodePort/Host` use `@` and the last `:`, so they do + not depend on the scheme. They still work. + - `ClusterBootstrapGuard.SeedPattern` was `^akka(?:\.[a-z0-9]+)?://`, a single dot segment. + **Finding F3:** it did not match `akka.ssl.tcp`, so with TLS on the guard would have parsed no + seeds and silently never engaged. Fixed in this change. + - `ClusterBootstrapCoordinator` uses Akka's own `Address.Parse`, which handles the TLS scheme. + - Overview's `LeaderResolver.HostOf` (scadaproj) parses after `@`, so it does not depend on the + scheme either. + +### H5: Site to OPC UA and MxGateway + +- **OPC UA** uses OPC UA SecureChannel, not TLS. `OpcUaEndpointConfig.SecurityMode` defaults to + `None`. The values are `None`, `Sign` and `SignAndEncrypt`. `AutoAcceptUntrustedCerts` defaults to + false, and when on it logs a man-in-the-middle warning (`RealOpcUaClient`). The application PKI + stores come from `ScadaBridge:OpcUa:{TrustedIssuer,TrustedPeer,RejectedCertificate}StorePath`. + `OpcUaGlobalOptions` says empty paths "fall back to a default under Path.GetTempPath()". A temp + path is the wrong home for a production trust list, so the production profile should set these + explicitly. The rig simulators run `--unsecuretransport --autoaccept` (`deployment.md` section 3.3). +- **MxGateway.** Each connection carries `Endpoint` (default `http://localhost:5000`), `UseTls`, + `CaFile` and `ServerName` (`Commons/Types/DataConnections/MxGatewayEndpointConfig.cs`). These are + passed straight to `MxGatewayClientOptions` (`RealMxGatewayClient.ConnectAsync`). TLS already works + per connection, with a private-CA file. **No ScadaBridge code change is needed.** Enabling it + depends on the gateway serving TLS (its `…Http__Url=https://…`; `deployment.md` mxgw section). + +### H6 and H7: relay and health + +- `InboundApiOptions.StandbyForwardPeerEndpoints` accepts `http` or `https` and is used through a + named `HttpClient` (`InboundAPI/ServiceCollectionExtensions.cs`) with default validation. It flips + when the central surface flips. +- The site metrics and health listener (:8084) is HTTP/1.1 by design, because Prometheus and the + Overview dashboard poll it. It carries no credentials (`/health/*` is anonymous). Making it TLS is + optional, and section 11 lists it as an owner decision. + +## 3. Threat summary (why each hop matters) + +| Hop | Credential or data exposed in cleartext today | Integrity or authentication gap | +|---|---|---| +| H1 | session cookie (when `RequireHttpsCookie=false`), LDAP password at login, Inbound API `sbk_` keys | none at the transport layer | +| H2 | operator LDAP password (Basic) or bearer token | none at the transport layer | +| H3 | per-site PSK (bearer), deployments incl. encrypted secret artifacts, audit rows | the PSK authenticates the caller, but it is replayable by anyone on the path | +| H3b | LocalDb API key, all replicated site rows | as H3 | +| H4 | every cluster message | **no authentication at all**: a reachable host can join | +| H5 | OPC UA values; MxGateway API key + writes | per connection; secure modes exist but are off by default | + +The PSK work (#96 and its predecessors) limits who may *call* the control plane. It does nothing for +confidentiality, and a key captured on the wire is valid until it is rotated. H4 is the weakest +link: unauthenticated cluster membership. + +## 4. Recommended TLS mode per hop + +### 4.1 Kestrel: HTTPS with ALPN (H1, H2, H6, H7 and the gRPC listeners) + +- Add a node-level certificate option, `ScadaBridge:Node:Tls`, with the same shape as + `RemotingTls`: `CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName` + + `StoreLocation`. Apply it with `listenOptions.UseHttps(cert)` to every listener the node binds. + - **Central HTTP surface:** `Http1AndHttp2` on one TLS port. ALPN negotiates h2 for gRPC-capable + clients and HTTP/1.1 for browsers. This is the family pattern: HistorianGateway runs one + `Kestrel:Endpoints:Https` with `Http1AndHttp2`. + - **gRPC ports (central :8083, site :8083):** `Http2` over TLS. Clients negotiate h2 by ALPN, and + no h2c remains. + - **Site :8084:** optional (section 11). +- Close **F1**: `ParseHttpBindPorts` must return the scheme as well as the port, and an `https` entry + must bind TLS. With no certificate configured it must **refuse to boot**, so it never downgrades + silently. +- Keep TLS configuration in code, as the existing listeners are, not in `Kestrel:Endpoints`. The + explicit-endpoints comment in `Program.cs` already documents one surprise from mixing the two + sources. A config-section endpoint added on top of code-bound endpoints is a second surprise + waiting to happen. It is unverified whether Kestrel unions the two sources or replaces one with + the other. +- HSTS (`UseHsts`) only after an environment is confirmed HTTPS-only. Section 11 has the decision. + +### 4.2 Traefik (rig): TLS entrypoint, re-encrypted backend + +- Add a `websecure` entrypoint on :443 (host `9443`), with a file-provider `tls.certificates` entry + that holds the lab certificate. Redirect `web` to `websecure`. +- Backend: in the production profile, Traefik dials `https://scadabridge-central-{a,b}:5000`, with a + `serversTransport` whose `rootCAs` is the lab CA. TLS runs end to end and the same Kestrel TLS path + is exercised on the rig. Plaintext on the internal docker network is an acceptable *rig-only* + shortcut, but it would leave F1 and §4.1 untested on the rig. +- Keep `ForwardedHeaders` trusting Traefik. Set `RequireHttpsCookie` back to its default, `true`, on + every TLS-fronted node. +- Turn off `api.insecure` (the dashboard) or bind it to loopback in any non-rig use. +- gRPC stays off Traefik, as it is today. Nodes dial each other directly. + +### 4.3 gRPC: TLS with certificate validation; PSK retained + +- Endpoints become `https://host:8083`: the site rows `GrpcNodeAAddress`/`GrpcNodeBAddress`, + `CentralGrpcEndpoints`, and `LocalDb:Replication:PeerAddress`. +- Clients validate the server chain and hostname with `SocketsHttpHandler` defaults. No + `RemoteCertificateValidationCallback` override and no "accept any" switch is to be added. + **Hostnames in endpoints must be SANs on the server certificate.** +- **Keep the PSK.** TLS gives confidentiality and server authentication. The PSK stays the per-site + *client* identity, which central's interceptor keys per-site authorization on. With TLS the PSK + stops being replayable from the wire. `WithSiteCredentials` now composes it over + `ChannelCredentials.SecureSsl` for https (section 9). +- Client-certificate mTLS on gRPC is **not** recommended for the first step. It would duplicate the + PSK's job and add a client-certificate distribution path to every site. Section 11 lists it as an + owner option. + +### 4.4 Akka.Remote: DotNetty SSL, one certificate per node, mutual authentication + +- Set `ScadaBridge:Cluster:RemotingTls:Enabled = true` with this node's own certificate. + `RequireMutualAuthentication = true` (the default). Chain validation is on and cannot be turned off + through ScadaBridge configuration. The `suppress-validation` key is always emitted as `false`. +- All seeds use `akka.ssl.tcp://scadabridge@host:port`. `ClusterOptionsValidator` refuses a + TLS-enabled config that still has an `akka.tcp` seed. Without that check, the pair would retry + joins forever with no startup error. +- `ValidateCertificateHostname = false` at first, which is Akka's default; the trust anchor is a + dedicated issuing CA. Turn it on once every `NodeHostname`/seed host is confirmed as a SAN on the + peer's certificate. +- If the CA is a broad corporate CA, chain trust alone admits *any* host holding a certificate from + that CA. In that case add issuer or subject pinning through `DotNettySslSetup.CustomValidator`. That + is a code change in `GetOrCreateActorSystem`: `ActorSystem.Create` with a `BootstrapSetup`, which + section 9 does not include. Section 11 has the decision. +- Both nodes of a pair flip together. A TLS node and a plaintext node cannot talk to each other. This + matches the existing rule that a site pair stops and starts together (LocalDb Phase 2). + +### 4.5 Site to OPC UA and MxGateway + +- **OPC UA:** set `SecurityMode = SignAndEncrypt` per connection, wherever the server offers it + (OtOpcUa does). Set explicit `ScadaBridge:OpcUa:*StorePath` directories under the node's data + root, not the temp default. Trust the server certificates explicitly. `AutoAcceptUntrustedCerts` + stays false. +- **MxGateway:** use an `https://` endpoint with `UseTls = true`, `CaFile` (the issuing CA) and + `ServerName` matching the gateway certificate. This needs no code change, only the gateway serving + TLS. + +## 5. Certificates: issuance, storage, rotation + +### 5.1 One certificate per node + +- **Subject Alternative Names:** the node's `NodeHostname` exactly as it appears in seeds and gRPC + endpoints (the FQDN on wonder, e.g. `wonder-app-vq03.zmr.zimmer.com`), plus the short name. On the + rig, the container name (`scadabridge-site-a-a`). +- **Extended Key Usage:** `serverAuth` **and** `clientAuth`. Akka mTLS makes every node both a TLS + server and a TLS client. Kestrel and gRPC need `serverAuth`. +- One certificate per node serves both Kestrel and Akka on that node. On the co-located wonder boxes, + the central and site services on one box can share the box certificate. Both services run on the + same host name, so the SANs are identical. + +### 5.2 Issuance + +- **wonder:** this needs a CA that the boxes trust. Whether a corporate PKI exists (for example AD + Certificate Services with auto-enrolment) and is available to these boxes is **not knowable from + the code**. It is open decision D1. +- **Rig:** use a lab CA generated by a new script, `docker/tls/gen-lab-ca.sh` (openssl, idempotent, + output gitignored). It issues one certificate per container. Both the CA and the leaves are + clearly marked lab-only, like the rig's other DEV-ONLY credentials. +- **Trust on Linux containers:** Akka's chain validation and `SocketsHttpHandler` both use the OS + trust store. So the rig image must install the lab CA. Mount it into + `/usr/local/share/ca-certificates/` and run `update-ca-certificates` at container start. Do not + bake it into the image, because the image is shared with non-rig uses. + +### 5.3 Where certificates and keys live + +| Environment | Certificate + private key | Secret material | +|---|---|---| +| wonder (Windows) | **`LocalMachine\My`, referenced by thumbprint** (`CertificateThumbprint`; `StoreLocation = local-machine`). No key file on disk. The private key ACL grants read to the service account. | none: the store holds the key | +| wonder, fallback | `.pfx` outside the content root (same rule as `Secrets:SqlitePath`: absolute, not under `app\`, which an upgrade replaces) | password as `${secret:SB-TLS-PFX-PASSWORD}`, resolved from the Secrets store by the pre-host expander | +| rig (Linux) | `.pfx` on a read-only bind mount (`/certs/.pfx`) | DEV-ONLY password in compose env, like the mesh PSKs | + +Storing the PFX bytes themselves in the Secrets store is **not recommended**. Akka's HOCON path wants +a file, and Kestrel would need a materialised copy anyway. The Secrets store is the right home for +the **password**. + +Do not log the HOCON document. With a file certificate it contains the PFX password, which +`BuildRemotingTlsBlock` escapes but does not redact. `GetOrCreateActorSystem` logs only individual +fields today, and `akka.log-config-on-start` is off by default. Both must stay that way. + +### 5.4 Rotation + +- **Leaf rotation under the same CA:** the peer validates the chain, not a pinned leaf. A node can + therefore get a new leaf and restart on its own, without coordinating with its partner: + - Kestrel picks up the new certificate on restart. A `ServerCertificateSelector` could make that + hot later. + - Akka.Remote reads its certificate only at `ActorSystem` start, so it needs a restart. + - On a pair, restart one node at a time. A one-node restart is ordinary failover. A pair-together + stop is required only for the plaintext/TLS flip, not for a leaf rotation. +- **CA rotation:** first distribute a trust bundle with both the old and the new CA to every node. + Then rotate the leaves. Then remove the old CA. +- **Expiry monitoring:** add a startup log and a `/health/ready` degraded entry when the node + certificate expires in fewer than N days. That is follow-on work. Who owns renewal is decision D9. + +## 6. Impact on the docker rig and the wonder boxes + +### 6.1 Rig (`docker/`, `docker-env2/`) + +- New: `docker/tls/` (CA script, gitignored output), Traefik `websecure` + certificates, + `RemotingTls` blocks and `akka.ssl.tcp` seeds in each node's appsettings, `https` site addresses in + `seed-sites.sh`, `https` `CentralGrpcEndpoints` and LocalDb PeerAddress, and CA trust in the + containers. +- Host ports: add 9443 for Traefik TLS. Existing 9000 stays open (redirect) until the profile is the + default. +- Scripts that probe `http://localhost:900x`, the CLI recipes and the Playwright suites need a + base-URL switch. Their certificate trust must come from the lab CA, never from "ignore errors". +- Rig posture, mirroring the LocalDb rollout: **site-a on TLS, site-b and site-c on h2c**, so both + states are proven side by side. Central must dial both: per-site endpoints make that natural. + +### 6.2 wonder (DEV vd03, QA vq03/vq04) + +- No Traefik. Kestrel terminates TLS itself on the central port (:8085 today; whether to move to 443 + is decision D4). +- Certificates in `LocalMachine\My` (if PKI is available, D1/D2). Thumbprints go in + `appsettings.{Central,Site}.json`, which are preserved across binary swaps per the deployment + records. +- The site rows in the shared ConfigDb change to `https://wonder-app-vq0X…:8083` through the CLI + `site update`. That is a data change, reversible the same way. +- **Remote access caveat:** under the 2026-09-28 WinRM scope, certificate installation counts as a + deploy step and is allowed only as part of a deploy the user asked for. +- Overview: when the central surface goes TLS, its registry `BaseUrl`s become `https`. Its leader + matching does not depend on the scheme (section 2, H4). + +## 7. Staged rollout + +Each stage is independently reversible (section 8) and proven before the next one starts. + +| Stage | What | Where | Gate | +|---|---|---|---| +| 0 | **This change:** design + default-off groundwork (Akka TLS block, https-capable PSK channels, guard regex) | code only | unit + real-transport tests green; default HOCON byte-identical | +| 1 | Code: `ScadaBridge:Node:Tls` for Kestrel; close F1; the AuditLog pull clients take the endpoint (after #42 lands); optional cert-expiry health entry | code only, default off | unit + real-Kestrel TLS tests; default Kestrel binding unchanged | +| 2 | Rig lab CA + Traefik `websecure` (re-encrypting backend) + central Kestrel TLS | docker | browser + CLI over https; Secure cookie observed; plaintext :5000 refused once flipped | +| 3 | Rig site-a: gRPC https (both directions + LocalDb PeerAddress) and Akka TLS on the site-a pair | docker site-a only | section 10 rig checks; failover drill on site-a; site-b/c unchanged | +| 4 | Rig: central pair Akka TLS; then all sites | docker | full failover drill (`docker/failover-drill.sh`) | +| 5 | DEV vd03 (single node: Akka TLS is self-only; gRPC + Kestrel TLS) | wonder DEV | post-deploy https `/health/ready`, CLI, MES test machines | +| 6 | QA vq03/vq04 pair | wonder QA | pair stop/start together; failover drill in a window | + +Migration detail for the central gRPC port: during stage 3, central must answer both TLS sites and +h2c sites on site-to-central calls. A single Kestrel port is either TLS or not. Stage 1 therefore +adds an **optional second central gRPC listener** (for example `CentralGrpcTlsPort`). Sites move to +it one at a time, and the h2c port is removed once no site uses it. + +The central-to-site direction has no such problem, because each site address is its own row. + +## 8. Rollback + +Every switch is configuration, and every default is off: + +- **Akka:** set `RemotingTls:Enabled=false` and revert the seeds to `akka.tcp`. Stop and start the + pair **together**. +- **gRPC:** revert the site rows to `http://` (`site update`), revert `CentralGrpcEndpoints` and + `PeerAddress`, then restart. The PSK is untouched throughout, so auth never lapses during rollback. +- **Kestrel and Traefik:** revert `ASPNETCORE_URLS` and the node `Tls` block, and revert the Traefik + files. +- **Cookie:** `RequireHttpsCookie=true` must stay **true** while the surface is TLS. Rolling the + surface back to HTTP is the only case where setting it false again is justified. +- **HSTS:** browsers keep enforcing HTTPS for the max-age after rollback. This is why HSTS is enabled + last, with a short max-age first (decision D10). +- **No schema change** is involved at any stage. On wonder, the previous `appsettings*.json` is + already preserved in `app.bak.` by the standard deploy recipe. + +## 9. Groundwork shipped with this design (default OFF) + +1. **`ScadaBridge:Cluster:RemotingTls`** (`ClusterRemotingTlsOptions` in ClusterInfrastructure). + - Fields: `Enabled`, `CertificatePath`/`CertificatePassword` or `CertificateThumbprint`/ + `StoreName`/`StoreLocation`, `RequireMutualAuthentication` (default true), + `ValidateCertificateHostname` (default false). + - `AkkaHostedService.BuildRemotingTlsBlock` renders `enable-ssl` + `ssl { … }` inside + `dot-netty.tcp`, **only when `Enabled`**. + - `ClusterOptionsValidator` (TLS on only) requires: + - exactly one certificate source; + - a legal store location; + - `akka.ssl.tcp://` on every seed. + - **Byte-identity proof:** before the change, the rendered HOCON was captured for 12 + configurations: `auto-down` / `leader-auto-down` / `keep-oldest`, bootstrap guard on and off, + Central and Site. After the change all 12 were re-rendered and diffed: identical. A permanent + test (`Disabled_block_with_fields_is_byte_identical_to_default`) pins it: a disabled block with + every field set renders exactly the default document. +2. **https-capable PSK channels.** `ControlPlaneCredentials.WithSiteCredentials(..., endpoint)` + composes over `ChannelCredentials.SecureSsl` for an `https` endpoint. For anything else, including + callers that do not pass the endpoint, it keeps the previous insecure composition with the + explicit opt-in, unchanged. `SitePairChannelProvider`, `SiteStreamGrpcClient` and + `CentralChannelProvider` pass their endpoint. The three AuditLog pull clients are deliberately + untouched (F2). With today's `http` endpoints, behaviour is unchanged. +3. **`ClusterBootstrapGuard.SeedPattern`** accepts any number of scheme segments, so `akka.ssl.tcp` + parses (F3). +4. **Not changed:** `Program.cs`, `SiteKestrelSetup.cs`, Traefik and every appsettings and compose + file. Kestrel binding is identical by construction. + +Tests: +- `HoconRemotingTlsTests` (5): includes booting a real TLS `ActorSystem` from a generated + certificate. A mutation check (forcing `enable-ssl = false`) turns it red with `akka.tcp`. +- `ClusterRemotingTlsValidatorTests` (8). +- `ClusterBootstrapGuardTests` (+2 for `akka.ssl.tcp`). +- `ControlPlaneTlsEndToEndTests` (6). These use real Kestrel with `Http2` over TLS (ALPN), the real + `ControlPlaneAuthInterceptor`, and a pinned test certificate: + - the correct PSK is accepted over TLS; + - a wrong PSK is still `PermissionDenied` over TLS; + - an untrusted server certificate fails with an `AuthenticationException` cause; + - the pre-fix composition throws on https; + - the h2c composition is unchanged on http. +- `HttpsEndpointChannelTests` (9). These check that all three production builders accept `https`. A + mutation check (disabling the TLS branch) turns the three construction tests red. + +## 10. Test plan for stages 1–6 + +**Unit and in-process:** extend the stage-0 pattern to the Kestrel `Tls` block: +- certificate loading from a file and from the store (the store path is Windows-only, so skip it + elsewhere); +- F1: an `https` URL with no certificate must refuse to boot; +- ALPN: a real Kestrel `Http1AndHttp2` TLS listener serves an HTTP/1.1 GET and an h2 gRPC call on + one port. + +**Rig, per hop** (each with a negative control, so a check that "passes" is shown to be able to fail): + +- **H1:** `curl --cacert lab-ca.pem https://localhost:9443/health/ready` returns 200. Login over + https sets the cookie `Secure` (read the `Set-Cookie`). `http://localhost:9000` redirects. Traefik + to backend is re-encrypted: a plaintext probe to `central-a:5000` is refused once flipped. +- **H2:** CLI against the https URL with the lab CA trusted succeeds. With the CA *not* trusted, it + fails with a certificate error and never falls back to http. +- **H3:** + - `openssl s_client -alpn h2 -connect site-a-a:8083` shows TLS 1.2 or later and `h2`. + - An h2c client against the TLS port fails. + - Central to site commands, streams, audit pulls and site to central heartbeats all flow for + site-a over TLS while site-b stays h2c. + - The PSK is still enforced: a wrong key over TLS gives `PermissionDenied`. + - The #96 rotation drill passes over TLS. +- **H3b:** LocalDb `LocalDbReplicationConnected` is true on the site-a pair over https. + `MaxBatchBytes` behaviour is unchanged. +- **H4:** + - The pair forms with `akka.ssl.tcp` addresses (`/health/ready` shows memberCount 2). + - **Negative:** a third container with a certificate from a *different* CA cannot join. It gets a + handshake failure in its log and is never Up. + - **Negative:** a container with *no* certificate cannot join, which shows mTLS is enforced. + - Expired-certificate node: refused. + - Failover drill timings within the existing `StableAfter` budget. A TLS handshake on reconnect + must not push detection past it. +- **H5:** one OPC UA connection on `SignAndEncrypt` with an explicitly trusted server certificate. + One MxGateway connection with `UseTls` and `CaFile`. + +**Rotation drill:** replace one node's leaf certificate under the same CA, restart that node only, +and confirm the pair re-forms. Then do a CA roll with a two-CA trust bundle. + +**wonder (stages 5–6):** run the standard deploy verification, but over https: `/health/ready` on +both centrals and sites, CLI `site list`, the inbound API probe recipe and the MES test machines. +Then run a QA failover in an agreed window. + +## 11. Open decisions for the owner + +Each decision has a recommendation, marked **(rec)**. + +- **D1. PKI availability on the wonder network.** Is there a corporate CA, such as AD CS with + auto-enrolment, that will issue certificates to `wonder-app-*` with `serverAuth` + `clientAuth`? + The code cannot show this. + - **(rec)** If yes, use it. If no, use a ScadaBridge-dedicated private CA managed offline, with its + root installed into `LocalMachine\Root` on each box. +- **D2. Where the key lives on Windows.** + - **(rec)** `LocalMachine\My` by thumbprint. The private key ACL grants read to the service account. + - Alternative: a PFX outside the content root, with its password in the Secrets store. +- **D3. The ScadaBridge service account on the wonder boxes.** It decides the private key ACL. The + deployment records show Windows services but not the account. +- **D4. The production front door.** + - **(rec)** Kestrel terminates TLS directly, as wonder works today. + - Alternatives: Traefik, IIS/ARR, or an F5/hardware load balancer. + - Also decide whether to keep port :8085 or move to 443. +- **D5. Has the revisit trigger been met?** Are the current wonder networks wholly inside the plant + security boundary? If yes, this stays design-only and nothing is flipped. +- **D6. Client auth on gRPC.** + - **(rec)** PSK over TLS. + - Alternative: add client-certificate mTLS. +- **D7. Akka trust scope.** + - **(rec)** If the issuing CA is broad (corporate-wide), add issuer or subject pinning through + `DotNettySslSetup.CustomValidator`. That is a code change. + - If the CA is ScadaBridge-dedicated, chain trust is enough. + - Also decide when to turn on `ValidateCertificateHostname`. +- **D8. Site :8084 metrics and health over TLS.** Prometheus and Overview would need the CA. Today the + listener carries no credentials. + - **(rec)** Leave it HTTP for the first rollout. +- **D9. Certificate lifetime, renewal cadence, and who owns renewal and expiry alerts.** +- **D10. HSTS.** Once it is on, browsers refuse plain HTTP for the max-age. + - **(rec)** Enable it last, with a short max-age first. +- **D11. Rig Traefik backend.** + - **(rec)** Re-encrypt to Kestrel. + - Alternative: terminate TLS at Traefik with plaintext on the docker network. +- **D12. OPC UA and MxGateway.** Move OtOpcUa connections to `SignAndEncrypt`, and serve mxgw over + TLS. These depend on those services' own rollout. +- **D13. SQL Server and LDAP** (out of scope here). The rig uses `TrustServerCertificate=true` and + GLAuth with `Transport=None`. Production equivalents need their own decision. diff --git a/docs/requirements/Component-ClusterInfrastructure.md b/docs/requirements/Component-ClusterInfrastructure.md index 4c073e22..f299c710 100644 --- a/docs/requirements/Component-ClusterInfrastructure.md +++ b/docs/requirements/Component-ClusterInfrastructure.md @@ -244,6 +244,7 @@ Each node is configured with: - **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site). - **gRPC port** (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport. - **Local storage paths**: SQLite database locations (site nodes only). +- **Akka.Remote TLS** (opt-in, `ScadaBridge:Cluster:RemotingTls`, default **off**; ScadaBridge#64): when `Enabled`, `BuildHocon` renders `akka.remote.dot-netty.tcp.enable-ssl` with this node's own certificate (`CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName`/`StoreLocation`), mutual authentication on by default and chain validation always on (no `suppress-validation` switch). Node addresses become `akka.ssl.tcp://…`, so every seed must use that scheme — `ClusterOptionsValidator` refuses an `akka.tcp` seed with TLS on. Both nodes of a pair flip together. With TLS off the HOCON is byte-identical to before. Akka.Remote has no other authentication, so mTLS is the only way to stop an arbitrary reachable host joining the pair. Design and rollout: `docs/plans/2026-11-09-tls-production-profile-design.md`. ## Windows Service