docs(plans): TLS production profile design for Traefik, Kestrel, gRPC and Akka remoting (ScadaBridge#64)

Per-hop current state with code evidence (browser/Traefik/Kestrel, CLI,
central<->site gRPC + PSK, LocalDb sync, Akka remoting, OPC UA /
MxGateway), the recommended mode per hop (Kestrel HTTPS + ALPN, gRPC over
TLS with validation and the PSK retained, Akka DotNetty SSL with a cert
per node and mutual auth), certificate issuance/storage/rotation, rig and
wonder impact, a staged rollout, rollback, a test plan and 13 open owner
decisions. Findings: the central ASPNETCORE_URLS re-bind drops the
scheme (an https URL binds plaintext); Akka.NET has no secure-cookie, so
mTLS is the only remoting authentication. Component-ClusterInfrastructure
documents the new RemotingTls option.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/plans/2026-11-09-tls-production-profile-design.md

@@ -0,0 +1,536 @@
+# TLS production profile: Traefik, Kestrel, gRPC and Akka remoting (design, 2026-11-09)
+
+**Status: DESIGN, plus opt-in groundwork that is OFF by default (ScadaBridge#64, register row 128).**
+No deployed configuration changes. No environment is switched to TLS by this document or by the code
+that ships with it. The revisit trigger in the register is unchanged: the first deployment onto a
+network that is not wholly inside the plant security boundary.
+
+Groundwork in the same change (section 9): an opt-in Akka.Remote TLS block
+(`ScadaBridge:Cluster:RemotingTls`), a fix that lets the control-plane PSK credentials work on an
+`https://` gRPC endpoint, and a fix to the bootstrap guard's seed parser for `akka.ssl.tcp`. With
+the defaults, the Akka HOCON document is byte-identical to before (12 rendered documents compared
+by SHA). Kestrel configuration is not touched.
+
+Every "current state" claim below cites the code it was read from. Anything about the wonder
+environment that the code cannot show is listed as an **open owner decision** in section 11, not
+assumed.
+
+---
+
+## 1. Hops in scope
+
+| # | Hop | Transport today | Listener / client |
+|---|---|---|---|
+| H1 | Browser to Central UI (and Inbound API callers) | HTTP/1.1, plaintext | Traefik `web` :80 to Kestrel :5000 (rig); Kestrel :8085 directly (wonder) |
+| H2 | CLI to `/management` | HTTP, plaintext; Basic or bearer token | same Kestrel HTTP surface as H1 |
+| H3 | Central and site gRPC control plane | h2c (HTTP/2 without TLS) + per-site PSK | site :8083, central :8083 |
+| H3b | LocalDb pair sync (site node to site node) | h2c + bearer API key | shares the site :8083 listener |
+| H4 | Akka remoting within a pair | DotNetty TCP, plaintext, no authentication | central :8081, site :8082 |
+| H5 | Site to external OPC UA server | OPC UA SecureChannel; `None` by default | DCL `RealOpcUaClient` |
+| H5b | Site to MxGateway | gRPC; h2c by default, TLS when configured | DCL `RealMxGatewayClient` |
+| H6 | Standby central to active central (Inbound API relay) | HTTP, plaintext | `StandbyForwardPeerEndpoints` |
+| H7 | Prometheus / Overview to `/health/*`, `/metrics` | HTTP/1.1, plaintext | site :8084, central HTTP surface |
+
+H6 and H7 follow the same Kestrel decision as H1. SQL Server and LDAP connections are out of scope
+here. They are noted in section 10.
+
+## 2. Current state, with evidence
+
+### H1: Browser to Traefik to Central UI
+
+- **Rig Traefik is HTTP only.** `docker/traefik/traefik.yml` declares one entrypoint,
+  `web: address: ":80"`, and `api: insecure: true`. `docker/traefik/dynamic.yml` routes
+  `PathPrefix(/)` to `http://scadabridge-central-a:5000` and `http://scadabridge-central-b:5000`,
+  health-checked on `/health/active`. Host port `9000:80` (`docker/docker-compose.yml`).
+- **Traefik exists only on the docker rigs.** The wonder deployment records show centrals served
+  directly by Kestrel on `:8085`, with no load balancer in front
+  (`docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md`;
+  `scadaproj/deployment.md` section 7.1, port 8085 "ScadaBridge Central: CentralUI + Management +
+  Inbound API"). QA users address `vq03:8085` or `vq04:8085` by name.
+- **Kestrel central HTTP surface.** `Program.cs` (Central branch) reads `ASPNETCORE_URLS`, falling
+  back to `http://+:5000`, and re-binds each port with `options.ListenAnyIP(httpPort, ...)` and
+  `HttpProtocols.Http1AndHttp2`. It does this because the explicit gRPC listener switches Kestrel
+  into explicit-endpoints mode.
+- **Finding F1: the scheme in `ASPNETCORE_URLS` is silently dropped.** `Program.ParseHttpBindPorts`
+  keeps only the port (its own remarks: "only the port matters"), and the re-bind never calls
+  `UseHttps`. So `ASPNETCORE_URLS=https://+:443` today binds **plaintext** HTTP on 443, with no
+  startup error. Any TLS profile must close this. Either honour the scheme (bind TLS with a configured
+  certificate) or refuse to boot on an `https` URL that has no certificate.
+- **Session cookie.** `SecurityOptions.RequireHttpsCookie` defaults to `true`, which sets the
+  cookie's `Secure` flag (`src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs`). Both rig centrals
+  set it to `false` (`docker/central-node-*/appsettings.Central.json`). When it is false the code logs
+  a startup warning and uses `SameAsRequest` (`Security/ServiceCollectionExtensions.cs`). The value on
+  the wonder boxes is not recorded in the repo; see section 11.
+- **Forwarded headers.** `ForwardedHeadersSetup` honours `X-Forwarded-For | X-Forwarded-Proto` from
+  configured proxies only, and is disabled by default. The rig enables it for `172.16.0.0/12`. This
+  matters for TLS that terminates at a proxy: with `X-Forwarded-Proto: https` trusted, the app sees
+  the request as HTTPS, so the session cookie is `Secure` even under `SameAsRequest`.
+
+### H2: CLI to management
+
+- `ManagementHttpClient` takes any absolute `http` or `https` URL
+  (`CLI/Commands/CommandHelpers.cs`: "Expected an absolute http/https URL"). It exchanges
+  credentials for a bearer token at `POST /auth/token`, falling back to HTTP Basic
+  (`CLI/ManagementHttpClient.cs` header remarks). Over `http` both credential forms travel in
+  cleartext.
+- It uses a plain `SocketsHttpHandler { ConnectTimeout, AllowAutoRedirect = false }`, with no
+  certificate-validation override anywhere in the CLI. So an `https` URL already gets OS-trust-store
+  validation. **No CLI code change is needed for TLS.** Operators change the URL, and the box must
+  trust the issuing CA.
+
+### H3: Central and site gRPC control plane
+
+- **Site listener.** `SiteKestrelSetup.Configure` binds `ListenAnyIP(grpcPort)` with
+  `HttpProtocols.Http2` only (h2c), plus `ListenAnyIP(metricsPort)` with `Http1AndHttp2`. Neither
+  calls `UseHttps`.
+- **Central listener.** `Program.cs` binds `ListenAnyIP(centralGrpcPort)` with `Http2` only (h2c).
+  The comment states: "gRPC does NOT go through Traefik (HTTP/1 only); sites reach this port by
+  container name".
+- **Endpoint validation already allows https.** `GrpcEndpointRules` accepts "an absolute http:// or
+  https:// URL" (`Commons/Validation/GrpcEndpointRules.cs`). The site addresses are DB rows
+  (`docker/seed-sites.sh` sets `--grpc-node-a-address "http://scadabridge-site-a-a:8083"`). The
+  site-to-central list is `ScadaBridge:Communication:CentralGrpcEndpoints` in each
+  `appsettings.Site.json`.
+- **Client builders.** These build channels:
+  - `SitePairChannelProvider.BuildChannel`: central to site commands.
+  - `SiteStreamGrpcClient`: central to site streams.
+  - `CentralChannelProvider`: site to central control.
+  - `GrpcPull{AuditEvents,SiteCalls,TraceEvents}Client`: central to site audit pulls, in AuditLog.
+
+  All of them build a `SocketsHttpHandler` with keepalive, then call `.WithMessageLimits()` and
+  `.WithSiteCredentials(...)`.
+- **PSK.** `ControlPlaneCredentials.WithSiteCredentials` attaches `Authorization: Bearer <psk>` and
+  `x-scadabridge-site` as call credentials (`Communication/Grpc/ISitePskProvider.cs`). On the server,
+  `ControlPlaneAuthInterceptor` (site) and `CentralControlAuthInterceptor` (central) are fail-closed.
+  Central resolves one key per site from `ScadaBridge:Communication:SitePsks` or the secret
+  `SB-GRPC-PSK-<siteId>` (`Host/SitePskProvider.cs`). Gitea #96 (`3a31dd2d`) added in-place rotation:
+  `SitePskRotationInvalidator`, plus `PskRejectionRefreshHandler`, which drops the cached key on a
+  `PermissionDenied`/`Unauthenticated` response.
+- **Finding F2: the PSK composition could not run over TLS. Fixed for 4 of 7 builders in this
+  change.** The credentials were always composed over `ChannelCredentials.Insecure`, with
+  `UnsafeUseInsecureChannelCallCredentials = true`. Grpc.Net.Client 2.76 refuses that on an `https`
+  address, with the error "Channel is configured with insecure channel credentials and can't use a
+  HttpClient with a 'https' scheme". So every https endpoint threw at channel construction. In
+  `SitePairChannelProvider` that throw is caught and the address logged as malformed, so a TLS site
+  would have been **silently undialable**. The class comment claimed TLS "requires no change to this
+  code"; it was wrong. The new test
+  `ControlPlaneTlsEndToEndTests.Https_channel_with_insecure_composition_throws_at_construction` pins
+  the failure. Section 9 describes the fix. **Still open:** the three AuditLog pull clients. Each
+  needs a one-argument change (`WithSiteCredentials(_pskProvider, siteId, endpoint)`). They were left
+  untouched on purpose because AuditLog is being changed concurrently for #42. Until they get the
+  argument, audit pulls from an `https` site fail at channel build.
+
+### H3b: LocalDb pair sync
+
+- Sync rides the site's :8083 listener. The dialling half's `LocalDb:Replication:PeerAddress` is
+  `http://scadabridge-site-a-b:8083` (`docker/site-a-node-a/appsettings.Site.json`).
+- The LocalDb library (`ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.DialPeer`) builds
+  `GrpcChannel.ForAddress(PeerAddress, …)` with **no channel credentials**, and sends its key as an
+  `Authorization` metadata header. An `https` PeerAddress therefore works unchanged, using
+  OS-trust-store validation. But **a site's :8083 cannot be TLS for central and h2c for its peer**:
+  the pair's PeerAddress must flip in the same step as the listener.
+
+### H4: Akka remoting within a pair
+
+- `AkkaHostedService.BuildHocon` emits `akka.remote.dot-netty.tcp { hostname, port }` and a transport
+  failure detector. It emits **no SSL keys**, so Akka's reference default `enable-ssl = false`
+  applies. Ports are central 8081 and site 8082; the rig publishes them to the host (`9011:8081`,
+  `9021:8082`, and so on).
+- **No authentication.** Akka.Remote 1.5.62 (`Directory.Packages.props`) has no shared-secret or
+  "secure cookie" mechanism; the strings `secure-cookie` and `require-cookie` do not exist in
+  `Akka.Remote.dll`. `untrusted-mode` exists but is off, and it only filters system messages and
+  actor selections. Anyone who can reach :8081/:8082 can join the cluster or message its actors.
+  **Mutual TLS is the only authentication Akka.Remote offers.** The issue text says "Akka
+  `enable-ssl` + secure cookie"; the cookie half does not exist in Akka.NET.
+- What Akka.Remote 1.5.62 supports (read from its embedded reference config and strings):
+  - `enable-ssl`
+  - `ssl.certificate.{path,password,flags}`, or `{use-thumbprint-over-file, thumbprint, store-name,
+    store-location}`
+  - `ssl.suppress-validation`
+  - `ssl.require-mutual-authentication`, **default true**
+  - `ssl.validate-certificate-hostname`, default false
+  - a programmatic `DotNettySslSetup` with a `CustomValidator`, plus helpers for chain, issuer,
+    subject, hostname and pinned-certificate validation
+
+  At startup it refuses a certificate without an accessible private key.
+- **The address scheme changes under TLS.** With SSL on, node addresses are `akka.ssl.tcp://…`. The
+  new test `HoconRemotingTlsTests.Enabled_document_boots_a_tls_transport_with_akka_ssl_tcp_addresses`
+  checks this by booting a real `ActorSystem` on the emitted document. Effects on code that parses
+  seeds:
+  - `StartupValidator.SeedNodeIsSelf` and `SeedNodePort/Host` use `@` and the last `:`, so they do
+    not depend on the scheme. They still work.
+  - `ClusterBootstrapGuard.SeedPattern` was `^akka(?:\.[a-z0-9]+)?://`, a single dot segment.
+    **Finding F3:** it did not match `akka.ssl.tcp`, so with TLS on the guard would have parsed no
+    seeds and silently never engaged. Fixed in this change.
+  - `ClusterBootstrapCoordinator` uses Akka's own `Address.Parse`, which handles the TLS scheme.
+  - Overview's `LeaderResolver.HostOf` (scadaproj) parses after `@`, so it does not depend on the
+    scheme either.
+
+### H5: Site to OPC UA and MxGateway
+
+- **OPC UA** uses OPC UA SecureChannel, not TLS. `OpcUaEndpointConfig.SecurityMode` defaults to
+  `None`. The values are `None`, `Sign` and `SignAndEncrypt`. `AutoAcceptUntrustedCerts` defaults to
+  false, and when on it logs a man-in-the-middle warning (`RealOpcUaClient`). The application PKI
+  stores come from `ScadaBridge:OpcUa:{TrustedIssuer,TrustedPeer,RejectedCertificate}StorePath`.
+  `OpcUaGlobalOptions` says empty paths "fall back to a default under Path.GetTempPath()". A temp
+  path is the wrong home for a production trust list, so the production profile should set these
+  explicitly. The rig simulators run `--unsecuretransport --autoaccept` (`deployment.md` section 3.3).
+- **MxGateway.** Each connection carries `Endpoint` (default `http://localhost:5000`), `UseTls`,
+  `CaFile` and `ServerName` (`Commons/Types/DataConnections/MxGatewayEndpointConfig.cs`). These are
+  passed straight to `MxGatewayClientOptions` (`RealMxGatewayClient.ConnectAsync`). TLS already works
+  per connection, with a private-CA file. **No ScadaBridge code change is needed.** Enabling it
+  depends on the gateway serving TLS (its `…Http__Url=https://…`; `deployment.md` mxgw section).
+
+### H6 and H7: relay and health
+
+- `InboundApiOptions.StandbyForwardPeerEndpoints` accepts `http` or `https` and is used through a
+  named `HttpClient` (`InboundAPI/ServiceCollectionExtensions.cs`) with default validation. It flips
+  when the central surface flips.
+- The site metrics and health listener (:8084) is HTTP/1.1 by design, because Prometheus and the
+  Overview dashboard poll it. It carries no credentials (`/health/*` is anonymous). Making it TLS is
+  optional, and section 11 lists it as an owner decision.
+
+## 3. Threat summary (why each hop matters)
+
+| Hop | Credential or data exposed in cleartext today | Integrity or authentication gap |
+|---|---|---|
+| H1 | session cookie (when `RequireHttpsCookie=false`), LDAP password at login, Inbound API `sbk_` keys | none at the transport layer |
+| H2 | operator LDAP password (Basic) or bearer token | none at the transport layer |
+| H3 | per-site PSK (bearer), deployments incl. encrypted secret artifacts, audit rows | the PSK authenticates the caller, but it is replayable by anyone on the path |
+| H3b | LocalDb API key, all replicated site rows | as H3 |
+| H4 | every cluster message | **no authentication at all**: a reachable host can join |
+| H5 | OPC UA values; MxGateway API key + writes | per connection; secure modes exist but are off by default |
+
+The PSK work (#96 and its predecessors) limits who may *call* the control plane. It does nothing for
+confidentiality, and a key captured on the wire is valid until it is rotated. H4 is the weakest
+link: unauthenticated cluster membership.
+
+## 4. Recommended TLS mode per hop
+
+### 4.1 Kestrel: HTTPS with ALPN (H1, H2, H6, H7 and the gRPC listeners)
+
+- Add a node-level certificate option, `ScadaBridge:Node:Tls`, with the same shape as
+  `RemotingTls`: `CertificatePath` + `CertificatePassword`, or `CertificateThumbprint` + `StoreName`
+  + `StoreLocation`. Apply it with `listenOptions.UseHttps(cert)` to every listener the node binds.
+  - **Central HTTP surface:** `Http1AndHttp2` on one TLS port. ALPN negotiates h2 for gRPC-capable
+    clients and HTTP/1.1 for browsers. This is the family pattern: HistorianGateway runs one
+    `Kestrel:Endpoints:Https` with `Http1AndHttp2`.
+  - **gRPC ports (central :8083, site :8083):** `Http2` over TLS. Clients negotiate h2 by ALPN, and
+    no h2c remains.
+  - **Site :8084:** optional (section 11).
+- Close **F1**: `ParseHttpBindPorts` must return the scheme as well as the port, and an `https` entry
+  must bind TLS. With no certificate configured it must **refuse to boot**, so it never downgrades
+  silently.
+- Keep TLS configuration in code, as the existing listeners are, not in `Kestrel:Endpoints`. The
+  explicit-endpoints comment in `Program.cs` already documents one surprise from mixing the two
+  sources. A config-section endpoint added on top of code-bound endpoints is a second surprise
+  waiting to happen. It is unverified whether Kestrel unions the two sources or replaces one with
+  the other.
+- HSTS (`UseHsts`) only after an environment is confirmed HTTPS-only. Section 11 has the decision.
+
+### 4.2 Traefik (rig): TLS entrypoint, re-encrypted backend
+
+- Add a `websecure` entrypoint on :443 (host `9443`), with a file-provider `tls.certificates` entry
+  that holds the lab certificate. Redirect `web` to `websecure`.
+- Backend: in the production profile, Traefik dials `https://scadabridge-central-{a,b}:5000`, with a
+  `serversTransport` whose `rootCAs` is the lab CA. TLS runs end to end and the same Kestrel TLS path
+  is exercised on the rig. Plaintext on the internal docker network is an acceptable *rig-only*
+  shortcut, but it would leave F1 and §4.1 untested on the rig.
+- Keep `ForwardedHeaders` trusting Traefik. Set `RequireHttpsCookie` back to its default, `true`, on
+  every TLS-fronted node.
+- Turn off `api.insecure` (the dashboard) or bind it to loopback in any non-rig use.
+- gRPC stays off Traefik, as it is today. Nodes dial each other directly.
+
+### 4.3 gRPC: TLS with certificate validation; PSK retained
+
+- Endpoints become `https://host:8083`: the site rows `GrpcNodeAAddress`/`GrpcNodeBAddress`,
+  `CentralGrpcEndpoints`, and `LocalDb:Replication:PeerAddress`.
+- Clients validate the server chain and hostname with `SocketsHttpHandler` defaults. No
+  `RemoteCertificateValidationCallback` override and no "accept any" switch is to be added.
+  **Hostnames in endpoints must be SANs on the server certificate.**
+- **Keep the PSK.** TLS gives confidentiality and server authentication. The PSK stays the per-site
+  *client* identity, which central's interceptor keys per-site authorization on. With TLS the PSK
+  stops being replayable from the wire. `WithSiteCredentials` now composes it over
+  `ChannelCredentials.SecureSsl` for https (section 9).
+- Client-certificate mTLS on gRPC is **not** recommended for the first step. It would duplicate the
+  PSK's job and add a client-certificate distribution path to every site. Section 11 lists it as an
+  owner option.
+
+### 4.4 Akka.Remote: DotNetty SSL, one certificate per node, mutual authentication
+
+- Set `ScadaBridge:Cluster:RemotingTls:Enabled = true` with this node's own certificate.
+  `RequireMutualAuthentication = true` (the default). Chain validation is on and cannot be turned off
+  through ScadaBridge configuration. The `suppress-validation` key is always emitted as `false`.
+- All seeds use `akka.ssl.tcp://scadabridge@host:port`. `ClusterOptionsValidator` refuses a
+  TLS-enabled config that still has an `akka.tcp` seed. Without that check, the pair would retry
+  joins forever with no startup error.
+- `ValidateCertificateHostname = false` at first, which is Akka's default; the trust anchor is a
+  dedicated issuing CA. Turn it on once every `NodeHostname`/seed host is confirmed as a SAN on the
+  peer's certificate.
+- If the CA is a broad corporate CA, chain trust alone admits *any* host holding a certificate from
+  that CA. In that case add issuer or subject pinning through `DotNettySslSetup.CustomValidator`. That
+  is a code change in `GetOrCreateActorSystem`: `ActorSystem.Create` with a `BootstrapSetup`, which
+  section 9 does not include. Section 11 has the decision.
+- Both nodes of a pair flip together. A TLS node and a plaintext node cannot talk to each other. This
+  matches the existing rule that a site pair stops and starts together (LocalDb Phase 2).
+
+### 4.5 Site to OPC UA and MxGateway
+
+- **OPC UA:** set `SecurityMode = SignAndEncrypt` per connection, wherever the server offers it
+  (OtOpcUa does). Set explicit `ScadaBridge:OpcUa:*StorePath` directories under the node's data
+  root, not the temp default. Trust the server certificates explicitly. `AutoAcceptUntrustedCerts`
+  stays false.
+- **MxGateway:** use an `https://` endpoint with `UseTls = true`, `CaFile` (the issuing CA) and
+  `ServerName` matching the gateway certificate. This needs no code change, only the gateway serving
+  TLS.
+
+## 5. Certificates: issuance, storage, rotation
+
+### 5.1 One certificate per node
+
+- **Subject Alternative Names:** the node's `NodeHostname` exactly as it appears in seeds and gRPC
+  endpoints (the FQDN on wonder, e.g. `wonder-app-vq03.zmr.zimmer.com`), plus the short name. On the
+  rig, the container name (`scadabridge-site-a-a`).
+- **Extended Key Usage:** `serverAuth` **and** `clientAuth`. Akka mTLS makes every node both a TLS
+  server and a TLS client. Kestrel and gRPC need `serverAuth`.
+- One certificate per node serves both Kestrel and Akka on that node. On the co-located wonder boxes,
+  the central and site services on one box can share the box certificate. Both services run on the
+  same host name, so the SANs are identical.
+
+### 5.2 Issuance
+
+- **wonder:** this needs a CA that the boxes trust. Whether a corporate PKI exists (for example AD
+  Certificate Services with auto-enrolment) and is available to these boxes is **not knowable from
+  the code**. It is open decision D1.
+- **Rig:** use a lab CA generated by a new script, `docker/tls/gen-lab-ca.sh` (openssl, idempotent,
+  output gitignored). It issues one certificate per container. Both the CA and the leaves are
+  clearly marked lab-only, like the rig's other DEV-ONLY credentials.
+- **Trust on Linux containers:** Akka's chain validation and `SocketsHttpHandler` both use the OS
+  trust store. So the rig image must install the lab CA. Mount it into
+  `/usr/local/share/ca-certificates/` and run `update-ca-certificates` at container start. Do not
+  bake it into the image, because the image is shared with non-rig uses.
+
+### 5.3 Where certificates and keys live
+
+| Environment | Certificate + private key | Secret material |
+|---|---|---|
+| wonder (Windows) | **`LocalMachine\My`, referenced by thumbprint** (`CertificateThumbprint`; `StoreLocation = local-machine`). No key file on disk. The private key ACL grants read to the service account. | none: the store holds the key |
+| wonder, fallback | `.pfx` outside the content root (same rule as `Secrets:SqlitePath`: absolute, not under `app\`, which an upgrade replaces) | password as `${secret:SB-TLS-PFX-PASSWORD}`, resolved from the Secrets store by the pre-host expander |
+| rig (Linux) | `.pfx` on a read-only bind mount (`/certs/<node>.pfx`) | DEV-ONLY password in compose env, like the mesh PSKs |
+
+Storing the PFX bytes themselves in the Secrets store is **not recommended**. Akka's HOCON path wants
+a file, and Kestrel would need a materialised copy anyway. The Secrets store is the right home for
+the **password**.
+
+Do not log the HOCON document. With a file certificate it contains the PFX password, which
+`BuildRemotingTlsBlock` escapes but does not redact. `GetOrCreateActorSystem` logs only individual
+fields today, and `akka.log-config-on-start` is off by default. Both must stay that way.
+
+### 5.4 Rotation
+
+- **Leaf rotation under the same CA:** the peer validates the chain, not a pinned leaf. A node can
+  therefore get a new leaf and restart on its own, without coordinating with its partner:
+  - Kestrel picks up the new certificate on restart. A `ServerCertificateSelector` could make that
+    hot later.
+  - Akka.Remote reads its certificate only at `ActorSystem` start, so it needs a restart.
+  - On a pair, restart one node at a time. A one-node restart is ordinary failover. A pair-together
+    stop is required only for the plaintext/TLS flip, not for a leaf rotation.
+- **CA rotation:** first distribute a trust bundle with both the old and the new CA to every node.
+  Then rotate the leaves. Then remove the old CA.
+- **Expiry monitoring:** add a startup log and a `/health/ready` degraded entry when the node
+  certificate expires in fewer than N days. That is follow-on work. Who owns renewal is decision D9.
+
+## 6. Impact on the docker rig and the wonder boxes
+
+### 6.1 Rig (`docker/`, `docker-env2/`)
+
+- New: `docker/tls/` (CA script, gitignored output), Traefik `websecure` + certificates,
+  `RemotingTls` blocks and `akka.ssl.tcp` seeds in each node's appsettings, `https` site addresses in
+  `seed-sites.sh`, `https` `CentralGrpcEndpoints` and LocalDb PeerAddress, and CA trust in the
+  containers.
+- Host ports: add 9443 for Traefik TLS. Existing 9000 stays open (redirect) until the profile is the
+  default.
+- Scripts that probe `http://localhost:900x`, the CLI recipes and the Playwright suites need a
+  base-URL switch. Their certificate trust must come from the lab CA, never from "ignore errors".
+- Rig posture, mirroring the LocalDb rollout: **site-a on TLS, site-b and site-c on h2c**, so both
+  states are proven side by side. Central must dial both: per-site endpoints make that natural.
+
+### 6.2 wonder (DEV vd03, QA vq03/vq04)
+
+- No Traefik. Kestrel terminates TLS itself on the central port (:8085 today; whether to move to 443
+  is decision D4).
+- Certificates in `LocalMachine\My` (if PKI is available, D1/D2). Thumbprints go in
+  `appsettings.{Central,Site}.json`, which are preserved across binary swaps per the deployment
+  records.
+- The site rows in the shared ConfigDb change to `https://wonder-app-vq0X…:8083` through the CLI
+  `site update`. That is a data change, reversible the same way.
+- **Remote access caveat:** under the 2026-09-28 WinRM scope, certificate installation counts as a
+  deploy step and is allowed only as part of a deploy the user asked for.
+- Overview: when the central surface goes TLS, its registry `BaseUrl`s become `https`. Its leader
+  matching does not depend on the scheme (section 2, H4).
+
+## 7. Staged rollout
+
+Each stage is independently reversible (section 8) and proven before the next one starts.
+
+| Stage | What | Where | Gate |
+|---|---|---|---|
+| 0 | **This change:** design + default-off groundwork (Akka TLS block, https-capable PSK channels, guard regex) | code only | unit + real-transport tests green; default HOCON byte-identical |
+| 1 | Code: `ScadaBridge:Node:Tls` for Kestrel; close F1; the AuditLog pull clients take the endpoint (after #42 lands); optional cert-expiry health entry | code only, default off | unit + real-Kestrel TLS tests; default Kestrel binding unchanged |
+| 2 | Rig lab CA + Traefik `websecure` (re-encrypting backend) + central Kestrel TLS | docker | browser + CLI over https; Secure cookie observed; plaintext :5000 refused once flipped |
+| 3 | Rig site-a: gRPC https (both directions + LocalDb PeerAddress) and Akka TLS on the site-a pair | docker site-a only | section 10 rig checks; failover drill on site-a; site-b/c unchanged |
+| 4 | Rig: central pair Akka TLS; then all sites | docker | full failover drill (`docker/failover-drill.sh`) |
+| 5 | DEV vd03 (single node: Akka TLS is self-only; gRPC + Kestrel TLS) | wonder DEV | post-deploy https `/health/ready`, CLI, MES test machines |
+| 6 | QA vq03/vq04 pair | wonder QA | pair stop/start together; failover drill in a window |
+
+Migration detail for the central gRPC port: during stage 3, central must answer both TLS sites and
+h2c sites on site-to-central calls. A single Kestrel port is either TLS or not. Stage 1 therefore
+adds an **optional second central gRPC listener** (for example `CentralGrpcTlsPort`). Sites move to
+it one at a time, and the h2c port is removed once no site uses it.
+
+The central-to-site direction has no such problem, because each site address is its own row.
+
+## 8. Rollback
+
+Every switch is configuration, and every default is off:
+
+- **Akka:** set `RemotingTls:Enabled=false` and revert the seeds to `akka.tcp`. Stop and start the
+  pair **together**.
+- **gRPC:** revert the site rows to `http://` (`site update`), revert `CentralGrpcEndpoints` and
+  `PeerAddress`, then restart. The PSK is untouched throughout, so auth never lapses during rollback.
+- **Kestrel and Traefik:** revert `ASPNETCORE_URLS` and the node `Tls` block, and revert the Traefik
+  files.
+- **Cookie:** `RequireHttpsCookie=true` must stay **true** while the surface is TLS. Rolling the
+  surface back to HTTP is the only case where setting it false again is justified.
+- **HSTS:** browsers keep enforcing HTTPS for the max-age after rollback. This is why HSTS is enabled
+  last, with a short max-age first (decision D10).
+- **No schema change** is involved at any stage. On wonder, the previous `appsettings*.json` is
+  already preserved in `app.bak.<ts>` by the standard deploy recipe.
+
+## 9. Groundwork shipped with this design (default OFF)
+
+1. **`ScadaBridge:Cluster:RemotingTls`** (`ClusterRemotingTlsOptions` in ClusterInfrastructure).
+   - Fields: `Enabled`, `CertificatePath`/`CertificatePassword` or `CertificateThumbprint`/
+     `StoreName`/`StoreLocation`, `RequireMutualAuthentication` (default true),
+     `ValidateCertificateHostname` (default false).
+   - `AkkaHostedService.BuildRemotingTlsBlock` renders `enable-ssl` + `ssl { … }` inside
+     `dot-netty.tcp`, **only when `Enabled`**.
+   - `ClusterOptionsValidator` (TLS on only) requires:
+     - exactly one certificate source;
+     - a legal store location;
+     - `akka.ssl.tcp://` on every seed.
+   - **Byte-identity proof:** before the change, the rendered HOCON was captured for 12
+     configurations: `auto-down` / `leader-auto-down` / `keep-oldest`, bootstrap guard on and off,
+     Central and Site. After the change all 12 were re-rendered and diffed: identical. A permanent
+     test (`Disabled_block_with_fields_is_byte_identical_to_default`) pins it: a disabled block with
+     every field set renders exactly the default document.
+2. **https-capable PSK channels.** `ControlPlaneCredentials.WithSiteCredentials(..., endpoint)`
+   composes over `ChannelCredentials.SecureSsl` for an `https` endpoint. For anything else, including
+   callers that do not pass the endpoint, it keeps the previous insecure composition with the
+   explicit opt-in, unchanged. `SitePairChannelProvider`, `SiteStreamGrpcClient` and
+   `CentralChannelProvider` pass their endpoint. The three AuditLog pull clients are deliberately
+   untouched (F2). With today's `http` endpoints, behaviour is unchanged.
+3. **`ClusterBootstrapGuard.SeedPattern`** accepts any number of scheme segments, so `akka.ssl.tcp`
+   parses (F3).
+4. **Not changed:** `Program.cs`, `SiteKestrelSetup.cs`, Traefik and every appsettings and compose
+   file. Kestrel binding is identical by construction.
+
+Tests:
+- `HoconRemotingTlsTests` (5): includes booting a real TLS `ActorSystem` from a generated
+  certificate. A mutation check (forcing `enable-ssl = false`) turns it red with `akka.tcp`.
+- `ClusterRemotingTlsValidatorTests` (8).
+- `ClusterBootstrapGuardTests` (+2 for `akka.ssl.tcp`).
+- `ControlPlaneTlsEndToEndTests` (6). These use real Kestrel with `Http2` over TLS (ALPN), the real
+  `ControlPlaneAuthInterceptor`, and a pinned test certificate:
+  - the correct PSK is accepted over TLS;
+  - a wrong PSK is still `PermissionDenied` over TLS;
+  - an untrusted server certificate fails with an `AuthenticationException` cause;
+  - the pre-fix composition throws on https;
+  - the h2c composition is unchanged on http.
+- `HttpsEndpointChannelTests` (9). These check that all three production builders accept `https`. A
+  mutation check (disabling the TLS branch) turns the three construction tests red.
+
+## 10. Test plan for stages 1–6
+
+**Unit and in-process:** extend the stage-0 pattern to the Kestrel `Tls` block:
+- certificate loading from a file and from the store (the store path is Windows-only, so skip it
+  elsewhere);
+- F1: an `https` URL with no certificate must refuse to boot;
+- ALPN: a real Kestrel `Http1AndHttp2` TLS listener serves an HTTP/1.1 GET and an h2 gRPC call on
+  one port.
+
+**Rig, per hop** (each with a negative control, so a check that "passes" is shown to be able to fail):
+
+- **H1:** `curl --cacert lab-ca.pem https://localhost:9443/health/ready` returns 200. Login over
+  https sets the cookie `Secure` (read the `Set-Cookie`). `http://localhost:9000` redirects. Traefik
+  to backend is re-encrypted: a plaintext probe to `central-a:5000` is refused once flipped.
+- **H2:** CLI against the https URL with the lab CA trusted succeeds. With the CA *not* trusted, it
+  fails with a certificate error and never falls back to http.
+- **H3:**
+  - `openssl s_client -alpn h2 -connect site-a-a:8083` shows TLS 1.2 or later and `h2`.
+  - An h2c client against the TLS port fails.
+  - Central to site commands, streams, audit pulls and site to central heartbeats all flow for
+    site-a over TLS while site-b stays h2c.
+  - The PSK is still enforced: a wrong key over TLS gives `PermissionDenied`.
+  - The #96 rotation drill passes over TLS.
+- **H3b:** LocalDb `LocalDbReplicationConnected` is true on the site-a pair over https.
+  `MaxBatchBytes` behaviour is unchanged.
+- **H4:**
+  - The pair forms with `akka.ssl.tcp` addresses (`/health/ready` shows memberCount 2).
+  - **Negative:** a third container with a certificate from a *different* CA cannot join. It gets a
+    handshake failure in its log and is never Up.
+  - **Negative:** a container with *no* certificate cannot join, which shows mTLS is enforced.
+  - Expired-certificate node: refused.
+  - Failover drill timings within the existing `StableAfter` budget. A TLS handshake on reconnect
+    must not push detection past it.
+- **H5:** one OPC UA connection on `SignAndEncrypt` with an explicitly trusted server certificate.
+  One MxGateway connection with `UseTls` and `CaFile`.
+
+**Rotation drill:** replace one node's leaf certificate under the same CA, restart that node only,
+and confirm the pair re-forms. Then do a CA roll with a two-CA trust bundle.
+
+**wonder (stages 5–6):** run the standard deploy verification, but over https: `/health/ready` on
+both centrals and sites, CLI `site list`, the inbound API probe recipe and the MES test machines.
+Then run a QA failover in an agreed window.
+
+## 11. Open decisions for the owner
+
+Each decision has a recommendation, marked **(rec)**.
+
+- **D1. PKI availability on the wonder network.** Is there a corporate CA, such as AD CS with
+  auto-enrolment, that will issue certificates to `wonder-app-*` with `serverAuth` + `clientAuth`?
+  The code cannot show this.
+  - **(rec)** If yes, use it. If no, use a ScadaBridge-dedicated private CA managed offline, with its
+    root installed into `LocalMachine\Root` on each box.
+- **D2. Where the key lives on Windows.**
+  - **(rec)** `LocalMachine\My` by thumbprint. The private key ACL grants read to the service account.
+  - Alternative: a PFX outside the content root, with its password in the Secrets store.
+- **D3. The ScadaBridge service account on the wonder boxes.** It decides the private key ACL. The
+  deployment records show Windows services but not the account.
+- **D4. The production front door.**
+  - **(rec)** Kestrel terminates TLS directly, as wonder works today.
+  - Alternatives: Traefik, IIS/ARR, or an F5/hardware load balancer.
+  - Also decide whether to keep port :8085 or move to 443.
+- **D5. Has the revisit trigger been met?** Are the current wonder networks wholly inside the plant
+  security boundary? If yes, this stays design-only and nothing is flipped.
+- **D6. Client auth on gRPC.**
+  - **(rec)** PSK over TLS.
+  - Alternative: add client-certificate mTLS.
+- **D7. Akka trust scope.**
+  - **(rec)** If the issuing CA is broad (corporate-wide), add issuer or subject pinning through
+    `DotNettySslSetup.CustomValidator`. That is a code change.
+  - If the CA is ScadaBridge-dedicated, chain trust is enough.
+  - Also decide when to turn on `ValidateCertificateHostname`.
+- **D8. Site :8084 metrics and health over TLS.** Prometheus and Overview would need the CA. Today the
+  listener carries no credentials.
+  - **(rec)** Leave it HTTP for the first rollout.
+- **D9. Certificate lifetime, renewal cadence, and who owns renewal and expiry alerts.**
+- **D10. HSTS.** Once it is on, browsers refuse plain HTTP for the max-age.
+  - **(rec)** Enable it last, with a short max-age first.
+- **D11. Rig Traefik backend.**
+  - **(rec)** Re-encrypt to Kestrel.
+  - Alternative: terminate TLS at Traefik with plaintext on the docker network.
+- **D12. OPC UA and MxGateway.** Move OtOpcUa connections to `SignAndEncrypt`, and serve mxgw over
+  TLS. These depend on those services' own rollout.
+- **D13. SQL Server and LDAP** (out of scope here). The rig uses `TrustServerCertificate=true` and
+  GLAuth with `Transport=None`. Production equivalents need their own decision.