ScadaBridge/docs/plans/2026-11-09-tls-production-profile-design.md

TLS production profile: Traefik, Kestrel, gRPC and Akka remoting (design, 2026-11-09)

Status: DESIGN, plus opt-in groundwork that is OFF by default (ScadaBridge#64, register row 128). No deployed configuration changes. No environment is switched to TLS by this document or by the code that ships with it. The revisit trigger in the register is unchanged: the first deployment onto a network that is not wholly inside the plant security boundary.

Groundwork in the same change (section 9): an opt-in Akka.Remote TLS block (ScadaBridge:Cluster:RemotingTls), a fix that lets the control-plane PSK credentials work on an https:// gRPC endpoint, and a fix to the bootstrap guard's seed parser for akka.ssl.tcp. With the defaults, the Akka HOCON document is byte-identical to before (12 rendered documents compared by SHA). Kestrel configuration is not touched.

Every "current state" claim below cites the code it was read from. Anything about the wonder environment that the code cannot show is listed as an open owner decision in section 11, not assumed.

---

1. Hops in scope

#	Hop	Transport today	Listener / client
H1	Browser to Central UI (and Inbound API callers)	HTTP/1.1, plaintext	Traefik web :80 to Kestrel :5000 (rig); Kestrel :8085 directly (wonder)
H2	CLI to /management	HTTP, plaintext; Basic or bearer token	same Kestrel HTTP surface as H1
H3	Central and site gRPC control plane	h2c (HTTP/2 without TLS) + per-site PSK	site :8083, central :8083
H3b	LocalDb pair sync (site node to site node)	h2c + bearer API key	shares the site :8083 listener
H4	Akka remoting within a pair	DotNetty TCP, plaintext, no authentication	central :8081, site :8082
H5	Site to external OPC UA server	OPC UA SecureChannel; None by default	DCL RealOpcUaClient
H5b	Site to MxGateway	gRPC; h2c by default, TLS when configured	DCL RealMxGatewayClient
H6	Standby central to active central (Inbound API relay)	HTTP, plaintext	StandbyForwardPeerEndpoints
H7	Prometheus / Overview to /health/*, /metrics	HTTP/1.1, plaintext	site :8084, central HTTP surface

H6 and H7 follow the same Kestrel decision as H1. SQL Server and LDAP connections are out of scope here. They are noted in section 10.

2. Current state, with evidence

H1: Browser to Traefik to Central UI

• Rig Traefik is HTTP only. docker/traefik/traefik.yml declares one entrypoint, web: address: ":80", and api: insecure: true. docker/traefik/dynamic.yml routes PathPrefix(/) to http://scadabridge-central-a:5000 and http://scadabridge-central-b:5000, health-checked on /health/active. Host port 9000:80 (docker/docker-compose.yml).
• Traefik exists only on the docker rigs. The wonder deployment records show centrals served directly by Kestrel on :8085, with no load balancer in front (docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md; scadaproj/deployment.md section 7.1, port 8085 "ScadaBridge Central: CentralUI + Management + Inbound API"). QA users address vq03:8085 or vq04:8085 by name.
• Kestrel central HTTP surface. Program.cs (Central branch) reads ASPNETCORE_URLS, falling back to http://+:5000, and re-binds each port with options.ListenAnyIP(httpPort, ...) and HttpProtocols.Http1AndHttp2. It does this because the explicit gRPC listener switches Kestrel into explicit-endpoints mode.
• Finding F1: the scheme in ASPNETCORE_URLS is silently dropped. Program.ParseHttpBindPorts keeps only the port (its own remarks: "only the port matters"), and the re-bind never calls UseHttps. So ASPNETCORE_URLS=https://+:443 today binds plaintext HTTP on 443, with no startup error. Any TLS profile must close this. Either honour the scheme (bind TLS with a configured certificate) or refuse to boot on an https URL that has no certificate.
• Session cookie. SecurityOptions.RequireHttpsCookie defaults to true, which sets the cookie's Secure flag (src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs). Both rig centrals set it to false (docker/central-node-*/appsettings.Central.json). When it is false the code logs a startup warning and uses SameAsRequest (Security/ServiceCollectionExtensions.cs). The value on the wonder boxes is not recorded in the repo; see section 11.
• Forwarded headers. ForwardedHeadersSetup honours X-Forwarded-For | X-Forwarded-Proto from configured proxies only, and is disabled by default. The rig enables it for 172.16.0.0/12. This matters for TLS that terminates at a proxy: with X-Forwarded-Proto: https trusted, the app sees the request as HTTPS, so the session cookie is Secure even under SameAsRequest.

H2: CLI to management

• ManagementHttpClient takes any absolute http or https URL (CLI/Commands/CommandHelpers.cs: "Expected an absolute http/https URL"). It exchanges credentials for a bearer token at POST /auth/token, falling back to HTTP Basic (CLI/ManagementHttpClient.cs header remarks). Over http both credential forms travel in cleartext.
• It uses a plain SocketsHttpHandler { ConnectTimeout, AllowAutoRedirect = false }, with no certificate-validation override anywhere in the CLI. So an https URL already gets OS-trust-store validation. No CLI code change is needed for TLS. Operators change the URL, and the box must trust the issuing CA.

H3: Central and site gRPC control plane

• Site listener. SiteKestrelSetup.Configure binds ListenAnyIP(grpcPort) with HttpProtocols.Http2 only (h2c), plus ListenAnyIP(metricsPort) with Http1AndHttp2. Neither calls UseHttps.

• Central listener. Program.cs binds ListenAnyIP(centralGrpcPort) with Http2 only (h2c). The comment states: "gRPC does NOT go through Traefik (HTTP/1 only); sites reach this port by container name".

• Endpoint validation already allows https. GrpcEndpointRules accepts "an absolute http:// or https:// URL" (Commons/Validation/GrpcEndpointRules.cs). The site addresses are DB rows (docker/seed-sites.sh sets --grpc-node-a-address "http://scadabridge-site-a-a:8083"). The site-to-central list is ScadaBridge:Communication:CentralGrpcEndpoints in each appsettings.Site.json.

• Client builders. These build channels:

  • SitePairChannelProvider.BuildChannel: central to site commands.
  • SiteStreamGrpcClient: central to site streams.
  • CentralChannelProvider: site to central control.
  • GrpcPull{AuditEvents,SiteCalls,TraceEvents}Client: central to site audit pulls, in AuditLog.

  All of them build a SocketsHttpHandler with keepalive, then call .WithMessageLimits() and .WithSiteCredentials(...).

• PSK. ControlPlaneCredentials.WithSiteCredentials attaches Authorization: Bearer <psk> and x-scadabridge-site as call credentials (Communication/Grpc/ISitePskProvider.cs). On the server, ControlPlaneAuthInterceptor (site) and CentralControlAuthInterceptor (central) are fail-closed. Central resolves one key per site from ScadaBridge:Communication:SitePsks or the secret SB-GRPC-PSK-<siteId> (Host/SitePskProvider.cs). Gitea #96 (3a31dd2d) added in-place rotation: SitePskRotationInvalidator, plus PskRejectionRefreshHandler, which drops the cached key on a PermissionDenied/Unauthenticated response.

• Finding F2: the PSK composition could not run over TLS. Fixed for 4 of 7 builders in this change. The credentials were always composed over ChannelCredentials.Insecure, with UnsafeUseInsecureChannelCallCredentials = true. Grpc.Net.Client 2.76 refuses that on an https address, with the error "Channel is configured with insecure channel credentials and can't use a HttpClient with a 'https' scheme". So every https endpoint threw at channel construction. In SitePairChannelProvider that throw is caught and the address logged as malformed, so a TLS site would have been silently undialable. The class comment claimed TLS "requires no change to this code"; it was wrong. The new test ControlPlaneTlsEndToEndTests.Https_channel_with_insecure_composition_throws_at_construction pins the failure. Section 9 describes the fix. Still open: the three AuditLog pull clients. Each needs a one-argument change (WithSiteCredentials(_pskProvider, siteId, endpoint)). They were left untouched on purpose because AuditLog is being changed concurrently for #42. Until they get the argument, audit pulls from an https site fail at channel build.

H3b: LocalDb pair sync

• Sync rides the site's :8083 listener. The dialling half's LocalDb:Replication:PeerAddress is http://scadabridge-site-a-b:8083 (docker/site-a-node-a/appsettings.Site.json).
• The LocalDb library (ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.DialPeer) builds GrpcChannel.ForAddress(PeerAddress, …) with no channel credentials, and sends its key as an Authorization metadata header. An https PeerAddress therefore works unchanged, using OS-trust-store validation. But a site's :8083 cannot be TLS for central and h2c for its peer: the pair's PeerAddress must flip in the same step as the listener.

H4: Akka remoting within a pair

• AkkaHostedService.BuildHocon emits akka.remote.dot-netty.tcp { hostname, port } and a transport failure detector. It emits no SSL keys, so Akka's reference default enable-ssl = false applies. Ports are central 8081 and site 8082; the rig publishes them to the host (9011:8081, 9021:8082, and so on).

• No authentication. Akka.Remote 1.5.62 (Directory.Packages.props) has no shared-secret or "secure cookie" mechanism; the strings secure-cookie and require-cookie do not exist in Akka.Remote.dll. untrusted-mode exists but is off, and it only filters system messages and actor selections. Anyone who can reach :8081/:8082 can join the cluster or message its actors. Mutual TLS is the only authentication Akka.Remote offers. The issue text says "Akka enable-ssl + secure cookie"; the cookie half does not exist in Akka.NET.

• What Akka.Remote 1.5.62 supports (read from its embedded reference config and strings):

  • enable-ssl
  • ssl.certificate.{path,password,flags}, or {use-thumbprint-over-file, thumbprint, store-name, store-location}
  • ssl.suppress-validation
  • ssl.require-mutual-authentication, default true
  • ssl.validate-certificate-hostname, default false
  • a programmatic DotNettySslSetup with a CustomValidator, plus helpers for chain, issuer, subject, hostname and pinned-certificate validation

  At startup it refuses a certificate without an accessible private key.

• The address scheme changes under TLS. With SSL on, node addresses are akka.ssl.tcp://…. The new test HoconRemotingTlsTests.Enabled_document_boots_a_tls_transport_with_akka_ssl_tcp_addresses checks this by booting a real ActorSystem on the emitted document. Effects on code that parses seeds:

  • StartupValidator.SeedNodeIsSelf and SeedNodePort/Host use @ and the last :, so they do not depend on the scheme. They still work.
  • ClusterBootstrapGuard.SeedPattern was ^akka(?:\.[a-z0-9]+)?://, a single dot segment. Finding F3: it did not match akka.ssl.tcp, so with TLS on the guard would have parsed no seeds and silently never engaged. Fixed in this change.
  • ClusterBootstrapCoordinator uses Akka's own Address.Parse, which handles the TLS scheme.
  • Overview's LeaderResolver.HostOf (scadaproj) parses after @, so it does not depend on the scheme either.

H5: Site to OPC UA and MxGateway

• OPC UA uses OPC UA SecureChannel, not TLS. OpcUaEndpointConfig.SecurityMode defaults to None. The values are None, Sign and SignAndEncrypt. AutoAcceptUntrustedCerts defaults to false, and when on it logs a man-in-the-middle warning (RealOpcUaClient). The application PKI stores come from ScadaBridge:OpcUa:{TrustedIssuer,TrustedPeer,RejectedCertificate}StorePath. OpcUaGlobalOptions says empty paths "fall back to a default under Path.GetTempPath()". A temp path is the wrong home for a production trust list, so the production profile should set these explicitly. The rig simulators run --unsecuretransport --autoaccept (deployment.md section 3.3).
• MxGateway. Each connection carries Endpoint (default http://localhost:5000), UseTls, CaFile and ServerName (Commons/Types/DataConnections/MxGatewayEndpointConfig.cs). These are passed straight to MxGatewayClientOptions (RealMxGatewayClient.ConnectAsync). TLS already works per connection, with a private-CA file. No ScadaBridge code change is needed. Enabling it depends on the gateway serving TLS (its …Http__Url=https://…; deployment.md mxgw section).

H6 and H7: relay and health

• InboundApiOptions.StandbyForwardPeerEndpoints accepts http or https and is used through a named HttpClient (InboundAPI/ServiceCollectionExtensions.cs) with default validation. It flips when the central surface flips.
• The site metrics and health listener (:8084) is HTTP/1.1 by design, because Prometheus and the Overview dashboard poll it. It carries no credentials (/health/* is anonymous). Making it TLS is optional, and section 11 lists it as an owner decision.

3. Threat summary (why each hop matters)

Hop	Credential or data exposed in cleartext today	Integrity or authentication gap
H1	session cookie (when RequireHttpsCookie=false), LDAP password at login, Inbound API sbk_ keys	none at the transport layer
H2	operator LDAP password (Basic) or bearer token	none at the transport layer
H3	per-site PSK (bearer), deployments incl. encrypted secret artifacts, audit rows	the PSK authenticates the caller, but it is replayable by anyone on the path
H3b	LocalDb API key, all replicated site rows	as H3
H4	every cluster message	no authentication at all: a reachable host can join
H5	OPC UA values; MxGateway API key + writes	per connection; secure modes exist but are off by default

The PSK work (#96 and its predecessors) limits who may call the control plane. It does nothing for confidentiality, and a key captured on the wire is valid until it is rotated. H4 is the weakest link: unauthenticated cluster membership.

4. Recommended TLS mode per hop

4.1 Kestrel: HTTPS with ALPN (H1, H2, H6, H7 and the gRPC listeners)

• Add a node-level certificate option, ScadaBridge:Node:Tls, with the same shape as RemotingTls: CertificatePath + CertificatePassword, or CertificateThumbprint + StoreName
  • StoreLocation. Apply it with listenOptions.UseHttps(cert) to every listener the node binds.
  • Central HTTP surface: Http1AndHttp2 on one TLS port. ALPN negotiates h2 for gRPC-capable clients and HTTP/1.1 for browsers. This is the family pattern: HistorianGateway runs one Kestrel:Endpoints:Https with Http1AndHttp2.
  • gRPC ports (central :8083, site :8083): Http2 over TLS. Clients negotiate h2 by ALPN, and no h2c remains.
  • Site :8084: optional (section 11).
• Close F1: ParseHttpBindPorts must return the scheme as well as the port, and an https entry must bind TLS. With no certificate configured it must refuse to boot, so it never downgrades silently.
• Keep TLS configuration in code, as the existing listeners are, not in Kestrel:Endpoints. The explicit-endpoints comment in Program.cs already documents one surprise from mixing the two sources. A config-section endpoint added on top of code-bound endpoints is a second surprise waiting to happen. It is unverified whether Kestrel unions the two sources or replaces one with the other.
• HSTS (UseHsts) only after an environment is confirmed HTTPS-only. Section 11 has the decision.

4.2 Traefik (rig): TLS entrypoint, re-encrypted backend

• Add a websecure entrypoint on :443 (host 9443), with a file-provider tls.certificates entry that holds the lab certificate. Redirect web to websecure.
• Backend: in the production profile, Traefik dials https://scadabridge-central-{a,b}:5000, with a serversTransport whose rootCAs is the lab CA. TLS runs end to end and the same Kestrel TLS path is exercised on the rig. Plaintext on the internal docker network is an acceptable rig-only shortcut, but it would leave F1 and §4.1 untested on the rig.
• Keep ForwardedHeaders trusting Traefik. Set RequireHttpsCookie back to its default, true, on every TLS-fronted node.
• Turn off api.insecure (the dashboard) or bind it to loopback in any non-rig use.
• gRPC stays off Traefik, as it is today. Nodes dial each other directly.

4.3 gRPC: TLS with certificate validation; PSK retained

• Endpoints become https://host:8083: the site rows GrpcNodeAAddress/GrpcNodeBAddress, CentralGrpcEndpoints, and LocalDb:Replication:PeerAddress.
• Clients validate the server chain and hostname with SocketsHttpHandler defaults. No RemoteCertificateValidationCallback override and no "accept any" switch is to be added. Hostnames in endpoints must be SANs on the server certificate.
• Keep the PSK. TLS gives confidentiality and server authentication. The PSK stays the per-site client identity, which central's interceptor keys per-site authorization on. With TLS the PSK stops being replayable from the wire. WithSiteCredentials now composes it over ChannelCredentials.SecureSsl for https (section 9).
• Client-certificate mTLS on gRPC is not recommended for the first step. It would duplicate the PSK's job and add a client-certificate distribution path to every site. Section 11 lists it as an owner option.

4.4 Akka.Remote: DotNetty SSL, one certificate per node, mutual authentication

• Set ScadaBridge:Cluster:RemotingTls:Enabled = true with this node's own certificate. RequireMutualAuthentication = true (the default). Chain validation is on and cannot be turned off through ScadaBridge configuration. The suppress-validation key is always emitted as false.
• All seeds use akka.ssl.tcp://scadabridge@host:port. ClusterOptionsValidator refuses a TLS-enabled config that still has an akka.tcp seed. Without that check, the pair would retry joins forever with no startup error.
• ValidateCertificateHostname = false at first, which is Akka's default; the trust anchor is a dedicated issuing CA. Turn it on once every NodeHostname/seed host is confirmed as a SAN on the peer's certificate.
• If the CA is a broad corporate CA, chain trust alone admits any host holding a certificate from that CA. In that case add issuer or subject pinning through DotNettySslSetup.CustomValidator. That is a code change in GetOrCreateActorSystem: ActorSystem.Create with a BootstrapSetup, which section 9 does not include. Section 11 has the decision.
• Both nodes of a pair flip together. A TLS node and a plaintext node cannot talk to each other. This matches the existing rule that a site pair stops and starts together (LocalDb Phase 2).

4.5 Site to OPC UA and MxGateway

• OPC UA: set SecurityMode = SignAndEncrypt per connection, wherever the server offers it (OtOpcUa does). Set explicit ScadaBridge:OpcUa:*StorePath directories under the node's data root, not the temp default. Trust the server certificates explicitly. AutoAcceptUntrustedCerts stays false.
• MxGateway: use an https:// endpoint with UseTls = true, CaFile (the issuing CA) and ServerName matching the gateway certificate. This needs no code change, only the gateway serving TLS.

5. Certificates: issuance, storage, rotation

5.1 One certificate per node

• Subject Alternative Names: the node's NodeHostname exactly as it appears in seeds and gRPC endpoints (the FQDN on wonder, e.g. wonder-app-vq03.zmr.zimmer.com), plus the short name. On the rig, the container name (scadabridge-site-a-a).
• Extended Key Usage: serverAuth and clientAuth. Akka mTLS makes every node both a TLS server and a TLS client. Kestrel and gRPC need serverAuth.
• One certificate per node serves both Kestrel and Akka on that node. On the co-located wonder boxes, the central and site services on one box can share the box certificate. Both services run on the same host name, so the SANs are identical.

5.2 Issuance

• wonder: this needs a CA that the boxes trust. Whether a corporate PKI exists (for example AD Certificate Services with auto-enrolment) and is available to these boxes is not knowable from the code. It is open decision D1.
• Rig: use a lab CA generated by a new script, docker/tls/gen-lab-ca.sh (openssl, idempotent, output gitignored). It issues one certificate per container. Both the CA and the leaves are clearly marked lab-only, like the rig's other DEV-ONLY credentials.
• Trust on Linux containers: Akka's chain validation and SocketsHttpHandler both use the OS trust store. So the rig image must install the lab CA. Mount it into /usr/local/share/ca-certificates/ and run update-ca-certificates at container start. Do not bake it into the image, because the image is shared with non-rig uses.

5.3 Where certificates and keys live

Environment	Certificate + private key	Secret material
wonder (Windows)	LocalMachine\My, referenced by thumbprint (CertificateThumbprint; StoreLocation = local-machine). No key file on disk. The private key ACL grants read to the service account.	none: the store holds the key
wonder, fallback	.pfx outside the content root (same rule as Secrets:SqlitePath: absolute, not under app\, which an upgrade replaces)	password as ${secret:SB-TLS-PFX-PASSWORD}, resolved from the Secrets store by the pre-host expander
rig (Linux)	.pfx on a read-only bind mount (/certs/<node>.pfx)	DEV-ONLY password in compose env, like the mesh PSKs

Storing the PFX bytes themselves in the Secrets store is not recommended. Akka's HOCON path wants a file, and Kestrel would need a materialised copy anyway. The Secrets store is the right home for the password.

Do not log the HOCON document. With a file certificate it contains the PFX password, which BuildRemotingTlsBlock escapes but does not redact. GetOrCreateActorSystem logs only individual fields today, and akka.log-config-on-start is off by default. Both must stay that way.

5.4 Rotation

• Leaf rotation under the same CA: the peer validates the chain, not a pinned leaf. A node can therefore get a new leaf and restart on its own, without coordinating with its partner:
  • Kestrel picks up the new certificate on restart. A ServerCertificateSelector could make that hot later.
  • Akka.Remote reads its certificate only at ActorSystem start, so it needs a restart.
  • On a pair, restart one node at a time. A one-node restart is ordinary failover. A pair-together stop is required only for the plaintext/TLS flip, not for a leaf rotation.
• CA rotation: first distribute a trust bundle with both the old and the new CA to every node. Then rotate the leaves. Then remove the old CA.
• Expiry monitoring: add a startup log and a /health/ready degraded entry when the node certificate expires in fewer than N days. That is follow-on work. Who owns renewal is decision D9.

6. Impact on the docker rig and the wonder boxes

6.1 Rig (docker/, docker-env2/)

• New: docker/tls/ (CA script, gitignored output), Traefik websecure + certificates, RemotingTls blocks and akka.ssl.tcp seeds in each node's appsettings, https site addresses in seed-sites.sh, https CentralGrpcEndpoints and LocalDb PeerAddress, and CA trust in the containers.
• Host ports: add 9443 for Traefik TLS. Existing 9000 stays open (redirect) until the profile is the default.
• Scripts that probe http://localhost:900x, the CLI recipes and the Playwright suites need a base-URL switch. Their certificate trust must come from the lab CA, never from "ignore errors".
• Rig posture, mirroring the LocalDb rollout: site-a on TLS, site-b and site-c on h2c, so both states are proven side by side. Central must dial both: per-site endpoints make that natural.

6.2 wonder (DEV vd03, QA vq03/vq04)

• No Traefik. Kestrel terminates TLS itself on the central port (:8085 today; whether to move to 443 is decision D4).
• Certificates in LocalMachine\My (if PKI is available, D1/D2). Thumbprints go in appsettings.{Central,Site}.json, which are preserved across binary swaps per the deployment records.
• The site rows in the shared ConfigDb change to https://wonder-app-vq0X…:8083 through the CLI site update. That is a data change, reversible the same way.
• Remote access caveat: under the 2026-09-28 WinRM scope, certificate installation counts as a deploy step and is allowed only as part of a deploy the user asked for.
• Overview: when the central surface goes TLS, its registry BaseUrls become https. Its leader matching does not depend on the scheme (section 2, H4).

7. Staged rollout

Each stage is independently reversible (section 8) and proven before the next one starts.

Stage	What	Where	Gate
0	This change: design + default-off groundwork (Akka TLS block, https-capable PSK channels, guard regex)	code only	unit + real-transport tests green; default HOCON byte-identical
1	Code: ScadaBridge:Node:Tls for Kestrel; close F1; the AuditLog pull clients take the endpoint (after #42 lands); optional cert-expiry health entry	code only, default off	unit + real-Kestrel TLS tests; default Kestrel binding unchanged
2	Rig lab CA + Traefik websecure (re-encrypting backend) + central Kestrel TLS	docker	browser + CLI over https; Secure cookie observed; plaintext :5000 refused once flipped
3	Rig site-a: gRPC https (both directions + LocalDb PeerAddress) and Akka TLS on the site-a pair	docker site-a only	section 10 rig checks; failover drill on site-a; site-b/c unchanged
4	Rig: central pair Akka TLS; then all sites	docker	full failover drill (docker/failover-drill.sh)
5	DEV vd03 (single node: Akka TLS is self-only; gRPC + Kestrel TLS)	wonder DEV	post-deploy https /health/ready, CLI, MES test machines
6	QA vq03/vq04 pair	wonder QA	pair stop/start together; failover drill in a window

Migration detail for the central gRPC port: during stage 3, central must answer both TLS sites and h2c sites on site-to-central calls. A single Kestrel port is either TLS or not. Stage 1 therefore adds an optional second central gRPC listener (for example CentralGrpcTlsPort). Sites move to it one at a time, and the h2c port is removed once no site uses it.

The central-to-site direction has no such problem, because each site address is its own row.

8. Rollback

Every switch is configuration, and every default is off:

• Akka: set RemotingTls:Enabled=false and revert the seeds to akka.tcp. Stop and start the pair together.
• gRPC: revert the site rows to http:// (site update), revert CentralGrpcEndpoints and PeerAddress, then restart. The PSK is untouched throughout, so auth never lapses during rollback.
• Kestrel and Traefik: revert ASPNETCORE_URLS and the node Tls block, and revert the Traefik files.
• Cookie: RequireHttpsCookie=true must stay true while the surface is TLS. Rolling the surface back to HTTP is the only case where setting it false again is justified.
• HSTS: browsers keep enforcing HTTPS for the max-age after rollback. This is why HSTS is enabled last, with a short max-age first (decision D10).
• No schema change is involved at any stage. On wonder, the previous appsettings*.json is already preserved in app.bak.<ts> by the standard deploy recipe.

9. Groundwork shipped with this design (default OFF)

1. ScadaBridge:Cluster:RemotingTls (ClusterRemotingTlsOptions in ClusterInfrastructure).
   • Fields: Enabled, CertificatePath/CertificatePassword or CertificateThumbprint/ StoreName/StoreLocation, RequireMutualAuthentication (default true), ValidateCertificateHostname (default false).
   • AkkaHostedService.BuildRemotingTlsBlock renders enable-ssl + ssl { … } inside dot-netty.tcp, only when Enabled.
   • ClusterOptionsValidator (TLS on only) requires:
     • exactly one certificate source;
     • a legal store location;
     • akka.ssl.tcp:// on every seed.
   • Byte-identity proof: before the change, the rendered HOCON was captured for 12 configurations: auto-down / leader-auto-down / keep-oldest, bootstrap guard on and off, Central and Site. After the change all 12 were re-rendered and diffed: identical. A permanent test (Disabled_block_with_fields_is_byte_identical_to_default) pins it: a disabled block with every field set renders exactly the default document.
2. https-capable PSK channels. ControlPlaneCredentials.WithSiteCredentials(..., endpoint) composes over ChannelCredentials.SecureSsl for an https endpoint. For anything else, including callers that do not pass the endpoint, it keeps the previous insecure composition with the explicit opt-in, unchanged. SitePairChannelProvider, SiteStreamGrpcClient and CentralChannelProvider pass their endpoint. The three AuditLog pull clients are deliberately untouched (F2). With today's http endpoints, behaviour is unchanged.
3. ClusterBootstrapGuard.SeedPattern accepts any number of scheme segments, so akka.ssl.tcp parses (F3).
4. Not changed: Program.cs, SiteKestrelSetup.cs, Traefik and every appsettings and compose file. Kestrel binding is identical by construction.

Tests:

• HoconRemotingTlsTests (5): includes booting a real TLS ActorSystem from a generated certificate. A mutation check (forcing enable-ssl = false) turns it red with akka.tcp.
• ClusterRemotingTlsValidatorTests (8).
• ClusterBootstrapGuardTests (+2 for akka.ssl.tcp).
• ControlPlaneTlsEndToEndTests (6). These use real Kestrel with Http2 over TLS (ALPN), the real ControlPlaneAuthInterceptor, and a pinned test certificate:
  • the correct PSK is accepted over TLS;
  • a wrong PSK is still PermissionDenied over TLS;
  • an untrusted server certificate fails with an AuthenticationException cause;
  • the pre-fix composition throws on https;
  • the h2c composition is unchanged on http.
• HttpsEndpointChannelTests (9). These check that all three production builders accept https. A mutation check (disabling the TLS branch) turns the three construction tests red.

10. Test plan for stages 1–6

Unit and in-process: extend the stage-0 pattern to the Kestrel Tls block:

• certificate loading from a file and from the store (the store path is Windows-only, so skip it elsewhere);
• F1: an https URL with no certificate must refuse to boot;
• ALPN: a real Kestrel Http1AndHttp2 TLS listener serves an HTTP/1.1 GET and an h2 gRPC call on one port.

Rig, per hop (each with a negative control, so a check that "passes" is shown to be able to fail):

• H1: curl --cacert lab-ca.pem https://localhost:9443/health/ready returns 200. Login over https sets the cookie Secure (read the Set-Cookie). http://localhost:9000 redirects. Traefik to backend is re-encrypted: a plaintext probe to central-a:5000 is refused once flipped.
• H2: CLI against the https URL with the lab CA trusted succeeds. With the CA not trusted, it fails with a certificate error and never falls back to http.
• H3:
  • openssl s_client -alpn h2 -connect site-a-a:8083 shows TLS 1.2 or later and h2.
  • An h2c client against the TLS port fails.
  • Central to site commands, streams, audit pulls and site to central heartbeats all flow for site-a over TLS while site-b stays h2c.
  • The PSK is still enforced: a wrong key over TLS gives PermissionDenied.
  • The #96 rotation drill passes over TLS.
• H3b: LocalDb LocalDbReplicationConnected is true on the site-a pair over https. MaxBatchBytes behaviour is unchanged.
• H4:
  • The pair forms with akka.ssl.tcp addresses (/health/ready shows memberCount 2).
  • Negative: a third container with a certificate from a different CA cannot join. It gets a handshake failure in its log and is never Up.
  • Negative: a container with no certificate cannot join, which shows mTLS is enforced.
  • Expired-certificate node: refused.
  • Failover drill timings within the existing StableAfter budget. A TLS handshake on reconnect must not push detection past it.
• H5: one OPC UA connection on SignAndEncrypt with an explicitly trusted server certificate. One MxGateway connection with UseTls and CaFile.

Rotation drill: replace one node's leaf certificate under the same CA, restart that node only, and confirm the pair re-forms. Then do a CA roll with a two-CA trust bundle.

wonder (stages 5–6): run the standard deploy verification, but over https: /health/ready on both centrals and sites, CLI site list, the inbound API probe recipe and the MES test machines. Then run a QA failover in an agreed window.

11. Open decisions for the owner

Each decision has a recommendation, marked (rec).

• D1. PKI availability on the wonder network. Is there a corporate CA, such as AD CS with auto-enrolment, that will issue certificates to wonder-app-* with serverAuth + clientAuth? The code cannot show this.
  • (rec) If yes, use it. If no, use a ScadaBridge-dedicated private CA managed offline, with its root installed into LocalMachine\Root on each box.
• D2. Where the key lives on Windows.
  • (rec) LocalMachine\My by thumbprint. The private key ACL grants read to the service account.
  • Alternative: a PFX outside the content root, with its password in the Secrets store.
• D3. The ScadaBridge service account on the wonder boxes. It decides the private key ACL. The deployment records show Windows services but not the account.
• D4. The production front door.
  • (rec) Kestrel terminates TLS directly, as wonder works today.
  • Alternatives: Traefik, IIS/ARR, or an F5/hardware load balancer.
  • Also decide whether to keep port :8085 or move to 443.
• D5. Has the revisit trigger been met? Are the current wonder networks wholly inside the plant security boundary? If yes, this stays design-only and nothing is flipped.
• D6. Client auth on gRPC.
  • (rec) PSK over TLS.
  • Alternative: add client-certificate mTLS.
• D7. Akka trust scope.
  • (rec) If the issuing CA is broad (corporate-wide), add issuer or subject pinning through DotNettySslSetup.CustomValidator. That is a code change.
  • If the CA is ScadaBridge-dedicated, chain trust is enough.
  • Also decide when to turn on ValidateCertificateHostname.
• D8. Site :8084 metrics and health over TLS. Prometheus and Overview would need the CA. Today the listener carries no credentials.
  • (rec) Leave it HTTP for the first rollout.
• D9. Certificate lifetime, renewal cadence, and who owns renewal and expiry alerts.
• D10. HSTS. Once it is on, browsers refuse plain HTTP for the max-age.
  • (rec) Enable it last, with a short max-age first.
• D11. Rig Traefik backend.
  • (rec) Re-encrypt to Kestrel.
  • Alternative: terminate TLS at Traefik with plaintext on the docker network.
• D12. OPC UA and MxGateway. Move OtOpcUa connections to SignAndEncrypt, and serve mxgw over TLS. These depend on those services' own rollout.
• D13. SQL Server and LDAP (out of scope here). The rig uses TrustServerCertificate=true and GLAuth with Transport=None. Production equivalents need their own decision.