Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
ScadaBridge Env2 Docker Infrastructure
A second Docker deployment of a minimal ScadaBridge cluster topology, designed to run concurrently with the primary docker/ stack so the Transport (#24) feature can be exercised end-to-end across two real environments.
See docs/plans/2026-05-24-second-environment-design.md for the design rationale.
Cluster Topology
┌───────────────────┐
│ Traefik LB :9100 │ ◄── CLI / Browser
│ Dashboard :8181 │
└────────┬──────────┘
│ routes to active node
┌──────────────────────┼──────────────────────────────┐
│ Env2 Central Cluster │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ env2-central-a │◄──►│ env2-central-b │ │
│ │ Web UI :9101 │ │ Web UI :9102 │ │
│ │ Akka :9111 │ │ Akka :9112 │ │
│ └────────┬─────────┘ └─────────────────┘ │
│ │ │
└───────────┼─────────────────────────────────────────┘
│ Akka.NET Remoting
▼
┌────────────────────┐
│ Env2 Site-X │
│ (Env2 Site X) │
│ │
│ node-a ◄──► node-b│
│ Akka :9121 :9122 │
│ gRPC :9123 :9124 │
└────────────────────┘
Port Allocation
Env2 host ports are the primary's ports + 100. Both stacks can run simultaneously.
| Node | Container Name | Host Web | Host Akka | Host gRPC | Internal |
|---|---|---|---|---|---|
| Traefik LB | scadabridge-env2-traefik |
9100 | — | — | 80 (proxy), 8080 (dashboard host:8181) |
| Central A | scadabridge-env2-central-a |
9101 | 9111 | — | 5000 (web), 8081 (Akka) |
| Central B | scadabridge-env2-central-b |
9102 | 9112 | — | 5000 (web), 8081 (Akka) |
| Site-X A | scadabridge-env2-site-x-a |
— | 9121 | 9123 | 8082 (Akka), 8083 (gRPC) |
| Site-X B | scadabridge-env2-site-x-b |
— | 9122 | 9124 | 8082 (Akka), 8083 (gRPC) |
Shared Infrastructure
Env2 attaches to the existing scadabridge-net Docker bridge network and reuses these primary infra containers:
| Service | Container | What env2 uses it for |
|---|---|---|
| MS SQL | scadabridge-mssql |
Env2-specific databases ScadaBridgeConfig2 / ScadaBridgeMachineData2 |
| LDAP | scadabridge-ldap |
Authentication (same test users) |
| SMTP | scadabridge-smtp |
Notification capture in Mailpit (env2 emails distinguishable by FromAddress) |
| OPC UA | scadabridge-opcua |
Simulated tags for site-x data connections |
| REST API | scadabridge-restapi |
External REST API testing |
Commands
First-Time Setup
# 1. Make sure primary infra is up (creates scadabridge-net, scadabridge-mssql, etc.)
cd infra && docker compose up -d && cd ..
# 2. Build image + create env2 databases + deploy env2 containers
bash docker-env2/deploy.sh
# 3. Seed env2's single test site (first-time only)
bash docker-env2/seed-sites.sh
The seed also creates a native alarm source demo: a MxAlarmDemo template with a
GalaxyAlarms native alarm source bound to the shared MxGateway connection, plus a
deployed MxAlarmDemo-1 instance on site-x. This exercises the read-only native alarm
mirror end-to-end — mirrored alarms appear live in the Central UI Debug View with severity
and condition badges. Verify the binding with:
scadabridge --url http://localhost:9100 template native-alarm-source list --template-id <id>
After Code Changes
bash docker-env2/deploy.sh
The Docker build is shared with the primary stack — if you've just run docker/deploy.sh, the env2 build hits a fully cached image.
Running Both Stacks Concurrently
bash docker/deploy.sh # primary
bash docker-env2/deploy.sh # env2
Both UIs are now reachable:
- Primary: http://localhost:9000
- Env2: http://localhost:9100
Teardown
bash docker-env2/teardown.sh
Containers stop, volumes (data + logs) preserved. To also drop the env2 databases:
docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
-Q "DROP DATABASE ScadaBridgeConfig2; DROP DATABASE ScadaBridgeMachineData2;"
CLI Access
dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
--url http://localhost:9100 \
--username multi-role --password password \
template list
View Logs
docker compose -f docker-env2/docker-compose.yml logs -f
docker logs -f scadabridge-env2-central-a
Test Users
Same as primary (env2 shares LDAP). See infra/glauth/config.toml and primary docker/README.md.
Transport Testing Workflow
See docs/plans/2026-05-24-second-environment-verification.md for the manual golden-path checklist.
What's Different from Primary
- Single site (
site-x) instead of three (site-a/b/c). - Host port range 91XX vs primary 90XX.
- Container names prefixed
scadabridge-env2-. - Databases
ScadaBridgeConfig2/ScadaBridgeMachineData2on the sharedscadabridge-mssql. Transport.SourceEnvironment="docker-cluster-env2"(stamped into exported bundle manifests).- Distinct
Security.JwtSigningKey(sessions cannot cross envs).