Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.
- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
SAME in-process message the ClusterClient path carries, Asks the existing
CentralCommunicationActor (zero handler-logic changes), encodes the reply via
the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).
- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
(verify the Bearer token against the key for the site in the required
x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
own-key — a genuinely different model. Fail-closed on every branch: missing or
blank header, unresolvable key, and mismatched token all → PermissionDenied,
never pass-through. One public constructor only (the explicit-prefix ctor is
internal), pinned by a reflection test — a second public ctor makes
Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.
- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
(default 8083, symmetric with sites), mirroring the Site branch. Additive to
central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
MapGrpcService. Port range-validated by NodeOptionsValidator.
- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
so a later task can exercise it.
Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
The Auth/Config normalization made ScadaBridge:InboundApi:ApiKeyPepper a hard
Central-only startup requirement (>=16 chars), but the local dev composes never
supplied it, so deploy.sh's freshly-built image crash-looped both Central nodes
on ConfigPreflight validation. Add a clearly-marked DEV-ONLY, insecure pepper
inline to each cluster's Central environment (distinct per environment). These
are NOT real secrets — production injects a true per-env secret out-of-band per
docs/operations/inbound-api-key-reissue.md; the inline values exist only so the
local docker / docker-env2 clusters start.
Renames the 13 SCADALINK_* runtime env vars → SCADABRIDGE_*, the ScadaLink__
.NET config keys → ScadaBridge__, the stale ScadaLink.Host.exe assembly name
→ ZB.MOM.WW.ScadaBridge.Host.exe, the scadalink_app SQL login → scadabridge_app,
and residual identifiers/comments/docs. Migration records (prior rename
tooling/design, DB-rename helper, this scrub script) carved out.
Adds tools/scrub-scadalink-refs.sh.
Switch site host to WebApplicationBuilder with Kestrel HTTP/2 gRPC server,
add GrpcPort/keepalive config, wire SiteStreamManager as ISiteStreamSubscriber,
expose gRPC ports in docker-compose, add site seed script, update all 10
requirement docs + CLAUDE.md + README.md for the new dual-transport architecture.
Add ActiveNodeHealthCheck that returns 200 only on the Akka.NET cluster
leader, enabling Traefik to route traffic to the active central node and
automatically fail over when the leader changes. Also fixes AkkaClusterHealthCheck
to resolve ActorSystem from AkkaHostedService (was always null via DI).
Central now resolves site Akka remoting addresses from the Sites DB table
(NodeAAddress/NodeBAddress) instead of relying on runtime RegisterSite
messages. Eliminates the race condition where sites starting before central
had their registration dead-lettered. Addresses are cached in
CentralCommunicationActor with 60s periodic refresh and on-demand refresh
when sites are added/edited/deleted via UI or CLI.
Multi-stage Dockerfile with NuGet restore layer caching, per-node appsettings
with Docker hostnames, shared bridge network with infra services, and
build/deploy/teardown scripts. Ports use 90xx block to avoid conflicts.
Joseph Doherty