The Native Alarm Source Overrides card had no bulk affordance — the CSV path
shipped CLI-only (instance native-alarm-source import --file), so the UI could
only retarget one source at a time inline.
Adds a second InputFile on that card, mirroring the attribute importer's UX
(hidden input behind a button-styled label, 512 KB cap, success/error alert
with the per-line error list, toast). Parsing reuses the SHARED
NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the
new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same
batch rules the server enforces in
ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must
resolve, must not be template-locked, and may appear at most once; any error
rejects the whole file and applies nothing.
Semantics match the CLI: merge, not full replace — sources absent from the
file keep their existing override, a blank field keeps the inherited value,
and an all-blank row clears that source's override (equivalent to the CLI's
all-null override row, without leaving a dead row behind).
Persistence reuses the inline editor's path: SaveNativeOverride's upsert body
is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the
import commits the validated batch in one SaveChangesAsync — no new server
method, no duplicated parsing.
Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field
inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source
and parser-error rejection, plus structural pins on the InputFile wiring.
Docs: Component-CentralUI.md native-alarm-source card gains the import bullet.
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).
Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.
Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
The artifact apply (DeploymentManagerActor.HandleDeployArtifacts) was
upsert-only, so deleting an external system (or shared script, DB connection,
data connection) centrally never removed the site's SQLite row — a deleted
external system stayed callable from site scripts forever. Central always
ships the COMPLETE set of each artifact class (ArtifactDeploymentService
GetAll* snapshots; the wire's presence-tracking wrapper lists preserve
null-vs-empty), so the site now applies upsert-then-reconcile: after storing
the incoming set, SiteStorageService.DeleteRowsExceptAsync removes any stored
row absent from it, per artifact table. A null list still means 'field not
shipped' and touches nothing.
Runtime cleanup rides along: a reconciled-away shared script is unregistered
from the compiled SharedScriptLibrary (a stale delegate would stay callable
until restart), and a removed data connection is evicted from the DCL hash
cache and its live connection actor stopped via the previously-caller-less
RemoveConnectionCommand — both on the actor thread via the extended
ApplyArtifactDataConnectionsToDcl message. All four tables are
RegisterReplicated, so the deletes reach the standby as ordinary CDC row
tombstones.
Tests: storage-level reconcile per table (incl. empty-set-deletes-all and
idempotency) in ArtifactStorageTests; actor-level pins in
DeploymentManagerActorTests (orphan delete, null-set no-op, library
unregistration, DCL stop for the removed connection only). Docs:
Component-DeploymentManager + Component-SiteRuntime record the
full-set/reconcile semantics.
Design review 2026-08-01: (1) MES relevance = dedicated severity band 900-999
(script constants, real IsFlaggedForMES predicate); (2) MESReceiver version
DROPPED — CvdReactor is the only implementation, router returns a clean
not-supported error elsewhere; (3) Description = Message else AlarmTypeName;
(4) enrich the native mirror NOW with an additive AckTime (proto +
native_alarm_state + DCL stamping); (5) script names match the endpoints;
(6) shared ReactorAlarms on both sides, suffix behavior as planned;
(7) existing MES API key, no new roles. Plan is now ready to execute.
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.
Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.
Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck
selected by cluster leadership (review 01 [High]): leadership is address-ordered
and diverges from singleton placement after a restart, and during a partition
both sides compute themselves leader so Traefik served both. Health 0.3.0 makes
the shared check age-based — it is now this host's own rule, promoted — so the
private copy is deleted and central registers the shared type.
ActiveNodeEvaluator, the "THE single definition of active node" this repo already
maintained, now delegates to the shared ClusterActiveNode rather than
re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp,
the inbound-API gate and the /health/active tier on one rule, and it means the
rule is shared with OtOpcUa, which had independently written a third copy.
Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering
trade-off is recorded at the PackageReference.
SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the
rule — it is a thin adapter over the site's own IClusterNodeProvider, which
scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering
the shared check directly would have to re-derive the site role and would lose the
property that the tier and singleton placement come from the same provider.
Central stays unscoped: every central member competes for one active slot.
No behaviour change on any node — same rule, one implementation instead of two.
Verified: Host.Tests 439/439, Communication ActiveNode 2/2.
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.
Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.
Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and
/metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site
node whether it was ready or which half of the pair was active. The family
overview dashboard probes every instance the same way, and this is the one gap.
Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so
the composition-root tests that build this graph actually cover them) and mapped
by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside
/metrics:
akka-cluster [Ready] the shared AkkaClusterHealthCheck. Also carries the
cluster-view data (leader/memberCount/...) the dashboard
reads, free with ZB.MOM.WW.Health 0.2.0.
localdb [Ready] NEW SiteLocalDbHealthCheck — SELECT 1 through the
registered ILocalDb. Site has no EF context; central's
DatabaseHealthCheck<ScadaBridgeDbContext> is central-only.
Replication state rides along as data ENRICHMENT only:
replication is default-OFF, so failing on it would mark
every correctly-configured node unready.
active-node [Active] NEW SitePairActiveNodeHealthCheck, delegating to
IClusterNodeProvider.SelfIsPrimary. Deliberately NOT
central's OldestNodeActiveHealthCheck: that one calls
SelfIsOldest(cluster) with no role argument and would
compute "oldest" across the wrong member set on a mesh
carrying more than one site.
Anonymous, as central's are: the site pipeline runs no authentication middleware
and has no FallbackPolicy, so nothing extra was needed.
Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free).
Tests: SiteHealthCheckTests builds the REAL site container and activates every
registration through its factory (exact-set names, one tier tag each, resolved
types, central-only checks absent behind a positive control) plus behaviour for
both new checks. SiteHealthEndpointTests boots the real site Program over
WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting
MapZbHealth would leave every registration test green while site nodes 404'd.
Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.
Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI,
DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra
containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a
shipped appsettings key already exists for it, consumer, purpose, potential
values, and required-ness. Compiled from GetEnvironmentVariable call sites, the
Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1,
and all appsettings*.json.
Code review of a1abbff7 found two edge cases: a discovery/advertised URL with no
explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an
IPv6 literal host could lose/double its brackets. Omit the port when neither URL
carries one; bracket an IPv6 host only when missing. +4 tests (12 total).
An OPC UA server advertises its base address from its own config, not the route
the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT
hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl
verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and
the connect failed. RealOpcUaClient now swaps the advertised authority for the
reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path;
no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).
'Pattern 4: Integration Routing' — RouteIntegrationCallAsync →
SiteEnvelope(IntegrationCallRequest) → SiteCommunicationActor integration
handler — was plumbed end to end but connected at neither end: no
producer (zero callers) and no handler (AkkaHostedService never
registered LocalHandlerType.Integration). It was an early scaffold the
architecture routed around — the brokered External→Central→Site→Central
round-trip is served by the Inbound API's routed-site-script path (the
RouteTo* verbs, driven from CommunicationServiceInstanceRouter), which is
live, tested, and shares IntegrationTimeout.
Decision (#32): delete. Removed the IntegrationCall{Request,Response}
messages, RouteIntegrationCallAsync, the SiteCommunicationActor receive
block + _integrationHandler field + LocalHandlerType.Integration, and the
four tests that covered them (2 actor, 2 message-contract, 1 dispatcher-
reject, 1 mapper-reject). KEPT IntegrationTimeout — it is the live
timeout for the RouteTo* verbs. Updated the exclusion-narrative comments
(proto/mapper/dispatcher), design §4, the components doc timeout table,
and marked the known-issue RESOLVED.
Full solution build clean (0/0); Communication 634 + Host 421 green.
Net -172/+20 across 14 files. Not in the gRPC proto (was deliberately
excluded there), so no wire-format change.
Rebuilt scadabridge:latest from main @ 8524a7f7 and recreated only the
env2 containers. All three gate checks pass on site-x:
1. both site nodes boot with the key (StartupValidator fail-closed →
reaching 'Application started' proves the key present);
2. control-plane PSK auth: no-header / wrong-key ⇒ PermissionDenied,
correct key ⇒ success, on both nodes (:9123, :9124);
3. LocalDb unaffected (local-only; 0 errors, healthy boot).
Bonus: central registers site-x online via gRPC heartbeat; no real
ClusterClient/receptionist (only the benign ClusterClientSiteAuditClient
label, same as the primary rig). Noted a seed-data gap (ScadaBridgeConfig2
dbo.Sites is empty) — orthogonal to the transport.
TestAccountSid was 'ACtest123' (AC + 6 chars). The management create
path validates the Account SID against ^AC[0-9a-fA-F]{32}$
(ManagementActor.cs:2205, added 2026-07-10, commit 40088a21) while the
fixture predates the guard (2026-06-19) — so the create was rejected and
the config-card + secret-non-leak assertions have been red/inert since.
Use AC + 32 hex (AC00000000000000000000000000000001) so the create
succeeds and the downstream Auth-Token-non-leak assertion actually runs
again. Unit-level RepositoryCoverageTests keep their short SIDs — they
construct SmsConfiguration directly and never hit the validated path.
BundleImporter.ApplyAsync opened a user-initiated transaction directly,
but the central ConfigurationDb context is configured with
EnableRetryOnFailure. SqlServerRetryingExecutionStrategy rejects
user-initiated transactions, so every bundle import threw on any real
SQL Server ('...does not support user-initiated transactions.'). The
whole Transport suite ran on the in-memory provider (no retrying
strategy, BeginTransaction is a no-op) so it never surfaced.
Extract the transactional apply into ApplyMergeAsync and drive it via
_dbContext.Database.CreateExecutionStrategy().ExecuteAsync, so the
strategy owns the BeginTransaction -> apply -> Commit unit as one
retriable block. The delegate resets per-attempt state (change tracker
+ ApplyMergeAsync rebuilds its own summary/resolution accumulators) so a
retried attempt cannot double-apply; a rollback failure is captured and
surfaced on the BundleImportFailed audit row exactly as before. Post-
commit side effects (ScriptArtifactsChanged publish, session zero/
remove) moved outside the retriable delegate so they run once.
Regression test: BundleImporterRetryingStrategyTests runs the import on
SQLite with a retrying execution strategy (RetriesOnFailure == true,
arming the same guard as production). It fails with the exact production
error on the pre-fix code and passes after. Existing rollback/apply
contracts unchanged (104 integration + 154 unit tests green).
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.
Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
coexistence flags; the CentralTransportMode/SiteTransportKind enums
gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.
Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).
Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.
Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.
Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.
Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
(SoakNotify instances survived the recreate as Enabled)
Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
(SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
(notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes
ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.
Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
(oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
whole dead window, 216==216 distinct — zero loss, zero dupes
Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.
- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
(the _siteClients lookup + ClusterClient.Send with the reply-to sender
preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
ControlPlaneCredentials, per-command deadlines set EQUAL to today's
CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
(flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
may have run), background failback probe to the preferred node with 1s→60s
doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.
Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.
Decisions worth recording:
- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
handlers with the exact same "handler not available" replies; the parked
handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
proxy — pinned by a dispatcher test that asserts the target is the parked probe
and NOT the dm proxy.
- Sender preservation intact. The actor's command handlers became thin
DispatchCommand delegations that still Forward (central Ask → reply routes
straight back); the existing SiteCommunicationActorTests pass unchanged, which
is the regression guard for that plumbing. UnsubscribeDebugView keeps its
fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
synthetic UnsubscribeDebugViewAck so a unary RPC still answers.
- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
CommitLeave; the gRPC service returns the ack, then schedules the real
Cluster.Leave — so a caller reaching the very node about to leave still gets its
ack instead of a broken stream. The actor path keeps today's coupled
resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
immaterial). Proven at both levels: a dispatcher test asserts the ack is built
before CommitLeave runs, and a TestServer test asserts the recorded seam order
is resolve-then-leave.
- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
one-public-ctor invariant and its test stay green.
Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.
Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).
Contract decisions worth knowing:
- Nullable COLLECTIONS ride in per-collection wrapper messages
(DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
and that distinction is live at the site — the same silent-data-loss class the
transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
TargetValueEncoded, where "wait for the empty string" is a real target, so it
carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
no stock wrapper). Enum translation is an explicit switch in both directions —
never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
return values, attribute values, tag read/write values) as a type-tagged union
so a boxed value keeps its runtime CLR type, as it does today under Akka's
type-preserving JSON serializer. Dates ride as invariant round-trip strings,
not Timestamp, which would silently normalise away DateTime.Kind and
DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
than reusing sitestream's AlarmStateUpdate, which flattens values to display
strings — right for a live stream, lossy for a snapshot the UI treats as
authoritative. The encoder omits an AlarmStateChanged.Condition that already
equals the record's derived default, so computed alarms round-trip exactly
(record equality compares the nullable backing field, not the property).
Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).
Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.
Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.
Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).
Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.
- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
including the exact sender-forwarding that routes central's reply straight
back to the waiting Ask. Default when no transport is injected -> behaviour
unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
sticky failover + background failback (1s-doubling-cap-60s), PSK +
x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
today's Ask timeouts. Cross-node retry ONLY on provably-unsent
connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
(validator: required when transport=Grpc). Host selects the impl; the
ClusterClient is created only on the Akka path.
Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.
- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
SAME in-process message the ClusterClient path carries, Asks the existing
CentralCommunicationActor (zero handler-logic changes), encodes the reply via
the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).
- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
(verify the Bearer token against the key for the site in the required
x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
own-key — a genuinely different model. Fail-closed on every branch: missing or
blank header, unresolvable key, and mismatched token all → PermissionDenied,
never pass-through. One public constructor only (the explicit-prefix ctor is
internal), pinned by a reflection test — a second public ctor makes
Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.
- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
(default 8083, symmetric with sites), mirroring the Site branch. Additive to
central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
MapGrpcService. Port range-validated by NodeOptionsValidator.
- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
so a later task can exercise it.
Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.
`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.
Decisions worth recording:
- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
telemetry actor already builds those messages, so a second copy would fork one
wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
null and empty genuinely differ here — SiteHealthCollector emits
`ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
null as "not reported", not as "reported empty". Same reasoning drives the
BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
offset). Lossless in practice — every producer stamps UTC — and documented +
asserted rather than left implicit.
SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.
Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.
Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
Joseph Doherty