Interface goes to Commons.Interfaces.Services and the state record to
Commons.Messages.Health, so CentralCommunicationActor no longer needs the
HealthMonitoring namespace. Implementation stays in HealthMonitoring.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The live AlarmStateUpdate already carried ack_time (24) but the Debug View
snapshot DTO did not, so snapshot-seeded alarm rows showed a null ack time
until the next live update. Additive google.protobuf.Timestamp ack_time = 21
(absent => null, including from a pre-#84 site), vendored C# regenerated with
docker/regen-proto.sh sitecommand, mapped both ways in SiteCommandDtoMapper.
The native golden sample now sets AckTime (the computed one leaves it null),
so the round-trip golden pins both the present and the absent decode.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ScriptCallResult and RouteToCallResponse gain trailing Truncated/LogsTruncated,
filled from the site FlowTraceCollector after Complete(); RouteToCallResponseDto
fields 8/9 carry them (sitecommand regen). FlowDebugRunner ORs them into
FlowDebugRunResult on the script entry, and on the method entry through a new
optional truncation sink on RouteHelper.WithTraceSink (carried through every
With* clone). Previously a site run whose collector dropped rows came back as a
silently short list. Older sites send false.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Central's Debug View live stream and Alarm Summary cache always saw false
because neither wire message had the field. Additive proto fields, vendored
C# regenerated with docker/regen-proto.sh (sitestream + sitecommand), mapped
both ways in StreamRelayActor/SiteStreamGrpcClient and SiteCommandDtoMapper.
The live stream publishes Alarm Actor events, which never carried the flag,
so InstanceActor.HandleAlarmStateChanged now stamps it before retaining and
publishing. A pre-#60 site sends the proto3 default. No UI badge yet.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Each cached client records when it was last handed out; a periodic sweep (half of the new
CommunicationOptions.GrpcClientIdleTtl, default 30 min, 0 disables) disposes and removes entries
unused for the TTL that have no live subscription (SiteStreamGrpcClient.ActiveSubscriptionCount).
Lookup and the sweep's check-and-remove share one lock so a handed-out client is never disposed
under its caller; an evicted client is re-created on next GetOrCreate.
Comment/XML-doc changes only:
- add summaries to ~10k test methods, fixtures, fakes and helpers
(+ <param> on theories, <returns> on async Task methods,
<inheritdoc/> on interface implementations the checker can resolve)
- strip ~2.6k project-tracking references (WP*, P*-*, row-NN, Task N,
#NN, Component-0NN) from comments, keeping the explanatory text;
reword false positives in comments only
- real docs on fakes implementing BCL/library interfaces
(IDisposable, ILogger, IOptionsMonitor, IAsyncLifetime)
- reword the last 5 src/ false positives (CWE-1236, failures-1,
year-0001, M365)
Only non-comment edits: four one-line nested classes split across lines
so members can carry docs. Build 0 errors / 0 warnings.
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.
Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Flow Debugger (design 2026-09-05 §5.3/§5.4): AttributeValueUpdate fields 7/8 stamped by
StreamRelayActor and parsed by SiteStreamGrpcClient (older frames => null/false). New
PullTraceEvents unary RPC on SiteStreamService (same PSK-gated prefix as PullAuditEvents):
a point read of the site audit SQLite by ExecutionId OR ParentExecutionId
(ISiteAuditQueue.ReadByExecutionIdsAsync, json_extract on the codec's camelCase keys, new
IX_audit_event_occurred index, 64-id / 2000-row caps) that moves no cursor and flips no
ForwardState. Central IPullTraceEventsClient/GrpcPullTraceEventsClient mirror the audit pull
client (endpoint resolution, NodeB failover, fault => empty) and register beside it.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.2/§5.4): site_command.proto gains trace=7 on
RouteToCallRequestDto, child_execution_id=6 + repeated sitestream.AuditEventDto trace_events=7
on RouteToCallResponseDto, trace=6 on RouteToSetAttributesRequestDto (imports
Protos/sitestream.proto the way central_control.proto does). Mapper round-trips them; an
empty trace_events list maps to null so an untraced/older site reads as 'not traced'.
Vendored C# regenerated via docker/regen-proto.sh (csproj untouched); golden samples extended.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
P8-8: replace the remaining wall-clock sleep-then-assert instances outside
Communication.Tests' Medium-wave sweep with the same row-40 house pattern —
continuous polling (AssertHoldsFor) for ABSENCE assertions instead of a single
end-of-window sample, which cannot distinguish "stayed quiet throughout" from
"hadn't happened yet when the one sample landed" and cannot catch a value that
flips wrong and back before that sample.
- DataConnectionActorBatchTests: QualityCounters_ArePushedOnTransitionOnly_AndCoalesced
(:206/:214 class) converted to AssertHoldsFor; ResolutionProbes_BackOff...
(:181/:184, a genuine real-time backoff-RATE assertion, not a sync-primitive
stand-in) converted to AssertRoundsWithinWindow, which enforces the ceiling
continuously (fails fast on a no-backoff regression) while the floor still
necessarily waits out the window.
- SiteAlarmLiveCacheServiceTests (:452-454, :470-472): converted to the
project's own AssertHoldsFor (mirrors SiteAlarmAggregatorActorTests', same
assembly).
- StoreAndForwardServiceTests (:891-892): async AssertHoldsForAsync variant
(the class runs on the async test thread).
- KpiHistoryRecorderActorTests (:1043-1049): two skip-path FoldCount==0
absence checks converted.
Each helper is a private per-file copy, matching the Medium-wave T17
precedent (SiteAlarmAggregatorActorTests.AssertHoldsFor) rather than a shared
TestSupport utility. Proof of the mechanism: a standalone (non-repo) harness
run 100 trials of a real background thread flipping a boolean true then back
to false inside a 300ms window — Thread.Sleep(300)+one sample missed the
violation 100/100 times (false pass), continuous 20ms polling caught it
100/100 times. Most of the actual conversions here guard monotonic counters
(their single-sample variant would eventually see the final value too), so
each converted test carries an inline honesty note on what specifically the
conversion buys for that assertion (fail-fast vs. true flip-back detection).
P8-9: ScadaBridgeTelemetryTests.EmitHelpers_DoNotThrow called
SetQueueDepthProvider(() => 5) against the process-global row-32 provider
slot with no clear and no [Collection] — a landmine for any future
same-process reader of scadabridge.store_and_forward.queue.depth. Fixed with
compare-and-clear in a finally via the existing (production, untouched)
ClearQueueDepthProvider. Searched this assembly and the whole test tree for a
second same-process reader that asserts on the value: none exists today
(MetricsEndpointTests scrapes /metrics but only asserts on the response
shape, and the only value-asserting test, QueueDepthGaugeTests, lives in
StoreAndForward.Tests — a separate test-host process the leak cannot reach),
so no [Collection] was added; the test's doc comment records the search and
flags it for re-check if that changes.
Test-only change, zero production edits. Full suites green: DCL.Tests
342/342, Communication.Tests 729/729, StoreAndForward.Tests 135/135,
KpiHistory.Tests 54/54, Host.Tests 522/522. Each converted test additionally
looped 20x in isolation with zero flakes.
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.
Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.
P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.
Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.
Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
fails with ResourceExhausted against the old ordering; the overshoot is driven
deterministically by parking both openers on the ServerCallContext token read,
which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.
The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.
The gauge test polls baseline+1, but the process-global gauge's baseline is
captured once: a parallel class's streams closing after capture lowers the
gauge below baseline+1 forever. Survey confirmed only SiteStreamGrpcServerTests
and CleanupVerificationTests drive the real SubscribeInstance handler that
moves the counter; both now share the SiteStreamSubscriberGauge collection
(row-96 pattern). 4/4 full-suite runs green, durations unchanged.
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
Six sleeps across four test files stood in for an observable edge. Each preceded an
ABSENCE assertion (or "settled" before one), so they failed toward false-pass rather than
flake: the assertion could not distinguish "the mechanism ran and correctly stayed quiet"
from "the mechanism had not run yet".
ScriptDeadlineAtEnqueueTests (two sleeps, the worst offender)
- :105 Thread.Sleep(2s) — now two explicit edges: the run must be observably QUEUED
(scheduler.QueueDepth >= 1, never checked before), then the wait polls until the 1 s
budget has elapsed, bailing early if the run leaves the queue, and asserts it is STILL
queued afterwards. The test now proves the design claim it was named for — queue wait
burns the run's budget — instead of assuming it.
- :128 Thread.Sleep(2s) + Assert.Equal(0, DetachedThreadCount) — replaced by a positive
control: a second script that genuinely wedges a worker, on the same scheduler, same
1 s timeout + 1 s grace, armed strictly later. Its stuck report is the edge; once it
lands, every watchdog armed at or before it has provably finished its grace delay. The
absence is then asserted per script name, and the detach counter is pinned at 1 (the
wedged run) rather than 0 — which a counter that never moves also satisfied.
DeploymentWarmThenGateTests (:117 plus two sibling Task.Delay(500) startup waits)
- The manager calls SetInstanceCounts unconditionally when its startup config load
completes, so a report counter on the test collector is the startup edge. The
empty-site zero is now the manager's own observation instead of an untouched field.
SiteAlarmLiveCacheServiceTests (:195, :216)
- Both sampled IsLive once after a fixed sleep. Now they poll continuously across three
linger windows on a new teardown observable (the mock client counts subscription
cancellations), so a stop at any instant inside the window is caught, and each closes
with a positive control proving that counter moves.
DataConnectionActorTests (:435)
- A Task.Delay(200) "settle" after an AwaitCondition on a connect counter. The counter
only proves ConnectAsync was CALLED; the actor is still Connecting until it processes
the piped result. Now waits for the Connected health publication that BecomeConnected
emits, so the subscribe under test provably runs against Connected rather than being
silently stashed. (The stash path keeps its own coverage in
WP6_ConnectingState_StashesSubscribeRequests.)
Negative controls (product broken in the working tree, old vs new test compared):
- Stuck-script watchdog disabled entirely: OLD test passed 2/2, NEW test failed. The
whole watchdog could be deleted and the old assertions were satisfied.
- Startup reporting a phantom instance count and taking longer than 500 ms: OLD passed
2/2, NEW failed (expected 0, actual 1).
- Linger cancellation disabled + the remaining-viewer guard dropped, so the stream is
torn down under a live viewer while IsLive keeps reporting true:
Resubscribe_Within_Linger OLD passed 2/2, NEW failed. (Last_Viewer_Leaving caught this
particular break in the old form too, via a later assertion — its strengthening is the
teardown observable and the positive control, not a false-pass fix.)
- DataConnectionActorTests :435 has no false-pass demo: the sleep guarded no assertion
there, and the change is determinism plus proving the claimed state.
Also fixes a latent race these timing changes exposed in
InstanceActorChildAttributeRaceTests: it resolved child actors single-shot immediately
after construction, but ResolveOne throws until the child's cell is published on the
parent AND the cell's Actor property stays null until the child's constructor has run on
its own mailbox thread. Neither is ordered against the test thread, so it failed both ways
under full-suite load (~1 run in 6 once the sleeps above stopped padding the schedule;
14/14 green on the unmodified tree, so it was latent, not new). Resolution now retries to
a bounded deadline; the assertion is unchanged.
Verification: SiteRuntime.Tests 8/8 green (605 tests), Communication.Tests 3/3 (717),
DataConnectionLayer.Tests 3/3 (319). Test-only change — no src/ modifications.
An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.
The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.
Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
(0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
the existing TimeoutSeconds "omit to preserve" convention. Both additive.
Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.
Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
sitecommand; csproj Protobuf item re-commented, no message types added or removed.
Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
appended so existing ones are untouched), apply path, and the legacy migrator's
explicit column list all carry them.
Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.
Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.
No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.
Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).
Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
scadabridge_site_connection_up never counted data connections. It counts open
central->site gRPC subscriber streams (Debug View + the site-wide alarm feed),
so it correctly reads 0 on a healthy site nobody is watching — and the name
invited reading it as device health. That misreading has already cost a live
debugging session: register row 48's 'connection_up is 0 on all six site
nodes' was healthy steady state, not the fault it was diagnosed as.
Two halves, per the row:
RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active
(helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and
the instrument gains the unit:"1" the other gauges already carried. Every
reference updated: the gRPC server, both test suites, Component-Communication.md,
components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test
now pins the new name AND asserts the old one is gone, so it cannot creep back.
NEW. scadabridge.site.data_connections.connected / .total — the first exported
data-connection signal (the state existed on the health report and in the CLI,
but never as a metric). connected < total is the alert condition; neither
number alone separates 'a device is down' from 'nothing is configured', which
is why the denominator ships with the numerator.
Sourcing is deliberate on two counts. It is fed by the DCL at its four state
transitions rather than derived from the health report, so the metric survives
a dead report pipeline — the exact failure the 'metrics stale' flag exists to
catch, and a derived gauge would go blind at the same moment. And it uses a
static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened
precedent) rather than a DI-injected holder: the DCL actors already take
ISiteHealthCollector through their constructors, and threading a second
dependency through both would ripple into every test that builds them. It also
keeps SiteHealthCollector untouched, which matters right now — another agent is
concurrently editing that file for the tag-resolution path.
Removal is wired too: an undeployed connection leaves the registry, so it stops
counting toward the total instead of latching connected < total on forever.
Both gauges read 0 on the standby (connections open on the active node only) —
healthy, documented, and called out in the docs so it does not become the next
site_connection_up.
No ObservedMeters allowlist change: all three instruments ride the existing
ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.)
Tests: Commons 711 passed (9 new, delta-based against a baseline with unique
per-test connection names, so they do not inherit the register row 32
QueueDepthGaugeTests static-state flake); Communication 717; DCL 319.
Root-caused from a full-suite-load-only failure of
GrpcStreamIntegrationTests.Pipeline_DuplicateCorrelationId_ReplacesStream:
ObjectDisposedException escaping SubscribeInstance.
SubscribeInstance's duplicate-prevention path cancelled AND disposed the
replaced stream's CancellationTokenSource. That CTS belongs to the replaced
handler's own `using var streamCts`, which is still running and still has to
read `streamCts.Token` — the Dispose raced that read. The race is pre-existing
and independent of R2: the same Dispose sat against the same first-token-read
when the handler still used `ReadAllAsync(streamCts.Token)`; it only ever
loses under enough scheduling pressure to land the replacement inside the
first stream's setup window.
Cancel only. The owning handler's `using` still disposes it exactly once on
every exit path, and cancellation is all replacement ever needed. The Cancel
is wrapped for the converse race (owner already finished and disposed),
mirroring CancelAllStreams().
Regression test makes the race deterministic by gating the first stream inside
its setup window — the _activeStreams entry is registered before Subscribe is
called, so the replacement always lands before the first stream reads its
token. Verified fail-before (ObjectDisposedException) / pass-after by
reinstating the Dispose as a negative control.
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
Deferred flake-pattern sweep of tests/ for the class fixed in c4caebe9 and
cfa6acbf — a bounded wait on observable A followed by a bare assert on an
observable B that the product only reaches strictly after A. Three clear
instances, each reproduced deterministically by delaying only the later step
and each re-verified green with that same delay still injected.
AlarmOnTriggerRuns_ShedAtTheSameCap_WithAnAlarmScopedSiteEvent gated on the
rate-limited shed site event and then asserted the shed COUNT bare.
AlarmActor.ShedAlarmRun increments the counter and only then emits the event,
and the event fires on the first shed only — so the gate observed Flap(4)'s
shed and ordered nothing with respect to Flap(5)'s, which is a separate
mailbox message with no observable of its own (an alarm on-trigger run has no
Ask caller to reply to, unlike ScriptActor.ShedRun, whose sibling test is
correctly ordered by its ScriptCallResult and is left alone). Deferring
Flap(5) by 2 s failed it with "Expected: 2 / Actual: 1". The count is now
ACCUMULATED across polls rather than re-read, because
SiteHealthCollector.CollectReport DRAINS the interval counters — a poll loop
that simply re-read it would consume the first shed and never reach 2.
EndToEnd_GrpcStubError_RowStays_Pending_NextTick_Succeeds gated on the
central row arriving and then asserted bare that the site SQLite row had left
Pending. SiteAuditTelemetryActor pushes via IngestAuditEventsAsync (which is
what writes the central row) and calls MarkForwardedAsync only after parsing
the ack. Delaying just that post-push step failed it with
"Assert.DoesNotContain() Failure: Filter matched in collection".
PreSnapshotBuffer_IsCapped_DropsOldest_AndCountsTheDrops gated on
"Count >= cap" and then asserted "Count == cap + 1" bare — a gate strictly
weaker than the assertion it guards, so it ordered nothing with respect to
the last event of a FlushBuffer loop that delivers one at a time. Parking
that loop after its 19,999th delivery failed it with
"Expected: 20001 / Actual: 20000".
Also hardens GrpcCentralTransportTests.WaitUntil, which returned silently on
timeout; today's single caller re-asserts immediately, so this only sharpens
the message rather than fixing a live flake.
Cleared with evidence, not guessed: SiteAlarmLiveCacheService's LingerStop
removes the site entry inside one lock, so IsLive and GetCurrentAlarms flip
atomically; and SiteReconciliationActor walks response.Gap with a sequential
foreach in which the asserted "Gone" log precedes the awaited "Good" row, the
inverse of this class.
Test-only; every ordering named above is correct as written.
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.
F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).
F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.
F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.
F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.
F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).
Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
Joseph Doherty