The Run-inbound-API-method pane rendered one text box per top-level schema
field and coerced the text on the page; an array/object field went through
JsonNode.Parse and, on failure, was SENT AS A STRING, while an empty box was
silently dropped. A required array-of-object parameter (MesMoveOut WorkOrders)
could not be entered at all — QA runs reached the site with "WorkOrders":[].
- The method pane now renders the same shared ParameterValueForm the script
pane uses (typed inputs, nested groups, add/remove list item editors,
$ref:lib: resolution, inline validation), keyed on the method id so a change
of method remounts it. The page keeps ONE value tree and BuildParameters()
serializes it as-is; the flat _inputs / Coerce() / SetInput path is deleted.
- Form | Raw JSON toggle: raw mode is a Monaco json editor for the whole
request body, given the method's root schema for completion, parsed and
schema-validated (InboundApiSchema.Validate) on every edit with the error
inline. Entering raw mode seeds the editor from the tree; leaving it
re-mounts the form over the last valid tree.
- Run is disabled while the form or the raw document is invalid (a required
field left empty disables Run before the first keystroke).
- localStorage per-method entry stores the value tree; on restore it is
validated against the schema and dropped if invalid, which also retires the
pre-#37 flat string-per-field entries.
- ParameterValueForm.ToJsonSchema / JsonElementToObject made internal static
for page reuse.
- Tests: 7 new bUnit tests (WorkOrders array-of-object round-trip to the
RunMethodAsync payload, required list-item gating, invalid raw JSON blocks
Run + shows parse/schema/non-object errors, valid raw JSON sent verbatim and
survives the switch back to the form, method switch resets, persisted tree
restored, legacy flat entry dropped); existing method-pane tests moved to
the form's #param-<field> ids and fill the required field before Run.
Playwright smoke selector moved to #param-Text.
- Docs: design doc, Component-CentralUI.md entry-points bullet, CLAUDE.md.
CentralUI.Tests 1425/1425 green; Playwright project builds.
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
Note the shared InstancePicker combobox (2026-09-08 design) replacing
the flat instance <select> on both pages, in Component-CentralUI.md's
Flow Debugger and Debug View sections and the CLAUDE.md Flow Debugger
bullet.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):
- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
client waits (F4), the 7-10 row estimate, the doubled central audit write per
routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
quoted: the method form, both CLI --trace-all-calls help strings, the CLI
README, the ApiMethod entity doc, the manual migration script and the two
Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
the tracking-store test race fixed alongside them (86451382).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.
TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.
Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.
Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.
OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.
No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Found in a browser Live run on the rig: the run drew API, DB and Notify nodes from
its inline rows; the 5 s pending-refresh re-read the tree through Replay before the
site's cached-call telemetry (12 s) had reached central, swapped the graph wholesale,
and the resolved DB node vanished (7 -> 6 steps) until a manual refresh. Polling had
already stopped because nothing was Pending.
MergeRefreshedGraph now unions the re-read rows into the run's known rows by EventId
(fresh copy wins), stores the union back on _raw and rebuilds through
FlowGraphBuilder, so a refresh only ever adds nodes. The union is exact: the inline
rows are the same AuditEvent objects the site writer persists (FlowTraceCollector).
Tests: new PendingRefresh_KeepsStepsTheLiveRunAlreadyHad_WhenReplayLagsBehindTelemetry
(fails on the old code). Two existing fixtures were internally inconsistent and only
passed under the old replace semantics: ParkedThenResolved built its Resolved outcome
with the Parked raw rows, and OutboundOutcome minted fresh event/correlation ids per
call, so no row could ever match across live and replay. Both now share rows per run.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls
property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger
runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs;
CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the
PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel
retention example), Component-Communication (§4 trace flag + inline events with proto field
numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor),
CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console
capture), design doc status → implemented. CLI README sections landed in Tasks 3/16.
Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire
relationships only, unchanged by a new page — no edit.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
scadaproj's umbrella index now keeps only short per-project summary rows,
parking long narrative in .claude/skills/sister-project-history/<project>.md
and shared-library CLAUDE.md files. Reflect that in the propagation rule so
future ScadaBridge index edits land in the right place.
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
A caller addressing a standby central node directly (or hitting Traefik inside its
/health/active probe window after a failover) used to get 503 STANDBY_NODE. The
standby now relays the request verbatim to the active peer over HTTP
(IStandbyRequestForwarder / HttpStandbyRequestForwarder) and replays the response;
method scripts and Route.To() still execute only on the active node.
- ScadaBridge:InboundApi:StandbyForwardPeerEndpoints (static list of the OTHER
central node's HTTP base URL; empty = old 503 behaviour) + StandbyForwardTimeout
(60s), both startup-validated.
- Peers tried in order: peer 503 stamped X-ScadaBridge-Standby, connection failure
or timeout → next peer; all fail → 503 ACTIVE_NODE_UNREACHABLE. A 503 without
the marker is the active node's own answer and is replayed.
- One-hop loop guard via X-ScadaBridge-Forwarded; 413 body cap now runs BEFORE
the active-node check so a standby never buffers/relays an oversized body.
- AuditWriteMiddleware skips its own ApiInbound row for a relayed request so a
call is audited once, on the node that executed it.
- Rig + env2 central nodes configured; spec (Component-InboundAPI.md → Standby
forwarding), Component-Host.md options row, CLAUDE.md updated.
- 20 new tests (filter decision + relay mechanics + validator).
Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.
Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.
Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.
The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.
dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.
Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.
Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.
Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.
Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.
Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".
No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md
Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
A healthy standby logged "Health check active-node with status Unhealthy …
'not the site pair primary (standby)'" at Error on every probe — every 10s on a
site node, and (discovered while fixing this) every 5s on a central standby,
which Traefik health-probes at that cadence. Correct-by-design state at an
alarming severity: it floods the log and trains operators to ignore ERR.
Mechanism: DefaultHealthCheckService logs one line per check per probe and picks
the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy.
Both active-node checks answer Unhealthy on the standby because 503 IS the
contract /health/active publishes. There is no IHealthCheckPublisher in this
repo, so each line is one inbound HTTP probe.
Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level
override could not work — Serilog's MinimumLevel.Override only RAISES a
per-source floor, and a LogEvent's Level is immutable, so neither an override
nor an enricher can rewrite Error as Information. Changing the check to report
Degraded was rejected: it would alter the HTTP status that /health/active
publishes, which is load-bearing for Traefik and the family dashboard.
The constraint (genuine failures must still log at Error) is preserved by keying
the predicate on the check NAME rather than the logger category: localdb,
akka-cluster, database and required-singletons keep their Error lines, and the
active-node check has no error path of its own — the active node always answers
Healthy — so the only event this can ever suppress is a standby reporting that
it is a standby. The verdict stays fully observable via /health/active's own
HTTP status, the readiness probe's cluster view, and the health dashboard.
Also corrects a stale CLAUDE.md claim found while verifying what consumes this
check: "/health/active is central-only (site nodes map no /health/* at all)" is
wrong — SiteServiceRegistration registers akka-cluster + localdb + the
role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls
MapZbHealth() on the HTTP/1.1 listener.
Fail-before evidence: with the filter behaviourally disabled,
StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the
other 9 tests in the file — the must-still-log constraints — pass either way,
which is what proves the filter is narrow rather than a blanket suppression.
Tests: Host 518 green (10 new).
LocalDb 0.3.0 (scadaproj branch localdb-per-table-snapshot) narrows the
snapshot-resync flag from one per-database bit to one per table, closing the
residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to v3
(__localdb_snapshot_state, upgraded in place on open); the wire stays compatible
with 0.1.x/0.2.x, negotiated by capability rather than by a lib_schema_version
bump — which is not available, since the handshake compares that field
fail-closed for equality.
SiteLocalDbSetup needed no restructuring: it already calls RegisterReplicated
once per table, which is exactly what per-table flagging keys off. The comment
now records that the loop shape is load-bearing, and what it buys — adding an
eleventh table to ReplicatedTables on an already-replicating site snapshots that
one table, where through 0.2.x the same edit re-streamed all eleven in full in
both directions. First boot is unchanged (all ten seed at once, so the flagged
set is every registered table and the library sends an ordinary full snapshot),
and upgrading the rig in place is a no-op (the ten are already ledgered, so
nothing seeds and nothing is flagged).
Suites: Host 490/490, SiteRuntime 604/604, StoreAndForward 134/134,
SiteEventLogging 76/76. Solution build clean.
topology-guide + CLAUDE.md LocalDb bullet updated; the umbrella scadaproj
CLAUDE.md travels in that repo's commit.
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.
Documentation + one log line, no behaviour change:
- topology-guide.md gains "Reading the replication backlog — the daily
site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
active node's PROCESS START, not a wall-clock hour, so it moves after every
failover), where it shows (replicated nodes only — not rig site-b/site-c),
the healthy signature (LocalDbReplicationConnected stays true, backlog
returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
Component-HealthMonitoring.md gains the two previously-undocumented
LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
expected transient backlog when a purge deleted rows on a replication-enabled
node, so the spike is correlatable in the log. Replication-awareness comes in
as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
matching the default that replication is opt-in and off.
Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.
Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
Final consistency sweep per plan §6: verified component docs against shipped
WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime
(recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor
references), TemplateEngine (BundleImporter watermark path), DeploymentManager
(phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm
poll, render coalescing), StoreAndForward (rate-limited drop logging), and
ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the
docs/components/ developer-reference set (SiteRuntime, SiteEventLogging,
InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue
for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for
stream graceful-completion reconnect, the required site audit DB path, honest
CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log
records the phase→commit map, gate results, adversarial-review tally, the
three test-flake root causes, and the nine-item residuals register.
F1: deploy/wonder-app-vd03/appsettings.Site.json (outside git, WP1.2's
StartupValidator gate applies live on next install/upgrade) was missing the
now-required AuditLog:SiteWriter:DatabasePath, added pointing at
E:\ApiInstall\ScadaBridge\site\data\auditlog.db alongside the file's
existing SiteEventLog/LocalDb paths; scanned deploy/ for other Site-role
appsettings with the same gap (none) and confirmed wonder does not pin
LocalDb:Replication:MaxBatchSize (F2 doesn't apply there).
F2: re-pin an explicit LocalDb:Replication:MaxBatchSize=64 on docker/site-a
node-a and node-b. MaxBatchBytes (2 MB default) only bounds the wire
message via the per-message split in SyncSession.PumpLoopAsync;
MaxBatchSize separately bounds the DB read page in
OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materializes the
whole page into memory before that split runs. Left at the 500 default, a
reconnect drain of worst-case config_json rows could transiently allocate
~35 MB per read even though every wire message stayed within budget.
Updated the CLAUDE.md LocalDb bullet to stop implying the row cap is fully
redundant with the byte budget (topology-guide.md has no matching claim).
F3: StoreAndForwardService's observer-queue onDropped callback logged a
Warning per dropped item, flooding logs at sweep rate for a stuck observer
with a large queue. LogObserverQueueDrop now logs once immediately on the
first drop of an episode, then throttles to at most one rollup Warning per
minute while drops continue, reporting the count dropped since the last
log; the cumulative ObserverQueueDroppedCount counter is unaffected.
Extended StoreAndForwardServiceTests with
ObserverQueue_ManyDropsInOneEpisode_LogsExactlyOneWarning, which floods the
bounded queue and pins exactly one drop-related Warning log for the
episode via a small CapturingLogger test double.
dotnet build ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors.
dotnet test StoreAndForward.Tests: 134/134 passed.
dotnet test Host.Tests: 490/490 passed.
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:
- Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
it is not scope-prefixed, why it is read-only, why placeholder rows are
included), the full ScriptAlarm shape, AckTime on the enriched
AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
builds its CDC triggers from the column list at registration time).
- Component-DataConnectionLayer.md already carried the AckTime section in the
first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
- Component-ScriptAnalysis.md: accessors returning domain types return the
SAME type on both surfaces, and the trust-model note that a deny-list needs
no entry for a new globals member -- only that its return type resolves in
a permitted namespace.
- Component-InboundAPI.md records the NEGATIVE decision: there is
deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
per-instance and lives on the site's Instance Actor, so the read goes
through a routed site script and the filtering happens where the data is;
central stays a thin router.
- CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
- The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
what was actually built (incl. the two choices that differ from the plan's
"or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
stay open -- they are deployed config and need a live rig.
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.
Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:
ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
→ SetStaticAttributeCommand.SourceExecutionId
→ AttributeValueChanged.SourceExecutionId (InstanceActor static-write path)
→ AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext
All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.
Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).
Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.
Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.
Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.
Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.
Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
coexistence flags; the CentralTransportMode/SiteTransportKind enums
gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.
Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).
Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.
Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.
Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.
- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
fails silently when broken, so it is enforced loudly. NOTE: the gitignored
deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
failure-detection timings, incl. a falsifiability control proving the OLD
peer-first ordering never forms.
Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
Joseph Doherty