Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.
Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.
Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
The gauge test polls baseline+1, but the process-global gauge's baseline is
captured once: a parallel class's streams closing after capture lowers the
gauge below baseline+1 forever. Survey confirmed only SiteStreamGrpcServerTests
and CleanupVerificationTests drive the real SubscribeInstance handler that
moves the counter; both now share the SiteStreamSubscriberGauge collection
(row-96 pattern). 4/4 full-suite runs green, durations unchanged.
Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.
Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).
11 clean passes, 1 pass with a caveat, 0 failures:
tag latency P50 0.88ms P99 4.57ms max 37.41ms (1.1M samples, end-to-end)
stream 900,675 delivered, 0 dropped at 100 live subscribers
health collect+ingest P99 0.31ms, 10/10 sites tracked
debug view P99 2.19ms, 264 completed, 0 timeouts
deploy 500 instances to a site in 2.6s
cpu 41% of ONE core = 2.9% of the box
memory working-set slope +8.83 MB/min
Three findings, reported rather than tuned away:
F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
positive slope ambiguous. The 1-hour run would settle it. Not tuned.
F2 (informational, by design) a deferred S&F backlog sits for one full
DefaultRetryInterval (28.9s measured) before anything drains --
EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
misread as slow drainage, so drain is reported as two numbers: retry wait,
then 3,533 msg/s of actual capacity.
F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
100.00% with zero drops while a peer lost 197,028/200,000 events entirely
within its own bounded channel. Mechanism recorded link by link.
Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.
Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).
Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.
Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Alarm Summary is now live-cache-driven via a transient per-site central
live alarm cache (ISiteAlarmLiveCache) fed by a site-wide, alarm-only
SubscribeSite gRPC stream (seed-then-stream), with the 15s poll retained
as fallback + NotReporting authority. No persisted central alarm store
([PERM]). Updated Component-CentralUI, Component-Communication (new §6.1
+ SubscribeSite RPC), M7 design (T13 delivered note), native-alarms
[PERM] note, stillpending follow-up note, CLAUDE.md, and moved register
row #10 to Resolved (left #22 in place for the sibling branch).
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Update Component-KpiHistory.md with the KpiRollupHourly schema, the third
recorder rollup tick + one-shot backfill, dual raw/rollup purge retention,
raw-vs-rollup query routing by RollupThresholdHours, per-metric gauge-vs-rate
aggregation (KpiMetricAggregationCatalog), the four new options + bounds, and
30d/90d trend windows. Append a Delivered-2026-07-10 note to the m6 design
plan, move register row #22 from Deferred to Resolved, and note rollups in the
CLAUDE.md KPI History bullet. Docs-only; no code changed.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Consolidates the reporter's poll cadence onto SqliteAuditWriterOptions instead of
the hard-coded 30 s constant, closing deferred-work register item #21 (the config-
shape cleanup its own XML doc flagged as a follow-up).
- SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30).
- Reporter reads it via IOptions; precedence explicit-override (tests) > configured
> 30 s default; a non-positive value falls back to the default. Corrected the
stale "hard-code / tunable in a follow-up" class-doc.
- Cadence tests + register updated (row 21 -> Resolved).
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Closes the operator parity gap the deferred-work register tracked as #12: native
alarm source overrides could only be set one-at-a-time, while attribute overrides
had a CSV bulk path. Adds an all-or-nothing CSV import for native sources.
- Commons: extract the RFC-4180 line splitter into a shared CsvLineSplitter
(refactor OverrideCsvParser onto it — no behavior change, pinned by its tests);
new NativeAlarmSourceOverrideCsvParser (header SourceName,Connection,
SourceReference,Filter; blank = inherited).
- Commons: NativeAlarmSourceOverrideEntry + bulk SetInstanceNativeAlarmSource-
OverridesCommand (auto-registered via the reflection command registry).
- ManagementService: Deployer-gated handler — flattens once, validates every
source resolves + is unlocked + no duplicates up front, then upserts the whole
batch under a single SaveChanges (true all-or-nothing txn). Added to the frozen
authorization matrix; dispatch-coverage guard passes.
- CLI: `instance native-alarm-source import --instance-id --file` (parity with
`instance import-overrides`) + README.
- Tests: native parser (Commons), CLI parse/entry mapping, 3 bulk-handler tests
(happy path single-commit, locked-source reject, unresolved-source reject).
Also corrects a stale XML-doc line in ScriptRuntimeContext (WaitForAttribute
quality-gated mode is shipped, not "planned") and updates the deferred-work
register: marks #7/#13/#15/#16/#20 verified-resolved, #12 as CLI/API-shipped with
only the Central UI upload affordance still pending.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Joseph Doherty